Just when I was not looking, two new official TPC-E results have been posted in the last week. IBM has a 3218.46 TPC-E score for an IBM System x3850 X5 that has four Intel Xeon E7-4870 processors, while HP has an 1881.76 TPC-E score for an HP ProLiant DL380p Gen8 system with two Intel Xeon E5-2690 processors.
What is notable about this is that the 3218.46 score for a four-socket Xeon E7-4870 system is significantly higher than we have seen for similar four-socket Xeon E7-4870 systems in the past. An especially good comparison is between an IBM System x3850 X5 that was submitted on June 27, 2011 and this latest result for an IBM System x3850 X5 system that was submitted on November 28, 2012. As you can see in Table 1, the newer submission for the same model server has a 12.4% higher score than the older submission. This is for the exact same model server, with the exact same number and model of processors. The first big difference that jumps out is that the newer submission is running SQL Server 2012 Enterprise Edition on top of Windows Server 2012 Standard Edition, while the older submission is running SQL Server 2008 R2 Enterprise Edition on top of Windows Server 2008 R2 Enterprise Edition.
Date | Model | Processor | Operating System | SQL Server Version/Edition | TPC-E Score |
6/27/2011 | System x3850 X5 | Xeon E7-4870 | Windows Server 2008 R2 Enterprise | SQL Server 2008 R2 Enterprise | 2862.61 |
11/28/2012 | System x3850 X5 | Xeon E7-4870 | Windows Server 2012 Standard | SQL Server 2012 Enterprise | 3218.46 |
Table 1: Comparing Two IBM System x3850 X5 TPC-E Submissions
Could this 12.4% performance jump be simply due to the newer operating system and the newer version of SQL Server? It is very possible that there were some low level improvements in Windows Server 2012 that work in conjunction with SQL Server 2012 to improve performance (similar to what we saw with Windows Server 2008 R2 combined with SQL Server 2008 R2). With Windows Server 2008 R2, Microsoft did some low-level optimizations so that they could scale from 64 logical processors to 256 logical processors. This work also benefitted smaller systems with fewer logical processors. I think it is likely that some similar work was done with Windows Server 2012, so that it could scale from 256 logical processors to 640 logical processors, so that might explain some of the performance increase. I have some questions in to some of my friends at Microsoft, trying to get some more detailed information about this possibility.
It is also possible that there were improvements in SQL Server 2012 all by itself that contributed to the performance increase. Another possibility is that the TPC-E team at IBM just did a much better job on this newer system. If you dive deeper into the two submissions, you will notice some other differences in the hardware and the environment for the test. The newer submission is a system with 2048GB of RAM and (126) 200GB SAS SSDs for database storage, with a 13.3TB initial database size, while the older submission is a system with 1024GB of RAM and (90) 200GB SAS SSDs for database storage, with a 11.6TB initial database size. As long as you have sufficient I/O capacity to drive the TPC-E workload, the TPC-E score is usually limited by processor performance, so I don’t really think that the RAM and I/O differences are that significant here.
What do you think about this? I would love to hear your opinions and comments!
One thought on “Two New TPC-E Submissions for SQL Server 2012”
I agree with you that the OS and database version don’t seem to explain the whole improvement. Going from Server 2008+SQL2008 to Server2012+SQL2012 only adds about a 5% boost to TPC-E results, based on my simple Excel regressions on the published results.
Likewise I don’t see the memory amount difference as being significant. The actual DIMMs used are different technologies, though; and while I think the memory throughput should be identical, there’s a chance that the different DIMMs yield different results.
My quick analysis points to the database structure as the source of the difference, but my simplistic model can’t say whether it’s due to physical differences (i.e. SSD quantity and JBOD type), or just the layout of the the database. My gut says it’s got to do with the hardware, because the guys running these tests wouldn’t put together more drives and change a "known good" configuration just for fun.
IBM and Fujitsu do seem to get slightly better than expected results on TPC-E than other vendors, so it’s possible they’ve found some additional tuning.