Oracle’s TPC Assertions Don’t Help Its Credibility

Oracle has been making much of its recent benchmark results. Its new TPC campaign may backfire, however; its deceptive assertions do it no credit, and obscure some interesting technical advances (such as its first use of flash technology) behind mislabeling and deliberate omission of important facts. The “benchmark wars” are far less active than they were in their heyday, when new leapfrogging results occurred quarterly, or even more often. TPC-C, the transaction processing measure, has long been understood to be a poor representation of today’s real transaction types. At various times, most of the DBMS vendors have stopped issuing them – but they come back when they think they can get a headline or two. Some hardware vendors have also been dismissive of the benchmark; in fact, until this one, Sun had been a skeptic for a number of years.

In practice, most production transaction processing requires DBMS features which are routinely turned off to achieve the breakthrough numbers that vendors like to tout. TPC-E was developed to correct some of these issues, but a quick glance at the top results shows that only Microsoft SQL Server results are available so far. Oracle and IBM DB2 have stayed with TPC-C.

Those busy benchmark days a few years ago were driven by competing hardware architectures as much as database providers, and it’s no surprise that recent interesting results are being driven by hardware again. Kickfire has grabbed attention with its hardware-based SQL processing to deliver some extraordinary results on TPC-H benchmarks, and column stores running in memory with “smart storage” are likely to roil the waters there some more in 2010. But the most visible TPC benchmark in Q4 was Oracle’s TPC-C, and not surprisingly, it was driven by the desire to tout a hardware play: Exadata v2. Unfortunately, the benchmark Oracle published is not an Exadata benchmark at all – not even V1 – and some of the interesting things about it are obscured by the manner of its description, unless you dig a little.

A quick glance at the published Top Ten TPC-C results as of December 13, 2009 (see http://www.tpc.org/tpcc/results/tpcc_perf_results.asp) shows that Oracle has indeed delivered the top recent result. “Recent” is important here: the report was submitted November 3, 2009, in time for Oracle Open World, but after the first ads ran. The report was preceded by a prime media buy (Wall Street Journal and elsewhere) in October that compared it to IBM’s DB2 benchmark from 15 months earlier and promised to beat it.

How that comparison was done publicly is instructive, both for what it says about the performance of the product, and for the manner in which Oracle promoted it. We’ve seen this style before; it can be characterized as at best overstated, and at worst deceptive. It’s based on the premise that the headline is what people remember – few people would read the fine print details if there were any.

But in fact, there wasn’t any fine print on the benchmark details in the advertising, and that takes its meaning into different territory. After paying the (very unusual) fine against Oracle imposed by the TPC, which explicitly forbids comparing unaudited, unsubmitted benchmarks to posted ones, Oracle subsequently issued ads which didn’t actually contain the tpmC or price/tpmC numbers – and weren’t about the Exadata machine at all. The fine was trivial – amounting to far less than any of the ads themselves cost. One is left to assume that it was just considered part of the cost of the campaign: as a founding and very active member of the Council, Oracle is certainly not unaware of the rules.

Digging a bit deeper (I’m not given to microanalysis of full reports anymore, but a few things jump out from the Executive Summary, compared to the one for IBM’s #2 result – both are only 4 pages), the actual results demonstrate several interesting things about the state of the art in transaction processing:

• Clustered systems have made great strides in delivering results. This is the only clustered system on the list, run on 12 systems; all the other results in the list are on non-clustered systems, and they are all behind in absolute numbers. Clustered architectures are now demonstrably capable of competing at the top end (if there were any doubt.).
• Clustered systems deliver a lot of capacity/power for less. Oracle’s benchmark ran on a Sun server with 48 processors, 384 cores and 3072 threads, a configuration with a total cost (including storage and software; more on that below) of $18M in late 2009. IBM’s had 32 processors and 64 cores, 128 threads, but it cost $17M in mid 2008. So from the system cost perspective, a little more money (relatively speaking) now buys more than an order of magnitude (24x) more threads. Again, perhaps not surprising, but one nice thing about the TPC is that it represents a stake in the ground; theoretically, anyone can buy this configuration at that price.
• The results fall far short of the hardware uplift. Though the news is good on the hardware side, the tpmC uplift: 7,646,486 for Oracle vs 6,085,166 for DB2, or 25%, seems almost trivial by comparison. Oracle’s statement that “Oracle and Sun were able to set the world record using eight times less hardware than IBM used for its largest benchmark” was one of the questionable choices of phrasing in their messaging about the results.
• DB2 clearly wins the “per core” comparison. Looked at on a tpmC per-core basis, Oracle got 20K/core, while IBM got 95K/core. I’d be careful about per-core licensing under such circumstances; the good news is that Oracle recently reduced their core multiplier, clearly seeing the value in pushing customers onto machines with more cores. (No doubt it helped the price/performance comparison for the benchmark too.)
• Memory drives performance, but not as much as one would hope.) There’s a sizable difference in memory: 6 Tb for Sun (at 512 Gb per node) and 4 Tb for IBM (64 Gb per core). A 50% difference, but not a 50% performance uplift. As system architectures become more effective at using multiple cores with dedicated memory, we should see a series of jumps in performance ahead.
• Solid state storage is transforming economics, if not yet performance: the Sun system used 686.6 Tb (with a good deal of new, high-performing solid state disk), which cost $8.4M; IBM, using older disk technology, used 746.5 Gb. at $11.5M. This is an extraordinary volume difference, and it suggests the economics of storage are shifting as rapidly as those of processors.  But again, all the added volume does not seem to have driven a proportional improvement in performance.

Bottom line: this suggests there is a very interesting set of skirmishes ahead, and substantial improvements in results I’d expect to come from both vendors. When IBM delivers benchmarks on its current DB2 release (the one discussed here was on DB2 9.5, not the current 9.7), current hardware, current storage – and perhaps using its own clustering pureScale technology, the game will really be afoot. One hopes IBM’s discussion of the results will be somewhat less fuzzy than Oracle’s. In the meantime, they can only stand by and watch the big headlines sink in; IBM rarely gets into you said/we said kinds of games. At its recent conferences for analysts, there was no talk of these issues to match the red meat attacks heard at OOW. They need to get their own benchmark to market. Soon. Watch also for a wildcard if VoltDB, the stealth project now in early trials, decides to publish an audited TPC-C.