Oracle’s TPC Assertions Don’t Help Its Credibility

Oracle has been making much of its recent benchmark results. Its new TPC campaign may backfire, however; its deceptive assertions do it no credit, and obscure some interesting technical advances (such as its first use of flash technology) behind mislabeling and deliberate omission of important facts. The “benchmark wars” are far less active than they were in their heyday, when new leapfrogging results occurred quarterly, or even more often. TPC-C, the transaction processing measure, has long been understood to be a poor representation of today’s real transaction types. At various times, most of the DBMS vendors have stopped issuing them – but they come back when they think they can get a headline or two. Some hardware vendors have also been dismissive of the benchmark; in fact, until this one, Sun had been a skeptic for a number of years.

In practice, most production transaction processing requires DBMS features which are routinely turned off to achieve the breakthrough numbers that vendors like to tout. TPC-E was developed to correct some of these issues, but a quick glance at the top results shows that only Microsoft SQL Server results are available so far. Oracle and IBM DB2 have stayed with TPC-C.

Those busy benchmark days a few years ago were driven by competing hardware architectures as much as database providers, and it’s no surprise that recent interesting results are being driven by hardware again. Kickfire has grabbed attention with its hardware-based SQL processing to deliver some extraordinary results on TPC-H benchmarks, and column stores running in memory with “smart storage” are likely to roil the waters there some more in 2010. But the most visible TPC benchmark in Q4 was Oracle’s TPC-C, and not surprisingly, it was driven by the desire to tout a hardware play: Exadata v2. Unfortunately, the benchmark Oracle published is not an Exadata benchmark at all – not even V1 – and some of the interesting things about it are obscured by the manner of its description, unless you dig a little.

A quick glance at the published Top Ten TPC-C results as of December 13, 2009 (see http://www.tpc.org/tpcc/results/tpcc_perf_results.asp) shows that Oracle has indeed delivered the top recent result. “Recent” is important here: the report was submitted November 3, 2009, in time for Oracle Open World, but after the first ads ran. The report was preceded by a prime media buy (Wall Street Journal and elsewhere) in October that compared it to IBM’s DB2 benchmark from 15 months earlier and promised to beat it.

How that comparison was done publicly is instructive, both for what it says about the performance of the product, and for the manner in which Oracle promoted it. We’ve seen this style before; it can be characterized as at best overstated, and at worst deceptive. It’s based on the premise that the headline is what people remember – few people would read the fine print details if there were any.

But in fact, there wasn’t any fine print on the benchmark details in the advertising, and that takes its meaning into different territory. After paying the (very unusual) fine against Oracle imposed by the TPC, which explicitly forbids comparing unaudited, unsubmitted benchmarks to posted ones, Oracle subsequently issued ads which didn’t actually contain the tpmC or price/tpmC numbers – and weren’t about the Exadata machine at all. The fine was trivial – amounting to far less than any of the ads themselves cost. One is left to assume that it was just considered part of the cost of the campaign: as a founding and very active member of the Council, Oracle is certainly not unaware of the rules.

Digging a bit deeper (I’m not given to microanalysis of full reports anymore, but a few things jump out from the Executive Summary, compared to the one for IBM’s #2 result – both are only 4 pages), the actual results demonstrate several interesting things about the state of the art in transaction processing:

  • Clustered systems have made great strides in delivering results. This is the only clustered system on the list, run on 12 systems; all the other results in the list are on non-clustered systems, and they are all behind in absolute numbers. Clustered architectures are now demonstrably capable of competing at the top end (if there were any doubt.).
  • Clustered systems deliver a lot of capacity/power for less. Oracle’s benchmark ran on a Sun server with 48 processors, 384 cores and 3072 threads, a configuration with a total cost (including storage and software; more on that below) of $18M in late 2009. IBM’s had 32 processors and 64 cores, 128 threads, but it cost $17M in mid 2008. So from the system cost perspective, a little more money (relatively speaking) now buys more than an order of magnitude (24x) more threads. Again, perhaps not surprising, but one nice thing about the TPC is that it represents a stake in the ground; theoretically, anyone can buy this configuration at that price.
  • The results fall far short of the hardware uplift. Though the news is good on the hardware side, the tpmC uplift: 7,646,486 for Oracle vs 6,085,166 for DB2, or 25%, seems almost trivial by comparison. Oracle’s statement that “Oracle and Sun were able to set the world record using eight times less hardware than IBM used for its largest benchmark” was one of the questionable choices of phrasing in their messaging about the results.
  • DB2 clearly wins the “per core” comparison. Looked at on a tpmC per-core basis, Oracle got 20K/core, while IBM got 95K/core. I’d be careful about per-core licensing under such circumstances; the good news is that Oracle recently reduced their core multiplier, clearly seeing the value in pushing customers onto machines with more cores. (No doubt it helped the price/performance comparison for the benchmark too.)
  • Memory drives performance, but not as much as one would hope.) There’s a sizable difference in memory: 6 Tb for Sun (at 512 Gb per node) and 4 Tb for IBM (64 Gb per core). A 50% difference, but not a 50% performance uplift. As system architectures become more effective at using multiple cores with dedicated memory, we should see a series of jumps in performance ahead.
  • Solid state storage is transforming economics, if not yet performance: the Sun system used 686.6 Tb (with a good deal of new, high-performing solid state disk), which cost $8.4M; IBM, using older disk technology, used 746.5 Gb. at $11.5M. This is an extraordinary volume difference, and it suggests the economics of storage are shifting as rapidly as those of processors.  But again, all the added volume does not seem to have driven a proportional improvement in performance.

Bottom line: this suggests there is a very interesting set of skirmishes ahead, and substantial improvements in results I’d expect to come from both vendors. When IBM delivers benchmarks on its current DB2 release (the one discussed here was on DB2 9.5, not the current 9.7), current hardware, current storage – and perhaps using its own clustering pureScale technology, the game will really be afoot. One hopes IBM’s discussion of the results will be somewhat less fuzzy than Oracle’s. In the meantime, they can only stand by and watch the big headlines sink in; IBM rarely gets into you said/we said kinds of games. At its recent conferences for analysts, there was no talk of these issues to match the red meat attacks heard at OOW. They need to get their own benchmark to market. Soon. Watch also for a wildcard if VoltDB, the stealth project now in early trials, decides to publish an audited TPC-C.

About Merv Adrian
Gartner Research VP, technology analyst and consultant, 30 years of industry experience, covering software mostly, hardware sometimes.

6 Responses to Oracle’s TPC Assertions Don’t Help Its Credibility

  1. Pingback: Tweets that mention Oracle’s TPC Assertions Don’t Help Its Credibility « Merv's Market Strategies for IT Suppliers -- Topsy.com

  2. Phil says:

    Do you really believe that IBM’s Power 595 TPC-C result is any less “skirmish”? IBM is the king of benchmarketing and I am glad, that once and for all, Oracle was able to easily prove that IBM’s Power/DB2 based solutions are just outdated.

    If you had looked further into the “details” of the published results and compare them to IBM’s fastest system, the Oracle-Sun TPC-C benchmark configuration and results demonstrated:

    * 26 % faster performance
    Oracle-Sun = 7,646,486 tpmC ; IBM = 6,085,166 tpmC
    * 7x better transaction response time
    Oracle-Sun = .168 seconds avg.; IBM = 1.22 seconds avg.
    * 16 % better price/performance
    Oracle-Sun = $2.36/tpmC; IBM = $2.81/tpmC
    * 8x fewer racks; less floor space
    Oracle-Sun = 9 racks; IBM = 76 racks
    * 10.7x better computational density
    Oracle-Sun = 849,610 tpmC/rack; IBM = 80,067 tpmC/rack
    * 4.6x less energy consumed
    Oracle-Sun = 73 KW per hour; IBM = 343 KW per hour
    * 5.9x better power/performance
    * Better availability with Oracle RAC and a cluster of 12 Sun T5440 servers; compared to a single IBM Power 595 server

    If you tally the “real costs” of running one solution over the other, you will see that the choice is clear. Oracle & Sun offer the far superior solution over IBM for running OLTP.

  3. Merv Adrian says:

    Thanks for your comment, Phil. I used the word “skirmishes,” a verb, not an adjective, in my post, so I have no idea what you’re getting at with your first sentence. I am glad Sun has moved past its unwillingness to participate in TPCs. It was good to see a Sun machine used for a record-setting TPC-H earlier this year with ParAccel, for example, although that result was withdrawn and it’s unlikely Sun will now stand up an expensive system for an Oracle competitor.

    If you read my piece closely in your haste to reproduce the published Oracle claims I linked to you saw that I acknowledged them. I said that it was significant to produce a demonstration of the power of a clustered solution, and the use of flash memory. I also specifically acknowledged the tpmC numbers, and the differences in price. I didn’t reproduce the “per rack” comparison as you did, again because Oracle had done that in the documents I pointed to. I focused instead on the number of cores and threads involved, both because I think it says something important about how well the software makes use of the resources available to it and because it had not been mentioned in the materials I cited.

    Kudos to Sun for the energy and space savings here – part of my point, which I hope was clear, is that architecture has moved ahead significantly since IBM’s 2008 machine, and Oracle has jumped to the top of the list by getting their benchmark to market quickly on a new generation of systems. It’s good to see a debate about these issues taking place again; I hope that as it unfolds it will clarify, rather than obfuscate, the important changes taking place. Memory architectures, multicore, flash memory, parallel code execution and the like make it possible for a new step function move up the DBMS performance ladder. I hope the players will engage with the same enthusiasm for advancing the state of the art that we have seen in past flurries of competing benchmarks.

  4. Frank says:

    Phil,
    Do you really think that a 25% throughput increase coming 15 months after the prior benchmark proves anything? The hardware advances alone account for far more than 25% in performance gains. Where have you been?

  5. Phil says:

    Hi Frank. Where have I been? I’ve been busy analyzing the latest Oracle-Sun systems and results.

    But actually Frank, I agree with you. 25% increase in 15 months is actually not all that great when you look at the big picture. But when you realize that IBM’s Power6 systems haven’t improved in that time frame and aren’t expected to until mid next year when Power7 finally arrives, then you realize that it is a big deal.

    What I find much more amazing than the performance gain, is the massive 10.7x better computational density and 5.9x better power/performance ratio. In todays datacenters, space and power consumption(and therefore heat) challenges are far more important metrics than performance gains alone and this Sun SPARC CMT technology really shows these improvements. Have you looked at the other Sun SPARC/Oracle world records they have published? From what I can see, the SPARC systems running a wide range of Oracle products are leading the top 7 commercial benchmarks in its category.

  6. Pingback: Additional Caveats Obscure Oracle’s TPC Benchmark « Merv's Market Strategies for IT Suppliers

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: