Additional Caveats Obscure Oracle’s TPC Benchmark

Since my piece on Oracle’s recent TPC-C was posted, interesting emails have pointed me to additional price/performance data, and I thought I’d offer a bit of that to my readers. One of the more interesting came from the admittedly biased Conor O’Mahony, a DB2 product manager for IBM. In his blog, Conor points out some interesting elements to Oracle’s pricing and support for the system tested. To wit: “the IBM result includes pricing for 24×7 support, upgrade protection, and perpetual licenses; the Oracle result does not include any of these features.” It turns out that Oracle uses a less costly, 3 year term license for the benchmark. After 3 years, the user has to re-up (or just buy a regular license.) The support piece is equally interesting; Oracle’s Incident Support offering – with up to 10 Web-based incident requests per server and no phone support or future upgrades – is used for the benchmark system pricing.

Benchmark critics like to point out that vendors use a lot of “benchmark specials.” Technical ones includes features being turned off, and special code inside the DBMS to optimize known tasks the benchmarks test. In both cases, understanding the tweaks is useful, because you may in fact be able to apply some of the “tricks of the trade” if your workload and policies permit. I’ve always believed that optimizing some of those known tasks did in fact advance the state of the art substantially over the early years of the RDBMS. But games with licensing and support are different – it’s rather unlikely that purchasers of systems that depend on high-level performance would make the choices above for production systems.

O’Mahony has another post that looks at a recent SAP benchmark with some similar issues to those I mentioned in my post about the TPC-C, with the addition of some “undocumented, unsupported features” that were employed. If SAP performance is one of your areas of interest you may want to check that post out as well.

I recognize that this post, and my earlier one, appear one-sided to IBM’s advantage. That is not my intent, and I welcome substantive comments delving further into the issues. My strongly held belief is that Oracle makes a good product; they are not the leader for nothing. And they have brought innovations to market ahead of their competitors in clustering and memory usage, among others. They deserve credit for that, and the benchmark results speak for themselves. That’s why I dislike seeing their value obscured by other issues like the ones described in these two posts.

About Merv Adrian
Gartner Research VP, technology analyst and consultant, 30 years of industry experience, covering software mostly, hardware sometimes.

2 Responses to Additional Caveats Obscure Oracle’s TPC Benchmark

  1. Pingback: Tweets that mention Additional Caveats Obscure Oracle’s TPC Benchmark « Merv's Market Strategies for IT Suppliers --

  2. Pingback: More miscellany | DBMS2 -- DataBase Management System Services

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: