It's probably too much to ask of fight-hungry hardware vendors, but here's my New Year's plea -- enough with the server performance benchmark rumbles already. How about a 2006 free of dual-core duels -- try saying that fast when you've had one eggnog too many! -- and benchmark brawls? And, while we're at it, please can the alliteration too.

Of course, I'm not expecting pugnacious IT players to cozy up to one another over the year ahead, just tone down all that stridency a notch or two.

As 2005 draws to a close, chip maker Advanced Micro Devices Inc. (AMD) has already reached for the server dual-core duel laurel wreath and, Napoleon-like, crowned itself the victor over bitter rival Intel Corp. If you must, you can check out all the chest-thumping at http://www.amd.com/duel.

Back in August, AMD challenged Intel to a performance showdown of their respective dual-core 64-bit processors for servers as judged by industry-standard benchmarks, notably those from the nonprofit organization Standard Performance Evaluation Corp. or SPEC for short.

This could be characterized as kicking a man while he's down, since everyone, even Intel, has acknowledged it lagged a ways behind AMD in actually introducing dual-core 64-bit chips for servers. Additionally, analysts and third-party vendors will readily tell you that AMD's dual-core Opteron chips outperform Intel's Xeon alternatives on a number of fronts.

Anyhow, wisely or unwisely, Intel chose not to strap on its dual-core guns and step into the ring with AMD, so as to speak. In fact, Intel never fully acknowledged the challenge's existence. Perhaps, the thinking at Intel ran along the lines of: Ignore the AMD glove at your feet so eventually they'll repocket it and move on.

But guess again, here comes another challenge, slyly repackaged as Sun Microsystems Inc. versus Dell Inc., but it's really pitting AMD against Intel once again. Despite much speculation to the contrary, Dell has still yet to commit to using AMD's chips, remaining an Intel-only shop. So, if you challenge Dell, you effectively challenge Intel.

Once again, the challenger, in this case, Sun is already claiming victory, pitting its dual-core Opteron-based Sun Fire X4100 server against an unspecified dual-core powered Dell machine in what Sun's calling a "benchmark brawl." Dell has until Jan. 31, 2006, to respond to Sun and hopefully the company will adopt the Intel approach and bring this sort of thing to an end.

All this kind of silliness begs the question: just how useful are benchmarks? Particularly when one vendor, step forward Sun, has recently seen fit to create its own performance metric dubbed SWaP for space, watts and performance.

Computing benchmarks are so highly specific, citing the performance of one particular hardware configuration of processor, memory and machine running a particular software application, that it's hard to have them mirror real-life conditions faced by users. How many servers do you know running a single application?

In talking to a couple of folks at the major user groups, it's clear that benchmarks are best seen as just one more statistic for potential customers to mull over, not the be-all and end-all the likes of AMD and Sun are suggesting.

"The problem is, unless you are benchmarking your own work, it is hard to decide what value the benchmark has," Robert Rosen, president of independent IBM Corp. user group Share, wrote in an e-mail response to questions. "Usually it is just another data point and it allows you to compare two different systems."

Smoot Carl-Mitchell, vice president of the board of directors of independent Hewlett-Packard Co. user group Encompass, agrees with Rosen.

"Benchmarks like SPEC have some use as a general measure of relative processor performance," he wrote in an e-mail response to questions. "However, given that the processor is but a small piece of an overall computing system, it is of limited usefulness when making a selection between processor types." The choice of which system to buy rests on other factors including the level of support offered by a particular vendor rather than mere system performance, according to Carl-Mitchell.

Over the past decade, benchmarks have become less valuable mostly due to hardware performance becoming less of an issue, according to Rosen. "There are still cases where benchmarking real work is valuable but the cost of doing that is so high (both preparing the benchmark and conducting the test) that fewer and fewer people do it," he wrote. "It is cheaper to over buy on hardware performance than to do the benchmark."

Rosen dubs the dual-core duel and benchmark brawl as "interesting to watch," but not of major interest to most enterprise users he's talked with.

"I find them amusing at times, but I really do not pay a lot of attention to them," Carl-Mitchell wrote. "Just because a vendor claims to have a faster system does not always mean the system is the one best suited to a specific environment."

So, there you have it, AMD and Sun.

How about sitting down right now and penning a New Year's resolution? Actually, here's one I wrote for you guys earlier: "We resolve not to engage in excessive benchmark bragging in 2006 and we will abstain from any kind of performance duel, brawl, battle royal, affray or rumpus." Happy New Year, I hope.

IDG News Service

• Email
• Print
• Share
  • Slashdot
  • Digg
  • Stumble
  • Reddit
  • Newsvine