Measuring how fast is really, really fast

by Tom Henderson

April 3, 2001 —

In computing, there are two perennial testing issues. They address speed and compatibility. When you consider how many facets there are to a modern PC and its communications infrastructure, testing performance and compatibility can be a wickedly difficult science.

In the early days of IBM clones, compatibility tests boiled down to whether or not a PC could run Lotus 1-2-3 out of the box and support Microsoft's Flight Simulator. System performance was measured by obscure but discrete metrics such as how many Fibonacci numbers could be calculated.

As the PC benchmarking game evolved, several trade magazines attempted to develop hardware and network performance measurements while vendors took a crack at their own measurements. But because most were developed in a vacuum, few standard test methodologies existed.

While performance and compatibility measurements at the desktop are important, networking PCs now also raises questions about system throughput and elasticity from end to end. There's good news and bad news about systems testing. Real-world performance numbers have traditionally been hard to come by. However, recent benchmarking efforts do seem to provide performance numbers that are much closer to the real world because they use tunable simulations rather than pristine -- even bleached -- platforms.

Two prominent (although largely vendor-driven) organizations exist: Standard Performance Evaluation Corporation (SPEC) and Benchmark and Performance Corp. (BAPCo).

SPEC provides a number of tests that simulate characteristics ranging from battery consumption to the number of Java browser transactions that can be performed per unit of time. SPEC Web99 and SPEC JBB2000 are two examples of platform simulators that measure the madness that network and Web operations centers go through.

While SPEC derives measurements from simulations, BAPCo uses real-world applications that have been "frozen together" in amalgamations to measure desktop characteristics and Internet content creation via its SYSmark2000 tool. A user's Internet experience is now measured by BAPCo's WebMark2001, which can be useful in the quest to measure the ever-important impatience point of Web performance.

I'd be remiss in talking about benchmarking without mentioning NetBench, ServerBench and WebBench from eTestingLabs. These tools, while not as honed as those from BAPCo and SPEC, deliver meaningful but confined analysis of their respective targets.

All transactional benchmarks today have a staggering variety of tuning options. The number that's often proffered as a vendor-competitive metric may have an enormous amount of confining characteristics used in testing to deliver the optimal result. The devil is in the details, and most benchmarks now include full disclosure statements. When determining whether or not The Number has any meaning in your network, make sure you read all the fine print.

Network World