Let the benchmark brouhaha begin

Intel, Apple Computer, Advanced Micro Devices and others will promote tests over the next few months purporting to show that their products are vastly superior to those from the competition--though the tests are often for tasks no sane person would consider.

"You want to run 'EverQuest' on 32 two-way, rack-mounted servers simultaneously and render a documentary on marine migration? The XPR 2500 outscores them all," goes a typical sales pitch.

Apple has new Power Mac G5 desktops coming out in August, while AMD plans to release its Athlon64 chip for desktops in September. Intel, meanwhile, is expected to introduce Madison, the third version of its Itanium processor, next week and to launch Prescott, a Pentium 4 with a face-lift, in the second half of the year.

Apple CEO Steve Jobs kicked off the benchmark season by claiming that the new G5s, which come with chips from IBM that can run 32-bit and 64-bit software, leave Intel-based PCs in the dust.

Apple conducted a series of Standard Performance Evaluation Corporation (SPEC) tests to compare the two. In the floating-point (fp) test, which measures how well a computer performs calculations that include decimal points, Apple's dual-2GHz processor G5 scored 840, while Dell Computer's Dimension 8300 with a single 3GHz Pentium 4 scored 693. In the SPEC integer mathematics (int) test, Apple's dual-processor machine scored 800, while Dell's scored only "slightly higher" with 889, according to the Mac maker.

It took an Intel about a half hour to respond. The chipmaker didn't comment directly, but pointed people in the direction of the Web site maintained by the board that manages the SPEC results. This showed Intel's 3GHz Pentium 4 scoring 1213 on the floating-point test and the 3.2GHz Pentium 4 scoring 1252--or about 50 percent better than the two-processor Apple system. On the integer test, the 3GHz Pentium 4 scored 1164, while the 3.2GHz scored 1221.

Did Apple lie? No, it probably got the results it said it did on the Dell machine, but the system was probably submerged in maple syrup or powered by a humidifier.

Other inconsistencies in Apple's tests appeared in a quick review of the Cupertino, Calif.-based company's white papers. In the hard-drive access test, Apple's machine uses Serial ATA, a high-speed interconnect technology, while the Dell computer is saddled with older technology, even though Serial ATA is available for it.

Chipmaker AMD didn't respond with its own performance claims for the Athlon64, but last week, Hewlett-Packard accidentally posted specifications on its site for a computer that inadvertently revealed specs for the upcoming chip. It was a very AMD thing to happen.

Gleaming the cube
Did Apple lie? No, it probably got the results it said it did on the Dell machine, but the system was probably submerged in maple syrup or powered by a humidifier. Everyone manipulates benchmarks and performance claims. Business Applications Performance Corporation (BAPCo), an independent benchmarking organization, once maintained its headquarters at Intel.

Then there was the time that Apple claimed its G4 Cube desktop was a supercomputer. What it failed to mention is that it was a supercomputer in the view of U.S. government regulations concerning exports to Pakistan and other budding nuclear nations. Those export regulations have since changed.

This year, the underhanded behavior should be in full throttle. Apple and AMD are faced with declining market share. Both companies, however, have come out with products that will rival Intel's. Any advantage they can obtain--test results, customer wins--will suffice.

Some of these claims will be somewhat ridiculous. Both Apple and AMD will play up the fact that their products run 64-bit software, but neglect to point out that there will be very little software that can take advantage of that feature. But they will likely be able to show distinct advantages for their new products, which will send Intel into spin mode. (In addition, IBM is providing technical assistance to both companies, which will prevent them from falling too far behind Intel.)

Invite Michael Kanellos into your in-box Senior department editor Michael Kanellos scrutinizes the hardware industry in a weekly column that ranges from chips to servers and other critical business systems. Enterprise Hardware every Wednesday.

One of the more interesting stories could be what happens at Pixar, the animated-film production company founded by Apple chief Jobs. "After running our RenderMan benchmarks, we can now say that the Power Mac G5 is the fastest desktop in the world," said Ed Catmull, Pixar's president, in a statement on Monday.

What Catmull failed to mention was that Pixar, at the moment, is largely an Intel shop. The company replaced Sun Microsystems servers with Intel-based RackSaver servers inside its render farm earlier this year. Most of the desktops at the company are also based on Intel chips, sources say. "Finding Nemo" was largely created on Pentium 4 systems, two sources said.

Intel executives will likely casually drop that factoid in different presentations this year. However, if Pixar is adopting Apple--and Catmull's comments seem to indicate that it is--Jobs will no doubt tout that fact.

Either way, it will be an interesting fall.