Apple's benchmarks put to the test

		Reader Resources Apple Power Mac G5 Tech Trends Watch

In his keynote speech Monday at the company's annual developer conference, Apple CEO Steve Jobs introduced the aluminum-encased Power Mac and showed a variety of industry benchmarks that placed the machine ahead of 3GHz Dell machines using Pentium 4 and Xeon processors. Jobs also showed the Power Mac, built around an IBM chip dubbed the G5, outperforming the Dell on various applications.

Although Apple has won praise for the new Macs as a substantial improvement over prior machines, some have criticized the choices made in putting together the rival Dell machine for use in tests developed by the Standard Performance Evaluation Corporation (SPEC). Apple's tests, which were conducted by third-party testing firm Veritest, used the same GCC compiler for both machines, with the Dell boxes running the Linux operating system. Critics charge that much higher benchmarks can be achieved using the Windows OS and an Intel-optimized compiler, rather than GCC.

"It wasn't really a fair test," said Gartner analyst Martin Reynolds, who said that the Dell machines are capable of producing scores 30 percent to 40 percent higher than those produced under Apple's methodology. "The reason this happened is Apple had a third party go out and test a Dell under less than optimal conditions."

In response, an Apple representative said it wanted to compare hardware performance, so it made sense to use the same compiler on the Mac and the Dell. The SPEC benchmark tests measure the performance of the hardware and the compiler.

"We set out to conduct and report as fair a comparison as we could," Greg Joswiak, Apple's VP of hardware marketing, said in an interview on Tuesday. "SPEC measures a combination of hardware and compiler. The only way to do a hardware-to-hardware comparison is to normalize the compiler."

Peter Glaskowsky, editor-in-chief of Microprocessor Report, said a company could get better benchmark results using a Dell machine with Intel and Microsoft compilers than with a Linux machine and GCC compiler. However, he also noted that Intel's chips perform disproportionately well on SPEC's tests because Intel has optimized its compiler for such tests.

Meanwhile, some technology-focused Web sites also questioned the choice of compiler and other details in the Apple-commissioned tests, in particular some of the hundreds of settings used with the competing machines. Joswiak said that the Power Mac settings were representative of how the final machines will ship, even though a few settings did differ from the way current prototypes are configured. As for the Intel-based PCs, he noted that some of the settings that have been criticized were chosen because they actually improved the performance.

The controversy highlights the notoriously thorny issue of computing benchmarks, which have long been criticized for not measuring typical performance and not providing an accurate basis for comparison.

In addition to the benchmarks released by Apple, Jobs on Monday also showed demonstrations in which the new Power Macs outperformed the Dell by greater than 2-to-1 ratios on several programs, including Adobe's popular Photoshop image editing software.

Reynolds says he has no reason to contest those claims. "In the absence of other information, the application benchmarks look quite credible," Reynolds said.

Those usage tests may also be more important than synthetic benchmarks, he said. "The SPEC benchmarks aren't that relevant anymore. People now are looking for things like multimedia (performance) and content management."

Glaskowsky agreed, saying that in programs such as Photoshop, the G5 should routinely outperform the Pentium 4 and other processors because the chip has additional floating point units and other performance enhancements that Photoshop can take advantage of.

"If you are a Photoshop (user), Apple has a very strong story," Glaskowsky said. "I think Apple ought to be sticking to that."

In any case, Reynolds said that it is clear Apple has gotten a significant boost from the G5, an architectural change that should help Apple for some time.

"They were kind of looking a little tired on the processor side," he said, noting that the new systems offer competitive performance with Intel-based machines. Reynolds said he was also encouraged by Apple and IBM's commitment to ship 3GHz systems within 12 months.