AMD: Chip test was altered to favor Intel

In a presentation designed to influence writers and editors at computer publications, AMD alleges that a recently released version of the Sysmark benchmarking tool, from industry consortium the Business Applications Performance Corporation, has been revised to downplay the strengths of AMD's Athlon XP processors while emphasizing tests that portray Intel's Pentium 4 in a favorable light. AMD also confirmed that it joined BAPCo earlier this year, with the aim of influencing the way next year's Sysmark is formulated.

The media campaign steps up AMD's efforts to ensure that its processors compare favorably with Intel's in the public eye, but some observers say both companies' rhetoric should be taken with more than a grain of salt.

Benchmarks create statistics that are supposed to make it easier to compare one processor's performance to that of another. Sysmark figures indicate a chip's performance in running particular applications, such as office-productivity software, 3D rendering tools and Web applications. Since last year, AMD has been on a renewed crusade to convince users to pay attention to performance measures such as benchmarks, which are supposedly more accurate than judging chips by their clock speed alone. Athlon chips generally sport lower clock speeds than Pentiums, even if they offer the same real performance.

In its presentation, AMD claims that the way tasks within Sysmark 2002 have been chosen clearly favors Intel's chip architecture. Fourteen tasks that most favored AMD chips were taken out, AMD claims. "Those accounted for most of our advantage over the Pentium 4," an AMD representative said.

The most dramatic change was the way scores for Internet content creation were calculated, AMD said. Compared with scores from Sysmark 2001, Intel chips scored slightly higher, while the score of AMD chips plummeted 20 percent.

An AMD representative said the company had joined BAPCo in order to have "a more direct interest" in the way the software was formulated. "We want to look at the reasons behind what they did when discussing Sysmark 2003 and have our say as well," the representative said. BAPCo has a long association with Intel. Not only was the chipmaker one of the founding members of the organization, BAPCo's headquarters were located at Intel's headquarters for years. BAPCo was formed in 1995 by Intel, Compaq Computer, Dell, IBM and several trade publishers. CNET Networks, the publisher of News.com, is a member of the organization.

BAPCo, for its part, says that it ensures objectivity by building its benchmarks around retail software and focusing on actual PC users' level of usage. Each participating company receives one vote on the benchmark once it is developed.

Intel said that other benchmarks besides Sysmark have confirmed the Pentium 4's performance. "The vast majority have said that the Pentium 4 2.8GHz is the highest-performing processor," said an Intel representative.

Most benchmarking sites are getting around the issue by using the older tests, along with a battery of other tests for gaming, multimedia and other applications. In general, the latest 2.8GHz Pentium 4 outperforms the Athlon XP 2600+ but not by a huge margin.

Some have suggested that the change came about because of typical standards-body lethargy. A member of the body brings up a proposal. No one really examines it deeply, and before anyone realizes it, the proposal has been enshrined.

Some observers have suggested that AMD may be exaggerating its portrayal of the changes to Sysmark 2002. Thomas Pabst, creator of the influential Tom's Hardware Guide Web site, said this week that laboratory tests did not show conclusively that the benchmarks had been changed to favor Intel.

According to some industry analysts, the argument is practically academic, because there is no universally agreed-on standard for measuring real-world performance. "Benchmark standards are supposed to be objective, so it shouldn't matter who's in bed with whom. It should be irrelevant, but it's not," said analyst Andy Brown of IDC.

Benchmarks could be more convincing if all of the important parties involved agreed to a set of rules, Brown said. "If there were independent benchmarks, conforming to certain criteria, I think perhaps it would be a bit easier for the industry to plead its case," he said.

AMD has known about the discrepancy for months and briefed analysts in the spring on the issue. The company, however, scuttled plans to go public with its findings, choosing instead to join BAPCo, said several sources.

Kevin Krewell, an analyst at the Microprocessor Report, further added that the Athlon did unusually well on the SysMark 2001 benchmark. The chief problem, he said, has been BapCo's secrecy about what elements make up their tests. "It has never been clear what is going into the tests and why."

Still, there's no getting around that AMD did worse on the tests. "There is an unequal negative impact on Athlon," said Dean McCarron, principal analyst at Mercury Research.