Nvidia accused of fudging tests
A software maker that specializes in tools for measuring PC performance says the graphics chip giant tweaked its drivers to inflate test results.
Saratoga, Calif.-based Futuremark on Friday said in a statement that Nvidia tweaked software needed to run its new GeForce FX 5900 processor to distort performance in Futuremark's 3DMark 03 testing application. Futuremark is one of the leading independent providers of software and services for performing PC "benchmark" tests.
The company said drivers--software files that govern how a component interacts with the rest of the PC--for the new Nvidia chip were altered to detect activity characteristic of a benchmark and adjust performance accordingly.
"Recently, there have been questions and some confusion regarding 3DMark 03 results obtained with certain Nvidia" products, Futuremark said in the statement. "We have now established that Nvidia's Detonator FX drivers contain certain detection mechanisms that cause an artificially high score when using 3DMark 03."
A representative at Nvidia questioned the validity of Futuremark's conclusions. "Since Nvidia is not part of the Futuremark beta program (a program which costs of hundreds of thousands of dollars to participate in), we do not get a chance to work with Futuremark on writing the shaders like we would with a real applications developer," the representative said. "We don't know what they did, but it looks like they have intentionally tried to create a scenario that makes our products look bad."
Nvidia is amid a hard-fought battle with rival ATI Technologies to claim the performance lead in PC graphics processors. After years of Nvidia dominating both in market share and performance, ATI took the speed lead last year with new versions of its Radeon chips. After settling manufacturing issues that resulted in ongoing delays of product launches, Nvidia hoped to regain the performance lead this year with the 5900.
Dean McCarron, an analyst with Mercury Research, said benchmark results are valuable for graphics chip makers if they want to claim general bragging rights for the fastest chip. But the results hold little sway with the hard-core PC game players who make up the bulk of the customer base for high-end graphics chips.
"What happens is that in the gamer market, the buyers are fairly sophisticated," McCarron said. "They're looking at real-world results--how the product performs on the games they're interested in playing. The overall benchmarks, like what Futuremark does, are more influential for the less sophisticated part of the market."
McCarron said manipulation of benchmark testing is common in the PC industry. "The best advice is to pick an application you're going to use a lot and find someone who's done testing using that," he said.