Microsoft's new IE9 triggers speed-test squabble

Microsoft has released a seventh test version of Internet Explorer 9 the company says is better at "real-world" Web-based JavaScript programs. But with it has come a rival's accusation that Microsoft essentially engaged in the storied computer industry practice of benchmark engineering--designing technology to be fast on an artificial speed test.

The seventh IE9 platform preview comes with new improvements to its Chakra JavaScript engine. And Microsoft asserted Chakra is engineered to do better on actual Web site tasks rather than narrow benchmarks.

"Over the last few weeks, we've been tuning the JavaScript engine for more of the patterns we've found in real-world sites. Based on the progress since the last platform preview, we're releasing an updated platform preview build," said Dean Hachamovitch, Microsoft's IE general manager, in a blog post yesterday.

The company has been creating a range of Web applications to let people test all the browsers that figure in today's fierce competition. And with the new version, Microsoft signaled its growing impatience with one test of JavaScript, the SunSpider benchmark--both with how basic its tests are and how limited it is compared to the full range of browser performance challenges.

"Microbenchmarks and real-world Web patterns have little in common," Hachamovitch said. "We think people should evaluate browser performance with real-world scenarios. Real-world scenarios involve using all the subsystems in the browser together rather than looking at single subsystems in isolation...real-world Web sites use JavaScript to respond to user input, manipulate strings, and move objects around the screen, and much more."

Microsoft has griped about oversimplified browser benchmarks before, but that didn't stopped it from prominently publishing results that now show it faster than Google's Chrome, Opera Software's Opera, Mozilla's Firefox, and Apple's Safari.

But also this week, Mozilla took issue with one aspect of IE and JavaScript: "dead code elimination."

This idea involves scrutinizing software before it runs and eliminating parts that are mere busywork--flawed code programmers don't always realize they use. One aspect of SunSpider, a test called math-cordic, "is written exactly in a way that triggers this general optimization," Hachamovitch said.

One objection came from Mozilla JavaScript engineer Rob Sayre, who filed an IE9 bug that led to Hachamovitch's explanation. Sayre, though, wasn't satisfied by that explanation, he indicated in a blog post.

His analysis showed that IE9 is using dead-code elimination for some tasks involving a number of mathematical operators. "Curiously, the IE9 team has chosen not to eliminate more expensive operations such as multiplication or division. The set of analyzed operators do happen to be the ones that appear in the math-cordic benchmark," Sayre said.

Then came his damning conclusion, essentially that Microsoft had tuned Chakra just for a good SunSpider result.

"I don't think it's possible to consider IE9's implementation of dead code elimination to be a serious general-purpose optimization. It seems clear that it hasn't been tested on anything other than SunSpider," Sayre said.

Added Mike Shaver, Mozilla's vice president of engineering, in a tweet, "Disappointed in it's based on real-world' DCE [dead code elimination] nonsense from MSFT; wish they'd just admitted starting with SS [SunSpider] benchmark and making mistakes." And he offered a challenge: "Treasure hunt: find a pattern in any site on the internet that is eliminated by the #IE9 DCE, other than math-cordic!"

Benchmark engineering is an ages-old practice in the computer industry. For example, with prominent server database test, TPC-C, server makers would attach absurdly large numbers of hard drives in an effort to achieve high scores. The design represent real-world constraints customers faced.

Alternatives to SunSpider exist--for example, Google's V8 benchmark and Mozilla's new Kraken. But even those are narrow tests that don't exercise many aspects of browser performance such as page-loading speed, graphics, and user interface responsiveness.

The situation frustrates Chris Wilson, who for years worked on IE but recently joined Google to become Web platform advocate and developer at Google.

Wilson "would rather see collaboration across vendors and developers on suite of tests that span DOM [the document object model interface browsers use to interact with Web pages] as well as pure JS [JavaScript]," he said in a tweet. He added that he prefer people "back away" from a single numeric JavaScript benchmark score as a performance claim, "frankly, because it has little to do with real-world user performance."