It began innocently enough--a promotional blog post about Microsoft's newest version of Internet Explorer 9. But in less than a week, it became an illustration of just how rapidly misinformation moves through a hot and increasingly important corner of the software market.
The browser market was already competitive a year ago, but the arrival of IE9, currently in beta testing, has added even more energy to the competition. That's because Microsoft has placed support for a host of modern Web technologies front and center, transforming IE from a drag on the Internet into an ally in developers' efforts to bring everyone a more powerful Web.
IE6, nearly a decade old but still widely used, has saddled Microsoft with a reputation for browser neglect. So what could be more eye-catching than the news that IE9 not only is back in the game, but that it leapfrogged the competitors?
Alas, as with many good stories, the truth turns out to be less dramatic and but harder to find. The IE9-is-better idea triggered teeth-gnashing on a Web standard mailing list and a hastily published disclaimer by the World Wide Web Consortium that's developing many of those Web standards.
The episode shows the difficulties of keeping track of a browser market in the throes of breakneck change. Even those creating the yardsticks to measure the progress struggle to keep up.
IE9 and standards support
What happened with IE9? Last week, . At the same time, browser makers had begun submitting test results to a new W3C suite of HTML5 standards compliance tests.
When Microsoft discussed the new IE9 version on its IEBlog, the company included a link to the tests, calling them "an early version of the W3C's Official HTML5 Test Suite Conformance Results."
Shortly after came the headlines: "W3C Says IE9 Is Currently the Most HTML5-Compatible Browser" from Slashdot. "IE9 Outperforms Other Browsers for HTML5 Compliance" from ReadWriteWeb. "IE 9 Beats Chrome, Firefox, and Safari on Official HTML 5 Test" from Windows IT Pro.
Then came the alarmed reaction by those creating the standards. The W3C page, it turned out, was preliminary at best.
"This test suite is vastly incomplete. Publishing unverified results of a vastly incomplete test suite without a big fat warning is extremely silly. Why was this done?" said Anne van Kesteren, an Opera developer who works on standards issues, in a mailing list posting Tuesday.
Added Ian Hickson, the Google employee who's editing the HTML5 specification, "I agree with Anne that it's rather pointless to be publishing results for this test suite. Realistically speaking the test suite isn't even 0.1 percent complete yet."
Shortly after, the disclaimer arrived on the Web page with the results: "The HTML5 test suite is still being developed. The number of tests and the results on these tests will change. The results in this document may be updated, replaced, or obsoleted by others documents at any time. It is inappropriate to cite those results as other than work in progress and unstable."
In a presentation this week, leaders of the HTML5 effort said there are 215 tests included in the suite right now, with more than 600 on the way, and that the group needs "a lot more tests!"
So declaring IE9 the winner on the test at this stage is something like saying England is better than the United States because it's got lusher lawns. The full range of tests aren't yet available.
Browser makers are scrambling to attract and retain users. As new features emerge, a golden age of Web demos has begun with helicopter games, ruffling curtains, exploding videos, computer aquariums, virtual reality, jiggling balls, and blooming flowers.
But what if you want something more authoritative? A natural way to get a handle on the chaos, of course, is to measure whose browser can run Web-based programs the fastest, or build Web pages to tally who supports what new features. Such seemingly neutral areas are where the problems are cropping up.
Now, though, the browser world has begun moving to new tests. The only problem with calling winners in the browser race, though, is that the new tests really aren't settled down yet. The W3C tests are just one illustration.
SunSpider stemmed from the WebKit project behind Apple's Safari browser and, more recently, Google's Chrome and any number of mobile browsers. Dissatisfaction is brewing with what's now a relatively elderly test, however.
"Because of...all the progress each browser vendor has made over the last several years, SunSpider is no longer particularly useful as a JS benchmark," Asa Dotzler in a blog post last week. "This is kind of obvious when you see that all of the top scores are pretty much tied. One one-hundredth of a second (across 26 tests) separates the slow from the fast, and that's just not particularly meaningful."
Benchmarking is ever an imperfect science, to be sure. It's hard to measure the full breadth of computing chores, hard to weight tests toward the important components, hard to account for different hardware and network constraints, hard to factor in the latest technology. To draw a parallel from the auto industry, miles-per-gallon ratings for cars are reasonably useful--until.
Unfortunately, it'll take time for the testing community to converge on new tests, even assuming the standards settle down enough to create meaningfully complete tests in the first place.
So for the time being, we'll all have to live with inconvenient ambiguities in browser testing. Sounds like good practice for the real world.