Life beyond JavaScript: Google's abuzz over RoboHornet test

Google last night unveiled an early version of RoboHornet, a general-purpose browser speed test that company engineers hope will shed light on something besides just JavaScript.

But Microsoft, maker of Internet Explorer, has derided the tests as not useful.

There are plenty of JavaScript speed tests, Google's new Octane among them, and for good reason: the programming language is used to turn static Web pages into interactive Web apps. But there's more to fast browsing, and Google hopes others will get involved to flesh out RoboHornet with a full suite of tests.

"It's a living, dynamic benchmark that aims to use the collective efforts of the Web development community and ultimately get browser vendors to fix real-world performance pain points," said RoboHornet leader Alex Komoroske in a Google+ post.

Paul Irish, another Google engineer, said RoboHornet will exercise browser facets including scrolling; Canvas, a 2D area for graphics; the DOM (Document Object Model), a means to let JavaScript programs modify the appearance of a Web page; and SVG (Scalable Vector Graphics), a format suited to graphics such as logos that can be resized easily.

Various subsets of the full test suite can be selected to exercise particular areas. To run the alpha version of RoboHornet, you'll have to disable any blocking of pop-up windows for the site.

RoboHornet is an open-source project on GitHub, and Google is seeking involvement from others.

Additions to the benchmark code itself have come from Google staff including Erik Arvidsson and Paul Lewis, but the RoboHornet committee includes members from Mozilla, Facebook, and Sencha. And it's not just about browser makers, but also those who build Web sites; other representatives come from SmugMug and Google Spreadsheets.

John-David Dalton, a new Microsoft employee, has worked on the benchmark and was on the RoboHornet committee, but he no longer is participating, Microsoft said.

Komoroske said the benchmark as it stands today is in its early stages: "RoboHornet is still in a very early alpha state, but that's where you come in. It's up to you to propose and vote for performance issues you care about, helping shape the future of the benchmark and effectively defining the areas that browser vendors will invest in making run faster!"

In a blog post today, Microsoft pooh-poohed RoboHornet.

"Members of our engineering team took a look at the benchmark and found that RoboHornet isn't all that representative of the performance users might encounter on real-world sites. Like all micro-benchmarks, RoboHornet is a lab test that only focuses on specific aspects of browser performance," said Roger Capriotti, director of Internet Explorer marketing.

The real Web should be used to measure performance, he said: "Internet Explorer is built from the ground up to perform incredibly well on Web sites, not lab micro-benchmarks."

In an update, Microsoft elaborated on its position regarding benchmarks:

The patterns inside the benchmark are corner case and not representative of real world performance. For example, one RoboHornet test draws a line with 2,500 vertices into a canvas 10,000 times. According to product instrumentation, 99.884+ percent of the canvas lines drawn in the world have only 2 points, and very few canvas applications draw more than a couple lines per frame. RoboHornet could not be farther away from measuring real-world patterns. At the same time, if they change RoboHornet to measure the real-world scenarios, the difference will not be perceivable. Synthetic benchmarks are never the right choice for the customer.

Google declined to comment, saying the project is an "independent benchmark," but did pass along a statement from Ryan Grove, a steward of RoboHornet and Web developer at photo-sharing site SmugMug:

"Feedback like this is extremely valuable in this early alpha phase of the project, but RoboHornet aims to be an independent benchmark driven by Web developers, not a marketing tool for browser vendors. We're eager to work with people from every part of the Web ecosystem to accomplish this goal."

Chrome programmers hardly ignore real-world Web sites. Automated page cycler testing checks each new change made to Chrome to see if page-load times slow down.

Improving Web performance is important. For one thing, people spend a lot of time in their browsers, and they tend to be more active when Web sites are responsive. For companies like Google, that means more searches, more search advertising opportunities, and therefore more revenue.

For another, better performance means Web apps can compete better against native apps for iOS, Android, Windows, OS X, or other operating systems. That helps programmers rely more on the universal Web foundation rather than crafting multiple versions of their software for each device.

Update, 9:17 a.m. PT: Adds information on Microsoft blog post criticizing RoboHornet. Updated 2:01 p.m. PT to reflect that John-David Dalton is a new Microsoft employee and is no longer on the RoboHornet committee and to add further Microsoft comment, and to add comment from the RoboHornet steward.