Doing a number on Web site traffic

"We had a hard case telling people how big we were and how many unique visitors we had. They didn't believe us. We didn't show up on people's radars," said Jason Feffer, former vice president of operations at MySpace who is now president and chief executive at opinion-forums site SodaHead.com. "It was frustrating that advertisers wanting to advertise to that demographic would go to Friendster when we were 10 times bigger than them."

That changed fairly quickly for MySpace. But the problem persists for many other Web sites competing for advertising dollars in an environment where the difference between being ranked fourth or fifth in a category can directly impact the bottom line. Being undercounted by the audience measurement firms costs publishers advertising deals and threatens their ad-based businesses.

For instance, MLB.com, the site for Major League Baseball, has repeatedly complained that audience figures reported for the site by Nielsen Online are much lower than those shown by MLB.com's internal data. League subsidiary MLB Advanced Media has asked Nielsen Online to stop reporting on its traffic because of the discrepancy.

How big of a difference is there? Nielsen Online reported MLB.com with about 6.2 million unique visitors in December while MLB.com recorded 19.4 million unique visitors during that month. For 2007, Nielsen Online's figures were one quarter those recorded by MLB.com, according to MLB.com.

"That's like a pilot of a plane landing in the wrong continent," says Bob Bowman, chief executive of MLB Advanced Media. "There are probably some advertisers or agencies that actually believe the numbers and are steering their ad buys elsewhere."

Nielsen Online is responding to MLB.com's concerns, said Mainak Mazumdar, vice president of measurement science and research service MegaPanel at Nielsen Online. "The best way to address this is to understand what the methodology is and the audit will show that."

An old but tenacious conflict
This isn't a new problem, but, it is hoped, it is one that will be addressed this year. The Interactive Advertising Bureau, which represents more than 300 Web publishers, including CNET Networks, asked ComScore and Nielsen Online to submit to audits by the Media Rating Council, a nonprofit group that approves audience ratings systems. In a letter sent last April to the heads of the two companies, IAB Chief Executive Randall Rothenberg complained that they were "still relying on panels, a media-measurement technique invented for the radio industry exactly seven decades ago, to quantify the Internet."

"To persist in using panels that undercount or ignore the diverse populations that are the future of consumer marketing is to deny marketers the insights they need to build their businesses," Rothenberg wrote. "And it certainly appears to us as if they are being undercounted or disregarded, for our members' server logs continue to diverge starkly from your companies' sample-based assessments, by 2x to 3x magnitudes, in some cases far beyond any legitimate margin of sampling error."

Nielsen Online and ComScore say the audits are on schedule to be completed by the end of the year, said David Doty, a marketing and public relations senior vice president at IAB. Meanwhile, the IAB is also working on guidelines for publishers for measuring their own traffic. Those guidelines should come out in the second quarter, Doty said.

The IAB has become a "referee to an increasingly ugly street brawl," said Tim Hanlon, executive vice president at Denuo, the media futures arm of ad firm Publicis Group.

Third-party audience measurement provides a measurement independent of Web publishers who have financial incentive to inflate their numbers to ad buyers.

The numbers will never be exactly the same because the methods of measurement are different and they measure different things. ComScore and Nielsen Online use data gleaned from samples of Internet users who have monitoring software installed on their computers, while publishers are looking at their actual server log data. Panels give a glimpse into who is visiting a site; server data shows exact visits to the site.

To some, the debate comes down to which is more accurate--a sample panel whose stats are projected out to represent a wider population, as with television and radio panels, or server data that shows the number of visits to a site, the number of page views, and the time spent?

Publishers complain that the ComScore and Nielsen Online panels have inadequate representation from international and Macintosh users and that panel data shortchanges Web sites that have AJAX and Flash content, such as those with videos and widgets, which changes dynamically without the page being refreshed. They also argue that panel data doesn't accurately measure peoples' surfing activity at work and ignores cell phone-only households.

The last point hits home for sites like MLB.com, which gets a lot of traffic from people sneaking peeks on sports scores and news during breaks at the office when they are supposed to be working.

Representatives at Nielsen Online and ComScore say that server log data will always be inflated because one person surfing on different computers from home and work can be recorded as two people, just like someone using two different browsers visiting the same site can. In addition, they point out that some people delete cookies (user data stored on their Web browser), which allows for one person to be counted as multiple users.

Server log data shows information for only one site, while panel data shows traffic trends across the Web, said Josh Chasin, ComScore chief research officer. Advertisers and agencies want to see how traffic is at comparable sites before they choose where to buy ad space, he said.

But then there are also wild variations between ComScore and Nielsen Online's panel data, which further muddies the waters.

Better information for advertisers
The IAB doesn't expect the audits and guidelines to neatly resolve the problem, but it is expected to provide more transparency as to exactly how the figures are reached so marketers can make more informed decisions about where they want to advertise, said the IAB's Doty.

Doty suggested that both types of data could be used to come up with the most accurate and comprehensive measurement. "I could see server-side data being about (traffic) numbers and the panel side being about measuring behavior, demographics, and ethnographics," he said. "There can be a convergence."

He wasn't alone in making that prediction. Bill Cook, senior vice president of research and standards at the Advertising Research Foundation, said invisible tags embedded in Web pages that track user activity could be used in conjunction with user panels.

Google Analytics, which uses Web tags, tracks closely to internal Web publisher server data and could fill in the gap between the publisher and third-party statistics. "The IAB is saying these guys need to be up to date with what's happening on the Web if they are going to be reporting on it," said Brett Crosby, group product marketing manager for Google Analytics.

Nielsen Online has developed a new product, called VideoCensus, that incorporates both server log data and panel measurements, which also use embedded tags for measuring video streams.

A company called Quantcast also could offer an answer. The company offers rankings and figures using a hybrid approach that combines traffic data directly from 22,000 publishers with panel-based data from more than one million people, including survey-based information.

"People are looking for an alternative and a better data set," said Adam Gerber, Quantcast chief marketing officer.

Another company, Compete, is trying to take on ComScore and Nielsen Online by using data from a panel of consumers who are paid to take surveys, and from traffic data from ISPs. However, no data is directly gathered from publishers, as it is with Quantcast.

In the meantime, everyone is looking to the IAB to call the play.

"The solution is to come up with some sort of plan, like they did with magazines, to audit the true circulation or true traffic," said MLB's Bowman. "We're beseeching them to do it right."