Net privacy and the myth of self-regulation

Consider the case of ComScore Networks, one of several new online market research companies to emerge within the last year providing access to pure "clickstream" data on Web surfers. The company has signed up about 1.5 million members by offering software to speed up Web performance. In exchange, it demands complete access to all of the information that streams over the browser, peering in on everything from Web pages visited to secure transactions and even personal banking records.

"This is like Carnivore," says Richard Smith, a privacy expert at the Privacy Foundation, referring to controversial eavesdropping technology developed by the FBI. "The government wants the same sort of information under new anti-terrorist bills."

ComScore isn't alone. A handful of other market researchers including Plurimus and Compete offer similar data to their customers through partnerships with Internet service providers and companies that quietly offer data-scraping services along with free software downloads.

All of these companies insist that their services are on the up-and-up and meet with current generally accepted privacy standards.

In a recent interview, ComScore Chairman Gian Fuloni told me, "We fully disclose how we use our data," adding that the company's privacy policy was vetted by lawyers at Wilson Sonsini, one of the country's pre-eminent technology law firms.

Compete's vice president of business development, Reed Cundiss, also defended his company's commitment to consumer privacy. He said all of its partners must have the ability in their privacy policy to track and resell clickstream data. "We contractually obligate our partners not to give us any personally identifiable information," he said.

Plurimus did not return phone calls seeking comment. On its Web site, the company says its data "is scrubbed clean of any individual identity by an independent third party...The third-party anonymization process ensures the complete privacy of individual users."

If these companies are in full compliance with current privacy regulations--and I do not doubt that they are--then we need some new regulations.

In order to be effective, disclosure must be matched with publicity. But companies often appear eager to have their painfully crafted policies ignored and left unread. Disclosure in privacy policies all too often looks like confession, namely a secretive attempt to gain absolution for one's sins.

ComScore, Plurimus and Compete operate slightly differently, but all have access to the same kind of raw data.

Plurimus gets clickstream data from ISPs. On its Web site, the company says it "assimilates data" from 3.5 million online subscribers. It does not disclose its partners.

Compete's Cundiss said the company gets its data from more than 9 million people through ISPs and so-called browser companions--software that hooks up to a browser and records usage. Compete currently has 10 partners, all of which have privacy policies that allow for the collection and sale of clickstream data, according to Cundiss. He said the company does not disclose its partners due to contractual restrictions.

ComScore works differently but achieves the same result. The company runs a script that redirects all of the data that goes into your browser through its own computers, much like an ISP, and caches data so it loads quicker.

How invasive is this? ComScore Chairman Fulgoni said that his company has examined the online banking records of some of its customers in order to verify household income information provided during the sign-up process.

Privacy questions aside, these companies promise significantly better Web stats than have generally been available before. Current audience measurement leaders Nielsen/NetRatings and Jupiter Media Metrix have long been criticized for providing incomplete and contradictory data, in part due to the relatively small sizes of their panels.

By boosting the research pool to millions of people and offering wider geographic and demographic samples, the new challengers believe they can offer more comprehensive data that is less subject to statistical skews.

Potential clients such as Yahoo say they are watching these types of services, although they are not yet convinced of all the claims. Asked about ComScore, the company's director of global market research, Anke Audenaert, says Yahoo is "looking at this, but we don't know enough yet" to make a firm determination.

While the promise of better Web statistics is compelling, these services and their partners do not appear to offer consumers a level playing field, even if they do play by the rules. Nielsen and Jupiter pay their panels for the right to peek in on their behavior, but ComScore effectively gets the same information for free, offering a carrot that is not very compelling.

After testing ComScore's surfing enhancement for a week on a broadband connection at work, I can't say I noticed a difference in Web performance. It's possible that dial-up users will see a bigger benefit, although the tradeoff would still appear to favor the company and not consumers. That could be a problem for ComScore long term, especially as more consumers sign up for broadband service.

Plurimis and Compete, which rely on ISP data, are even more troubling, given their failure to publish the identities of their partners. Barring new disclosure rules, it's likely that these kinds of data-sharing deals will quietly become more widespread, especially now that the Federal Trade Commission has switched its stance on privacy under the leadership of Timothy Muris and no longer backs new laws to guarantee consumer privacy rights.

That means we may have to rely on nothing more than the self-interest of corporations to provide more information voluntarily. So far, at least, that's not a strategy the industry has voluntarily warmed up to. Companies sell the data, but they are not interested in advertising that they're doing it. The reason is obvious: People don't like having someone looking over their shoulders online.