Why Google puts privacy second

European regulators sent shock-waves through the search engine industry earlier this week, when they proposed significantly tighter rules for logging data. If the EU adopts the proposed rules, Google, Yahoo and Microsoft will have to significantly reduce the amount of time they keep identifying search logs, and will have to start treating IP addresses as personally identifiable data -- something that Google has been particularly vocal against.

Google has recently engaged in a major public relations effort to try and make a credible argument for keeping log data. The company has trotted out respected employee researchers to try and make the case that deleting such data will hurt search results. When all of their claims are analyzed, however, one thing becomes clear: It's all about the money (and the clicks).

Google has a genuine need to retain detailed log information on one kind of user: Those who click on ads. However, in order to avoid creating a situation where only clickers lose their privacy, the company logs data on all searchers instead. That is, the privacy of millions is threatened, to protect the incentive for users to click on ads.

Over the last few months, a number of Google's engineers have issued public statements on the company's public policy blog to defend its much criticized log data retention policies. The company claims that the data can be used to hunt down malware, to catch people defrauding its advertising system, and can be used to improve search results, especially for localized results.

Google claims that accurate logging data can improve localized searches. This data is then used to intelligently respond to searches, such that a search for "GM" will result in General Motors related information for an American search user, yet someone in France be presented with information on "Guerre Mondiale" (World War).

What Google has done here, is attempt to muddy the waters of the debate. Yes, accurate logging data improves localized searches. However, the company does not need to retain the exact network address (known as an IP address) of each and every search. Instead of tracking my searches by my network address, 129.53.136.23, the company could instead log that I came from San Francisco, California. That, in itself, would be more than enough information in order to help it localize and improve search results.

Avoiding disincentives

Of all the excuses that Google's puppets have presented for retaining search logs, there is only 1 case where Google actually has a legitimate need to store information that identifies the individual user, and network address: advertising clicks.

Google is an advertising company first, and a search engine second. Sometimes, we forget this, but Google has a lot of bills to pay. After all, those free meals and massages for employees have to be paid for somehow.

Google displays text advertisements on all of its web search results pages. Advertisers, for the most part, pay per click. That is, every time a user clicks on one of the ads, Google charges an advertiser a few cents (or dollars, depending on the search term). Because of the amounts of money at play, this tends to attract criminals wishing to defraud the system. Thus, it is not terribly surprising that Google wishes to retain information on the user who clicked.

What is most interesting to note though, is that if a user does not click on one of Google's web advertisements, the only credible reason for retaining detailed search information becomes moot. If a user doesn't click, they can't possibly be engaged in fraud, and thus there is no reason to retain identifying information on the user's search.

Were Google to institute an information needs based logging policy, it would find itself in a curious position: users who clicked on advertisements would have detailed logs retained for months, if not years, while users who didn't click on ads would quickly have any identifying information scrubbed from logs, and replaced with more generalized info.

The obvious problem with such a scenario would be that of incentives, especially once the policy was made public. Users would lose their privacy each time they clicked on an advertisement. Unfortunately for the company, this is exactly the wrong kind of message to send. It wants to encourage users to click on its text ads, not to provide incentives for customers to skip them.

Thus, in order to not create that situation, and to avoid the disincentive to click on ads, Google logs data on every search, by every user. And because of this, we all suffer -- even those users who never even see ads, because they use technologies like AdBlockPlus and CustomizeGoogle.

Disclaimer: In 2006, worked as a summer intern in Google's click fraud team. Shuman Ghosemajumder, Google's "Business Product Manager for Trust & Safety" and the person claiming that search logs prevent fraud worked in the same team.

None of the information in this blog post involves confidential company information.

I was awarded a Google fellowship in both 2006 and 2007, for $5000 each time. Finally, I just returned from a Scholar Retreat in San Francisco, which the company paid for.