Bing is better than Google? Bunkum, says Freakonomics

Googling is one of those habits.

You know, like flossing in the living room or changing your underpants every other day.

Yet Microsoft's search engine has Binged on and on that it is, in fact, better. Its own Web site insists that real people prefer Bing's results by a ratio of almost 2 to 1.

Such a claim tends to invite skepticism.

The vast minds at Freakonomics decided to take their skepticism to another level -- the one of actually trying to replicate the Bing results.

Their expectations were that people might not be able to tell the difference between the two searches. Their findings were quite searching.

Using a similar sample size to Microsoft's own claimed research (around 1,000), Freakonomics enlisted the help of four Yale law students -- people of obviously unimpeachable integrity.

The next step was, just as with the Bing It On Challenge, to perform five searches and see which results respondents preferred.

In the experiment, Freakonomics found that 53 percent chose the Google results, 41 percent chose Microsoft's, and 6 percent just couldn't make their minds up.

This might be summarized as not exactly 2 to 1. And not exactly in favor of Bing.

Freakonomics didn't freak out. It tried some other experiments to see if it could find parameters along which there was a clear preference for Bing. It found none. It found a clear preference for Google or, at best, a tie.

In conclusion, it described Microsoft's claims as "fishy."

I contacted Microsoft to ask whether it, too, smelled a little rotting clam in its claims.

Bing's behavioral scientist, Matt Wallaert, offered this statement: "The professor's analysis is flawed and based on an incomplete understanding of both the claims and the Challenge. The Bing It On claim is 100 percent accurate, and we're glad to see we've nudged Google into improving their results. Bing It On is intended to be a lightweight way to challenge people's assumptions about which search engine actually provides the best results. Given our share gains, it's clear that people are recognizing our quality and unique approach to what has been a relatively static space dominated by a single service."

On reading this, I found myself relatively static.

What about the Freakonomics analysis was flawed? What didn't it understand about Microsoft's claims? And what makes the challenge "lightweight?" Is this scientific language "not entirely serious?" Or perhaps even "not at all scientific?"

I confess that I have twice tried the Bing It On challenge. Once when the site came out. And once outside a newly opened Microsoft store.

Both times, Google won.

Does that mean I am abnormal? Or might it mean that the Freakonomics research reflects a greater reality than does Bing's own?

Updated, 2:41 p.m. PT: Bing's Wallaert offered me some further elucidation, which almost smacked of irritation. He said:

Professor Ayres seems a bit confused about the concept of statistical power (he claims that our sample of 1,000 doesn't have sufficient power, yet gets significant results with a much smaller sample size of 300) but more fundamentally, he doesn't seem to understand the important ethical differences between a controlled experiment and a marketing Web page. Specifically, Ayres is bothered that we don't release the data from the Bing It On site on how many times people choose Bing over Google. The answer here is pretty simple: We don't release it because we don't track it. Microsoft takes a pretty strong stance on privacy and unlike in an experiment, where people give informed consent to having their results tracked and used, people who come to BingItOn.com are not agreeing to participate in research; they're coming to a marketing Web site. So not only would it be unethical to collect data from the site to make the claims, it would be highly unscientific to make a claim based on an uncontrolled environment that has no guarantee of a representative sample, where people are free to try and game it one way or another, and that is covered in Bing branding. Perhaps Ayres simply assumed that the ethics of data collection and the science of search aren't important; at Bing, we believe they are.

Marketing vs. Research. You take your pick.