Google Caffeine: Newsier, by a nose

Google is running a public experiment of a new search technology, code-named Caffeine (link takes you to the beta search engine). There are visible changes in search results, but they are so subtle that you likely won't notice differences between the old Google and the beta Google unless you run queries side by side.

Caffeine is a test version of the Google indexing and ranking system, not an upgrade of the Google user interface. For many queries that I ran, the top entries were identical, and result pages look the same.

For example, when searching for "Google Caffeine," the top 10 results were the same with one minor exception: The fourth and fifth results were reversed between today's Google and Caffeine. Results on following search pages diverged more, but are obviously a lot less important, since most people use the results from the first page of search hits.

In searching for a news item, "Great America roller coaster," the results were different this morning, with the old Google neglecting Monday's news about people getting stranded 40 feet up when the coaster jammed. In later testing, the results converged toward the same set of hits, and both Google and Caffeine had the news item at the top of the page.

In a broader test, "Health Care Obama," the top two results were the same. The third result was a news item on both versions, although it was from CNN on the current Google and from The New York Times on Caffeine. The Caffeine results diverged substantially from current Google after that, showing results from less well-known sources like TheHealthCareBlog and RealClearPolitics. On the current Google, only one result on the first page was from a non-mainstream news site.

In a query for product information, "Nikon d300 review," results differed between the old and beta Google more noticeably, although I wouldn't say that the new beta Google was any better than the old Google.

Google execs say Caffeine searches execute more quickly, but I noticed no speed difference in anecdotal testing. Caffeine reports different numbers of results between the two versions, but since most people don't go beyond the first page or two of results for most queries, that number has no real bearing on the quality of hits.

One thing I am hoping to see on Google Search eventually, but did not see in Caffeine, is more sensitivity to real-time results from sources like Twitter. Bing has a very slight advantage in this regard, pulling in recent Twitter posts from some users if you search for their twitter handle (e.g, "@rafe"). No standard search engines have yet figured out how to incorporate news that comes in via Twitter, however. See also: Facebook gets Twitter-like search.

The bottom line: While it's good to see Google working to improve search quality, the changes visible in the Caffeine test so far are more curious than eye-opening. Currently, Google is a very good search engine. Caffeine might be slightly better in some cases, but it does not yet appear to be a breakthrough in quality.

For more on what's behind Caffeine, read Google's Caffeine: A jolt to search rankings?