Sizing up the search competition

In the search engine duel between Google and Overture Services, it equals bragging rights.

Search kingpin Overture last week touted the newly extended index of its subsidiary AlltheWeb as larger than that of Google, the Web's undisputed search leader. To answer visitor queries, AlltheWeb mines nearly 3.2 billion unique pages, it said Thursday of last week--roughly 68 million more pages than Google listed in its index at that time.

But since then, Mountain View, Calif.-based Google has quietly leaped ahead again, expanding its database to more than 3.3 billion Web documents by Thursday this week, according to its home page. A Google representative confirmed the change.

"Google raised the number on its home page to accurately reflect the number of Web pages it offers consumers," a representative wrote in an e-mail. The search company's worldwide index now includes 3.3 billion Web documents, 800 million Usenet pages and 400 million images.

The contest is only the latest between Google and Overture and reflects sharp competition in the industry. Search providers have long played copycat on index size and features, but the stakes have gone up, as Web search has become a big moneymaker for Web portals and search specialists alike. The search ad business is expected to ring up sales of $2 billion this year, according to research.

Google and Overture are the top two Web search companies, in serving both algorithmic and commercial search results to consumers' queries, researchers say. The business of selling targeted advertising links within search results has simultaneously transformed the industry and revived the online ad market.

That has prompted consolidation: Yahoo recently said it would acquire Pasadena, Calif.-based Overture for about $1.7 billion. And in the last year, Overture, a commercial search pioneer, bought the Web search assets of AltaVista and Fast Search & Transfer (including AlltheWeb), to compete with Google in its front yard.

The revenue up for grabs has also set off a power war over search distribution partners. For example, Google has won America Online business away from Overture; and Overture has won the hand of Freeserve away from Google.

How big is size?
Index size is just another measure of prominence, some analysts say. The number of documents in a database is one of the three pillars of a search formula, alongside relevancy and freshness of results, related to a query. For example, Safa Rashtchy, an analyst at U.S. Bancorp Piper Jaffray, downplays the importance of size in search.

"By itself, the index size isn't that important," said Rashtchy, who added that to carry weight, a search engine should have between 2 billion and 3 billion documents. "What makes a bigger difference is the relevancy algorithm--it has to be fresh, it has to be relevant."

Google has endeared itself to millions of Web surfers through the relevancy of its search results. Rashtchy said among all players, Google is still the leader in that area, with Yahoo-owned Inktomi and AlltheWeb closing in. As a result, Fast's AlltheWeb (Overture), among others, is focusing on relevance, too.

AlltheWeb has not only added 1 billion documents to its index, it has also improved the relevance algorithms it uses for ranking Web pages by their authority, according to Overture. Tim Mayer, Overture's vice president of Web search products, would not give specifics on the relevancy tweaks but said visitors will see vast improvements in coming weeks.

"This is a demonstration of Overture investing in the acquisitions that it made previously," Mayer said.

Some search executives say size is a hollow claim. Paul Gardi, head of search for AskJeeves' Teoma, has said it's difficult to find many high-quality documents on the Web to search, because many of them are bogus pages spammers set up. What's more important is developing technology to efficiently catalog Web pages by their importance and relevance to communities.

Still, AlltheWeb and Google have kept challenging each other on index size. The two companies have leapfrogged each other in recent years, making regular additions to their searchable databases. For example, in December 2001, Google raised its index to 2 billion documents, with another billion pages of images and newsgroups.

At that time, Google co-founder Larry Page said: "To search our collection of 3 billion documents by hand, it would take 5,707 years, searching 24 hours per day, at one minute per document. With Google, it takes less than a second."

Today, the top search contenders are still ensuring that their document size is up to snuff. When asked if it had upped its document size as a result of AlltheWeb's update, a Google representative replied: "No. Google updates its homepage number on a periodic basis throughout the year, and we've planned this increase for months."