The Google gods

By Stefanie Olsen
Staff Writer, CNET News.com
October 31, 2002, 4:00 a.m. PT

Patrick Ahern has witnessed the power of Google--and the difficulties of trying to do business without it.

Data Recovery Group, where he is president, would typically come up around the fourth listing on Google's popular search engine last year. Then in January, when Google removed the company from its listings without explanation, Data Recovery saw a 30 percent drop in business.

"When you're No. 4 that plays well; when you fall off, you tend to lose phone traffic. And if you don't have the right relationship with Google to find out what you could have done wrong, you're out of luck," Ahern said, noting that this can have a dangerous domino effect. "If you're not ranked in Google, Yahoo won't list you. It's incestuous."

In the dot-com shakeout, Google has not only survived but reigns supreme. Web surfers have flocked to the service, effectively voting it the best search engine around. So powerful has Google become that many companies view it as the Web itself: If you're not listed on its indexes, they say, you might as well not exist. And if you don't advertise on Google or otherwise curry favor, critics add, you may never find out what it takes to get a prominent listing.

		News.Commentary The search goes on Most companies, Forrester says, already own a search engine--one that doesn't work.

"So many people are dependent on Google's free editorial traffic that it's like food out of their mouths to lose ranking," said Danny Sullivan, who runs Searchenginewatch.com. "Search engines are not in the business of supporting people's companies. But if they are going to provide editorial, they need to provide support. These are some of the issues they face."

Only a few years ago, when traffic was more evenly distributed among search services, Web sites clamored to get free listings from familiar names such as Lycos and AltaVista. But the dot-com collapse led portals and search engines to sell sponsored links and license editorial results from a third party--a market that Google, based in Mountain View, Calif., has come to dominate since its founding in 1998.

Leading portals and Internet service providers such as Yahoo, America Online and EarthLink have turned to Google to power their searches because of its simple, straightforward style and consistency for serving up germane results. Such companies typically mix Google's results with listings of their own to maintain some autonomy and uniqueness on the Web. It is these deals that have catapulted Google ahead of competitors and sent Web site operators scrambling for prominent listings in its search results.

"It's not like we've put all our eggs in one basket--it's just that there is no other basket," said Greg Boser, president of Web marketing consultancy WebGuerrilla, which helps companies improve their visibility in search engines.

In many ways, this is no exaggeration. Google averages about 15 million visitor hours each month, compared with Yahoo search at 6 million hours, according to Sullivan. Search hours are calculated by factoring the number of site visitors by the average number of minutes each spends at the site.

Yahoo recently renewed its partnership with Google for an unspecified term, a deal some doubted would happen because of the search darling's growing threat. Yahoo has even given Google more editorial voice and played down its own directory results. Search engine experts say that in comparison tests, a search on Google and on Yahoo varies little for popular terms like Britney Spears or NFL.

Google software engineer Matt Cutts said that the search engine business is healthier and more competitive than ever, with many niche providers and international forces. "What we worry about is providing the best results to users; we don't worry about market share," he said. "That will all work itself out."

Exactly how the system works itself out, however, is what worries people in the industry. Boser and others are concerned that paid advertisers get crucial advice that maximizes their editorial listings.

"As you're spending money with them as an advertiser, that spending does not buy you position; that spending buys you advice on how to get position--a byproduct of the advertising sport," said Data Recovery's Ahern, a client of WebGuerrilla.

Ahern learned that the term "data recovery" was a valuable commodity in paid search, worth as much as $10 per click. As a result, he was forced to spend several thousand dollars a month on pay-per-click advertising on Google and rival Overture to compensate for losing up to 85 percent of his traffic earlier this year. Ahern no longer buys Google ads because his top position has been restored, but it has been an expensive process.

The depth of concern about the search industry's practices was made clear at a conference earlier this year, where participants stressed the need for policies to protect Web sites dependent on the engines.

Among the proposals were calls for fair and consistent spam-reporting policies in which Google and others reply to all complaints, not just those from advertisers. Also suggested were standards for a formal review system that helped sites understand why they're not listed and thereby give them information necessary to improve their chances.

Craig Silverstein, Google's chief technology officer, denies that advertisers get preferential treatment in its editorial listings. On the issue of information about changes in rankings, he said it is impractical to provide support for everyone, considering that the company indexes nearly 2.1 billion pages. But Google is examining its system, well aware that growing criticism could damage its credibility with the public at large.

In an effort to beat Google at its own game, some marketers have adopted guerrilla tactics that try to manipulate search rankings. This has led to a proliferation of "link farms"--elaborate linking schemes designed to manipulate one of Google's only publicized algorithms, PageRank, which factors a site's popularity based on the number of Web pages that link to that site.

Many industry executives speculate that Google changed its search algorithm in September to combat link farms, which caused a shift in listings and a lower number of documents served up for any given query. The changes have thrown Webmasters into a frenzy, as shown in thousands of messages on industry sites such as SearchEngineForums.

Daniel Brandt, who publishes the watchdog site Google-watch.org, complains that PageRank is the equivalent of a popularity contest that favors major, established Web sites. "We don't know which comes first--whether Google is reflecting popularity or if it's creating popularity," he said.

Silverstein asserts that Google does not provide preferential treatment to advertisers in text results or target smaller sites for delisting. Google alters its algorithms all the time to improve search quality, he said, and sites might have seen changes recently because of a "fresh crawl" that updates up to 3 million documents more regularly.

As for tactics such as spam and link farms, Google says those just make accurate searches more difficult for surfers. That, in turn, could affect Google's credibility at a time when trust is more important to the company than ever.

"There's a constant battle between abusive technologies and providing relevant search results," Silverstein said. "When you use something as a tool and you don't have control over it, that's an issue, an issue of trust. You need to be able to trust us that we are acting in the best interests of our user."

Others dismiss much of the complaints about Google as conspiracy theory, saying it should not be required to baby-sit sites that have simply fallen victim to the Internet's evolution.

"Google is a great scapegoat when people get mad. But it's about protecting relevancy," said Jessie Stricchiola, president of Alchemist Media, which helps companies improve traffic to their Web sites. "The cannibalization of search results has the potential to become problematic, because you have one chance with 10 distribution channels. But that's the nature of how the Web is evolving--that's not Google's fault."