The changing face of search engines

Abbott, head of Web marketing for Integramed.com, which sells infertility treatments, considers obtaining an editorial listing in Yahoo's directory "a waste of time," given that 95 percent of his site's traffic comes from pay-per-click advertising in search networks Overture and Google.

"I paid for reviews with all the directories at the beginning of my marketing initiative (in 1999), but being in a directory is meaningless now unless you can rank on the first page," he said. "I have complete control with pay-per-click advertising--I can get to the top, I control the headline and message--whereas with a directory listing, it's what (Yahoo editors) decide to write."

Once the primary road signs to navigating the Internet, directories have moved to the shoulder. They are being displaced by algorithmic search tools and commercial services that many people--Abbott among them--now believe do a better job in satisfying Web surfers and advertisers. The transformation is bringing to an end an altruistic era of human editors, who once wielded significant clout in driving traffic to Web sites through recommendations made without regard for commercial considerations.

The transition has sparked a power shift in the search world that is forcing directory leader Yahoo to reinvent its search business to better compete with an uprising of algorithmic and commercial search providers, most notably Google and Overture Services. In response, Yahoo over the past year has continued to distance itself from its noncommercial directory roots, adding paid search links from Overture, demoting directory listings on its search pages to results provided by Google and scooping up algorithmic search provider Inktomi.

The recent flurry of activity at Yahoo has company watchers wondering what the future holds for the portal's search tools, and what place, if any, there might be for its once dominant directory.

"In October, Yahoo made the directory secondary to Google," said Danny Sullivan, editor of the industry newsletter Search Engine Watch. "Suddenly the value of getting listed in Yahoo seemed to disappear. Now, if you're not listed with Yahoo, it may not matter."

Is Yahoo losing its religion?
Clearly, it pays to be in the paid search business. Yahoo's deal with Overture has helped it achieve three consecutive quarters of profitability and has allowed management to boost financial expectations. Yet the success in partnering with commercial search providers raises the inevitable question, why pay dozens or hundreds of people to search the Web when another company wants to pay you to do the same?

Searching for answers This chart shows the search services provided by various Web destinations. In many cases, sites use tools developed by other companies. ("Yes" means the company created its own tool.)

Directory

Algorithmic

Paid

Yahoo

Yes

Google

Overture

Google

ODP

Yes

Yes

MSN

LookSmart

Inktomi

Overture, LookSmart

Overture Services

No

Inktomi Altavista, Fast*

Yes

LookSmart

Yes

WiseNut

Yes

Open Directory Project (ODP)

Yes

No

No

AltaVista

No

Yes

Yes

AskJeeves

Yes

Teoma

Google

* Coming soon

Yahoo keeps the operations of its search editors close to its vest. Company executives will not comment on how many editors it employs. A Yahoo representative said "on average" the company employs "a building full" of directory editors.

At least one search engine marketer has said that Yahoo has scaled back on its directory editors slowly over recent months, giving people new duties or emphasizing paid search listings.

But Srinija Srinivasan, Yahoo's vice president and editor in chief, denied the company has recently laid off or redeployed members of its editorial staff.

"The state of search technology thankfully has improved," Srinivasan said in an interview. "That said, we firmly believe there continues to be a gap between the best technology and what we can provide incrementally with the human experience."

Conceived by co-founders Jerry Yang and David Filo in a Stanford trailer in 1994, much of Yahoo's popularity was built on the directory's ability to give order and organization to the unruly Web. As legend has it, Yahoo was developed by Yang and Filo as a way to categorize their favorite sumo wrestling Web sites. Even the company name--originally the acronym "Yet Another Hierarchical Officious Oracle"--highlighted its directory roots.

Unlike the other search competitors that emerged in the mid-1990s, such as Excite, Lycos, Infoseek and AltaVista, Yahoo did not develop its technology to crawl through millions of Web sites. Instead, it hired humans to manually search the Web to find, organize and review sites about thousands of topics. Yahoo's editorial team became an emblem of the Internet's rise where legions of college graduates would do the heavy lifting to help Web newbies find what they want.

Yahoo did not rely exclusively on its directory, signing partnerships over the years with algorithmic search engines such as AltaVista, Inktomi and Google to provide backup results. But up until October, these third parties were never the centerpiece of Yahoo's search results.

		News.Commentary Yahoo's challenge To match Google, Yahoo must modify its newly acquired Inktomi search engine to factor in editorial judgment.

In late 1999, Yahoo began to tinker with its coveted directory service. Under then Yahoo CEO Tim Koogle, the company launched a new fee plan, requiring sites to pay $299 a year--$600 for adult sites--in order to be considered for inclusion in its directory listings. If accepted, companies would be required to pay the fee annually to retain their listing.

Yahoo does not break out results for this business, but said it has no plans to discontinue it.

While many analysts called the move to paid inclusion overdue at the time, others believe the decision hurt the directory's credibility.

"One has to wonder how the economic interests of search is messing with the altruistic agenda of the directory," said Lance Loveday, president of Web marketing consultancy Closed Loop Marketing.

Altruism vs. cash
Yahoo isn't the only directory facing criticism these days.

Search engine marketers also point to the Open Directory Project (ODP) as an example of how far directories have fallen behind algorithmic search providers, both in terms of the reach and quality of the results they provide.

		Special report The Google gods Does the search engine's power threaten the Web's independence?

AOL Time Warner-owned Netscape runs the ODP, which launched in June 1998 under the name NewHoo. It uses about 210,000 volunteer editors to catalog the Web, many of whom are search engine marketers. Though Google and AOL both draw on this directory for specialized searches, the service is thought to be plagued with troubles. Hardware failures over the winter holidays caused the directory to be out of commission for several months, for example.

Elisabeth Osmeloski, a search engine marketer and a volunteer editor with ODP, said that more than 50 percent of the sites submitted for review are spam links, causing a major sap to volunteers' time. "The ODP has a huge backlog from bad submissions; there are sites waiting for two years to be reviewed. It would be better if Netscape got (more) behind it," she said.

Bob Keating, editor in chief of the ODP, said that it's fighting the good fight against the encroachment of the profit motive into what is rightfully an editorial process.

"We're trying to combat the commercialization of search," said Keating, whose ODP has a catalog of 3.8 million sites, compared to some 4 billion for Google. "A lot of Web directories have gone the other direction. As search gets even more commercialized, the Open Directory is the only one that's left that's really grounded in the original concept of the Net--that it's an information source and not a money-making vehicle."

He added that the ODP has checks and balances to stop spammers from controlling the directory.

LookSmart, which launched in October 1996 and was originally backed by Reader's Digest, started as a directory of the top sites in any category on the Web. It employed hundreds of editors and writers to handpick sites. Now, it employs about 100 editors, but they largely review commercial sites that have bought into the directory, which is licensed to Microsoft's MSN. It also runs a noncommercial directory called Zeal.com that is staffed with about two or three editors and a team of volunteers.

Many small sites say they still see value in a listing in directories like Yahoo's, LookSmart's and the ODP because such links are given weight by Google's PageRank, a system for evaluating the popularity of a Web page. Google rides on the back of human-screening of Web sites, and many people see directory links as an easy step on the road to popularity in major search engines.

Tim Mayer, vice president of Fast Web Search, which was recently acquired by Overture, said that directories are largely treated like any other link on the Web, and some may be thought of as more authoritative than others. But he said the influence of the directories has faded as they have become more commercial.

"The more authoritative the site or directory is that is linking to your site, the more weight given in the link popularity," said Mayer, adding that link popularity is only one feature of many in the relevance algorithm. Still, he said, it's less important "in many search engines in the past years as directories have moved from purely editorial to pay-for-play."