Web search results still have human touch

Alongside technological progress in the search and navigation area came an unexpected twist in 1999: a surge in Web sites' reliance on search results compiled by human editors rather than software robots.

"This was the year that the humans won," said Danny Sullivan, editor of SearchEngineWatch.com. "If you look at the '95-'96 time frame, you had one major search service, Yahoo, that used human beings to categorize sites while the others were trying to use technology to do the same thing. But now with six out of the top 10 services, the main results you get will be by people."

Five years into the life of the commercial Web, search remains one of the medium's most vexing problems. Web searchers complain that it's difficult to find what they're looking for since most engines turn up thousands, if not hundreds of thousands, of mostly irrelevant results.

Innovation in the search and directory space can only benefit consumers, according to some analysts.

"They've got a long way to go until search is something people like you and I are happy with using," said Kris Tuttle, managing director of SoundView Technologies. "There's a lot of work to be done in search and navigation in the next few years, and hopefully that will result in delivering a better experience."

Directories, offering mostly editorially selected and categorized results, saw two major shifts this year. The more influential was the rise of America Online's Open Directory Project, an open source directory based on information compiled by volunteer editors. Dulles, Va.-based AOL acquired the Open Directory Project with Netscape, which had acquired it with NewHoo.

The open-development directory is based on the type of open source software development projects that gave rise to the Linux operating system and the Apache Web server. Products of open source efforts typically are available for free and licensed to the Internet at large.

While many if not most open-development projects launched by corporations--including Mozilla.org, sponsored by Netscape and AOL to develop the Communicator Web browser--have failed to rally volunteers in the volumes that the grassroots Linux and Apache efforts have, the Open Directory Project has distinguished itself as one of the Internet's most successful open-development projects with corporate roots.

Under the slogan "Humans do it better," the Open Directory Project has grown at Internet speeds. When Netscape acquired NewHoo in November 1998, the project had 4,700 contributors and catalogued 84,000 Web sites. By October 1999 there were 16,500 contributors and about 1 million Web sites.

Perhaps more significantly, some of Netscape's and AOL's fiercest portal competitors have validated the idea by adopting the Open Directory Project. It is now the default directory on AOL's proprietary online service, AOL.com; Netscape's Netcenter; Lycos; Lycos' search site HotBot; AT&T WorldNet; AltaVista; InfoSpace; and more than 100 other sites, according to Netscape.

Waltham, Mass.-based Lycos' decision to adopt a human-generated directory was particularly significant, according to SearchEngineWatch.com's Sullivan.

"That was an earthquake in the industry," Sullivan said. "Users don't perceive it that way, but to have one of the oldest crawler-based services say they're going to use humans was huge. It didn't get the attention it deserved."

The Open Directory Project wasn't the only directory to advance this year. Editorially produced LookSmart in February became the directory provider for MSN, Microsoft's Web portal.

One search pioneer produced a directory generated not by humans, but by data analysis technology. San Mateo, Calif.-based Inktomi, market leader in outsourcing search results to the major portals, this year launched its Directory Engine, which automates the task of creating Web directories and lets companies using it tailor the categories to their needs.

Customers for the Directory Engine include SAP, which will use it for its mySAP business-to-business portal; the Financial Times of London; iWon.com; and Aeneid.

But machine- and human-generated directories are not necessarily incompatible: Inktomi last month announced a deal with LookSmart, which is adopting the Directory Engine to use in combination with its editorially produced product for narrowly focused, or "vertical," portals.

Directories were not alone in reinventing themselves in 1999. The other half of the search equation--search results from automated Web crawling technology, or "spiders"--experienced a small boom in innovative approaches to refining query results.

This year brought the launch of Google.com, whose citation-analysis system ranks search results based on how many sites link to them. Founded last year by Stanford University computer science graduates Sergey Brin and Larry Page, Google this year boasted a $25 million funding round.

Google competes with search technology provider Direct Hit, which debuted last year and recently filed for an IPO. Direct Hit refines search results based on popularity, looking at what links from previous results searchers have chosen.

Another more recent entry into the refined search arena is Yep.com, produced by WebsideStory and based on information gleaned by the company's HitBox tracker tool.

While start-ups nipped at Inktomi's heels, the market leader spent the year readying its own popularity-based search results software and other "relevance boosting" technologies. Inktomi says it has implemented technologies similar to both Google's linking analysis and Direct Hit's popularity analysis. In addition to looking at which sites users select from search results, Inktomi's caching business gives it a vast resource to analyze what sites are popular with users.

Another approach tried this year was to increase the number of Web pages catalogued. Norwegian company Fast Search & Transfer launched with the promise of indexing 200 million of the Web's estimated 800 million pages, almost twice the 110 million pages indexed by Inktomi.

If market leader Inktomi's numbers tell a story, demand for search skyrocketed this year. Inktomi processed 2.9 billion queries in the third quarter, up from 1.35 billion queries for the third quarter of 1998.

Despite new challenges, 1999 solidified Yahoo's lead among directories and navigation portals, SoundView Technologies' Tuttle said. Inktomi maintained its lead in outsourcing its search engine to the major consumer portals, and Verity emerged as the leader providing search capability to corporate sites.