Inktomi gives Web sites control--for a price

The introduction Wednesday of Inktomi's Index Connect lets large Web sites pay to make sure they're included in the company's search index. The service gives sites control over how often their sites are indexed and what specific pages are included.

"In the past, search engines always have had a relationship with people who are searching, but the relationship on the content side has been pretty random," said Troy Toman, vice president and general manager of Inktomi's search solutions division. "It's a hit-and-miss proposition when you're spidering the Web on a random basis."

Most search engines work by crawling the Web with spiders, or software programs that continually surf the Web, indexing pages and words.

The problem Index Connect proposes to solve is that many Web pages undergo frequent and regular change.

"We refresh our index every 30 days," Toman said. "We can now say, instead of that, why not set up your relationship so that you can send us a data feed every time you update a page, and we can add that as short as every 15 minutes to an hour?"

Index Connect brings the search pioneer a step closer to some of its rivals whose "pay-for-placement" business models are based on taking money from content providers in return for prime placement in the search results. A search for "sports car" on such a site, for example, would return sports car dealers in an order determined by how much money they paid for their placement.

Critics fault such schemes for corrupting the integrity of search results. Search purists prefer that results be weighted according to relevance, rather than a company's willingness to cut the search engine a check.

Inktomi acknowledged that its new service would weight its index toward paying content providers but was quick to distance itself from the pay-for-placement schemes.

Index Connect "does potentially have an impact on the index, with more content coming from paying clients," Toman said. "But our results and relevance algorithms are exactly the same. This is paid inclusion, giving sites the opportunity to be considered for a search. But we are still very democratic when we select search results."

Inktomi last year introduced Search/Submit, which lets sites pay to be searched on a regular basis. But that service--which Toman described as "hugely successful"--was aimed at smaller sites, or those submitting fewer than 1,000 Web page addresses. Index Connect is designed to offer a similar service to the large sites.

Index Connect gives sites more say in the number of pages that are indexed and the frequency with which they are updated. It also allows some sites to submit pages directly, rather than have Inktomi's powerful spiders come in and crawl around--which can have the effect of slowing a site down.

San Francisco news site "KQED had prohibited our search bots because it was slowing performance, so now they actually feed us the information from their site," Toman said. "Now you can have our spider on a timer or send us a data feed with all the pages on it that you want included."

KQED, The Hunger Site and other nonprofit sites can use Index Connect free of charge.

Search engine analyst Danny Sullivan, editor of SearchEngineWatch.com, said paid submissions are not necessarily bad. But he warned that search engines would have to work to ensure equal access and representation in the index.

"The key thing is to ensure that important content that should be in the index is also included, regardless of payment," said Sullivan. "Otherwise, the search engine index isn't as comprehensive or useful to searchers. Given this, Inktomi's statement that it is to begin offering deeper inclusion to some nonprofit sites for free is a good sign. It means that perhaps the commercial portion of the Web can to some degree subsidize the important nonprofit portion."

For-profit sites pay a set-up fee, then pay on a cost-per-click basis that ranges from a nickel to 50 cents per click.

For-profit customers so far include eBay, Epinions, LookSmart, RollingStone.com and Virage.