Spying an intelligent search engine

Innovation in Web search using artificial intelligence may lead to the day when you could expect the Web to do the tedious tasks for you.

While most would agree that Google has set the current standard for Web search, some technologists say even better tools are on the horizon thanks to advances in artificial intelligence.

Search is like oxygen for many people now, and considering Google's breakthroughs in Web document analysis, supercomputing and Internet advertising, it can be easy to think this is as good as it gets. But some entrepreneurs in artificial intelligence (AI) say that Google is not the end of history. Rather, its techniques are a baseline of where we're headed next.

Proponents of AI techniques say that one day people will be able to search for the plot of a novel, or list all the politicians who said something negative about the environment in the last five years, or find out where to buy an umbrella just spotted on the street. Techniques in AI such as natural language, object recognition and statistical machine learning will begin to stoke the imagination of Web searchers once again.

"This is the beginning for the Web being at work for you in a smart way, and taking on the tedious tasks for you," said Alain Rappaport, CEO and founder of Medstory, a search engine for medical information that went into public beta in July.

"The Web and the amount of information is growing at such a pace that it's an imperative to build an intelligent system that leverages knowledge and exploits it efficiently for people," he added.

Medstory is not alone. Other young companies such as stealth start-up Powerset and image specialist Riya are also looking to turn arcane computing techniques into business success stories.

Unlearning "keywordese"
In the eyes of a search engine, the Web is essentially a body of words on billions of pages, along with the hyperlinks that connect the words. One of Google's big breakthroughs was to link those words efficiently, measuring relevance by the appearance of words on a page, and the number of hyperlinks pointing to that page, or its popularity.

As a rule, search engines don't understand the words--they're merely programmed to match keywords that are more significant on a page, closer together or linked more often from other pages. So in essence, when someone types in "woolly mammoth," he or she sends the search engine on a wild goose chase for those words, not the animal.

As a result, search engines miss the nuance of the human language. For example, Google might render a simple search for "books by children" by scouting for the pages that include the words "books" and "children," but it would eliminate the so-called stop word--in this case, "by"--because stop words are those that occur on almost every page. Yet those stop words occur so often because they are important to the meaning of a phrase. "Books by children" is different from "books about children," and different still from "children's books."

Barney Pell Barney Pell

Barney Pell, founder of a yet-to-be-launched AI search engine, calls the restrictive language of search engines "keywordese."

"Search engines try to train us to become good keyword searchers. We dumb down our intelligence so it will be natural for the computer," said Pell, whose company, Powerset, is based in Palo Alto, Calif.

"The big shift that will happen in society is that instead of moving human expressions and interactions into what's easy for the computer, we'll move computers' abilities to handle expressions that are natural for the human," he said.

Powerset, which hasn't divulged its launch date yet, is using AI to train computers not just to read words on the page, but make connections between those words and make inferences in the language. That way a search engine could think through and redefine relevance beyond the most popular page or the site with the most occurrences of keywords entered in a search box.

Featured Video