Yahoo BOSS + Twitter + Google App Engine = fresh news
A Yahoo engineer has combined a Yahoo search service with Twitter to offer a new way to present news that's both fresh and pertinent.
- Shankland covered the tech industry for more than 25 years and was a science writer for five years before that. He has deep expertise in microprocessors, digital photography, computer hardware and software, internet standards, web technology, and more.
Here's Web 2.0 at its finest: A Yahoo programmer has combined his own project, Yahoo BOSS (Build Your Own Search Service), with Twitter and Google App Engine to create a new way to determine what news is both new and important.
The service, called TweetNews, presents Yahoo news search results in a different way, using results from the same search on Twitter to determine what should get high placement, according to a blog posting about it by BOSS engineer Vik Singh.
BOSS supplies Yahoo search results in a form that can be repackaged, processed, and published for free, though Yahoo asks for revenue sharing for popular services.
TweetNews combines human interest, as judged by Twitter users, with a measure of authority, as judged by publications that make the cut for Yahoo News search. The application also includes an expandable "related tweets" button that supplies links to people's Twitter references to the various news stories.
"Twitter as a ranking signal for search freshness may prove to be very useful if constructed properly," Singh said in the blog.
Here's a screenshot of the search in action, using the terms "hudson plane" to illustrate the news items Twitter users find most pertinent.
The application is publicly available as a service running on Google App Engine--not the first time Singh has demonstrated BOSS ideas on his main competitor's application hosting system. Google hosts applications on App Engine for free, but only within various limits, and Singh's However, the application exceeded its quota within a few hours of his posting.
The application isn't just a novel demo, though. It's an attempt to solve a challenging problem in determining what breaking news is most pertinent to people. Here's how Singh describes the challenge:
Freshness (especially in the context of search) is a challenging problem. Traditional PageRank style algorithms don't really work here as it takes time for a fresh URL to garner enough links to beat an older high ranking URL. One approach is to use cluster sizes as a feature for measuring the popularity of a story (i.e. Google News). Although quite effective IMO this may not be fast enough all the time. For the cluster size to grow requires other sources to write about the same story. Traditional media can be slow however, especially on local topics.
I remember when I saw breaking Twitter messages describing the California Wildfires. When I searched Google/Yahoo/Microsoft right at that moment I barely got anything (< 5 results spanning 3 search results pages). I had a similar episode when I searched on the Mumbai attacks. Specifically, the Twitter messages were providing incredible focus on the important subtopics that had yet to become popular in the traditional media and news search worlds.
What I found most interesting in both of these cases was that news articles did exist on these topics, but just weren't valued highly enough yet or not focusing on the right stories (as the majority of tweets were). So why not just do that? Order these fresh news articles (which mostly provide authority and in-depth coverage) based on the number of related fresh tweets as well as show the tweets under each. That's this service.