Culture

Netflix contest ends, with winner in doubt

The $1 million contest to improve Netflix's movie recommendation software ended with two teams in something of a photo finish, but the outcome remains in doubt.

Aug. 3, 2009 9:00 p.m. PT

2 min read

The $1 million contest for the Netflix Prize ended Sunday with two teams in the computing version of a photo finish. The outcome, it seems, remains in some doubt.

The competition, begun in 2006, really shifted into high gear in the last month, after an international team of statisticians, machine learning experts, and computer engineers declared that they had come up with algorithms that could improve the movie recommendations made by Netflix's internal software, Cinematch, by at least 10 percent.

The announcement by that team, BellKor's Pragmatic Chaos, set off a 30-day race, under the contest rules, for other teams to try to best them. Entries flowed fast and furious since then. And just minutes before the Sunday deadline, a team composed of other former teams, appropriately called the Ensemble, nudged ahead of BellKor's, on the public Web leaderboard.

So the Ensemble won, right? Not necessarily. In an e-mail message Sunday night, Chris Volinsky, a scientist at AT&T Research and a leader of the BellKor's team, said: "Our team is in first place as we were contacted by Netflix to validate our entry." And in an online forum, another member of the BellKor team, Yehuda Koren, a researcher for Yahoo in Israel, said his team had "a better test score than the Ensemble," despite what the rival team submitted for the leaderboard.

So is BellKor the winner? Certainly not yet, according to a Netflix spokesman, Steve Swasey. "There is no winner," he said.

A winner, Swasey said, will probably not be announced until sometime in September at an event hosted by Reed Hastings, Netflix's chief executive. The movie rental company is not holding off for maximum PR effect, Swasey said, but because the winner has not yet been determined.

The Web leaderboard, he explained, is based on what the teams submitted. Next, Netflix's in-house researchers and outside experts have to validate the teams' submissions, poring over the submitted code, design documents and other materials. "This is really complex stuff," Swasey said.

A leading member of the Ensemble, Domonkos Tikk, a Hungarian computer scientist, did not sound too hopeful. "We didn't get any notification from Netflix," Tikk said in a phone interview from Hungary. "So I think the chances that we won are very slight. It was a nice try."