IBM researchers show love for 'Jeopardy' champion Watson

SAN JOSE, Calif.--I'm going to just come out and admit it--I was rooting for the humans.

By "humans," of course, I mean Ken Jennings and Brad Rutter, two men who on the one hand are the greatest champions in the history of "Jeopardy" and who on the other just ended up getting their butts handed to them at the game by a computer that didn't even seem to know that Toronto isn't in the United States.

In case you were somehow in a cabin in the mountains with no Internet access and no TV over the last few weeks and don't know what I'm talking about, I'm referring of course to the latest IBM Grand Challenge--Big Blue's development of a supercomputer known as Watson that was intended to be able to beat the world's best "Jeopardy" players at a game centered around one of the biggest problems in computing: understanding and parsing natural language.

Over the last three days, Watson's battle against Jennings and Rutter played out on national TV in a two-game match. May the best, er, man win.

Though I wasn't able to be in the room at IBM Research in Yorktown Heights, N.Y., when the matches were played last month, I did get invited to the final night party this evening at IBM's Almaden Research Center here, and let me tell you, though I was in a roomful of actual human beings, not many of them shared my preference for a contestant with DNA. These folks were definitely in Watson's corner, tinny text-to-speech voice and all.

In the end, they all got the last laugh. As you've no doubt heard by now, Watson out and out dominated Jennings and Rutter, finishing the two games with a total of $77,147, more than the two humans' $24,000 (Jennings) and $21,600 (Rutter) combined.

So I guess Jennings' tongue-in-cheek comment "I, for one, welcome our new computer overlords," which he wrote along with his Final Jeopardy question, was somewhat appropriate.

Roomful of researchers
When I got the invitation to tonight's festivities, I accepted readily. I knew it would be a lot of fun to watch the prime-time conclusion of IBM's four-year effort in the same room as a large group of people who might actually be able to understand the complex science, technology, and math behind the Watson project.

Of course, many of those researchers brought their families with them to watch the final match, and some of the kids may have been more enthusiastic than any of the employees.

"My daughter feels 50 percent more geeky" than she used to, said James Kaufman, a research manager at the Almaden facility. To which the daughter, Sarah Kaufman, added, "I was pretty geeky to start with."

The evening began with coffee and popcorn outside the research center's auditorium, and on hand was a surprise guest--Eric Brown, Watson research manager at Yorktown Heights, who happened to be in town for another meeting and decided to stick around and regale the crowd with some tales of the project.

Standing outside the auditorium, Brown was answering questions and signing autographs. If you think about it, it's probably a pretty rare thing for an otherwise unknown IBM employee to be signing autographs, but here he was a rock star.

One kid came up to him for an autograph, and Brown said to him, "Are you going to be a computer scientist when you grow up? Because, you know, when you're a computer scientist, you get to go on TV. It's really quite glamorous."

To Brown, being able to watch the finale with a group of fellow researchers was a very different experience than being with members of the general public. Yet, he said that everywhere he's gone as an ambassador for the Watson project, he's been struck by the high level of public excitement. Still, here at Almaden, he knew that the audience would bring "a different eye" to the show.

Confidence
To me, one of the most interesting things about watching the two matches among Watson, Jennings, and Rutter was seeing the little display at the bottom of the screen in which viewers could see not only Watson's top three potential answers but also the percentage of confidence the computer had in each.

To Brown, that last element is one of the most important parts of the entire project. "What [Watson] really is," Brown said, "is a demonstration of the technology. And what we really want people to think about is [that Watson has to] come up with an answer buried in a [natural language] concept, not only the right answer, but confidence in the answer."

Strangely, there were times during the two matches when Watson's confidence in what turned out to be the right answer was extremely low. At least once, Watson pegged what turned out to be the right answer at just 12 percent confidence and didn't even bother to buzz in. That struck me as odd.

But to James Kaufman, that wasn't surprising, given that in order to come up with an answer, Watson had to balance several different algorithms. Most of the time it worked and quite well as evidenced by the computer's resounding victory. But sometimes the computer seemed very off-kilter.

Another thing that didn't surprise Kaufman was how well the computer did, even matched up against trivia powerhouses like Jennings and Rutter. "I know the guys [on the] Watson" team, Kaufman said, alluding to those researchers' across-the-board genius-level intelligence.

What did surprise Kaufman was how, depending on the topic at hand, the amount of time that Watson sometimes took to answer a question. "That struck me as almost human," he said. "It was almost hesitation."

Kaufman said that he really enjoyed being able to watch Watson take on the champions and do so well, even as the computer sometimes made silly mistakes, such as its answer of Toronto to the first-match Final Jeopardy question about U.S. cities. But mainly, Watson showed off what was an extremely impressive display of computing prowess, one that now has to be measured right up against IBM's Deep Blue's victory over chess world champion Garry Kasparov in 1997.

"I spoke to a colleague," Kaufman said, "who said that the goal [of the Watson project] was to create the computer on 'Star Trek.' They're moving the needle in that direction, and I think they did it."

'Marry that computer'

Throughout the evening, IBM Silicon Valley human resources manager Alexa MacDonald fired up the crowd with a series of IBM- and "Jeopardy"-related questions. One showed how closely the group had been paying attention to the project. When MacDonald asked what the correct answer to the first match Final Jeopardy question was--the very same question that Watson botched by responding with "Toronto," about half the room shouted out "Chicago!"

To Andy Hood, a technician specialist at Almaden who with a steady stream of fist-pumping cheers, loud exhortations to the screen, and general energetic support of Watson may have been the most enthusiastic member of the crowd, the outcome of the matches was never in doubt. "I believed that Watson was going to crush [Jennings and Rutter]," Hood said. "He's just got way more information available."

But teenager Sarah Kaufman may have been the one who Watson may most want to meet. "I want to marry that computer," she said in all seriousness after the computer had finished destroying its merely human opponents.