FiveThirtyEight.com's Nate Silver on life post-election

AUSTIN, Texas--If there was one name that stood out on the agenda of speakers at the South by Southwest Interactive (SXSWi) festival here this week, it was famed FiveThirtyEight.com blogger Nate Silver.

Known as a statistical wunderkind, his models predicted the final outcome of the 2008 presidential election to within .4 percent of the final popular vote. But more important to many Democrats who had their hopes for electoral victory dashed by George W. Bush in 2000 and 2004, FiveThirtyEight.com--which got its name from the total number of electoral votes available--was able to provide daily affirmation that Barack Obama was really winning, even when many were tempted to believe he would be overcome by Sen. John McCain.

Silver was SXSWi's keynote speaker on Sunday, and he and interviewer Stephen Baker of Business Week went onstage in front of an audience of about 2,000 fans, most of whom were there to hear Silver talk about the secret sauce behind his hugely popular blog.

What many might not know is that Silver first came to prominence not in the political realm, but in baseball, where he authored Baseball Prospectus, a well-regarded baseball statistics site. Many might see the connection between baseball and politics as far-fetched, but to people like Silver, it's a very direct path.

Still, before starting FiveThirtyEight.com, he wasn't entirely a political neophyte. Silver had already begun to make a name for himself in the liberal political blogosphere with a series of data-rich posts on DailyKos. When he began to recognize some significant holes in the national polling establishment, he decided to step in to fill the void.

After his keynote interview, Silver sat down with CNET News and talked about the election, how his site got started, and more about the philosophical similarities between baseball and politics.

Q: Many Democrats were emotionally tied to what you were doing, in the sense that your data kept them calm during the election. Did your own numbers keep you calm?
Nate Silver: Yeah, I think so. I'm just one of those people that likes to try and dissect a problem and once you started to dissect, some days you feel better about it. If I ever get cancer, the first thing I'll probably do is go on the Web and collect a bunch of data about different survival rates. I just feel better about things when I do them that way. It's a nerdy kind of thing to do.

Q: We were able to get up every day and look at the data and see what was going on. And this is not something you could do because it was your own data. How your own data affect how you felt about what was going on?
Silver: I wouldn't be frustrated by it if McCain or Obama picked up points on a particular day. Sometimes you get frustrated if you know that something you did reveals something about your model. When something doesn't feel right, and you go and make changes. And we made a lot of changes over the course of the campaign where, even as recently as two weeks before the election, we were tweaking little parameters, and what started out as a pretty simple system--taking weighted averages of polls--became much more complex over time. But, yeah, we were never saying we had the perfect answer. We were always trying to improve things as we went along.

Q: The blog had an overt liberal position, but you always said the statistics were objective. What kind of feedback, if any, did you get from conservatives?
Silver: We had a pretty good balance. We had probably about a 2-1 ratio in terms of liberal versus conservative readers, based on the comment threads. Now that we're not in an election, I think it's swung more toward the liberal side, both in terms of my writing and what people are reading about.

We try and be fair. That's the main thing, we try and be forthright. There's so much commentary from conservatives, also from liberals, that is just entirely disingenuous about certain things. It's a lot of cheerleading and cherry-picking of data. We're trying to present a case that by and large is a liberal's case, because it's my case. It's how I see the world. But we're trying to use data to do it where a lot of people just make bad arguments.

Q: How helpful was it to FiveThirtyEight.com to have the Minnesota election and the Mississippi runoff to keep buzz going after the election?
Silver: Definitely the Minnesota race, and also Alaska took awhile to resolve. But we were going to lose a lot of our traffic no matter what. Especially because we had had people clicking on the site 10 times a day or 10 times an hour waiting for the polling updates. So, those kind of eased the transition, especially Minnesota. But we're now at a point where I'm in some ways enjoying writing the blog more now because now there's all these other things I want to talk about.

Q: Was that the plan for after the election?
Silver: The plan was to do a little bit more with the Congress, where we started to analyze Congressional voting patterns, and we're doing a little bit of that. But my background is in economics, and fortunately, most of the major issues relate to economics, things like the stimulus and the bailout. But health care is all about economics, and so is cap-and-trade environmental policy. And so I think I'm fortunate that the issues that are most important now are things I can talk about.

National security and "values" issues, those are less amenable to the quantitative analysis that I do. So we're talking more about policy and less about politics than I expected.

Q: What are your traffic numbers now, and how does that compare to the campaign?
Silver: We're at about 800,000 uniques per month, and in October and November, we peaked at about 2.2 million. With visits, the difference is larger, because people were clicking so repeatedly on the site. But basically we're doing about the same now as we were in August, just before the conventions. You would like to still be getting 2 million visits on a particular day, but of course you're not going to get that. A lot of people tune out after the election.

Q: The artist Shepard Fairey never imagined that his Obama "Hope" poster would be such an icon. Did you expect your site to have as big an effect as it did?
Silver: No, I'm a pretty confident guy, and I thought that this is something I'm doing in part because people aren't doing it that well, but I didn't think there would be this kind of scale. You discover that Web traffic is very non-linear, and it's viral. You don't go from 1,000 to 2,000. You go from 1,000 to 10,000 and from 10,000 to 100,000. You have this multiple happening, kind of four times, on a logarithmic scale, and I didn't expect that. But there's a lot of luck here.

Q: What do you attribute your success to?
Silver: Being in the right place at the right time. Also, it's working hard, and having an interesting product. But that just gets you in the door. So from that point of view, a lot was luck. It's some of the media we did, or certain predictions we made, like for the North Carolina primary, where we said Obama would win comfortably, and he did, and a guy at Newsweek did an interview based on that. Especially with the media, it all kind of feeds back on itself, where your qualification to do media is doing other media.

Q: What was your process for working on your statistical models?
Silver: There's no one step that's all that complicated. You're building it bit by bit. It started when I was coming back from a trip to New Orleans and was in the airport, and thought, "Why hasn't anyone collected all these general election polls and seen whether Clinton or Obama is doing better against McCain?" That was the first stroke of insight. But you start doing that, and then I became frustrated by the fact that there are some polls that I knew weren't very good, but they were getting as much weight as good ones, so it became going back and seeing who's been most accurate over time. So it's literally been one step at a time. A lot of it is you sitting there at 4 in the morning with a can of Red Bull and a data-processing program.

Q: What software are you using?
Silver: I use Microsoft Excel. I use Excel in ways that it shouldn't probably be used, but it's a pretty flexible program if you know how to trick it out a little bit. And for the hard-core data processing, you need something else, and I use Stata.

Q: How are political campaigns and baseball connected?
Silver: There is kind of the same rhythm to political campaigns and baseball seasons. You have to be patient to appreciate it. In political campaigns that last two years, you have certain primaries, like Iowa and New Hampshire, that are very important. And two conventions, three or four debates, and the vice presidents being picked. Collectively, those took up about 20 days, whereas the campaign lasted about 700 days.

So there's not much high-impact stuff, and you have to follow it every single day, and understand how little threads become big threads, and how the picture develops very slowly. That's similar to baseball, which has a long season.

Q: There's also a partisan thing, like Yankees versus Red Sox and Democrats versus Republicans, right?
Silver: I think there is something to that. Political affiliations are kind of like picking sports teams. And that's why when we are expressing a political point of view we try not to make it personal. Some people think those who think differently than they do politically are evil people. That's not the most productive way of looking at the world.

Q: Were pollsters calling cell phone users?
Silver: At the end of the day, I think about half the firms were. Most of the national polling firms, like NBC/Wall Street Journal, and I think ABC News and Pew Research were. The better-financed polls are starting to do that. But it's going to become more and more important, because it's not like the thirtysomething or twentysomething generations are going to feel the need to have landlines. This is a real problem for the polling firms in the long term.