How CoverItLive failed users during iPad unveiling

A Steve Jobs keynote is technology journalism's Super Bowl equivalent. And as with the Super Bowl, they're best enjoyed in real time. Thus the healthy growth of the live-blogging platform CoverItLive, which enables journalists to file live reports that are transmitted, as they type them, to their online readers.

The company, started in 2007, has been growing well and winning the support of journalists not just in the technology realm, but in sports, politics, and other fields. It has become the largest live-blogging platform there is. The embedded CoverItLive live blog player is popping up on sites across the Web. (For a sample CoverItLive live blog, see this replay of a Google press announcement.)

But CoverItLive fell apart during the January 27 iPad announcement. Just as the event was getting started, incoming Apple fans were turned away from embedded CoverItLive live blogs on important sites like TUAW, MacWorld, and MacNN. Readers quickly abandoned many of these sites and headed to others, like Gdgt, that were using home-grown live-blogging tools. To stop readers from leaving, some sites, such TUAW, abandoned CoverItLive on the spot and began publishing frequent updates on their standard platforms. Regardless, it appeared to be a disaster for the small live-blogging company.

The Steve Jobs keynote also temporarily overwhelmed other sites, including CNET News and our sister site ZDNet (read what went wrong). CoverItLive competitors include services like Scribble Live and live video services like Qik and Justin.tv.

This was not CoverItLive's first failure during a Steve Jobs keynote. On January 15, 2008, during the MacBook Air announcement, the platform also collapsed. Since then, CoverItLive flourished nonetheless, winning over journalists in other fields, mostly in sports and politics, who started to use the product regularly. And then the tech sites started to come back.

Sites like MacNN used the service to great advantage. Publisher Monish Bhatia says that building his own live-blogging platform would have been too expensive, and that CoverItLive offered a good blend of features. MacNN used it about six times before the recent failure, he said. But, he told me, "I wish they were a paid service." He wanted a contract to fall back on with the company should anything go wrong.

Between the 2008 failure and the recent one, CoverItLive has not had a failure during a major live event. And many of these events, such as President Obama's Nobel Prize acceptance speech in December of 2009, got more live blog traffic (9 million views) than a Steve Jobs keynote had until then.

So what is it about Jobs' keynotes that is so toxic to CoverItLive? And how can the company regain the trust of tech journalists?

It started out so well
"We were having a great day," CoverItLive CEO Keith McSpurren told me about January 27, the day of the iPad announcement. Several high-profile sites had already embedded the CoverItLive viewer on their site, and by 10 a.m, when the event was set to officially begin, the writers in San Francisco's Yerba Buena Center were settled in an dready to start typing their live reports. In order to keep the service working for everyone, CoverItLive's servers had throttling code running that was supposed to slow down the real-time updating on readers' embedded players--just a little, but enough to level the load and keep everyone online. Amazon and Rackspace servers were spun up and ready to handle the traffic.

But the throttles didn't work. A massive influx of users all pressing the "play" buttons on their embedded CoverItLive viewers at the same time overwhelmed the service, causing the "hard door" overcapacity message to show up on sites. McSpurren said the situation was rectified within about 15 minutes, but by then readers had gone elsewhere.

McSpurren told me that of all the events people use CoverItLive for, Steve Jobs keynotes are unique. Other tech events are "rounding errors" in terms of traffic, and even huge non-tech sports and political events, while they may have more viewers than a Jobs keynote, don't cause the same strain on the CoverItLive systems.

"The problem isn't the size of the nightclub," McSpurren says, "It's getting people in the front door." Jobs keynotes are unique in that the fans all rush the system at the same time. The throttling system is supposed to keep the server load manageable as the CoverItLive systems set up sessions with millions of new users at once. McSpurren said that even during a huge sporting event, like the coverage of a Manny Pacquiao boxing match, there's not the huge rush. The need to throttle, he says, is 100 percent unique to a Jobs keynote.

After the event was over, publishers expressed their anger and concern. "The barbecueing was severe," McSpurren said. Publishers told him things like, "I felt stupid for having CoverItLive on my site."

"It was a horrible day for us as a company, and a horrible day for our users. We just didn't get out ahead of it enough," he said.

But McSpurren believes he can recover his standing with the tech sites. "Twitter got through several failures," he notes. "I don't believe failure is negligence. There aren't many entrepreneurs who go from zero to success without a massive failure at one point along the way."

Not all publishers are sold now, nor were they before. Ryan Block, founder of he gadget site Gdgt, told me with CoverItLive, "you're plugging that content into someone else's system. That's fine most of the time, but I've seen how irrationally angry people get when we go down during a live blog, and my assumption was that CoverItLive couldn't handle our load." Block is also not a fan of the way the embedded player looks to end users.

The fixes
To answer the call of publishers who are leery of relying on a service for which they don't have a contract and a service-level agreement, CoverItLive is launching new premium levels of service. There will be a premium free service that has ads in the player, a level up from that that lets publishers share in the advertising revenues, and finally a paid service with no ads. The ad-free, paid program will be sold by Pluck, and come with an SLA and 24-7 support and monitoring.

The free, ad-free service will still be available, but will get defeatured slightly and will have a cap on the maximum number of viewers it will support.

McSpurren told me he's not providing software to customers that they can run on their own hardware. However, he will wall off the paid and premium sites from the free users, hopefully to prevent systemwide failures of the type that plagued the this January's Steve Jobs keynote.

Some publishers, like MacNN's Bhatia, aren't sure they'll be back. His site lost a lot of money in advertising revenues, he says, and he's exploring other solutions, including a home-grown platform.

Gdgt's Block says, "I never really considered [CoverItLive] an option. Even though building a system that can stand up to a Jobsnote 100 percent perfectly is an insanely major hassle, if I'm going to go to the trouble to do live coverage, I want to make sure the medium is up to snuff, too."

While publishers (inlcuding CNET) debate the merits of trusting a service like CoverItLive compared to the hassle and expense of building their own live blog platforms, there's no doubt that the medium itself is growing. McSpurren says CoverItLive is responsible for more than 82 million page views a month. And live blog viewers don't just read a story in a minute or two and leave, as they do on most blogs. Rather, live blog watchers stick on a site for multiple minutes, sometimes more than an hour at a time. There is media gold in the platform; even if it remains a challenge to build a live blog system that's robust enough to handle the strain that today's users can put on it.

This story has been updated from the original, with information about TUAW's fall-back posting method.