Debate: Can the Internet handle big breaking news?
CNET News' Tom Krazit and Declan McCullagh debate whether the tendency of Web sites to stagger under high demand can be avoided, or is even that big of a problem.
It happens time and time again: when news breaks, the Internet slows.
It's quite obvious at this point that the Internet has muscled its way into the lives of anyone who needs information. And Michael Jackson's death Thursday had as great an impact on the Internet as anything in the history of the medium that didn't involve the World Trade Center.
Can a system that has trouble keeping up with ever-increasing demand for its services be considered a reliable source of information when a true crisis emerges? After an editor banished a budding argument between CNET News' Tom Krazit and Declan McCullagh from a company-wide mailing list, we decided to let them fight it out here.
Tom: How can any system that doesn't work precisely when people need it the most be considered the future of communications?
In a way, it took the death of perhaps the greatest entertainer of the last century to expose a key truth of this century: our new favorite communications tool, the Internet, buckles in times of crisis. News sites, including this one, were sluggish or completely offline at the peak of demand for information, forcing many to go back in time and flip on the television.
What if something really happens? How can companies trying to build information-related businesses on the Internet ever hope to supplant existing communications networks if they fail at the moment of truth? CNN's telecast didn't go down Thursday.
Declan: I think it's a little unfair to say the Internet "buckles in times of crisis." Sure, a few Web sites--Google News, The Los Angeles Times, TMZ, Yahoo, MSNBC--had slowdowns or outages. (That list includes our own CNET and CBS Interactive sites, which experienced serious problems for about half an hour.)
Some news Web sites slowing down or becoming unreachable for 30 minutes is not the same thing as the Internet "buckling." If an earthquake were to take out the trans-Pacific cable landings in California's Morro Bay, San Luis Obispo, and Grover Beach, if car bombs knocked out MAE East and MAE West, and if a hurricane laid low the cable landings in Long Island and New Jersey, that might--might!--qualify.
In fact, yesterday's sad news about Michael Jackson demonstrated not the vulnerability, but the resilience of the modern Internet ecosystem. True, a few sites were having problems. But The Los Angeles Times' report about Jackson's coma, and its subsequent report about his death, were picked up and mirrored widely. Even if you couldn't get through to the Times, you could get through to innumerable blogs and others news sites citing it. Or you could just wait a few minutes for the traffic to die down.
Was this really such an inconvenience?
Tom: Ok, I'll concede the point about the broader Internet: near as I could tell, ICanHazCheeseburger.com was performing like a champ yesterday.
But this is a systemic problem with the Internet, or perhaps put more accurately, the Web. The more people who demand the service provided by an information Web site, the harder it gets for that site to provide that information. CNN/MSNBC/et al don't buckle when millions of people change the channel to watch O.J. meander down a Los Angeles freeway or the opening salvos of the Iraq War.
In an online world where businesses are spending billions trying to shift information consumption patterns onto the Web, how can these outages be tolerated? You're right, it's very easy to navigate elsewhere if you can't find what you are looking for on Site A. But if you can't depend on Site A in times of crisis, you're not going to go back there in future times of crisis, hurting the reputation of that site as a reliable source of information.
Even Google was unable to handle the load. And if Google can't, nobody can. This is a serious problem for online businesses, especially as people continue to come online in emerging economies and with mobile devices.
Declan: I was using Google News pretty frequently during the time that Michael Jackson's fate was uncertain, and noticed no problems. Others, including some of our colleagues, did. I suspect that Google is using a different set of servers for Google News vs. its main search engine. So it's not so much that Google couldn't design a system to handle an unusual spike in traffic, but that it chose not to do so.
Let me put this argument another way: You said that the Internet "fail[s] at the moment of truth" but lauded "existing communication networks" that supposedly work just fine. Well, existing communication networks fail too. If more than a small fraction of telephone customers try to get a dial tone at once, there's a problem. Ever try to make a call on Mother's Day or with a cell phone at a conference? You're likely to get a fast busy signal or "all circuits are busy" message. Telephone companies could design for higher usage, but have chosen not to. They've figured out that the costs outweigh the benefits.
(Similarly, printed newspapers sell out very, very early on days like Election Day. Is this "fail[ing] at the moment of truth?")
It's really more of an economic than an engineering problem. Is it worth it to add an extra, say, threefold server and bandwidth capacity for that hour or so a year when it's needed? Or pay Akamai's overage charges? Probably not; the revenue may not cover the fees. So if your average rate is 100 users/sec, you might build for 1,000 users/sec max and then not be able to handle those once-a-year occasions when the rate is 5,000 users/sec.
An economist might say the solution to this situation is to ration by price. News pages might normally be free, but under times of high load, a micropayment would be charged. That way, the people who want or need the information the most would get it. Of course this means we need a micropayment infrastructure; I'm not holding my breath...
Tom: We're talking about how to respond to instant demand for information in the modern era. You're right, telephone networks can get overwhelmed. That's why we haven't used the telephone as the primary information source since "Thriller" was released.
Television doesn't get overwhelmed in these situations. The entire state of California could turn to CNN right now and nothing would flinch. If the entire state of California clicked on this story right now, our building might explode.
The Internet has choke points that will limit its ability to be the primary source of information to the world. Yet, companies continue to build businesses around the idea of the Internet as a dominant source of information to the world, neglecting the thorny networking problems that will only continue to get worse as traffic grows and our demand for real-time news increases.
Declan: Aha! I think we're nearing agreement.
We know that providing servers and purchasing bandwidth to handle millions of people an hour is expensive, and may not always scale well. One way to deal with this is to make it much easier for ad-supported news organizations to purchase overflow capacity; perhaps the additional revenue would justify the additional expense. If there's sufficient demand, I'm sure someone will come up with it if Akamai doesn't offer it already. Or news organizations could strip extraneous graphics off of their sites for that hour or so of peak usage--basically entering an emergency text-only mode. (Anyone still using the Lynx Web browser would love it!)
Another option is to recognize the limitations of the medium. Because radio and TV are broadcast, they'll always be more efficient at reaching hundreds of millions of people at once. So maybe CNN.com can't compete with CNN Headline News right now. But if the worst that happens is major news Web sites get a little slow for some 30 minutes a year, I'm not going to worry about The Death Of Online News; the Internet is robust and distributed enough that sufficiently important information about the next 9/11 attack will be distributed one way or another.
In other words, until we achieve technocratical perfection, there's nothing wrong with a bit of redundancy in our lives: keep that old transistor radio and some spare batteries around for a backup.
Tom: Seriously, we didn't even talk about the real Achillies Heel in this whole system: the power grid.