Some of Sprint's corporate customers went offline for hours yesterday after a software upgrade went awry.
SprintLink, which provides access to businesses and Internet service providers, accidentally loaded corrupted software during the 1 a.m. PT upgrade, according to company spokesman Charles Fleckenstein. As a result, 47 of the company's approximately 100 routers went down, cutting off access to ISPs and businesses.
"It was kind of miserable," said William Reveal of stricken ISP Internet Revealed in Moline, Illinois. "For it to be out for so long really hurt. We lost two of our business customers over this."
The outage lasted anywhere from a few hours to most of the day. By reinstalling the original software, SprintLink reactivated about half the routers by 6:30 a.m. PT, and all but two were back in service by 9:00 a.m. PT.
A router in Chicago took several hours more to repair because of weather-related power outages, and a Texas router was not restored to service until later in the evening.
Sprint declined to disclose how many or which customers were affected.
Sprint is not alone in dealing with outages, which strike ISPs every so often. But as more and more businesses come to depend on the Internet, reliable access is becoming increasingly important.
Reveal said this outage was not his first loss of service since signing up with Sprint two and a half years ago. One of Reveal.net's auxiliary T1 connections failed to work when the primary line went down in an incident two weeks ago, said Reveal, who added that he was uncertain of the cause of that loss of service.
But one thing he does know is that he will no longer be putting all his connectivity eggs in one basket.
"We are already looking at getting service from another provider for the second T1," he said. "It became painfully apparent to us that we needed to be 'multihomed' so that if the Sprint line goes down we have access through another provider. Sprint is going to lose some business from us."
Other businesses have learned the same lesson, and many have at least a backup service provider that can be used for email in case of an outage.
Service providers for businesses also are reinforcing their networks with backups. The more reliable the promises of connectivity, the higher the access costs.
Sprint had tested the Internet operating system software that caused the outage, said Fleckenstein, and the company is still trying to get to the bottom of the corruption. Sprint will attempt the upgrade again, no sooner than the coming weekend.
Sprint last month took a 30 percent interest in Internet service provider EarthLink in an attempt to beef up its consumer Internet subscriber base. Those customers were not affected by the outage, according to Fleckenstein.