Verio makes amends for outage

Verio is trying to make amends for an outage caused by failed generators in a facility hit by Hurricane Wilma earlier this week in south Florida that affected hundreds of customers across the United States.

The company sent letters to all customers affected by the outage offering one month of free e-mail and Web hosting. Verio, which hosts thousands of e-mail domains and Web sites for consumers as well as small and midsize businesses, offers a 99.9 percent service guarantee to customers. This week alone, Verio service was not functioning for some consumers for two and a half days. The company typically allows customers to make a request for a credit if the service guarantee is not met.

"Verio understands the severity of the situation," said Dennis Boyle, chief operating officer for Verio. "We don't want to ask customers to submit credit requests. These customers are important to us. And it's a big deal for us."

Boyle wouldn't comment on how much the credit will cost the company.

The Verio facility in Boca Raton, Fla., has been through several hurricanes in the past seven years, including four that hit Florida last year without any problems. It has five backup generators that provide power to the servers when commercial power is interrupted. But when a fuel pump stopped functioning on the generators Monday after Hurricane Wilma hit, the Verio site was shut down.

Service wasn't fully restored until around 10:30 pm PDT on Tuesday night. Verio customers as far away as Washington state experienced interruptions. During the outage, people were unable to receive e-mails and Web sites hosted in Boca Raton were unable to be reached.

None of the servers offering shared hosting in the Verio Boca Raton facility were mirrored in another location, something that could have kept e-mail and Web sites functioning. The main reason these services weren't replicated is because of cost, Boyle said. Adding that feature to the service would result in higher prices for customers, who pay between $12 and $50 per month to use Verio's e-mail and Web services that share common servers.

Analysts say it's important for Verio to appease customers affected by the outage. But they also say that the incident should serve as a wake-up call to businesses to take a more pro-active role in developing their own disaster-recovery plans.

"Customers need to understand the dynamics of the recovery system that their provider has in place," said Counse Broders, research director for network services at Current Analysis. "Whether it's a hurricane in Florida, an earthquake in California or a tornado in Dallas, a natural disaster can happen. And customers need to ask questions ahead of time and decide for themselves if they need to pay extra to have their data mirrored."

Even large companies that have their data mirrored by a provider or do it themselves often have the replicated data in a facility within the same geographic area. During the terrorist attacks in New York on Sept. 11, 2001, many financial institutions had duplicate servers mirrored in facilities across the river in New Jersey. But in the case of a widespread terrorist attack or a natural disaster, such as a hurricane, those facilities could also have been damaged or compromised, said John O'Keefe, senior analyst for managed services at Current Analysis.

"In general, geographic diversity in selecting hosting locations needs to be wider," O'Keefe said. "That should be standard for companies whose businesses rely heavily on stored data or hosted services like e-mail and e-commerce."

To ensure an incident like this won't happen again, Verio's Boyle said, the company is investigating ways to improve the reliability of its service during future disasters. It's reviewing everything from increasing the frequency of testing on the backup generators to assessing the cost of mirroring shared hosting data in other locations to even moving its servers out of Florida altogether.

"We're developing a plan to make sure this doesn't happen again," he said.