Some sites back online after NaviSite goof

NaviSite, which bought Web hosting company Alabanza in August, planned to move Alabanza's servers from Baltimore to NaviSite's headquarters in Andover, Mass. NaviSite warned customers that the sites would be down for about 12 hours on Saturday. Instead, they have been down for five days.

At least for some, things are finally looking up.

More than half of Alabanza's 850 servers and an estimated 60 percent to 65 percent of the customers' Web sites were back up as of Wednesday afternoon, said Rathin Sinha, chief marketing officer at NaviSite. He told CNET News.com that he could not say exactly how many customers or Web sites overall had been affected.

Alabanza had about 300 data center customers who resold hosting services to a "large number" of other companies, he said. At the time Alabanza was acquired by NaviSite, the company said it powered nearly 165,000 sites.

Basically, the company encountered technical problems with migrating data to the new location. The original plan was to physically relocate 200 servers and transfer data from about 650 other servers over the Internet using a dedicated high-bandwidth line, Sinha said.

But the online transfer was taking too long, so the company decided to physically move additional servers to save time, he said. Then, there were some unanticipated synchronization problems involving Address Resolution Protocol requests, which interfered with the transfer of the data to the new servers, he said.

In addition, the servers were overwhelmed with traffic as thousands of customers checked whether their Web sites were back up. That further slowed down the network and recovery process, according to Sinha.

"We have been sequentially bringing customers back up one at a time to validate both Quality Assurance (QA) and optimum performance levels," Mark Clayman, senior vice president of hosting services at NaviSite, wrote in a statement on the company's Web site on Wednesday morning.

But even though they may be back online, unhappy customers are threatening to switch to a different hosting provider.

Taking data into their own hands
Cynthia Brumfield, president of consulting firm Emerging Media Dynamics, had been planning to drive a convoy with other unplugged NaviSite customers to the company's headquarters. The plan was to get copies of data that they could put back online themselves with another provider. Then Brumfield's site came back up.

"But we're downloading the files as fast as we can in the event the sites go down again. That way we can quickly move to another hosting provider if need be," she wrote on her IP Democracy blog.

"There are a lot of lessons to be learned in all of this. First, always keep back-ups of everything away from your Web hosting provider. Secondly, losing access to e-mail, client projects, Web sites, and other Internet-related tools causes a form of insanity," Brumfield writes. "One person estimated that her business lost $20,000 in sales over the past four days, which is a lot of money to a small-business owner. More importantly, some of NaviSite's customers have lost customers themselves, which is a kind of permanent loss."

Erik Staley, president of biotech consulting firm Valicom, isn't one of the lucky ones. Three sites he owns were still dark on Wednesday. "I don't know if I'm down for a day, a week, a month," he said in a phone interview. "One of the biggest problems has been lack of communication and only very generalized responses from NaviSite about this."

Staley opened a Gmail account so his clients and others could contact him. "But you can't go to my site and send in requests and learn more about me," he said. "Just the appearance of running some sort of substandard operation reflects badly on my business."

Asked about complaints about poor customer service, NaviSite's Sinha said the company was sorry for the problems and was doing the best it could to ameliorate the situation.

"We got way beyond our normal customer service volume, so the best we could do was to really open up more (phone) lines," bring all the salespeople into the office to answer calls and outsource the customer support to India overnight, he said.

"We take it very seriously. This impacted so many people and so many businesses," Sinha said. "We're doing it 24/7. This company is trying to get this thing fixed."