Earthquake fears prompt business continuity

SAN FRANCISCO--On April 18, 1906, San Francisco was rocked by an earthquake that destroyed large swathes of the city and claimed the lives of more than 3,000 inhabitants of the Bay Area.

One resident, the author Jack London, wrote at the time: "San Francisco is gone. Nothing remains of it but memories...Its industrial section is wiped out. Its business section is wiped out. The factories and warehouses, the great stores and newspaper buildings...are all gone."

A century later, a bigger, bolder, rebuilt and more resilient San Francisco is more important than ever. Now it sits near the heart of the global IT industry and serves as a major world financial center. But it remains well aware of the terrible potential that exists along the San Andreas fault.

The vast skyscrapers downtown may now be built to withstand huge pressures, but what about the infrastructure and the systems that keep modern business ticking--and the people who must be able to access them? Business continuity and disaster recovery are serious issues for all organizations in the Bay Area.

Barry Cardoza, head of business continuity contingency planning and disaster recovery at Union Bank of California, which is based in downtown San Francisco, told Silicon.com that earthquakes are a major consideration in the area but only as part of a complex patchwork of worst-case scenario planning--which includes everything from pandemics to terrorist attacks and acts of civil disorder.

"You have disasters that you can see coming, and you've got disasters that you can't see coming," Cardoza said, "and an earthquake is an example of (the latter). And you don't know how bad it's going to be until it hits."

As such, the bank must have processes in place ahead of such an event to mitigate the threat. Simply reacting is not a strategy. The contingency department must also understand every aspect of the business and weigh downtime for each in terms of financial and reputational damage.

Stock traders, for example, might need to be at their most active in the wake of a disaster.

"If they can't react when the markets are impacted," Cardoza said, "then that could put you out of business. They might normally be turning over millions, but the losses could be in the billions."

Multiple data centers in diverse locations, mirrored sites that can take over at the flick of a switch, "hot sites"--where staff can walk in and start working exactly as they would if they were in their normal location--and a vast amount of redundancy are also key to business continuity planning, said Cardoza, who is a member of the Bay Area Recovery Coalition with a number of peers from other major banks.

"We have real-time mirroring between data centers," he added. "It's no longer a question of how long will it take us to back up from tape. It's a matter of minutes not hours once we switch over."

Another Bay Area business that must plan for any worst-case scenario is FCE Benefit Administrators, which handles health care claims. Tom Toot, IT manager at FCE, told Silicon.com an earthquake would present his own company with the challenges of business continuity and disaster recovery at the very moment its services might be most in demand.

"We need business continuity. It is critical for us," said Toot. "A few missed e-mails could prevent treatment being approved in a timely manner. I really don't want to say it's a matter of life or death, but it really could be."

A double challenge
Communications lines are critical, Cardoza said. "If we can't communicate with one another, then we can't mitigate the problem."

Senior executives all have locations at their disposal that are fitted with videoconferencing facilities, among other communications options.

"There is a lot of redundancy," said Cardoza, showing three desk phones that all connect via different lines. "If we can't go one route then we can go another. We have redundant carriers and redundant phones."

He said the bank is also looking at satellite, BlackBerrys, walkie-talkies and SMS as communication alternatives, as well as at sharing initiatives across the whole organization--which is headed by Tokyo-Mitsubishi Bank, an institution that must weigh similar tectonic concerns for its own region.

The only benefit afforded in earthquake planning is that which is known: where the fault lies and what areas it affects. Fires, tornados and hurricanes tend to be bound by less clearly mapped lines.

"I'd rather be here than in tornado country, or in the way of a hurricane," said Cardoza.

Cardoza said as such the bank is able to plan recovery sites outside the areas that would be affected--though he conceded nowhere is completely safe.

"A business recovery site must be as far away as possible, but not so far that people won't go there. And you want hardware to be as far away from the affected area as possible. It might not be a case of having site A and site B; you may also have site C and so on."

However, Cardoza said while he hopes his business continuity planning will never be tested, he said it is far from a bottomless pit for funds. He said much of the planning offers day-to-day strategic value, such as using data centers for load-balancing.

He said one important development, bringing in Citrix, whose product allows staff to work from home if locations became inaccessible, was actually a result of executives requesting to work from home in order to improve efficiency.

"It was only once I was speaking to Citrix I realized this is one of the greatest opportunities for resilience," Cardoza said. "We're looking for a situation which is win-win from a strategic point of view. I certainly never want to see these plans put to the test."

Will Sturgeon of Silicon.com reported from San Francisco.