Lessons: Keeping networks alive in New York
The communications breakdown at Ground Zero revealed the risks of concentrating networks in one place--but, after a century of haphazard growth, the system is impossible to uproot. Manhattan is a case study for cities facing similar problems worldwide.
Lessons: Keeping networks alive in New York
By Sandeep Junnarkar
Staff Writer, CNET News.com
August 28, 2002, 4:00 a.m. PT
NEW YORK--As architects submit proposals for rebuilding the World Trade Center, teams of engineers are working deep below the streets of Manhattan to construct a project of their own--one designed to keep the city connected to the rest of the world if disaster strikes again.
What they have found, however, is something many had feared all along: New York's concentration of key network interchanges in one place makes its communications exponentially more vulnerable, yet they have no choice but to continue working with it.
"The benefits of connecting all these networks together are so huge that decoupling them would be unrealistic," said Anthony Townsend, an associate professor at New York University's Taub Urban Research Center. As a result, he added: "Failures in one type of infrastructure system are likely to cascade into others because information technology and telecommunications have been linked within individual networks and between different networks like power, telecommunications and transportation."
This unsettling realization is forcing businesses to re-evaluate how to structure computer networks, data backup centers and links to the Internet. Companies, agencies and all manner of organizations are working on plans to connect their staffs to networks remotely in an emergency, all the while knowing that a single well-placed explosion could render their efforts futile.
If New York's situation seems impossible, that city is hardly alone. The concentration of networks was the product of a century of growth, an evolutionary process that every major metropolis has experienced. As a result, governments and businesses worldwide are observing the city's progress as a case study for ways to reinforce their own systems.
Diversification is key
Manhattan is an island crammed with many of the world's most powerful industries, all of which rely on digital networks to keep their global businesses running, but its systems are representative of major communications hubs everywhere. Last year's catastrophe exposed just how fragile all communications networks are, from the digital hubs that run American's cities right down to the most basic functions like using a telephone.
Wall Street firms of all sizes were shocked to discover that their well-laid plans for diversified networks and backup systems were little more than theories put to paper. Moreover, following the consolidation of the telecommunications industry, even phone companies did not have such critical information as the exact route of their networks.
The Bank of New York was one of many to learn of all this the hard way. The institution--more like a bank for banks--had multiple backup systems, dual-access in and out of each of its Manhattan buildings, and a resilient state-of-the-art network design known as "ring architecture." It still had a total communications breakdown.
"We had all the redundancies, only to find out that several central offices in the current configuration nationwide were connected to each other," said Donald Monks, senior vice president of the Bank of New York. "As a result of these dual connections, it's not buying you any redundancy from the position of failure of the senior central office. These problems were exposed by 9/11, and in some cases we think they still exist."
Faced with an exodus of companies that could erode its tax base, New York officials are reviewing various diversification proposals to reassure businesses that even if one network is destroyed, alternate systems can pick up the slack. One idea is to convert abandoned underground water pipes into fiber conduits.
"These huge pipes would be an excellent conduit for making the city's landline telecommunications infrastructure more robust," said Agostino Cangemi, the city of New York's deputy commissioner for technology and telecommunications. The cast-iron, high-pressure pipes run up and down every street below 34th Street and in Brooklyn.
The city also plans to harness its 20,000 light poles and municipal rooftops for wireless data and cellular communications. The previous mistake was to concentrate major antennas in one location at the World Trade Center: When the twin towers went down, so did all cellular communications in the area. Officials are already negotiating with Ricochet to resume its services from atop 3,000 Manhattan light poles.
Fiber all over the map
Cangemi said the city will also encourage the construction of so-called lateral fiber conduits. At present, shared fiber conduits run north and south under New York's avenues but not east and west along its streets.
"Each time a building on a side street needs a conduit, it costs about $200,000 to build a new one," Cangemi said. "This initiative is intended to help lower carrier costs associated with getting fiber into the building and to alleviate a reliance on copper connections to a single central office."
That subsidiary, which provides data processing and communications services for the securities industry, is fortifying its infrastructure with dedicated fiber optic lines it can track. The goal is to create a metropolitan-area network that guarantees geographic diversity of its routes.
Instead of linking to just two of the system's data centers, member firms will connect to the metropolitan network initially at a minimum of four points--two in Manhattan, one in neighboring Westchester County and one in New Jersey. The route between these connection points will consist of interlocking fiber-optic rings; if one part is broken, traffic is routed around the damage. The NYSE hopes to have this infrastructure in place by the end of this year.
The exchange is also pushing an aggressive move to use the Internet because of its greater flexibility and self-healing properties, said Roger Burkhardt, the chief technology officer at the NYSE. "A large part of what we are investing in is a move to modern technologies. We started the move to a common IP-based infrastructure about two years ago so we would have much more flexibility," he said.
Others are investing in early-warning technologies such as embedded chips that could detect anything from leaks along waterways to structural weaknesses at a power facility. Such sensors could also sense explosives or unauthorized entries to a building.
"My biggest problem is, how do you sort through all that information that comes from thousands of data points?" Zimmerman said. "False positives are a huge problem, sending the whole system out of whack. It could be a simple thing like a roach dropping in front of the sensor."
Technology as archeology
In many ways, engineers seeking to update the nation's communications infrastructure are doing battle with history. Most large U.S. cities have grown by building new technologies upon older ones. Because of this haphazard construction, ultramodern digital lines often run precariously beside waterways, sewer pipes and aging telephone lines.
"If you look back to the evolution of the telegraph and telephone, they largely followed existing infrastructure networks like railroads--and like the Interstate highway system followed the railroads," NYU's Townsend said. "These things don't come into being arbitrarily; they are laid down to reinforce existing economic links."
That industrial expediency can have serious consequences, as seen in last year's train derailment and fire in downtown Baltimore. The blaze brought down parts of WorldCom's UUNet network, interrupting Internet access along the Eastern corridor while causing scattered power outages.
Accidental damage to wires and cables by backhoes or other equipment during street work remains one of the most common reasons for floods, power outages and communication disruptions to this day. In addition, any efforts to remedy the situation are often exacerbated by rival companies unwilling or unable to share documentation that shows the exact location of all the wires.
"The level of competition and the tradition of secrecy as well as the fragmentation of the telco sector is a major reason no one has done a comprehensive inventory of telecommunications infrastructure before," Townsend said. "It continues to cause problems and scare people away from looking at it."
Not surprisingly, businesses are searching for ways to diversify their physical offices as well as their networks. But because communication lines are so heavily concentrated in Manhattan, they may need to set up auxiliary offices out of the area altogether to ensure their ability to use other networks and remain operational.
Credit Lyonnais, for example, thought it was safely out of reach on Sept. 11, having a branch on a different power grid in midtown Manhattan miles from its other office at Ground Zero. Still, the French bank's operations were disrupted.
The bank is setting up a backup center in SchlumbergerSema Global Recovery Services' facilities about 30 miles from the city.
Others point to a more obvious reason to keep offices farther apart from one another: In any type of disaster, related to terrorism or not, staffs will inevitably face physical obstacles that have nothing to do with the company itself.
"You can have the most hardened systems with motor generators, UPS (uninterruptible power supply), all sorts of network redundancies. But what if you can't physically get into the building?" said Gino Menchini, New York City's commissioner of technology. "Do you have a facility outside that area which is prepared with adequate technology where you can send people to work? I think that kind of facility is a value to anybody, whether it is a government office or a corporation." 
 Lessons: Keeping networks alive in New York
By Sandeep Junnarkar NEW YORK--As architects submit proposals for rebuilding the World Trade Center, teams of engineers are working deep below the streets of Manhattan to construct a project of their own--one designed to keep the city connected to the rest of the world if disaster strikes again.
What they have found, however, is something many had feared all along: New York's concentration of key network interchanges in one place makes its communications exponentially more vulnerable, yet they have no choice but to continue working with it. "The benefits of connecting all these networks together are so huge that decoupling them would be unrealistic," said Anthony Townsend, an associate professor at New York University's Taub Urban Research Center. As a result, he added: "Failures in one type of infrastructure system are likely to cascade into others because information technology and telecommunications have been linked within individual networks and between different networks like power, telecommunications and transportation." This unsettling realization is forcing businesses to re-evaluate how to structure computer networks, data backup centers and links to the Internet. Companies, agencies and all manner of organizations are working on plans to connect their staffs to networks remotely in an emergency, all the while knowing that a single well-placed explosion could render their efforts futile. If New York's situation seems impossible, that city is hardly alone. The concentration of networks was the product of a century of growth, an evolutionary process that every major metropolis has experienced. As a result, governments and businesses worldwide are observing the city's progress as a case study for ways to reinforce their own systems. Diversification is key Wall Street firms of all sizes were shocked to discover that their well-laid plans for diversified networks and backup systems were little more than theories put to paper. Moreover, following the consolidation of the telecommunications industry, even phone companies did not have such critical information as the exact route of their networks. The Bank of New York was one of many to learn of all this the hard way. The institution--more like a bank for banks--had multiple backup systems, dual-access in and out of each of its Manhattan buildings, and a resilient state-of-the-art network design known as "ring architecture." It still had a total communications breakdown. "We had all the redundancies, only to find out that several central offices in the current configuration nationwide were connected to each other," said Donald Monks, senior vice president of the Bank of New York. "As a result of these dual connections, it's not buying you any redundancy from the position of failure of the senior central office. These problems were exposed by 9/11, and in some cases we think they still exist."
Faced with an exodus of companies that could erode its tax base, New York officials are reviewing various diversification proposals to reassure businesses that even if one network is destroyed, alternate systems can pick up the slack. One idea is to convert abandoned underground water pipes into fiber conduits. "These huge pipes would be an excellent conduit for making the city's landline telecommunications infrastructure more robust," said Agostino Cangemi, the city of New York's deputy commissioner for technology and telecommunications. The cast-iron, high-pressure pipes run up and down every street below 34th Street and in Brooklyn. The city also plans to harness its 20,000 light poles and municipal rooftops for wireless data and cellular communications. The previous mistake was to concentrate major antennas in one location at the World Trade Center: When the twin towers went down, so did all cellular communications in the area. Officials are already negotiating with Ricochet to resume its services from atop 3,000 Manhattan light poles. Fiber all over the map "Each time a building on a side street needs a conduit, it costs about $200,000 to build a new one," Cangemi said. "This initiative is intended to help lower carrier costs associated with getting fiber into the building and to alleviate a reliance on copper connections to a single central office."
That subsidiary, which provides data processing and communications services for the securities industry, is fortifying its infrastructure with dedicated fiber optic lines it can track. The goal is to create a metropolitan-area network that guarantees geographic diversity of its routes. Instead of linking to just two of the system's data centers, member firms will connect to the metropolitan network initially at a minimum of four points--two in Manhattan, one in neighboring Westchester County and one in New Jersey. The route between these connection points will consist of interlocking fiber-optic rings; if one part is broken, traffic is routed around the damage. The NYSE hopes to have this infrastructure in place by the end of this year. The exchange is also pushing an aggressive move to use the Internet because of its greater flexibility and self-healing properties, said Roger Burkhardt, the chief technology officer at the NYSE. "A large part of what we are investing in is a move to modern technologies. We started the move to a common IP-based infrastructure about two years ago so we would have much more flexibility," he said. Others are investing in early-warning technologies such as embedded chips that could detect anything from leaks along waterways to structural weaknesses at a power facility. Such sensors could also sense explosives or unauthorized entries to a building.
"My biggest problem is, how do you sort through all that information that comes from thousands of data points?" Zimmerman said. "False positives are a huge problem, sending the whole system out of whack. It could be a simple thing like a roach dropping in front of the sensor." Technology as archeology "If you look back to the evolution of the telegraph and telephone, they largely followed existing infrastructure networks like railroads--and like the Interstate highway system followed the railroads," NYU's Townsend said. "These things don't come into being arbitrarily; they are laid down to reinforce existing economic links." That industrial expediency can have serious consequences, as seen in last year's train derailment and fire in downtown Baltimore. The blaze brought down parts of WorldCom's UUNet network, interrupting Internet access along the Eastern corridor while causing scattered power outages. Accidental damage to wires and cables by backhoes or other equipment during street work remains one of the most common reasons for floods, power outages and communication disruptions to this day. In addition, any efforts to remedy the situation are often exacerbated by rival companies unwilling or unable to share documentation that shows the exact location of all the wires. "The level of competition and the tradition of secrecy as well as the fragmentation of the telco sector is a major reason no one has done a comprehensive inventory of telecommunications infrastructure before," Townsend said. "It continues to cause problems and scare people away from looking at it." Not surprisingly, businesses are searching for ways to diversify their physical offices as well as their networks. But because communication lines are so heavily concentrated in Manhattan, they may need to set up auxiliary offices out of the area altogether to ensure their ability to use other networks and remain operational. Credit Lyonnais, for example, thought it was safely out of reach on Sept. 11, having a branch on a different power grid in midtown Manhattan miles from its other office at Ground Zero. Still, the French bank's operations were disrupted.
The bank is setting up a backup center in SchlumbergerSema Global Recovery Services' facilities about 30 miles from the city. Others point to a more obvious reason to keep offices farther apart from one another: In any type of disaster, related to terrorism or not, staffs will inevitably face physical obstacles that have nothing to do with the company itself.
"You can have the most hardened systems with motor generators, UPS (uninterruptible power supply), all sorts of network redundancies. But what if you can't physically get into the building?" said Gino Menchini, New York City's commissioner of technology. "Do you have a facility outside that area which is prepared with adequate technology where you can send people to work? I think that kind of facility is a value to anybody, whether it is a government office or a corporation." |
| ||||||
