CNET también está disponible en español.

Ir a español

Don't show this again

Tech Industry

Lessons: Keeping networks alive in New York

The communications breakdown at Ground Zero revealed the risks of concentrating networks in one place--but, after a century of haphazard growth, the system is impossible to uproot. Manhattan is a case study for cities facing similar problems worldwide.

     


     
    Lessons: Keeping networks alive in New York

    By Sandeep Junnarkar
    Staff Writer, CNET News.com
    August 28, 2002, 4:00 a.m. PT

    NEW YORK--As architects submit proposals for rebuilding the World Trade Center, teams of engineers are working deep below the streets of Manhattan to construct a project of their own--one designed to keep the city connected to the rest of the world if disaster strikes again.

    In a subterranean labyrinth of aging pipes and bundled wires stretching for miles in every direction, these engineers are trying to make this city's densely packed communications networks less susceptible to the kind of widespread outages caused by the Sept. 11 attacks.

    What they have found, however, is something many had feared all along: New York's concentration of key network interchanges in one place makes its communications exponentially more vulnerable, yet they have no choice but to continue working with it.

    "The benefits of connecting all these networks together are so huge that decoupling them would be unrealistic," said Anthony Townsend, an associate professor at New York University's Taub Urban Research Center. As a result, he added: "Failures in one type of infrastructure system are likely to cascade into others because information technology and telecommunications have been linked within individual networks and between different networks like power, telecommunications and transportation."

    This unsettling realization is forcing businesses to re-evaluate how to structure computer networks, data backup centers and links to the Internet. Companies, agencies and all manner of organizations are working on plans to connect their staffs to networks remotely in an emergency, all the while knowing that a single well-placed explosion could render their efforts futile.

    If New York's situation seems impossible, that city is hardly alone. The concentration of networks was the product of a century of growth, an evolutionary process that every major metropolis has experienced. As a result, governments and businesses worldwide are observing the city's progress as a case study for ways to reinforce their own systems.

    Diversification is key
    Manhattan is an island crammed with many of the world's most powerful industries, all of which rely on digital networks to keep their global businesses running, but its systems are representative of major communications hubs everywhere. Last year's catastrophe exposed just how fragile all communications networks are, from the digital hubs that run American's cities right down to the most basic functions like using a telephone.

    Wall Street firms of all sizes were shocked to discover that their well-laid plans for diversified networks and backup systems were little more than theories put to paper. Moreover, following the consolidation of the telecommunications industry, even phone companies did not have such critical information as the exact route of their networks.

    The Bank of New York was one of many to learn of all this the hard way. The institution--more like a bank for banks--had multiple backup systems, dual-access in and out of each of its Manhattan buildings, and a resilient state-of-the-art network design known as "ring architecture." It still had a total communications breakdown.

    "We had all the redundancies, only to find out that several central offices in the current configuration nationwide were connected to each other," said Donald Monks, senior vice president of the Bank of New York. "As a result of these dual connections, it's not buying you any redundancy from the position of failure of the senior central office. These problems were exposed by 9/11, and in some cases we think they still exist."

    Now, major players on Wall Street are demanding that telecommunications carriers guarantee in writing that the network route is well diversified--not just at the central office but throughout the whole infrastructure. Monks suggested that companies pay a fee for such documentation to bind it as a contractual agreement.

    Faced with an exodus of companies that could erode its tax base, New York officials are reviewing various diversification proposals to reassure businesses that even if one network is destroyed, alternate systems can pick up the slack. One idea is to convert abandoned underground water pipes into fiber conduits.

    "These huge pipes would be an excellent conduit for making the city's landline telecommunications infrastructure more robust," said Agostino Cangemi, the city of New York's deputy commissioner for technology and telecommunications. The cast-iron, high-pressure pipes run up and down every street below 34th Street and in Brooklyn.

    The city also plans to harness its 20,000 light poles and municipal rooftops for wireless data and cellular communications. The previous mistake was to concentrate major antennas in one location at the World Trade Center: When the twin towers went down, so did all cellular communications in the area. Officials are already negotiating with Ricochet to resume its services from atop 3,000 Manhattan light poles.

    Fiber all over the map
    Cangemi said the city will also encourage the construction of so-called lateral fiber conduits. At present, shared fiber conduits run north and south under New York's avenues but not east and west along its streets.

    "Each time a building on a side street needs a conduit, it costs about $200,000 to build a new one," Cangemi said. "This initiative is intended to help lower carrier costs associated with getting fiber into the building and to alleviate a reliance on copper connections to a single central office."

    Some companies are turning to cutting-edge technologies to strengthen their businesses. The venerable New York Stock Exchange, which lost connections when falling debris severed its digital lines, is building a more resilient infrastructure for its Securities Industries Automation subsidiary.

    That subsidiary, which provides data processing and communications services for the securities industry, is fortifying its infrastructure with dedicated fiber optic lines it can track. The goal is to create a metropolitan-area network that guarantees geographic diversity of its routes.

    Instead of linking to just two of the system's data centers, member firms will connect to the metropolitan network initially at a minimum of four points--two in Manhattan, one in neighboring Westchester County and one in New Jersey. The route between these connection points will consist of interlocking fiber-optic rings; if one part is broken, traffic is routed around the damage. The NYSE hopes to have this infrastructure in place by the end of this year.

    The exchange is also pushing an aggressive move to use the Internet because of its greater flexibility and self-healing properties, said Roger Burkhardt, the chief technology officer at the NYSE. "A large part of what we are investing in is a move to modern technologies. We started the move to a common IP-based infrastructure about two years ago so we would have much more flexibility," he said.

    Others are investing in early-warning technologies such as embedded chips that could detect anything from leaks along waterways to structural weaknesses at a power facility. Such sensors could also sense explosives or unauthorized entries to a building.

    Yet Rae Zimmerman, director of the Institute for Civil Infrastructure Systems at New York University, warns against information overload. Not only can this slow an operation's efficiency, but an elaborate security system can work against itself.

    "My biggest problem is, how do you sort through all that information that comes from thousands of data points?" Zimmerman said. "False positives are a huge problem, sending the whole system out of whack. It could be a simple thing like a roach dropping in front of the sensor."

    Technology as archeology
    In many ways, engineers seeking to update the nation's communications infrastructure are doing battle with history. Most large U.S. cities have grown by building new technologies upon older ones. Because of this haphazard construction, ultramodern digital lines often run precariously beside waterways, sewer pipes and aging telephone lines.

    "If you look back to the evolution of the telegraph and telephone, they largely followed existing infrastructure networks like railroads--and like the Interstate highway system followed the railroads," NYU's Townsend said. "These things don't come into being arbitrarily; they are laid down to reinforce existing economic links."

    That industrial expediency can have serious consequences, as seen in last year's train derailment and fire in downtown Baltimore. The blaze brought down parts of WorldCom's UUNet network, interrupting Internet access along the Eastern corridor while causing scattered power outages.

    Accidental damage to wires and cables by backhoes or other equipment during street work remains one of the most common reasons for floods, power outages and communication disruptions to this day. In addition, any efforts to remedy the situation are often exacerbated by rival companies unwilling or unable to share documentation that shows the exact location of all the wires.

    "The level of competition and the tradition of secrecy as well as the fragmentation of the telco sector is a major reason no one has done a comprehensive inventory of telecommunications infrastructure before," Townsend said. "It continues to cause problems and scare people away from looking at it."

    Not surprisingly, businesses are searching for ways to diversify their physical offices as well as their networks. But because communication lines are so heavily concentrated in Manhattan, they may need to set up auxiliary offices out of the area altogether to ensure their ability to use other networks and remain operational.

    Credit Lyonnais, for example, thought it was safely out of reach on Sept. 11, having a branch on a different power grid in midtown Manhattan miles from its other office at Ground Zero. Still, the French bank's operations were disrupted.

    "We always thought that we have two locations--one in midtown and one downtown--on separate power grids. We thought that was OK since we were mostly protecting against localized issues like fires or power outages," said George Levitt, the bank's chief information officer. "But after 9/11, you had to adjust your thinking: Maybe one should be a little further away, outside of Manhattan."

    The bank is setting up a backup center in SchlumbergerSema Global Recovery Services' facilities about 30 miles from the city.

    Others point to a more obvious reason to keep offices farther apart from one another: In any type of disaster, related to terrorism or not, staffs will inevitably face physical obstacles that have nothing to do with the company itself.

    "You can have the most hardened systems with motor generators, UPS (uninterruptible power supply), all sorts of network redundancies. But what if you can't physically get into the building?" said Gino Menchini, New York City's commissioner of technology. "Do you have a facility outside that area which is prepared with adequate technology where you can send people to work? I think that kind of facility is a value to anybody, whether it is a government office or a corporation." 





    Technology experts say Sept. 11 showed that more steps need to be taken to prepare any company or city for an emergency, including both electronic and physical attacks.

    "We have initiated a portal that will warn the IT directors of every state about cyberalerts and attacks. A year ago, we didn't do that--nor did we have the capability."
    --Matt DeZee, chief information officer, State of South Carolina

    "The 9/11 attacks illustrated how fragile and poorly engineered our cellular communications system is. It is incapable of dealing with crisis. (Emergency responders) didn't have a way to communicate with each other since their radio systems are incompatible, and as a result they used cellular phones. But only one in 20 cell phone calls were connected on Sept. 11. In emergency situations, it is not acceptable for them to be relying upon a public system that doesn't give them priority."
    --Anthony Townsend, associate professor, New York University

    "It is only prudent to expect cyberattacks as well as physical attacks. I hesitate to speak for other people, but I really believe that if your business relies on networking and connectivity, you need to have an extremely resilient infrastructure. That was the one thing people learned: It is not enough to think you've got it, you have to have a process whereby your providers can attest to the fact that you really do have the diversity you need."
    --Roger Burkhardt, chief technology officer, NYSE

    "You can have the most hardened systems with motor generators, UPS (uninterruptible power supply), all sorts of network redundancies. But what if you can't physically get into the building? What do you do? Do you have a facility outside that area which is prepared with adequate technology where you can send people to work? I think that kind of facility is a value to anybody, whether it is a government office or a corporation."
    --Gino Menchini, commissioner of technology, New York City

    "You have to worry about insiders damaging your networks. Now when we hire, we look at their tax data and drivers license. We are also re-evaluating our employee exit procedures to make sure their access to the networks is taken away."
    --Dan Lohrmann, chief security officer, State of Michigan


    U.S. vulnerable to data sneak attack

    Is privacy the next casualty?

    National security and the patent squeeze

    Is your cable guy a spy?

    The tech side of homeland defense

    IT pros: U.S. government at cyber risk

    Enlisting science in terrorism fight

    Want to be a cybercop? Uncle Sam needs you

    FBI digs deeper into the Web

    In FBI shift, cybercrime is a priority

    Security confab calls for U.S. spending

    Patriot Act draws privacy concerns

    Privacy vs. safety: Terrorist threat shifts priorities

    ISPs aid FBI in terrorist search