How a CTO faced his worst nightmare
After 9/11, anxious investors fretted whether the stock exchange could get back on its feet. Roger Burkhardt, chief technology officer of the NYSE, made sure they didn't need to wait long to learn the answer.
Blocks away from the devastation stood the august New York Stock Exchange. With the world suddenly turned upside down, anxious investors wanted to know whether the stock market's largest financial exchange would be able to open on September 17, the day when NYSE officials promised they would again ring the trading bell.
Thanks to the efforts of the NYSE's technology team and its chief technology officer, Roger Burkhardt, they were able to make good on that deadline. But whatever reassurance investors had in the NYSE's ability to stand up to the technical challenges of 9/11 has been subsequently overtaken by a new era of instability marked by fears of terrorist attacks, corporate accounting scandals, and a stock market meltdown.
CNET News.com recently spoke with Burkhardt about how his team got the exchange back on its feet in the aftermath of the terror attacks and the technology challenges the NYSE regularly faces because of the whipsaw movements of the stock market.
Q: After the attacks on September 11, when did you first step back into the NYSE?
A: We were back in that day and the next day. Although the exchange is only a block and a half away--I was able to see the towers out of my window--we were not actually damaged. Our trading floor was OK. We had two trading floors at the World Trade Center which had gone--all the people got out of there safely. From a raw facilities point of view we could have traded on the 11th and the rest of that week, but that clearly would not have been the right thing to do. The issues of getting back in were really that the authorities didn't want too many people downtown, and you needed a good reason to be there.
Were most of the trading firms in the area that connect with your systems all up and running by 9:30 am on Monday (September 17)? Were there any from outside or in the area unable to participate in trading that morning?
We had lost a lot of telephone lines that bring in data to our computer centers and also voice lines to the floor, which would have meant that we would not have had full access by all members. That raised some public policy issues, particularly for the retail investor; if their broker-dealer is the one who doesn't have connectivity, they would be disadvantaged.
| "I think September 11 was the biggest challenge that our technical team has had to face in recent years." |
There was a connectivity issue that affected not just our market, but all markets. There was also the fact that there were a number of firms that were scrambling to get into their back-up facilities. A number of large firms like Morgan Stanley and Merrill Lynch were affected. And then there were firms like Goldman Sachs, just down the street from here, who were like us in that their building was undamaged. In fact, the Merrill Lynch building was also undamaged, but they were just not allowed to come in because the authorities quite rightly wanted to focus on rescue operations. That affected all the markets. Clearly, if you want a market, you want it to be a fair market, with breadth of access. You don't want one retail investor to not be able to get through to sell or buy.
So by Monday, how did you manage to connect all the firms that connect to your systems?
We worked with member firms for the balance of that week to help them re-establish connectivity. We worked very closely with Verizon, whose staff did a tremendous job. We have a subsidiary called Securities Industry Automation Corporation. It's been around for over 25 years and provides data processing and communications capabilities for the securities industry. It was initially set up by the NYSE and the American Stock Exchange, but also provides services to a broader part of the industry--for example, market data systems for equities and options. It also is the collection point for all the post trade information for all instruments. What is important about that is that because so many of us use them, they have telephone lines coming in from everybody. They play this hub role where they can effectively use communications set up for one purpose in an emergency to recover something else.
9/11 was a terrible nightmare. Any other tech nightmares come to you while you sleep?
I think September 11 was the biggest challenge that our technical team has had to face in recent years. The impact was so broad and the conditions here were quite challenging. There were lots of distractions here. People had lost family and friends--and the teamwork they showed in the face of that was just tremendous.
What impact has the stock market downturn had on your trading volume and technology? How is the downturn influencing how much you plan to spend on technology?
Our volume is up year over year. I need to defer the spending question to the communications people because that is the practice here. What I will tell you is that we believe, as a management team, that technology is at the very core of our business. You'll hear that from Dick Grasso (the NYSE's chief executive) and the rest of the management team. Unlike a lot of the industry that was retrenching, we continue to invest in technology.
(Editor's note: According to the NYSE, the exchange spent around $2 billion on technology in the past decade. Although the company doesn't make public annual technology spending data or spending on individual projects, a spokesman said that its technology spending is projected to increase by 18 percent year over year in 2002.)
Has 9/11 led to a boost in your IT spending?
Even before September 11, our plans called for a modest increase. Sept 11 has added to that. We continue to invest in tech. I'll give you and example of why it is important. One of the things we did between September 11 and September 17 is we prepared to allow the American Stock Exchange, which could not get into its building, to trade equities from one of our five trading rooms. The reason we were able to do that is that we had invested in a very flexible thin-client trading technology, as had they, which was very centrally managed and flexible so that we basically redirected a whole set of resources to be used to the American Stock Exchange's applications, and it worked flawlessly. The SIAC technicians set this up over just a few days.
We, in fact, traded record volume for that week, if memory serves me right; we traded 2.3 billion on the 17th and over 10 billion for the week. You just have to keep investing in these kinds of technologies if you want to grow your business. And what we are finding is that not only are our volumes up, but the advent of decimalization has driven more transaction for the volume. (Editor's note: Decimalization is a switch to dollars-and-cents pricing instead of fractions.) We get a lot more quotes, a lot more message rates for the same volume. So we continue to make very significant investments.
A glitch or problem with technology at the NYSE wouldn't result in consumer complaints about lost shipments or something like that, but rather could lead to tragic financial losses for investors. So would you say that the NYSE is a "late adopter" of technology rather than an "early adopter"?
We certainly need to be a very careful adopter of technology. The trust of confidence that people have in our market depends on, among other things, a very, very outstanding record of availability and reliability. We need to be very careful in how we adopt technology, but I wouldn't say that leads us to be on the trailing edge as a user of technology.
Where does Linux figure in your plans?
We are in the thin-client space with Linux on the front end. Because we are very early into electronic transactions, we actually were forced to build things that if you started 10 years later you would have bought. We have a set of people who really know how to build this stuff--build middleware for example. If you were starting a stock exchange today, you wouldn't build middleware. You would just go buy it. We have a mixture of bought and what we built a number of years ago. That means we have the technical depth that we can be a careful adopter of technology without paying the price of being late to market.| "With the potential for cyber threats, the advice I get is, 'Don't tell anyone about anything we are using.'" |
I just used that as an example that we are not a trailing edge adopter. And I am a little sad about this because I enjoy talking about a bunch of technologies here from many great companies like HP, IBM and others. But with the potential for cyberthreats, the advice I get is, "Don't tell anyone about anything we are using. Don't tell people we are using Linux." So I am reluctant to get into our technology infrastructure, but I will tell you that our thin-client model is a very flexible model.
Our application servers are all off site at two data centers that run in active-active mode. They are always processing work, and either one is capable of taking the whole load, and the clients on the trading floor are very, very thin so you can boot them very fast. That is one of the things about Linux--you can recover very quickly. That was very relevant to September 11. It allowed us to A, support the American Stock Exchange, and B, it made it much easier to build our contingency trading floors by having that thin-client model.
To date, there really are no services as hyped by Microsoft, Sun, IBM and others. Do you think the actual services will live up to the hype? Are you looking into services for the NYSE?
We have been in a message-based market since the mid 70s. Some stock exchanges put terminals on people's desks to enter orders and receive reports. We never did that. We don't have any announcements around the use of Web services.
What's your take on the notion that adopting new technologies leads to greater productivity?
Our situation is interesting in that our volume has grown 30-fold, I would say in the last 20 years, but the number of people producing that volume is fixed by our corporate constitution. The number of members of the stock exchange, the ones that trade, is fixed by the corporate constitution. We are in an industry with continually aggressive growth and a completely fixed work force, so we are reliant on technology for the productivity we need to keep up with the market growth. The management team has also taken a very aggressive attitude toward bringing in new kinds of trading services, like automatic executions, bringing in a system to distribute the specialist books so that everyone can see the orders, and bringing in new forms of electronic executions for the large institutional investors. It is technology that makes that possible. It is absolutely what makes it possible.
What's been your biggest disappointment with Web-based technologies?
We have been pretty comfortable. I'll give you an example. When decimals came out, what happened is that the liquidity was spread over many more price points because the (Securities and Exchange Commission) mandated a penny increment. So what happened was that our specialists needed urgently to see a bigger window into their books. They had a system that showed them 8 price points. And they could page up and down. But when the markets go crazy that is a little slow. We needed to put, very urgently, a tool in their hands that showed them more information to help them deal with decimalization.
We were able to develop that tool and roll it out into production in six weeks using Web technology. There is no way we could change a mainstream application that quickly and be the careful adopters of technology we were talking about previously. The risks would be too great.
I see this thing as a kind of portfolio, and you have to choose what aspect of your portfolio to use for what purpose. And so we have found Web technologies to be very valuable inside our business, if you like.
