X

Inside Symantec's security bunker

What goes on behind the locked underground doors, as the company hunts down hacking attacks and tracks botnets?

Tom Espiner Special to CNET News
7 min read
In one of the rolling hills above Winchester, England, is a decommissioned nuclear bunker that houses Symantec's U.K. Security Operations Center.

The facility, built at enormous cost to British taxpayers at the end of the Cold War in the early 1990s, is now owned by the security company. The popular image of a bunker is a dank, rat-infested hole in the ground, but luckily for Symantec's team, the interior looks surprisingly like any other office.

The facility is home to Symantec's U.K. Managed Security Services team, whose main task is to filter and monitor data fed back from customers' intrusion prevention systems, firewalls and intrusion detection systems.

Symantec's bunker

The Winchester team analyzes some 1.5 billion lines of code per day, said Jeff Ogden, Symantec's director of managed security services for Europe, the Middle East and Africa. "We spend our lives gathering and analyzing information and intelligence," he said. "This is an enormous amount of information, and we're trying to pull it into a coherent state."

The managed security services team is located in a room glassed off from the main bunker, which has 15 workstations ranged in three rows of five. Four large flat-screen monitors, mounted on the wall, face the workstations. Sky News plays constantly in the background to help the team monitor the geopolitical situations that may affect the info-threat landscape.

Tight security
Access to the bunker is closed--even other Symantec personnel cannot enter the building without prior clearance. Any visits must be announced at least 24 hours in advance. Symantec customers must sign nondisclosure agreements before visiting.

Once inside, all employees must log in at a special workstation and must log out when leaving. Three external cameras have a 360-degree view of the building. A digital recorder keeps 30 days of backup. The bunker runs round the clock, staffed by a minimum of four and a maximum of 15 analysts.

Even the atmosphere inside is highly managed. It is pressurized to 1.5 pounds per square inch greater than outside air pressure, so air is constantly being forced out--handy if someone decides to drop an atomic bomb in the vicinity. In the event of a nuclear attack, the air can be filtered through charcoal, and there are still safeguards in place against a gas attack.

The bunker has features like a security alarm--two strips of black plastic with glowing red insides--that's activated if any unauthorized visitor steps inside the glassed-off internal perimeter, where the analysts work away. Get too close to the alarm and it bleeps and registers an intruder.

If anyone gets past that, there's one last line of defense to deal with. "That's when I appear with a baseball bat," said Gordon May, Symantec's facilities manager.

Globally, there are 120 million desktops and servers using Symantec's products, which all feed back samples of malicious code. The company uses basic agent technology to collect the information, or customers can choose to send in the information manually.

"We deploy a small agent onto the customer collection point--the firewall, or the syslog server. The agent is a small piece of software that collects, compresses, signs and encrypts the data before forwarding it to us," Ogden said.

The data process
Once the data has been collected, it is sent to Symantec where it is analyzed and, if there is any danger of attack, a report is speedily sent to the client. "If the situation is critical or an emergency, we pick the phone up and say to the customer 'You could be under attack,'" Ogden said.

All customer information is stored centrally and run through two filters: a "progressive threat model," which decides whether the code is a threat, and an "expert query engine." The expert query engine decides what the threat is targeting, where it's coming from and what the threat is. This code is then analyzed by a Symantec engineer and the incident classified according to its threat level:

• Informational: The client has been scanned by hackers, but no more action is required

• Warning: The client has been scanned and a vulnerability has been detected by hackers

• Critical: The client has been scanned, and vulnerable machines are being targeted

• Emergency: There is a possibility of code being deposited on vulnerable machines

During ZDNet UK's visit to the facility, an attempted distributed denial-of-service attack, launched using a botnet in Romania, was detected.

"We profile the threat by finding out where it's being launched from, who it's being aimed at and what it's trying to achieve," Ogden said.

On a wider network
The Security Operations Center's Winchester facility is part of Symantec's global network of information monitoring stations. Customer data is monitored in five centers. The other four are located in Sydney, Australia; Munich, Germany; Alexandria, Va.; and San Antonio.

The security operation centers work closely with Symantec's seven security response centers, located around the globe, in locations including the U.S., Canada, Ireland, Japan and Australia. Where the primary role of the operations center is to identify attacks against customers, the response centers work on a higher level and collate information from a wider variety of sources.

Along with monitoring viruses directly detected by customers, Symantec scans 25 percent of global e-mail traffic for malicious code. It has a number of "honeypot" e-mail boxes, which are accounts provided by ISPs. They are not used, so anything that ends up there is usually spam, Trojan horses, viruses or other forms of malicious software.

An attack quarantine system linked to the honeypot network captures such malicious code. "It is a virtual network that simulates servers, and so looks like a real network," said Art Wong, vice president of security response and managed security services at Symantec.

Symantec maintains a list of all the vulnerabilities found across its network, called Bugtraq. Wong said that it's both a clearing house and a database of vulnerabilities. This list is shared with other security vendors to speed up the process of issuing patches.

The threat of botnets
As a leading security vendor, Symantec is well-positioned to identify future threats. Some of the biggest offenders on the radar at the moment are botnets, which are extensive networks of compromised computers controlled by hackers. These botnets are usually used to launch distributed denial-of-service attacks, which effectively flood Web servers or e-mail boxes with traffic.

The growth of botnets is a major problem, with a 100 percent increase in the U.K. since 2004, according to Symantec. The company believes that right now, the U.K. contains the highest number of botnets in the world.

"Just over a third of the botnets we've seen are in the U.K.," said Wong, quoting figures from Symantec's Internet Security Report VIII, published in September 2005. This is higher than the U.S., which has traditionally had more botnets.

The high incidence of botnets in the U.K. probably has to do with the recent explosion in broadband usage and the fact that most U.K. home users wouldn't know if their computer was compromised, Wong suggested. "Maybe there's a slightly lower awareness level in Britain of botnets," he said. "The IP addresses could come from legitimate machines that have been compromised by hackers. Maybe the machines don't have patches, or are not running up-to-date anti-malware products. Plus, if you have 10,000 machines in a botnet, it's difficult to track back to each IP address."

Taking control
On average, it takes eight minutes for a new machine to be compromised when hooked up to the Web for the first time, according to Symantec tests on a Microsoft Windows PC not running XP Service Pack 2 or antivirus software.

There is a particular danger for businesses using the same network as a compromised machine, because once one machine has been infected behind the firewall, hackers can use it to infect others. "If attackers manage to infect a machine within an organization, they can profile additional machines within that subnet. Executable code can be injected onto other machines to profile the users," Ogden said.

Symantec does not tell those people with compromised IP addresses that their computers are being controlled by hackers, due to the sheer scale of the problem. "A botnet can consist of thousands of machines, and we just don't have the time to contact everyone. Our first priority is our customers," Ogden said.

However, when it comes to serious incidents, Symantec does support the police. But the company is keen to point out that it doesn't supply any direct details on customers. "The information we supply to our customers belongs to them, and it's up to them to provide information to law enforcement agencies regarding any suspect activity. When companies are targeted, it's the customer who initiates giving information about the offending individuals," Ogden said.

It also supports the police in its efforts to counter botnets. "In the U.K., the National Hi-Tech Crime Unit has been proactive in trying to close down botnet activity. We welcome any initiative which closes down botnets," Ogden said. "We have had some contact with the authorities in the past, and it works quite successfully."

If a company is the subject of an attack, Symantec recommends it goes to the police. Symantec will only go so far with chasing potential criminals. If an attack has been unsuccessful, they are unlikely to be hunted down, Ogden said.

"If we have controlled and closed down a particular threat to a customer, there's not a great deal of benefit in tracking down the individuals who mounted the attack," he said.

Tom Espiner of ZDNet UK reported from London.