X

Rare peek: Inside Symantec's security fortress

The security world doesn't trust the Web. Symantec wants to prove naysayers wrong.

Elinor Mills Former Staff Writer
Elinor Mills covers Internet security and privacy. She joined CNET News in 2005 after working as a foreign correspondent for Reuters in Portugal and writing for The Industry Standard, the IDG News Service and the Associated Press.
Elinor Mills
10 min read
Some of the most sensitive data on the Web is housed in these safes guarded by Symantec's Certificate Authority operations -- digital keys used for authenticating Web sites so people can trust that the site they are on is legitimate.
Some of the most sensitive data on the Web is housed in these safes guarded by Symantec's Certificate Authority operations -- digital keys used for authenticating Web sites so people can trust that the site they are on is legitimate. James Martin/CNET

MOUNTAIN VIEW, Calif. -- The journey to the heart of the operation reminded me of the late '60s TV show "Get Smart," where one heavily fortified door leads to another locked entryway followed by more complicated defenses in a seemingly never ending series of entry points requiring PINs, badges, and irises or fingers scans. I balked at the DNA test. Joking. Actually, I was just along for the exclusive tour, flanked by a group of engineers and executives with high-level security clearances.

This is the belly of Symantec's Certificate Authority operations, where the company creates digital certificates and keys that prove Web sites are who they say they are and not an impostor trying to steal your data or spy on you.

Picture the scene. There's a building with no signage tucked amid a cluster of beige buildings on the Symantec campus. Your generic office park, but one that houses vital data that pretty much anyone who surfs the Net comes into contact with in one way or another. Nestled within safety deposit boxes, hidden in nine safes, locked in a cage, housed in a secret room in the middle of the building are stored a million digital keys and cryptographic certificates.

You likely don't know they are there, but these digital keys are exchanged and verified behind the scenes in fractions of a second, the time it takes to open a Web site. Usually, the only visible representation showing this is going on is a green URL bar or padlock symbol at the top of the browser when you use "https" (Hypertext Transfer Protocol Secure), indicating that the communication is taking advantage of the SSL (Secure Sockets Layer) cryptographic protocol. Most Internet users take it for granted that when they click on a URL they are going to the site they intend to visit, but underlying that action is a complex infrastructure for assigning the digital equivalent of identity papers to companies, government agencies and organizations running Web sites that require a high level of trust. Without this assurance, people couldn't trust that the site they are visiting that advertises itself as their bank is really their bank.

There have been hiccups. A series of attacks on Certificate Authorities last year, including several that allowed people to get fraudulent SSL certificates, have some people questioning how trustworthy the system really is. A hacker breached the networks of several resellers of Certificate Authority Comodo's certificates and created fake Google, Yahoo, Microsoft and Skype certificates that would have allowed someone to impersonate secure versions of those sites. Then Dutch Certificate Authority DigiNotar was compromised and hundreds of fraudulent certificates were issued, including one that was used to try to trick Iranian Gmail users. In both cases, the fake certificates were revoked. Meanwhile, DigiNotar eventually went bankrupt. Separately, Certificate Authority GlobalSign had a public facing Web server breached but said none of its certificate infrastructure was affected.

To some, these incidents illustrate a core flaw in the system. "There are a very large number of certificate authorities that are trusted by everyone and everything," Peter Eckersley, senior staff technologist at the Electronic Frontier Foundation, told CNET after one of the attacks. "We have 1,500 master certificates for the Web running around. That's 1,500 places that could be hacked." Echoing the concerns, Mike Zusman of Web app security firm Intrepidus Group said at the time of the attacks: "These organizations act as cornerstones of security and trust on the Internet, but it seems like they're not doing basic due diligence that other organizations are expected to do, like the banks."

Inside Symantec's secret SSL Certificate Vault (pictures)

See all photos
Symantec's SSL Vault

Symantec wants to show the world that its high standards distinguish its Certificate Authority (CA) business from the rest of the pack. "Not all Certificate Authorities provide equal assurance, yet all are equally trusted by browsers," says Paul Meijer, senior director of enterprise security operations at Symantec, which bought VeriSign's identity and authentication business two years ago and is using its checkmark logo across all the Symantec brands. "It takes considerable resources to ensure that the security of the public and private keys are protected from hackers."

Symantec, for instance, uses 14 global data centers to issue digital certificates to browsers. They handle and verify 4.5 billion online lookups a day. Meijer opened the doors recently to one of its four main data centers that handle its Public Key Infrastructure, which Symantec boasts has military-grade technology based on Department of Defense standards for storing classified material. As expected, the front door is locked, guarded by one of the ubiquitous cameras and requiring a PIN to enter. Benign-looking bush planters serve as a barricade to block cars from driving up into the site. Inside, a guard on duty at all hours stops visitors (except he somehow missed the CNET photographer!) and requires photo identification and a signature of everyone entering the building. Subsequent doors have biometric readers, including fingerprints and Iris matches, for access, with fewer and fewer employees approved as you get closer to the center of the building with its double layers of metal mesh-enforced walls.

In the security operations center, behind a wire-enforced window, engineers monitor big and little screens that provide information on the performance of the system, display alerts if there is a problem with any of the servers and provide other duties that keep the system up and running 24/7. Down a corridor, through a fingerprint-protected biometric reader-ensconced door is the data center. Its chilly environs with racks and rows of servers look and feel like any old data center -- not much to look at. Entrance is restricted to personnel with a verified need-to-know classification and requires two-factor authentication including biometrics. The servers are secured behind military-grade locks.

At the far end, a special cabinet houses cryptographic tokens, also known as Hardware Security Modules (HSMs), that hold the private keys used to unscramble the ciphers for the public key. The two keys are needed to authenticate Web transactions, such as accessing a trusted site. Public keys for a company or Web site are embedded in Web browsers and stored online so anyone can access them. Private keys, which are kept private, are used by the key owner to unscramble the ciphertext that was created using the public key. Not necessarily the most intuitive concept. But it works seamlessly and when it doesn't you know you shouldn't trust that Web site. That's what most Internet surfers need to worry about.

Back to the tour. The cabinet is secured with a cool looking Piezoelectric gyro-generator lock that works by rotating the face. It requires a physical key and combination that only five people in the company know. None of those five employees can access the room without an authorized escort. Symantec prides itself on this separation of duties, which prevents against insider malfeasance but could prove problematic if all the authorized people were on the same flight during an airplane accident. The contents of this cabinet are so highly prized, it has no external hinges but instead has a contact magnet that sets off an alarm when the front is opened.

To ensure continued operations if anything goes wrong, Symantec has redundancy in its servers and its energy sources, which is fairly standard for a company safeguarding so much important data. In the event of a natural disaster, the city may run low on water and public transportation could be at a standstill, but the data center will still be bustling. In the case of a grid disruption a bank of batteries will pick up the slack until two five-megawatt diesel generators kick in, able to power the building for three to four days each, said Ralph Claar, senior manager of PKI Operations.

Key ceremony

Our group (thankfully) leaves the chilly data center room and is herded down a corridor to the hallowed Key Ceremony Room. This is where the magic happens and digital keys and certificates are created. The entrance requires a comic dance of Security Salsa with specific people's irises and PINs matched with badge access that has to be timed just right or risk triggering motion sensors inside and a trap door in the floor opens. Ok, that last part I made up. But the sequence of things does have to be just right or security is alerted, which is a good thing.

We enter and I'm unimpressed. Where is the pomp, the circumstance, the tapestry curtains and red velvet throne? Instead, there are a few ordinary looking office chairs around a bland conference table, a couple of Windows-based PCs and a log book that guests sign with a ballpoint pen. The room is air-gapped, meaning the computers do not have Internet access or a connection to the outside world. The machines are used to create the cryptographic keys and their digital certificate wrappers. I brace myself for the ceremony, but wait. There's another secret room. We are ushered through a door at the far end of the room, requiring another flurry of badges and biometrics. There the true jewels of the operation are kept.

This is "probably the most secure room on campus," says Meijer, in an understated way. Like Russian nesting dolls, there are vaults within vaults. Against the back wall sit 120 safety deposit boxes inside nine media storage safes within a burglar-proof cage with a special lock. Multiple keys are needed to open the locks, and specific people are entrusted with knowing the combinations.

In the safety deposit boxes are kept the most prized tokens carrying digital keys. They are slightly bigger than a credit card and contain a circuit board with seven chips. These tokens are used to cryptographically generate the pairs of public and private keys. Once a public key is generated, a certificate is created that serves as a wrapper for the public key and stores data such as who the key belongs to, what country they are located in, who issued the key, and how long it is valid for.

It gets more complicated. Each token has a password to unlock it to gain access to the root keys it stores. Root keys are the master public keys for certificate authorities and companies like banks that may serve as their own certificate authorities. They are kept offline in a safe, while corporate and other clients keep their respective versions of the root keys. Roots allow the companies to create subordinate public and private keys for themselves or others in what is known as a "chain of trust." In its vault, Symantec has root keys for 98 percent of the Fortune 100, according to Meijer. The tokens, which hold hundreds of key pairs, are tamperproof and meet federal government cryptographic processing standards, he says.

These tokens are tough nuts to crack because they hold so much sensitive data, literally the master keys enabling most of the Web commerce today. They have their own passwords, and the password bits are split into multiple pieces, called "shares." At least three of the shares are needed to unlock the token, sometimes more. Symantec employees are enlisted to serve as "shareholders" and each has a key to remove his or her particular secret share from a safety deposit box within the company vault. The shares are stored on USB thumb drives that are housed in plastic cases in the shape of keys. "You need six people to do anything with the public-private key pair," says Claar, whose math abilities are clearly beyond mine.

Each safe stores more than 1,000 tokens and more than 5,000 secret shares that are kept in numbered tamper-evident plastic bags and logged, with token numbers matched to numbers provided by the token manufacturer. Symantec's security consciousness has rubbed off on its token supplier and now that company is not only storing Symantec's firmware that is embedded on the tokens in a safe but also keeping it under video surveillance, according to Claar.

Accessing the Key Ceremony room at Symantec, which is protected by biometric security including iris scanners.
Accessing the Key Ceremony room at Symantec, which is protected by biometric security including iris scanners. Jamnes Martin/CNET

Finally, we are ready to see a simulation of a key creation ceremony. They are formal procedures that follow a script. Everything is videotaped. One person, the "operator," logs the steps so that the process can be audited and is admissible in court. Another person serves as the witness. The operator and witness follow written instructions and use proprietary Symantec software to create keys and certificates while a video camera records every keystroke and mouse click that is made. And it's not much to watch. This process can last anywhere from 20 minutes to multiple hours, depending on the size of the keys created. The longer the key length, such as 1,024 bits or 3,096 bits, the longer it takes to create (as well as the longer it takes to crack it.)

The most exciting part probably is when the token used to generate a key pair is inserted into a card reader connected to a PIN entry device that is connected to the computers where the operator is using the Certificate Authority application. The second most exciting part is probably when the key-shaped "shares" are inserted into the PIN device. But this happens quickly and then you are just back to watching two people sitting at computers for a while. Thankfully, we got the abbreviated version.

Like most things with security, there is nothing really all that exciting with the digital certificate infrastructure on which so much e-commerce depends, until something goes wrong. When fake Google certificates are used to try to deceive users in Iran, people see the cracks in the system and may feel less confident about using the Web for highly sensitive activities and data. And even if Certificate Authorities all beef up their security measures to the levels of Symantec's, there are still problems with the way the system operates. There is no automated process to revoke fraudulent certificates, no public list of certificates that have been issued, and no list of CA resellers or partners have been given a duplicate set of the master keys. And something very troubling -- there are no mechanisms to prevent fraudulent certificates from being issued by compromised companies or repressive regimes bent on surveillance.

"If too many other Certificate Authorities get attacked, the trust in e-commerce goes away," says Meijer. "The DigiNotar and Comodo attacks last year were big wake-up calls for the industry and it prompted our desire to want to go out and say we do a good job." At this point, self regulation should be adequate to get the lax players in line, he said. "We've been very big advocates within the CAB (Certification Authority/Browser) Forum to push for stronger requirements."