X

Damage control

Forget ironclad shields against Slammer-style attacks. Companies need to focus on limiting their pain.

8 min read
 
Damage control
 
'Slammer' attacks may become way of life for Net

By Robert Lemos
Staff Writer, CNET News.com
February 6, 2003, 12:00PM PT

Siebel Systems thought it had dodged the bullet.

After a round-the-clock weekend watch for any infection of the so-called SQL Slammer worm--also known as Sapphire and SQL Hell--that hammered other companies' networks, the software maker apparently had escaped with only minor incidents in its international offices.

 
The SQL Slammer worm took a toll on networks in several large companies and municipalities, starting early on a recent Saturday.
Company Impact
Bank of America Many ATMs unavailable on Saturday, some other financial services affected
Continental Airlines* Some flights delayed or canceled because of online ticketing and electronic check-in problems
Microsoft Windows XP can't be activated, gamers have trouble connecting to Asheron's Call 2 and Microsoft Network access is spotty
City of Seattle* Emergency 911 network wasn't working
Washington Mutual ATMs unavailable until Monday; other financial services affected

Sources: CNET News.com, *Associated Press

But the quiet wouldn't last. More than three days after the worm started spreading on the evening of Jan. 24, Slammer somehow got into Siebel's internal network and sent traffic skyrocketing.

"It tied up our network," said Mark Sunday, chief information officer for the San Mateo, Calif., manufacturer of e-business applications. "The quantity of network traffic generated was an order of magnitude greater than anything we had seen before."

The disturbing lesson: Regardless of what protective measures have been taken, no network can be considered secure. Companies deemed bastions of security--such giants as Bank of America, American Express and Microsoft, under its year-old Trustworthy Computing initiative--found their internal networks deluged with data from the Slammer attack.

Although the worm caused roughly $1 billion in damage by some estimates, its most significant casualty may be the perception that companies can remain secure by keeping up with software patches and other protective updates. The truth of the matter, security experts say, is that companies need to begin treating such attacks as inevitable and focus on limiting their damage, rather than expending every effort trying to create an ironclad perimeter.

"We have recognized over the last few years that you cannot prevent a virus," said Joe Hartmann, director of North American antivirus research for security software firm Trend Micro. "There will always be an entry point."

Many of last week's victims found that their internal networks were more vulnerable than they should have been. And with a worm like Slammer, a small crack in the security surrounding a company can mean days, if not weeks, of cleaning the infection from inside systems.

In Siebel's case, the worm wreaked havoc even though the company had moved the lion's share of its network infrastructure to two data centers in Utah. Slammer was still able to swamp the company's internal network at its San Mateo headquarters, limiting use of e-mail and other resources for more than 24 hours while security teams hunted down infected servers.


Special Report
Tracking Code Red

"This ranked right up there with Nimda and Code Red," said Peter Allor, director of operations for the Information Technology Information Sharing and Analysis Center, one of many federal agencies that watch for major threats to critical networks.

The SQL Slammer worm, at 376 bytes of computer code, is much smaller than either Code Red's estimated 4KB (4,096 bytes) or Nimda's 60KB (61,440 bytes). Exploiting a hole that had been announced and patched by Microsoft six months earlier to the day, the worm inundated other computers on the Internet with a copy of itself. The worm's small size meant that it could send itself out in a single data package, or packet, that automatically infected the victim by loading Slammer into memory.

That efficiency made Slammer the fastest-spreading worm to date, infecting 90 percent of all vulnerable servers in its first 10 minutes, according to a report by a coalition of researchers from University of California San Diego, Lawrence Berkeley National Labs, and Silicon Defense, a security consultancy.

Code Red, by contrast, had to first search for vulnerable servers and then send a copy of itself afterward. While the number of servers infected by Code Red doubled every 37 minutes, Slammer-infected servers doubled every 8.5 seconds. Code Red infected nearly 400,000 computers in July 2001, while Nimda proliferated tenaciously through corporate networks two months later.

We have recognized over the last few years that you cannot prevent a virus.Yet Slammer was far easier to clean up than Nimda: Restart a server, and it was gone. And the worm infected 200,000 Microsoft SQL servers, about half the number of machines hit by Code Red, according to data from Incidents.org.

Nevertheless, the effects of the SQL Slammer were more visible than those of its predecessors.

"This worm did something that we have not seen before in that the customer is seeing it," Allor said. In the past, a worm or virus would disrupt the internal operations of its victims but, most times, would not be noticed by anyone outside the companies.

"In this case, the customer was affected," Allor added. "People weren't getting dial tones; airplanes couldn't fly; ATMs weren't giving cash."

Those problems weren't caused by any particular feature of the self-replicating program; they were caused by the fact that the worm attacked a weak point in many companies' systems--the database.

We have recognized over the last few years that you cannot prevent a virus."You can have millions of users all over the world that need to get information from that database," said Phyllis Schneck, chairwoman of the national executive board of the FBI InfraGard, the security world's equivalent of the Rotary Club. "Those people might not be vulnerable, but because they have to interact with the database they are affected."

That's exactly what happened to Bank of America, whose automated teller machines suddenly stopped dispensing cash early Jan. 25. The reason: The sheer volume of data produced by servers infected with Slammer smothered databases in Bank of America's internal network.

"When a person uses an ATM, (the ATM) communicates with databases on our internal networks," said Lisa Gagnon, a spokeswoman for the bank. "That communication couldn't happen because our network was so congested."

Indeed, a single infected server churning out copies of the worm could theoretically congest the bandwidth of a 100Mbps Ethernet, according to analyses of the program.

Later that evening, most of the company's ATMs were back in service. The company located the entry point but, for security reasons, would give no details of how the worm got in.


Special Report
Year of the Worm

"Either the patch failed or we missed some servers when we applied the patch," Gagnon said. "Going forward we will analyze what we can do to make sure this doesn't happen again."

Easier said then done. Even technology powerhouse Microsoft wasn't fully prepared for the worm, despite a year-long focus on improving corporate and software security.

A string of e-mails circulated within Microsoft's internal information-technology groups, subsequently viewed by CNET News.com, portrays a chaotic scene as the company scrambled to fight the SQL Slammer's onslaught.

Slammer showed us that it's hard for everyone to keep up with patches, no matter
who you are. "All apps and services are potentially affected and performance is sporadic at best," Mike Carlson, director of data center operations for Microsoft's Information Technology Group, stated in an e-mail sent at 8:04 a.m. PST on Jan. 25. "The network is essentially flooded with traffic, making it difficult to gather details concerning the impact."

Rick Devenuti, Microsoft's chief information officer, said the software giant learned some hard lessons.

While the company had hardened its networks, it hadn't cut connections between buildings on all ports--the software addresses on which an application listens for data from the network. The worm found its way to a connected port and, because the buildings weren't isolated, was able to spread throughout the campus.

"It just takes one machine to get going," Devenuti said in an interview last week. "At any given point in time, it is hard to be 100 percent patched with any machine. We are working hard to make patch management easier, but 100 percent is a high bar, and in this case we are not there."

Many corporate network administrators insist that they can win the race to close the holes through which worms squirm, but a rising number of security veterans are questioning whether companies should continue to make "totally patched" the rallying cry of their defense strategies.

"Slammer showed us that it's hard for everyone to keep up with patches, no matter who you are," said Mary-Ann Davidson, chief security officer for database leader Oracle.

Like Microsoft, Oracle has struggled to prioritize the proliferation of patches so that system administrators can get the most critical ones in place first. Such a ranking is necessary to ease the burden on overworked administrators and to help companies gauge whether to spend time testing the patches to ensure that they don't break critical functions of the system they are supposed to protect.


Special Report
Patchwork security

"The last thing you want is your order management system to go down in the last quarter because you didn't get a chance to test a patch before applying it," Davidson said.

Microsoft knows that well. A month before Code Red, the company twice had to release a patch for its Exchange servers to fix problems that the software upgrade introduced into the applications. Then the software giant pulled a December patch for Windows NT after reports that repaired systems were crashing.

The problem is simply one of time, said Steve Solomon, CEO of security software maker Citadel Security Software. His company develops systems that automate the installation of patches--assistance that is welcomed by many taxed network managers.

"Right now," Solomon said of these overburdened administrators, "the task is so tedious and so time-consuming they can't keep up with the manual process." 

Flash worm
The SQL Slammer worm--also known as Sapphire and SQL Hell--introduced some new facets to infectious programs.

UDP: The worm sent itself out on the Internet via user datagram protocol (UDP) packets, one of two data types commonly used on the Internet (the other being the familiar transmission control protocol, or TCP, packets). UDP can be sent and forgotten; no reply is required, allowing the worm to keep sending data to random addresses without stopping. The result: The deluge of data overwhelmed network connections and doubled the number of infected computers every 8.5 seconds.

Small size: The worms itself consisted of 376 bytes of assembly code and is entirely self-contained. One package of data acts as a probe and infector, making the worm extremely efficient. Code Red first established a connection to a vulnerable computer and sent itself to that computer. The two-step process slowed down that worm.

Database infector: While there has always existed the possibility to infect databases, Slammer is the first computer worm to infect SQL databases on such an extensive scale. Most worms have attacked a more public target: Web servers and e-mail. The fact that so many database servers--an estimated 200,000--could be infected underscores that a skilled attacker could have compromised much of the data on those servers before the worm struck.

Memory resident: The worm didn't copy any files to the hard disk, it stayed in main memory (RAM). Like Code Red, the Slammer worm could be easily erased by shutting down the system, since data in main memory doesn't persist without power. Yet, operating in RAM also allowed the program to execute quickly.

Source: Analyses by eEye Digital Security, Incidents.org, and CAIDA.

Related news
Microsoft releases anti-Slammer tools

Microsoft may get 'Slammed' in Korea

Slammer--the first 'Warhol' worm?

Counting the cost of Slammer

Slammer may not feed on Microsoft alone

Microsoft fails Slammer's security test

Setbacks in search for worm author

Asia fingered as Slammer's birthplace

Worm exposes apathy, Microsoft flaws

Computer worm slows global Net traffic

Nimda winds down; companies recover

Code Red for security

Year of the Worm

Patchwork security

Editor: Mike Yamamoto
Copy editor: Peggy Gannon
Design: Ellen Ng
Production: Meghan McDowell