Internal memos show that the software giant hadn't patched its own network against the Slammer worm, causing many of its services to fail.
Code Red for security
Virulent worm calls into doubt
our ability to protect the Net.
"All apps and services are potentially affected and performance is sporadic at best," Mike Carlson, director of data center operations for Microsoft's Information Technology Group, stated in an e-mail sent at 8:04 a.m. PST Saturday to other members of Microsoft's operations groups. "The network is essentially flooded with traffic, making it difficult to gather details concerning the impact."
The messages put Microsoft in an awkward position: The company relies on customers to patch security flaws but the events of last weekend show that even it is vulnerable. In this case, Microsoft urged customers to fix a vulnerability in the SQL Server 2000 software, but it apparently hadn't taken its own advice. Moreover, despite its 1-year-old security push, the software giant still had critical servers vulnerable to Internet attacks.
"This shows that the notion of patching doesn't work," said Bruce Schneier, chief technology officer for network protection firm Counterpane Internet Security. "Publicly, they are saying it's not our fault, because you should have patched. But Microsoft's own actions show that you can't reasonably expect people to be able to keep up with patches."
For years, system administrators have complained about their inability to keep up with the steady stream of patches that have poured out of Microsoft and other software companies. In October, the software giant even raised the bar for what's considered a "critical" vulnerability, so that administrators wouldn't have to deal with so many patches that seemingly required immediate attention.
?Seems like every time I install a system patch, something else goes wrong with my system,? said Frank Beier, president of Web design firm Dynamic Webs. The designer said many system administrators won?t patch for many months, because they don?t trust Microsoft to fix the problem without breaking some other function of the software.
?In most cases, I'm better off just playing Russian roulette with the hackers until our servers are broken into,? he said.
In the case of SQL Slammer, it seemed that Microsoft had done it right. The company had informed customers six months earlier about a flaw and included patches in both a roll-up patch--a software update that includes all the latest patches--and in the company's latest service pack for Microsoft SQL Server 2000.
But even within Microsoft, something went wrong.
"At approximately, 10:00 p.m. (PST, Friday), traffic on the corporate network jumped dramatically, eventually bringing all services to a crawl," stated Carlson's memo. "The root cause appears at this time to be a virus attacking SQL."
Help and How-to
SQL Slammer worm
How to recognize and
prevent the virus.
On Saturday, the Microsoft's Windows XP Activation service was down, not because the servers were vulnerable, but because the company's internal network was inundated with junk data, Rick Devenuti, the chief information officer for the software giant, said in an interview Monday.
"We are not sure how the virus got into our network," he said.
That the company has SQL servers on the desktop is not surprising, he added. Many of its developers run the database on their PCs, and other test machines have vulnerable databases installed to replicate customer networks. Devenuti didn't know how the worm got into the system to affect those servers, however.
"It just takes one machine to get going," he said. "At any given point in time, it is hard to be 100 percent patched with any machine. We are working hard to make patch management easier. But 100 percent is a high bar and in this case we are not there."
News.com's Stephen Shankland contributed to this report.