X

Microsoft meets the hackers

special report In the name of education, the software giant invites security researchers to infiltrate Windows systems.

11 min read

"Blue Hat" summit meant to reveal ways of the other side

By Ina Fried
Staff Writer, CNET News.com
June 15, 2005 4:00AM PDT

REDMOND, Wash.--The random chatter of several hundred Microsoft engineers filled the cavernous executive briefing center recently at the company's sprawling campus outside Seattle.

Within minutes after their meeting was convened, however, the hall became hushed. Hackers had successfully lured a Windows laptop onto a malicious wireless network.

"It was just silent," said Stephen Toulouse, a program manager in Microsoft's security unit. "You couldn't hear anybody breathe."

"(Hackers are) not just a bunch of disaffected teenagers sitting in their mom's basement. These are professionals that are thinking about these issues."
--Noel Anderson
Wireless networking
engineer, Microsoft

The demo was part of an extraordinary two days in which outsiders were invited into the heart of the Windows empire for the express purpose of exploiting flaws in Microsoft computing systems. The event, which Microsoft has not publicized, was dubbed "Blue Hat"--a reference to the widely known "Black Hat" security conference, tweaked to reflect Microsoft's corporate color.

The unusual March gathering, a summit of sorts between delegates of the hacking community and their primary corporate target, illustrates how important security has become to the world's most powerful software company. Microsoft Chairman Bill Gates himself estimated earlier this year that the company now spends $2 billion a year--more than a third of its research budget--on security-related issues. Security has also become one of the main themes of the company's developer conferences, including last week's TechEd event, where Microsoft pitched security improvements in Windows to 11,000 attendees.

Blue Hat was significant for other, less tangible reasons as well. It provided a rare glimpse inside the netherworld of computer security, where the ethical lines are sometimes fuzzy in the technological arms race between network engineers and the hackers who challenge them. During the course of the event, each side witnessed for the first time the inner workings, culture and psychology of the other.

"I didn't know if we were going to end up with this massively adversarial experience or if this was going to be something of a collaborative mode between all of us," said Dan Kaminsky, one of the outsiders who presented at the conference. Like others in the hacker group--many of whom are known as "security researchers" in their professions--he noted that the relationship ended up being the collaborative sort.

Still, in such a charged atmosphere, it didn't take long for emotions to show.

Matt Thomlinson, whose job it is to help make Microsoft engineers create more secure code, noticed that some of the engineers were turning red, becoming obviously angry at the demo hacking incident. Yet as painful as the lesson was, he was glad to see the crowd of engineers taking things personally.

Thomlinson frequently makes similar entreaties to the engineers on the need for secure code, but he said his own lectures don't have the same effect. "It kind of hits people up here," Thomlinson said, pointing to his head. "Things are different when a group of programmers watches their actual code exploited. It kind of hits people in the gut."

For two days, Microsoft staffers took these body blows repeatedly as they learned of various exploits. On day one, several dozen executives, including some of the company's most senior ones, were exposed to this simulated wrath in a makeshift boot camp. Among the participants were Jim Allchin, Microsoft's Windows chief, and Brian Valentine, head of core Windows operating system development. The second day drew about 400 rank-and-file Windows engineers, including people who don't necessarily focus on security features in their day-to-day work.

"It is rare that I can present to the people who are both responsible for and capable of fixing the issues that I cover."
--HD Moore
Security researcher

Allchin is not just any high-ranking software executive: In the technology industry, his name has become largely synonymous with the Windows operating system he oversees. A strong supporter of Blue Hat, Allchin wanted the Windows group not just to hear about security issues, but to see them as well.

"I'd already been through lots of days of personal training on the tools that are used to do this," Allchin said about the work of the hackers. "I personally wanted to really do a deep dive and really understand from their perspective."

It was a relatively safe way to get the experience. In a world where "white hats" are the security do-gooders and "black hats" are the hard-core villains, the hackers at Blue Hat were hardly representative of the dark side; if they had any pigment at all, it was no more than a tinge of gray.

This could well be a significant reason Microsoft held the event--to woo an influential group that has the choice of reporting security flaws discreetly or going public with them. The software maker routinely preaches the benefits of what it calls "responsible disclosure."

To the researchers, Microsoft's motivation was less important than the opportunity to meet in person with those who hold the keys to the kingdom and explain why they do the things they do.

"It is rare that I can present to the people who are both responsible for and capable of fixing the issues that I cover," security researcher HD Moore said,

Continued ...


"Blue Hat" summit meant to reveal ways of the other side

(continued from previous page)

adding that he doesn't plan to change his practice of giving companies 30 days before going public with issues. "I still have no desire to play e-mail tag with the (security response team) for a year for every bug that I find."

But Moore did gain a better understanding of why it takes Microsoft so long to create patches and said his impression of the people who create the products have changed. "I still may not agree with their security policies and how they handle bug reports, but at least I know they actually believe what they are saying," he said.

"Things are different when a group of programmers watches their actual code exploited. It kind of hits people in the gut."
--Matt Thomlinson
Director of security
engineering, Microsoft

Others agreed. "They are taking this subject seriously. It was really cool to see," said Kaminsky, a security researcher who does work for telecommunications company Avaya. "At some point, there was a shift at Microsoft."

That shift began in earnest with a well-publicized memo written by Gates on the concept of "trustworthy computing" in 2002. Security had long been a concern at Microsoft, but the issue became imperative after several high-profile attacks exposed the degree of its vulnerabilities.

"The security faults we are seeing could end up bringing an end to the era of personal computing," Kaminsky said. "The ability to customize our computers is under attack from those who are customizing it against our will."

It was this kind of impassioned rhetoric that won respect even among some of the more wary Microsoft participants.

Noel Anderson, a wireless networking engineer on Microsoft's Windows team, became suspicious as soon as he walked into the hacking demo--and saw the giant wireless antenna at the front of the auditorium.

Anderson decided that he should leave his laptop turned off, an instinct that saved him the embarrassment of falling into the hackers' trap, even though the hackers focused on a demo laptop. But under different circumstances, he thought to himself, "I might have even fallen for that."

As a result, Anderson and his team walked away with some concrete ideas on how to make sure future versions of Windows are more resilient to wireless attacks. He also left the room with a new respect for the hackers behind the demonstration.

"It's not just a bunch of disaffected teenagers sitting in their mom's basement," he said. "These are professionals that are thinking about these issues."

The hackers, for their part, seemed equally impressed with the technical knowledge of the senior executives they encountered.

At one point, researcher Matt Conover was talking about a fairly obscure type of problem called a "heap overflow." When he asked the crowd, made up mostly of vice presidents, whether they knew about this type of issue, 18 of 20 hands went up.

"I doubt that there is another large company on this planet that has that level of technical competency in management roles," Moore said.

Yet regardless of the mutual admiration, some tense moments were inevitable during the confrontation.

Microsoft developers, for instance, were visibly uncomfortable when Moore demonstrated Metasploit--a tool that system administrators can use to test the reliability of their systems to intrusion. But Metasploit also includes a fair number of exploits, as well as tools that can be used to develop new types of attacks.

"You had these developers saying, 'Why are you giving the world these tools that make it so easy to do exploitation?'" Kaminsky said. They calmed down, he said, once the researchers were able to state their case.

"We do regression testing in the real world of software development," Kaminsky said. "If we say, 'This thing isn't going to break,' then we need to test that. What these tools give is the ability to do this kind of testing, to be able to say not just, 'We did the best we could,' but 'We tried stuff and nothing worked.'"

Nevertheless, he understands why not all Microsoft developers were satisfied with the explanation.

"I'm also sure Ford wasn't too happy with (Ralph) Nader's reports in the late '60s," he said. "What do you mean you are telling people our cars can blow up?"

"The ability to customize our computers is under attack from those who are customizing it against our will."
--Dan Kaminsky
Security researcher

By the end of the two days, those on both sides felt they had just scratched the surface and were more than willing to meet again.

And executives such as Toulouse and Anderson said they came to a better understanding of what makes hackers tick.

"We have conversations where we say an attacker might do this or an attacker might do that. Now there is a face to some of those guys," Anderson said. "They were just as much geeks as we were."

The next time a Blue Hat event is held, as promised by Microsoft, Kaminsky said he would jump at the chance to return--assuming Microsoft lets him back.

"I'll be there next time, no matter what," he said. "I have some really interesting and devious plans coming up." End box


"Blue Hat" summit meant to reveal ways of the other side

By Ina Fried
Staff Writer, CNET News.com
June 15, 2005 4:00AM PDT

REDMOND, Wash.--The random chatter of several hundred Microsoft engineers filled the cavernous executive briefing center recently at the company's sprawling campus outside Seattle.

Within minutes after their meeting was convened, however, the hall became hushed. Hackers had successfully lured a Windows laptop onto a malicious wireless network.

"It was just silent," said Stephen Toulouse, a program manager in Microsoft's security unit. "You couldn't hear anybody breathe."

"(Hackers are) not just a bunch of disaffected teenagers sitting in their mom's basement. These are professionals that are thinking about these issues."
--Noel Anderson
Wireless networking
engineer, Microsoft

The demo was part of an extraordinary two days in which outsiders were invited into the heart of the Windows empire for the express purpose of exploiting flaws in Microsoft computing systems. The event, which Microsoft has not publicized, was dubbed "Blue Hat"--a reference to the widely known "Black Hat" security conference, tweaked to reflect Microsoft's corporate color.

The unusual March gathering, a summit of sorts between delegates of the hacking community and their primary corporate target, illustrates how important security has become to the world's most powerful software company. Microsoft Chairman Bill Gates himself estimated earlier this year that the company now spends $2 billion a year--more than a third of its research budget--on security-related issues. Security has also become one of the main themes of the company's developer conferences, including last week's TechEd event, where Microsoft pitched security improvements in Windows to 11,000 attendees.

Blue Hat was significant for other, less tangible reasons as well. It provided a rare glimpse inside the netherworld of computer security, where the ethical lines are sometimes fuzzy in the technological arms race between network engineers and the hackers who challenge them. During the course of the event, each side witnessed for the first time the inner workings, culture and psychology of the other.

"I didn't know if we were going to end up with this massively adversarial experience or if this was going to be something of a collaborative mode between all of us," said Dan Kaminsky, one of the outsiders who presented at the conference. Like others in the hacker group--many of whom are known as "security researchers" in their professions--he noted that the relationship ended up being the collaborative sort.

Still, in such a charged atmosphere, it didn't take long for emotions to show.

Matt Thomlinson, whose job it is to help make Microsoft engineers create more secure code, noticed that some of the engineers were turning red, becoming obviously angry at the demo hacking incident. Yet as painful as the lesson was, he was glad to see the crowd of engineers taking things personally.

Thomlinson frequently makes similar entreaties to the engineers on the need for secure code, but he said his own lectures don't have the same effect. "It kind of hits people up here," Thomlinson said, pointing to his head. "Things are different when a group of programmers watches their actual code exploited. It kind of hits people in the gut."

For two days, Microsoft staffers took these body blows repeatedly as they learned of various exploits. On day one, several dozen executives, including some of the company's most senior ones, were exposed to this simulated wrath in a makeshift boot camp. Among the participants were Jim Allchin, Microsoft's Windows chief, and Brian Valentine, head of core Windows operating system development. The second day drew about 400 rank-and-file Windows engineers, including people who don't necessarily focus on security features in their day-to-day work.

"It is rare that I can present to the people who are both responsible for and capable of fixing the issues that I cover."
--HD Moore
Security researcher

Allchin is not just any high-ranking software executive: In the technology industry, his name has become largely synonymous with the Windows operating system he oversees. A strong supporter of Blue Hat, Allchin wanted the Windows group not just to hear about security issues, but to see them as well.

"I'd already been through lots of days of personal training on the tools that are used to do this," Allchin said about the work of the hackers. "I personally wanted to really do a deep dive and really understand from their perspective."

It was a relatively safe way to get the experience. In a world where "white hats" are the security do-gooders and "black hats" are the hard-core villains, the hackers at Blue Hat were hardly representative of the dark side; if they had any pigment at all, it was no more than a tinge of gray.

This could well be a significant reason Microsoft held the event--to woo an influential group that has the choice of reporting security flaws discreetly or going public with them. The software maker routinely preaches the benefits of what it calls "responsible disclosure."

To the researchers, Microsoft's motivation was less important than the opportunity to meet in person with those who hold the keys to the kingdom and explain why they do the things they do.

"It is rare that I can present to the people who are both responsible for and capable of fixing the issues that I cover," security researcher HD Moore said,

Continued ...


"Blue Hat" summit meant to reveal ways of the other side

(continued from previous page)

adding that he doesn't plan to change his practice of giving companies 30 days before going public with issues. "I still have no desire to play e-mail tag with the (security response team) for a year for every bug that I find."

But Moore did gain a better understanding of why it takes Microsoft so long to create patches and said his impression of the people who create the products have changed. "I still may not agree with their security policies and how they handle bug reports, but at least I know they actually believe what they are saying," he said.

"Things are different when a group of programmers watches their actual code exploited. It kind of hits people in the gut."
--Matt Thomlinson
Director of security
engineering, Microsoft

Others agreed. "They are taking this subject seriously. It was really cool to see," said Kaminsky, a security researcher who does work for telecommunications company Avaya. "At some point, there was a shift at Microsoft."

That shift began in earnest with a well-publicized memo written by Gates on the concept of "trustworthy computing" in 2002. Security had long been a concern at Microsoft, but the issue became imperative after several high-profile attacks exposed the degree of its vulnerabilities.

"The security faults we are seeing could end up bringing an end to the era of personal computing," Kaminsky said. "The ability to customize our computers is under attack from those who are customizing it against our will."

It was this kind of impassioned rhetoric that won respect even among some of the more wary Microsoft participants.

Noel Anderson, a wireless networking engineer on Microsoft's Windows team, became suspicious as soon as he walked into the hacking demo--and saw the giant wireless antenna at the front of the auditorium.

Anderson decided that he should leave his laptop turned off, an instinct that saved him the embarrassment of falling into the hackers' trap, even though the hackers focused on a demo laptop. But under different circumstances, he thought to himself, "I might have even fallen for that."

As a result, Anderson and his team walked away with some concrete ideas on how to make sure future versions of Windows are more resilient to wireless attacks. He also left the room with a new respect for the hackers behind the demonstration.

"It's not just a bunch of disaffected teenagers sitting in their mom's basement," he said. "These are professionals that are thinking about these issues."

The hackers, for their part, seemed equally impressed with the technical knowledge of the senior executives they encountered.

At one point, researcher Matt Conover was talking about a fairly obscure type of problem called a "heap overflow." When he asked the crowd, made up mostly of vice presidents, whether they knew about this type of issue, 18 of 20 hands went up.

"I doubt that there is another large company on this planet that has that level of technical competency in management roles," Moore said.

Yet regardless of the mutual admiration, some tense moments were inevitable during the confrontation.

Microsoft developers, for instance, were visibly uncomfortable when Moore demonstrated Metasploit--a tool that system administrators can use to test the reliability of their systems to intrusion. But Metasploit also includes a fair number of exploits, as well as tools that can be used to develop new types of attacks.

"You had these developers saying, 'Why are you giving the world these tools that make it so easy to do exploitation?'" Kaminsky said. They calmed down, he said, once the researchers were able to state their case.

"We do regression testing in the real world of software development," Kaminsky said. "If we say, 'This thing isn't going to break,' then we need to test that. What these tools give is the ability to do this kind of testing, to be able to say not just, 'We did the best we could,' but 'We tried stuff and nothing worked.'"

Nevertheless, he understands why not all Microsoft developers were satisfied with the explanation.

"I'm also sure Ford wasn't too happy with (Ralph) Nader's reports in the late '60s," he said. "What do you mean you are telling people our cars can blow up?"

"The ability to customize our computers is under attack from those who are customizing it against our will."
--Dan Kaminsky
Security researcher

By the end of the two days, those on both sides felt they had just scratched the surface and were more than willing to meet again.

And executives such as Toulouse and Anderson said they came to a better understanding of what makes hackers tick.

"We have conversations where we say an attacker might do this or an attacker might do that. Now there is a face to some of those guys," Anderson said. "They were just as much geeks as we were."

The next time a Blue Hat event is held, as promised by Microsoft, Kaminsky said he would jump at the chance to return--assuming Microsoft lets him back.

"I'll be there next time, no matter what," he said. "I have some really interesting and devious plans coming up." End box