The Heartbleed Internet security flaw has proved so bad that it just might wind up doing some good.
For two weeks now, IT experts around the world have scrambled -- with mixed success -- to come up with a fix before hackers exploited the bug to attack vulnerable servers. Corralling Heartbleed has proved to be so nettlesome a problem that several of the technology world's biggest companies are now banding together in what amounts to a common defense pact to better protect critical tech infrastructure.
The Core Infrastructure Initiative, announced Thursday morning and organized by the Linux Foundation, pulls talent and more than $1 million of support from Google, Facebook, Microsoft, Amazon Web Services, Rackspace, Cisco, Dell, Fujitsu, IBM, Intel, NetApp, and VMware. Developers working under the auspices of the Linux Foundation will then coordinate on open-source projects, checking code for security. The hope is that the new attention being lavished will prevent future bugs and also remedy a structural weakness in the development of open-source software in which the code libraries are largely maintained by unpaid volunteers.
Heartbleed is the fanciful nom-de-guerre given to a catastrophic OpenSSL vulnerability. OpenSSL is a widely used open-source tool that forms the backbone of Internet encryption and is deemed critical to Internet security. It's used by websites and hardware manufacturers to protect the data transfer of important customer information such as usernames, passwords, Social Security numbers, and credit card numbers.
After the world learned about Heartbleed's existence, thousands of website owners and tech firms began to patch their systems but not before attackers were able to breach a walled-off virtual private network by exploiting the Heartbleed vulnerability.
Despite the heightened attention to Internet security in the aftermath of the Heartbleed episode, experts caution against being lulled into believing that the Core Infrastructure Initiative signals an "all safe" sign.
Security on fire -- and not in a good way
Security researcher and strategist Josh Corman sees an analogy between Heartbleed and the Cuyahoga River fires that plagued Ohio for 100 years. Like Heartbleed, which is neither the worst nor the first major vulnerability to cause problems in OpenSSL, the Cuyahoga River fire of 1969 started off as just another instance of pollution in the waterway erupting into flames.
But thanks to the timely use of an older photo depicting more severe flames dancing on the river and subsequent political outrage, the 1969 fire became a rallying cry for the burgeoning environmental movement. It would eventually lead to the passage of the Clean Water Act in 1972.
Heartbleed, Corman says, is probably not the Internet's Cuyahoga River moment of 1969. But we are watching the river burn.
"I think attackers have taken notice, and blood is in the water," said Corman, the former director of security intelligence at Akamai and current chief technology officer at security firm Sonatype. "It's now open season on open source."
Heartbleed: Disease and symptom
While you can peg some level of blame on the researcher who admitted to accidentally missing the bug during his review of OpenSSL code, there were far more factors that contributed to Heartbleed than simple human error. Heartbleed infected OpenSSL because of the numerous challenges that have yet to be solved in developing heavily-used but often volunteer-driven open-source software.
"We use a lot of open-source software because we hope that, across all the entire userbase, somebody has spent the time to make sure their investment is protected," said security researcher Dan Kaminsky.
Perhaps the greatest stumbling block to open-source security, is finding the money to hire cryptographic programming experts to work on the code full-time. One of the goals of the Core Infrastructure Initiative is to ensure that critical projects have full-time engineers working on them, so on that end it's a step in the right direction.
"OpenSSL is critical infrastructure," said Justin Troutman, a North Carolina-based security and privacy researcher and book author, "but it's not funded that way."
Given how much of the Internet relies on OpenSSL, from major Silicon Valley tech firms to tiny startups building widgets to sell on Kickstarter, you'd think that they would take securing OpenSSL more seriously.
"Considering the millions that are generated over [Open]SSL, it needs millions," he said. "It's not just about funding the developers who maintain it, but bringing in new experts."
"The real issue," Corman agreed, "isn't hacking code or hacking devices, it's hacking incentive structures."
But there's simply not a lot of incentive for a cryptographic expert to volunteer his or her time on an enormous, complicated open-source project when a paid gig can offer not only money but a more favorable environment.
Chris Wysopal, the chief technology officer of code security verification firm Veracode, said that relying on altruistic developers is a terrible way to achieve effective cryptographic code when the market will pay a premium for that work.
"Any project that requires cryptography should have extra scrutiny," he said, noting that his company has "looked at a lot of software" and found the quality of cryptographic code in enterprise software and open-source software to be "about the same."
"We saw with OpenSSL that one guy wrote the code, and one guy looked at it, and that's how we have this problem. We need a more rigorous process, two or three people looking at the code," Wysopal said.
The kind of dedicated source of funding that OpenSSL requires, Troutman added, must come at least in part from the major tech firms that rely on the project. That means Google, Facebook, Microsoft, Yahoo, and others.
Right now, each company that's signed up for the Initiative has contributed $100,000 per year for three years. Let's assume that all goes to OpenSSL for now, along with what Troutman estimated OpenSSL's working capital of around $1 million per year. He speculated that it would take an annual operating budget of 10 times that to hire the kind of talented encryption experts that OpenSSL needs.
"Ten million dollars is nothing to balk at, it's a substantial amount that would get us going, but for the long term I don't know. Because it's critical infrastructure, it still demands a lot," he said.
For his part, Kaminsky doesn't expect the solution to be as simple as getting tech giants to cough up a couple million annually.
"The real game is figuring out new models that guarantee a necessary level of testing and review on the software we depend on," he said. "Whether that software is open or closed empirically has nothing to do with whether anyone's tested it or not."
Rethinking how we use open source
While a fleet of dump trucks filled with cash certainly wouldn't hurt sorting out the symptoms that led to Heartbleed, there's more to the overall problem than just a lack of adequate funding. If Kaminsky's right about the need to rethink the model of how open-source code gets tested, we're looking at several years of the tech equivalent of the Cuyahoga River burning before it can be put out.
A major reason why websites affected by Heartbleed got fixed so quickly was because Heartbleed is the first media-savvy vulnerability. It exposed and emphasized the importance of public awareness in getting companies to fix security problems.
Codenomicon, the team of security researchers who discovered the bug at the same time as Google researcher Neel Mehta, say that they intentionally built the kinds of media-friendly tools such as the logo, website, and a FAQ needed to expedite the Heartbleed news. It's a bug with its own heroic origin story.
That's not necessarily going to happen with other bugs, potentially worse than Heartbleed. Much like the Cuyahoga River fires, Wysopal said, "you'll see that there are other more severe vulnerabilities in OpenSSL. They weren't a media event. [Heartbleed] shows that now we're so much more in tune with how a vulnerability can impact our lives," he said.
The many open-source projects remain highly vulnerable. Code analysis firm Coverity's most recent annual study complements Wysopal's company's findings, that open-source projects are more secure than proprietary code, but only to a point. That belies the idea that open source is more secure simply because, or in hopes that, more people are looking at it.
"I found that 90 percent of modern applications are not written, they're assembled from open-source code," Corman said. "That's also true of highly-regulated industries, such as government and financial services."
That wasn't always the case. They used to run only proprietary code, but they began to change, Corman said, around 2008 and 2009 -- the same time as the global economic downturn.
He explained that they abandoned wholly proprietary code for a mix of open- and closed-source code because it was easier and cheaper to leverage existing code when they could.
But, Corman said, "shared dependence means shared attack surface." That's come back to bite every company that employed the versions of OpenSSL with the Heartbleed bug.
Wysopal agreed. "Things change pretty slowly because it takes so long to change the technology and deploy it. I am hopeful that we are getting better at building software," he said, but warned of situations like when giving your heating and air conditioning systems technician network access exposes more than 100 million Americans to possible credit card fraud, as it did in November's Target breach.
There are other burgeoning, smaller-scale yet important changes to how open source is being handled. In addition to the growing outrage over breaches, there are outfits like BuildItSecure.ly, which helps crowd-sourced "Internet of Things" projects include security standards and protocol. Corman is working on a more policy-focused nonprofit that he founded called I am the Cavalry, which advises experts and politicians on improving security protocol.
"We are so far away from a safe and stable state for open-source consumption. [Heartbleed] is not anomalous. We're very vulnerable, very prone," Corman said. "Chances are this is the tip of the iceberg."