How Web mail providers leave door open for NSA surveillance

Billions of supposedly private e-mail messages a day flow through unsecured links, where they can be snared in digital dragnets operated by the National Security Agency and other intelligence services.

Recent revelations about NSA surveillance -- including a top-secret document discussing "collection of communications on fiber cables and infrastructure as data flows past" -- have highlighted the ease with which government eavesdroppers can exploit the Internet's infrastructure. Another classified document, which the Guardian published Thursday, mentions network-based surveillance of Hotmail servers.

Over the last decade or so, Web mail providers began to turn on encryption to armor the connections between users' computers and Gmail, Yahoo Mail, Hotmail and other services. That form of protection against surveillance, which typically appears in a Web browser as an "https" connection accompanied by a padlock image, is viewed as generally secure and is used by banks as well. Google has offered it since 2004, and Yahoo finally followed suit this year.

But during the next step, when those e-mail messages are transferred from one company's servers to another's, they're rarely encrypted. An e-mail message that a Facebook user addresses to a Yahoo Mail user, for instance, will be delivered in an unencrypted form through a server-to-server connection that provides no protection against surveillance.

"The incentives aren't really there for companies to try to implement it," says Ashkan Soltani, an independent security consultant who has highlighted some of these security shortcomings on Twitter. That's the case even though, he says, enabling encryption is "a really easy thing to do."

A survey of top mail providers shows that Google is alone in using strong encryption, known as SMTP-TLS, to fully armor e-mail connections for its users, as long as the other company's server is willing to encrypt as well. SMTP-TLS also protects employee e-mail at security-conscious companies, large law firms, and sensitive government agencies including the NSA, the White House, and the Department of Homeland Security. (You can check on your own provider by typing in your e-mail address at CheckTLS.com.)

Unfortunately, those are the exceptions. Facebook, Hotmail, Yahoo Mail, and AOL Mail do not accept incoming e-mail in SMTP-TLS encrypted form, meaning hundreds of millions of users' private communications are vulnerable to monitoring. Both the sending and receiving servers must have encryption turned on for a secure connection to happen.

"My sense is that Google is the one large company that has demonstrated it cares about crypto," says Dan Auerbach, a staff technologist at the Electronic Frontier Foundation in San Francisco. "We think [encryption] should obviously be supported by all these mail servers."

One reason why so many mail providers don't encrypt server-to-server mail links using SMTP-TLS is that, unlike browser encryption, this security precaution would be invisible to users. And the fat pipes that backbone providers provide have historically been viewed as safe. (SMTP-TLS stands for Simple Mail Transfer Protocol Transport Layer Security. TLS was published as an Internet protocol in 1999.)

Adam Langley, a software engineer at Google, told CNET that "we do support TLS" for both inbound and outbound exchanges between mail servers. But, diplomatically, he declined to speculate on why many other companies do not. The company even offers its Google Apps users the high security choice of rejecting non-encrypted connections.

A Facebook spokesman said: "Facebook currently supports user-to-server encryption, but does not currently support server-to-server encryption as we have not seen wide adoption of the protocol. We are open to adoption to this or other protocols in the future as they are used by more services." A Yahoo representative said: "At Yahoo, we invest heavily in the security of our users and we're continually looking to enhance the security capabilities of our products." AOL did not respond to queries.

The potential privacy risks of server-to-server e-mail deliveries have been thrown into sharp relief by surveillance-related disclosures over the last two weeks from Edward Snowden, the former NSA contractor, and U.S. government officials. Snowden said in a Guardian online chat this week that e-mail and other Internet communications inside the United States are "ingested" by the intelligence agency's immense collection apparatus and that "Americans' communications are collected and viewed on a daily basis on the certification of an analyst rather than a warrant."

Web companies have offered blanket denials of allegations that they provided NSA eavesdroppers with "direct access" to their servers, and Google even challenged the U.S. government this week before the Foreign Intelligence Surveillance Court in a bid to clear its name.

A leaked NSA slide talking about "upstream" data collection from "fiber cables and infrastructure as data flows past" suggests that those companies are telling the truth: the NSA instead is tapping into Internet backbone links operated by companies such as AT&T, CenturyLink, XO Communications, Verizon, and Level 3 Communications -- and using that passive access to vacuum up unencrypted communications. Additional evidence comes from the classified directives released Thursday that discuss surveillance procedures and were signed by Attorney General Eric Holder.

Documents that came to light in 2006 in a lawsuit brought by the Electronic Frontier Foundation offer insight into the spy agency's relationship with Tier 1 Internet providers. Mark Klein, who worked as an AT&T technician for over 22 years, disclosed (PDF) that he witnessed domestic voice and Internet traffic being surreptitiously "diverted" through a "splitter cabinet" to secure room 641A in one of the company's San Francisco facilities. The room was accessible only to NSA-cleared technicians.

The New York Times revealed in 2009 that a secret NSA database, code-named PINWALE, archived foreign and domestic e-mail messages that analysts could search through "without warrants" as long as Americans' correspondence did not amount to more than 30 percent of any database search. PINWALE is the the NSA's main database for intercepted communications, while metadata is stored in a separate database called MAUI, and initial sorting is performed by a program called XKEYSCORE, according to the recent book "Deep State: Inside the Government Secrecy Industry."

Other mail providers that do not appear to permit SMTP-TLS links for e-mail delivery include AT&T, Earthlink, and Comcast. Apple, which did not respond to a request for comment, does not appear to support SMTP-TLS for server-to-server iCloud e-mail, though it does for user-to-server links. Fastmail.fm and Hushmail do support SMTP-TLS for automatic encryption of incoming mail. Oddly, the FBI does not for its own employees' incoming e-mail.

Yahoo, Microsoft, and Apple protect their own internal correspondence more carefully than they do their users' communications: their separate employee mail servers support incoming encrypted messages.

A Microsoft representative said the company does not support server-to-server SMTP-TLS for consumer products including Outlook.com and Hotmail.com. (Microsoft finished switching users from Hotmail to Outlook last month.)

Microsoft does enable encryption in some other situations. Those include Exchange ActiveSync, or when users choose the "SMTP send" option from Outlook.com, which was announced last month. SMTP send allows you to log in to Outlook.com, but actually send the message using your Yahoo Mail or Gmail account.

In addition, Microsoft enables server-to-server encryption for paying customers, including those using Office 365. The Department of Homeland Security, which has a 10-year relationship with Microsoft for technology services, has outsourced its mail to the mail.us.messaging.microsoft.com server, which does enable SMTP-TLS.

Even if a company don't support SMTP-TLS encryption between servers, other technologies exist to make data unreadable to government snoops. One is called S/MIME, but it's hardly popular. End-to-end encryption in the form of PGP or GnuPG is another choice. Those are viewed as some of the most secure options, but are also the most difficult to use.

"We don't know the extent to which the NSA or other intelligence agencies are reading people's mail," says Auerbach, EFF's staff technologist. "Companies not supporting encryption for the sending of e-mail leaves the door wide open for these agencies to do it, were they inclined to do so."

Disclaimer: McCullagh is married to a Google employee not involved with this issue.

Last updated at 9:15 a.m. PT with additional details