Los Angeles gets its Google Apps groove

The nation's second-largest city is close to overcoming security fears and jumping on the cloud computing bandwagon. Will other cities follow?

Elinor Mills Former Staff Writer
Elinor Mills covers Internet security and privacy. She joined CNET News in 2005 after working as a foreign correspondent for Reuters in Portugal and writing for The Industry Standard, the IDG News Service and the Associated Press.
Elinor Mills
9 min read

On August 11, Randi Levin, the chief information officer of the city of Los Angeles, stood before City Council members at a hearing of the information technology committee and made her case for why the nation's second-largest city should adopt Google Apps.

"The ability to get whatever information the city needs, whenever they need it, on whatever device they need it on will fundamentally change the way the city works and enhance productivity greatly," she said. "In a fiscal crisis it is difficult to find technology solutions that will save money without requiring a significant capital outlay to achieve those objectives."

Security concerns have kept many government agencies and large corporations away from Google Apps. That is starting to change. A number of small U.S. cities are using the suite and there are Google Apps pilots in more than a dozen federal agencies. If Los Angeles signs on, it would join the District of Columbia as one of the largest government adoptions.

Randi Levin, chief information officer of the city of Los Angeles and a key player in the city's move to adopt Google Apps. City of Los Angeles

Security experts and officials in other state and federal government departments tend to be wary about outsourcing the management and storage of highly sensitive data to an outside company.

Even in Los Angeles some persuasion was needed. Tony Cardenas, a councilman and chair of the IT committee, and a police official last month said they were worried that sensitive police investigations could be compromised if data were exposed somehow under Google's control. "Drug cartels would pay any sum of money to be aware of our progress on investigations," Cardenas told the Associated Press.

The concerns came to light after the poor e-mail practices of an employee at Twitter and an easy password reset mechanism at Yahoo gave a hacker access to sensitive Twitter documents stored on Google Apps.

While that breach had nothing to do with level of security of Google Apps, it prompted renewed scrutiny of the cloud computing service. It's unclear exactly what effect the exposure of financial and business plan data will have on Twitter. But the specter of confidential police records being hosted by a search company has some people concerned, particularly with a service that is just 3 years old and that added enterprise-level security enhancements 18 months ago.

Google Apps creates "a value proposition. No doubt about it," Mike Hamilton, chief information security officer of the city of Seattle, said in a phone interview after the LA proposal to adopt Google Apps made the news. Seattle uses the e-mail security outsourcing service Google offers through its Postini Message Security, and Hamilton said he is pleased with that service.

"My personal opinion is there's still some shaking out to do around this," he said. "Some bad things are going to happen before all this gets worked out and we don't want to be an early adopter."

During a chief security officer panel at the Black Hat security conference in Las Vegas last month, a group of CSOs from several corporations and one government agency said Google Apps was on their radar, but they didn't seem too eager to embrace it just yet.

"We are looking at those apps and have a number of pilots underway," said Bob Lentz, CSO for the U.S. Department of Defense. "There are a number of security requirements that have to be addressed."

LA concerns assuaged
The sentiment over the past month among LA officials about Google Apps has turned from righteous skepticism to cautious acceptance of the contract, under which systems integrator Computer Sciences Corp. will help the city deploy the Google technology. Cardenas hardly mentioned security during the August 11 committee meeting.

In what appeared to be mere preaching to the choir, Levin made some strong points in favor of the five-year, $2 million contract in her final comments before the committee.

"We own the data, not Google," she said, kicking off a list of key points. "Their security is better than ours," she added. And "cloud computing is safe."

Also, city employees would be using the version of Gmail that is designed for corporate and government customers, not the free consumer product, and they will benefit from electronic discovery, archiving, and disaster recovery functions that they don't have access to today, she said.

Meanwhile, Microsoft Office will remain on all desktop PCs that have it and future purchases will be evaluated for necessity, she added.

Officials from the police department and city attorney's office told the committee that their concerns were being resolved in negotiations with Google. The committee promptly approved the Google Apps adoption proposal, given certain caveats on Google customization for those departments and a later rollout for the police department. The proposal now goes to the budget and finance committee and then the full City Council for a vote in early September.

Specifically, 17,000 of the 30,000 total LA city workers would be migrating to Google Apps if the plan gets approved. The 13,000 workers in the police department and the city attorney's office would initially only use Gmail (instead of Groupwise) and not the other Google Apps, officials said.

In order to meet security requirements from the California Department of Justice (which is within the Attorney General's office), Google needs to either provide some additional level of background checks for people able to access the data or provide an encrypt option and allow city officials to hold the key, according to city officials.

Google also needs to add other functions to the e-mail service, such as auto-acknowledgment of receipt, Tim Riley, chief information office for the LA Police Department, said in an interview.

"We deliver 400,000 electronic subpoenas to our employees every year and there are requirements for the e-mail," he said. "When an officer opens his e-mail, there has to be acknowledgment back to the server that shows he opened it and was (officially) served."

Google has committed to resolving the issues, Riley said.

"Their security is better than ours...cloud computing is safe."
--Randi Levin, chief information officer, city of Los Angeles

As for productivity applications, the police department is in year three of a five-year contract with Microsoft for Office that covers about 8,500 PCs, Riley said.

"We have a number of Microsoft databases that are not currently accessible from Google Apps," he said. "That's not to say that down the line we wouldn't" consider Google Apps.

The LA City Attorney's Office has similar security constraints to the police department, and in addition it is required by the courts to use either Word or WordPerfect to file documents, an official said during the hearing.

"People are comfortable with Microsoft Office and there are compatibility issues," Ted Jordan, a city attorney said in an interview.

Meanwhile, "we are still talking to Google about risk management issues, indemnification, limits of liability, data security and disclosure and loss," Jordan said.

For Levin, the cost-benefit analysis is a no-brainer. Google provides enhanced collaboration and remote access ability and more storage and would replace an "antiquated e-mail system" that needs to be replaced, Levin said in an interview.

Because the Google productivity apps are less robust than the Microsoft apps and there are compatibility problems for workers using the two different programs, power users who need the high-end functions of Microsoft applications will still be able to use them, she said.

LA city officials have said the contract is projected to save the city $13.8 million, according to Matt Glotzbach, product management director for Google Enterprise.

More precisely, if the city were to pay for all the capabilities it will get with Google that it currently doesn't have--including archiving, automated electronic discovery and video conferencing--the cost could run as high as $50 million, Kevin Crawford, the city's assistant general manager, said in an interview.

The Google contract would mean the city won't need to dedicate at least 16 servers to run its current system and it can deploy that same number of staff members who operate and maintain it to other areas, he said.

Rebates are unique
But there is an added financial incentive for the city to move to Google. The contract the city is drawing up with Google is written so that any public agency within the state of California can amend it and adopt it for its own use, Crawford said. If 100,000 users sign up, the city will get $1.2 million in rebates, he said.

Crawford told committee members that his office had heard from more than 20 state agencies that are interested in using Google Apps under the contract terms.

"This is unheard of in technology contracts," Levin said during the hearing.

"My personal opinion is there's still some shaking out to do around this...Some bad things are going to happen before all this gets worked out and we don't want to be an early adopter."
--Mike Hamilton, chief IS officer, city of Seattle

Meanwhile, all the authentication for the single sign-on system is designed to go through the city's authentication system before it hits Google, Crawford said.

As far as concerns about reliability, the city's current in-house system was down about 300 times more than Google was in the last 12 months, he said.

"Google Apps has much higher reliability than on-premise systems tend to have," said Glotzbach, adding that the company promises paying customers uptime of greater than 99.9 percent. "We make redundant copies, multiple live copies, and other things most government agencies can't and don't do."

Outages with Google Apps in the past have been infrequent and short-lived and most did not affect paying customers, according to Google. But, still, they have happened and they make headlines.

Last month there was a four-hour outage in the Google App Engine application hosting infrastructure service. In February, business customers were affected by a 2.5-hour Gmail outage. A Google Docs outage hit them in July 2008, while a Google Apps outage affected some Gmail users in March. Google Docs, meanwhile, had a privacy glitch in March that Google said led to "inadvertent sharing" for a small fraction of documents.

Security concerns weren't enough to keep executives from large corporations like Genentech, Motorola's handset division, Johnson Diversey, and Fairchild Semiconductor from jumping on board.

Blazing the trail for government agencies is the District of Columbia, where deployment of Google Apps was completed in July 2008. The project was driven by DC Chief Technology Officer Vivek Kundra, who has since left to become chief information officer for the federal government, where next year's budget includes funding for cloud computing initiatives.

The District of Columbia paid $479,560 for the Enterprise Google Apps license and is leveraging Google Apps to create the city's intranet at a cost of $500,000, according to a city spokeswoman. As a result, DC saved $3.5 million by replacing an earlier plan to pay another company $4 million for the portal project, she said.

Google Apps is available to 38,000 DC city employees, 4,000 of whom are actively using it, she wrote in an e-mail response to questions. Gmail could potentially replace Microsoft Exchange there, "but this decision has not been made yet," she wrote.

"Our policy has been to put into Google Apps only information that could be transmitted via e-mail; at this time we are not placing sensitive data into Google Apps," the DC spokeswoman wrote. "The District mostly uses Google Apps collaboration tools (e.g. Google Docs, Sites, Video) which filled a void we had for Intranet collaboration tools."

Google is working with the governments to help them overcome their fears.

"Government, by definition, tends to be a little bit more conservative and generally with good reason," Glotzbach said. "But security experts in the city of LA have dug into it and found that in a number of ways the cloud can be more secure than existing in-house systems."

Security guru Bruce Schneier says that as far as cloud computing is concerned, resistance is futile in the long run.

"This is the future of computing and don't think for a minute that this isn't what people will be doing in a few years," he said in his question-and-answer session at the Defcon security conference late last month in Las Vegas.

"I think it's nutty to give up that much control," he said after mentioning Google Apps specifically. "The problem here is trust. They're going to go out of their way to secure their systems more than any user because their reputation is at stake."

Updated 4:52 p.m. PDT to add that Computer Sciences Corp. would be the systems integrator of the Google Apps technology for Los Angeles and that the proposal goes to the city's budget and finance committee before it goes to the full city council.