Microsoft's data centers growing by the truckload
Redmond is going beyond the traditional racks, instead having its servers delivered and run from a sealed container, a move that should cut costs and power demands.
Correction, 9:31 a.m. PDT: This story cited the wrong state for Microsoft's Quincy data center. That facility is located in Quincy, Wash.
Once upon a time, Microsoft used to fill its data centers one server at a time. Then it bought them by the rack. Now it's preparing to load up servers by the shipping container.
Starting with a Chicago-area facility due to open later this year, Microsoft will use an approach in which servers arrive at the data center in a sealed container, already networked together and ready to go. The container itself is then hooked up to power, networking, and air conditioning.
"The trucks back 'em in, rack 'em, and stack 'em," Chief Software Architect Ray Ozzie told CNET News. And the containers remain sealed, Ozzie said. Once a certain number of servers in the container have failed, it will be pulled out and sent back to the manufacturer and a new container loaded in.
It's just one way that Microsoft is trying to cope in a world where it adds roughly 10,000 servers a month.
"You contain your infrastructure but you also contain the heat that's generated from the servers," Arne Josefsberg, Microsoft's general manager of infrastructure, said in an interview this week. "We are working incredibly hard to improve the energy efficiency of our data centers."
Only a couple of years ago, Microsoft was adding capacity one server at a time, adding individual servers to racks and taking a couple of hours to wire in each new server.
"That's way too expensive, way too slow," said Josefsberg.
Microsoft also used to lease much of its space, until it realized that data centers were going to be a very big part of its future as more and more software moved into the cloud. A couple of years back, though, it found itself running tight on capacity and bought two San Francisco Bay Area data centers in which it had been leasing space.
Over the past 18 months, though, Microsoft has been on a buying--and building--spree. The company has opened a data center in Quincy, Wash., and will open the Chicago facility, as well as another in San Antonio, Texas, later this year. A facility is due to open in Dublin, Ireland next year.
Microsoft is close to announcing yet another data center, Josefsberg said. The software maker also has signed a memorandum of understanding to build a data center in Russia.
No more off-the-shelf hardware
Gone are the days in which Microsoft settled for off-the-shelf hardware to fill its server farms. These days, Microsoft is looking for servers designed to its exact needs. It's not just that Microsoft doesn't want servers that have keyboard or USB ports--it wants motherboards that don't even have the added wiring necessary to support those things that it will never use. Such moves eliminate cost, space, and power consumption.
"We are not physically building our servers, but there is very deep engagement (with the computer makers)," Josefsberg said.
Even a 1 percent or 2 percent reduction in power consumption makes a big difference, Josefsberg said. As it is, Microsoft is trying to cram a whole lot of gear into a small space. While server racks at a Web hosting facility might have power densities of 70 watts to 100 watts per square foot, things are packed far more tightly in the containers, which might be consuming in the thousands of watts of power per square foot.
The container approach is easiest to implement on the ground floor of a facility. In Chicago, for example, it will use containers on the first floor and more traditional racks on the second level. But Josefsberg said that, though it poses some logistical challenges, the company is also considering using multiple levels of containers at other sites, including at a Dublin, Ireland data center due to open next year.
What the servers are serving up
So just what are all these data centers doing? Outsiders got a glimpse into this thanks to a slide included as part of a video that Microsoft put on its Web site touting its environmental efforts. The chart shows search accounting for the vast number of the servers--nearly 80,000 or so--with Hotmail and Messenger distant runners-up in terms of server usage.
Josefsberg said the figures were accurate, but out of date, reflecting where things were at a year or 18 months ago.
"Search was a very large portion of our demand in fiscal year 2008," he said. "Going into this year it is still a very large proportion. It is now not as dominant as it was last year."
Microsoft is seeing new demands, he said, such things like consumer video and photo services as well as its collection of hosted enterprise services under the Microsoft Online moniker.
Josefsberg said his goal is to keep capacity a certain number of months ahead of where Microsoft's utilization is running. To do that, he said, takes some serious planning. Business unit heads who used to have to just create a forecast for revenue and headcount, now need to be able to predict how much server capacity they will need, or at least give Josefsberg the data he needs to make such calculations.
He points to things like Microsoft's work with the Olympics as indicative of the kinds of demands his data centers will see in years to come.
"One of the big drivers for us that I see is the move to IP-based delivery of rich video," Josefsberg said.
And not all of his problems will be solved just because Microsoft can now get its servers by the containerful. Microsoft has sophisticated "heat maps" that plot the best locations for new data centers based on everything from government policy to water supply to power prices. But in other areas, such as networking technology, Microsoft is counting on the industry to make some quantum leaps.
"When you think about large-scale data centers there are a number of limitations in the technology," he said. "Some of the network protocols were designed years ago...Some are 30 years old."