An inside look at the testing of Windows Phone 7
In part three of a behind-the-scenes look at the development of Microsoft's new phone software, Ina Fried takes a look at Redmond's massive testing operation.
Editor's note: this is the third and final part in a series of stories on the behind-the-scenes efforts by Microsoft to bring Windows Phone 7 to market.
REDMOND, Wash.--On a rather sunny August day deep within one of the many nondescript buildings that dot Microsoft's campus, a robot taps away at a prototype Windows Phone 7 to double-check that the screen is accurately reading touch input. Another robot, affectionately dubbed Wally, tests the accelerometer and other sensors built into the new phones. A building over, thousands of phones are going through a variety of stress tests, including both automated and hand-performed tasks.
That's a major shift for Microsoft, which in the past focused on making sure its software was bug-free and left much of the testing of final phones to the partners that make the actual Windows Phone devices.
"We really felt like our product quality ensuring system infrastructure wasn't good enough," said Darren Laybourn, general manager of test for Microsoft's phone unit. Laybourn and his team allowed CNET an exclusive view inside the testing labs as the company was putting the final touches on its software and helping its phone maker partners gain approval from various cell phone carriers. "We spent over $15 million on hardware...we've added hundreds of senior people to the team, we've completely revamped our processes."
Microsoft used its beefed-up testing methods as part of its effort to convince carriers that it was serious about getting back in the phone game.
"That was kind of my story," Laybourn said, showing a slide deck he used in pitches to AT&T and other cell service providers. "Here's why we suck less, basically."
Indeed, Windows Phone 7, which went on sale in the U.S. on Monday, is arguably one of the most heavily tested products to come out of Microsoft, save perhaps a full release of Windows or Office.
With Windows Phone 7, Microsoft felt it needed to do more of the testing work in-house, both to help speed the process of getting phones approved on various cellular networks and to ensure consistent quality. As part of that effort, Microsoft employed more than 4,500 actual phones and tens of thousands of virtual devices running on servers. That's up from about 60 phones that Microsoft used to test the prior generation of phones.
In all, Microsoft ran through more than 10 million hours of automated testing, or about 20,000 phone-hours per day.
Microsoft is of course, not alone in doing this kind of testing. Phone makers and carriers do this all the time. Even its software rival, Google, has been known to go through a similar approach, as the company did with the Nexus One--a process they documented in a video posted to YouTube.
Just in time?
With Windows Phone 7 though, Microsoft was in something of a hurry. After suffering through many delays and a complete "reset" of the project, the company had bet its reputation on delivering an operating system that would be on phones and on sale in time for this year's holiday season.
"It's taken a relatively long time for Windows Phones to get through trials in the past," Laybourn said. "We were on a fairly tight schedule this time for phones to get out. We would have to beat what most phones have actually ever done to meet our dates."
The company met its deadline, though to do so it did have to delay until next year its support for CDMA networks like Sprint and Verizon.
Despite all of the work and testing, though, it remains to be seen how much progress the company will make in its uphill battle to regain share in the cell phone business. Despite some positive reviews, Microsoft's phones appeared to be met with a mixed reception on their first day of sales at AT&T and T-Mobile stores in the U.S. The first Windows Phone 7 models went on sale in Europe a couple weeks ago, but Microsoft has yet to comment on sales.
Much of the testing is automated, with scripts simulating the kind of work that a person would do on his or her phone, in many cases not even making calls on a real network. Not all of the phone use can be automated, however.
"We actually have people on a real network doing real work," Laybourn said. The human testing came in handy earlier in the year when one model was producing photos that were just a bit too dark.
Microsoft also tapped its processes for testing Windows and created all kinds of automated telemetry that could notice whether, say, the battery was draining too fast and notify Microsoft (that feature has been turned off in the final phones that are going on sale to consumers.)
The company credits all of that testing with the fact that Windows Phone's battery life has gone from abysmal to something that Microsoft now feels is competitive with other phones on the market. Initially, users were getting as little as an hour and a half of battery life on the Samsung Taylor test devices. By the time Microsoft had finished testing and improving the code, battery life had been doubled, doubled again and then doubled yet again.
The physical testing of devices is just one element of what Microsoft did to prep for Windows Phone 7. Even before the company had any real code ready, Microsoft began focus group testing for the radically different look that it planned to employ. It was actually the second time Microsoft had done such work, having nearly completed focus group tests for a very different Windows Mobile 7 before Microsoft opted to scrap that work in favor of something that could better compete with the iPhone.
Starting last year, Microsoft began rapidly testing the new approach, initially having users comment on mock-ups and emulated versions of the software. It wasn't until the beginning of this year that they were able to test with actual devices running Windows Phone 7.
The company also did other types of testing on individual components of the product, such as the software-based keyboard, which has an uncanny ability to guess what word a user is typing after just a few letters--in some cases even when a user is off on several of the letters.
The 'fat finger' problem
Microsoft began with the premise that users would have the "fat finger" problem, where they were likely to mistype when entering letters on an onscreen keyboard.
"We want to consider that people are going to mistype the keys," said Parthasarathy Sundararajan, a member of the Windows Phone 7 test team. "We want to account for that."
Without changing the appearance of the onscreen keyboard, the Windows Phone software favors the user input that is most likely as a word is being typed. For instance, if someone has typed "accordio" the software could favor the next character being an "n" rather than a "b."
Tom Adams, another member of the testing team, was skeptical that he could get used to a software keyboard, having lived on a BlackJack 2 device, which had a hardware keyboard. Now, he said he doesn't even think of using a hardware keyboard. The real test, he knows, will come if he can convince his wife to go with a touch screen only phone.
"My wife has to have a keyboard," he said. "I'll be intrigued to see if she actually uses the keyboards."
The level of detail into which Microsoft went was surprising, down to researching the impact of the phone's various sounds. For instance, after some research, the company opted to have multiple touches on the software keyboard produce multiple different sounds as opposed to repeating the same tone over and over, a choice it said is both less pleasing and less tactile for the user.
The investment in testing was part of a broader effort by Microsoft to improve the quality of Windows Phone 7 over past company efforts. Another key part of the effort was Microsoft's decision to limit the variations that phone makers could have. To use Windows Phone 7 software, hardware makers agree to use one of a narrow set of specifications with room to modify only certain aspects of the device, such as whether it has a keyboard or not or which type of screen is used. That's a big contrast to past versions of the software where phone makers could change virtually everything from the chips that the phones ran on to the types of components and features included to the screen size and even the user interface.
"We had a wide open platform--unconstrained access to everything--you could change whatever you wanted," Laybourn said.
Microsoft hopes that the decision to narrow the options will make it easier for those who want to write programs, essentially ensuring that software that runs on one Windows Phone 7 device will work on any other Windows Phone 7 device.
"Android is really going down the path we were on which is they really can't promise anything to developers," Laybourn said.