Microsoft turns Live users into guinea pigs

A recruitment Web site launched Thursday gave some details of a project called the Experimentation Platform. The purpose of the platform is to enable Microsoft programmers, and eventually third-party developers, to try out new ideas quickly on Live users in controlled experiments, according to the site.

Microsoft confirmed that it is working on an experimentation platform. "We are always looking to rapidly innovate on our services and platform, and encouraging experimentation is important here," a representative for the software maker said. However, the representative declined to give further details.

The move should help generate features for Windows Live, a set of Internet-based services such as e-mail, blogging and instant messaging. Microsoft introduced its Live services, which will provide online versions of existing desktop applications and new Web-based tools, in beta earlier this year.

The Experimental Platform described in the recruitment site would be used to put such tools through their paces. Using randomized experimental designs, developers can expose a percentage of users to a new treatment and measure the effects. They can then run tests to determine whether the differences are statistically significant, and so establish causality. The platform will collect multiple metrics and provide analysis tools to improve insights about the value of different features to customers.

The platform has been designed to make sure users' experiences aren't disrupted, according to the Web site. How can it guarantee this? The experiments will be monitored while they are running, and if an application is underperforming, it will automatically switch over to the "control" application or a feature running in the experiment. As with any randomized experiment, users will never know if they are part of the experiment or if they are part of the control group.

The architecture of the testing platform encourages software providers to deploy applications more quickly, eliminating the need for extensive testing, according to Microsoft's online description. And because it is designed to have no adverse impact on customers, new technology or fixes to older problems that work well will be distributed automatically to more users, and ones that fail will be just as quickly discarded.