Why performance testing and public relations don't mix

Performance testing might not be the first thing you think of when you read product reviews, but it often plays an important role in helping to evaluate how some products stand up against their competition. CNET Labs takes performance testing very seriously, and it's sometimes a lot more difficult than you might think... When we encounter testing problems in the lab, we often turn first to the manufacturers to help us troubleshoot the issue. We like to give the vendors the opportunity to address the issues and offer explanations or solutions, in part to make sure we're not overlooking something or doing something wrong. If relevant, we'll mention the problems we encounter during testing within the respective product reviews.

One problem we've been struggling with for more than a month now is how Windows Vista manages to break one of our benchmarks--but only on some systems. CNET Labs' Multimedia multitasking test is a robust benchmark test that uses QuickTime Pro to encode a high-definition video, while iTunes encodes a bunch of audio files in the background. While this test works without a hitch on most of the Vista systems we've seen, a number of systems couldn't perform the QuickTime video encode. Curiously, some of these were rather high-end systems. We soon discovered these problem systems had trouble even just playing back standard-definition MOV files in QuickTime. Some couldn't even open the QuickTime application! We are well aware that the current versions of QuickTime and iTunes are not officially compatible with Vista, but our test had been running fine on the vast majority of Vista systems. We had a mystery on our hands!

We queried the Apple, Intel, and Microsoft public relations reps we regularly work with, to try to get to the bottom of this. Apple asked for more information and then we didn't hear back from them for an entire month! When we finally heard back from Apple, it was only to ask us for more information. (Ugh.) We've also been playing e-mail ping-pong with Microsoft for a month, but have yet to make any progress with the company on the problem. (Sigh.) Intel at least was quick to respond to our queries and to throw resources toward looking into the problem. (Yeah!) Of course, Intel had perhaps more motivation than the others, for as it turns out, much of the blame could be traced back to an Intel driver. Early on in our investigation, we discovered that all of the systems that exhibited this problem had RAID arrays using the Intel Matrix Storage Manager driver. Intel soon confirmed our suspicions that there was an incompatibility with its Intel Matrix Storage Manager driver for Vista and QuickTime 7.1.

Intel offered to send us an updated, prerelease version of the driver, so we could confirm the fix for ourselves. Before they would send us this new, prerelease driver, however, Intel wanted us to sign an NDA. The wording of the NDA was so ambiguous, however, that it could potentially be interpreted so as to limit CNET from publicly stating anything about Intel for the next five years without first getting Intel's permission. It is unlikely that censorship was Intel's intent; the apparent primary focus of the document was to protect Intel's intellectual property. But the boilerplate nature of the document opened the door to ambiguous legal interpretation. We didn't sign the document, and Intel didn't send us the driver.

Where does this leave us? With a test that still won't run on some systems and three major technology players still leaving CNET Labs hanging. We're hoping the problem will be resolved once Apple releases a version of QuickTime that is officially Vista compatible. Chances are, that's what Apple, Intel, and Microsoft are hoping for too. Rumor has it that we should see such a release from Apple very soon.