Microsoft OneCare did/did not improve in recent AV tests

After finishing dead last in a comparative antivirus test, Microsoft Windows Live OneCare recently garnered some positive press. The latest tests performed by AV-Comparatives.org seem to show an improvement, with OneCare moving up two places. While OneCare is certified by West Coast Labs and ICSA, it is the competitive independent antivirus testing results that mean more in terms of how well a product performs in the real world against real malware. Thus, some might argue that things are looking up for the nascent Redmond antimalware team.

That's until you look closer at the tests. AV-Comparatives performed two different tests, months apart, alternating between real-time detection and on-demand detection. The two tests are not the same. With real-time detection, a fully updated version of the antivirus product is exposed to a list of viruses (and their variants) currently in the wild. This is the test that OneCare failed back in February. On-demand tests use a smaller collection of viruses, usually viruses received since a predetermined freeze on signature file updates from the vendor. The idea here is to see if older signature files can detect newer malware. On this, the Microsoft AV product scored slightly higher in the latest (May 2007) results. We'll have to wait until the next real-time test to see if OneCare has improved or not. The FAQ section of the AV-compartives site provides more information on the testing process.

A blog on the McAfee site also goes into greater detail on this. Among the points made by McAfee researcher Joe Telafici is that results for proactive tests might vary because of distribution size (i.e. larger vendors will have to be more cautious with their heuristics than a smaller company in order to avoid false positives) and that with the on-demand test the signatures are frozen months beforehand, whereas a user who is only two days out of sync with the latest update will experience much better results. Telafici also includes several McAfee-authored essays on antivirus testing in his blog:

Comparing the comparatives
Counting spyware detections

Antivirus testing workshop in Reykjavik