Antivirus Test: A Quest for Nearly Objective Rankings

Editor's Note: Chaz Sowers wants reliable, indepedently tested antivirus software with few false positives. But what really constitutes an "independent" test? Unsatisfied with lab ratings, he built his own malware testbed and put 35 AV products through the paces. Here is the story behind one man's AV rankings; your results may vary.

I started my research with an online company that just recently rated the "top" 14 AV products. They promised "independent comparatives of antivirus software" while at the same time stating that "since 2008 [they charge] a fee for various services we provide."

Call me a skeptic. But when a testing lab accepts money from a company to test its product, I have to wonder about the independence of the findings.

Even assuming the test results are truly independent, this business model excludes smaller companies that are unable or unwilling to pay the testing fees. A quick search online found over 40 AV products, many from companies I had never heard of before. I wondered how the lesser known ones might fair against the better known ones.

Since I already have a day job (as senior security architect at Vangent Inc., provider of information management and strategic business process outsourcing services) and didn't accept money from anyone for these test results, I decided to share my independent and unbiased comparison of AV products.

The results may surprise you.

Testing methodology, disclaimer and other stuff
My testing methodology was as unbiased as I could make it. After all, I had a vested interest in finding the best AV solution for my own computer. Of course my testing falls short of the double-blind scientific method, but I think it holds up well for publication in mainstream media. Remember this above all: I was searching for an AV product that would identify and delete the highest number of the test malware that I have. My emphasis in testing was on a high number of detections and my testing penalized software that reported a large number of "false positives."

Software
I used a fresh install of Windows XP, running in a Sun Virtual Box virtual machine, to run all tests. The installation of Windows was fully patched and updated (including SP3) as of Jan. 8, 2009. Each AV program was copied to the main machine from a shared folder and was the only program on the virtual machine not part of a regular Windows install. The test data resided on a logical D:\ drive and consisted of 36,438 pieces of malware. All of the malware has been, or currently is, in the wild. The virtual machine was restored to the previous, pristine state after each test.

Hardware

Sign up for ITworld's Daily newsletter
Follow ITworld on Twitter @IT_world

• Email
• Print
• Share
  • Slashdot
  • Digg
  • Stumble
  • Reddit
  • Newsvine