Anti-virus testing guidelines coming early next year

Consumers should have more accurate information with which to compare security
software suites under a new set of software testing guidelines to be finalized
early next year.

Last week, security vendors and software testing organizations agreed during
a conference in Seoul to form the Anti-Malware Testing Working Group, which
will determine how best to conduct behavioral tests of security software, said
Andreas Marx, who works for AV-Test.org, a German antivirus software testing
group.

Behavioral tests are time-consuming but important since the style of test replicates
how PCs encounter malicious software on the Internet, such as through Trojan
horse programs in e-mail attachments or through browser exploits, Marx said.

Those tests are seen as superior to signature-based tests, in which the virus
detection engine is run against a batch of thousands of malware samples. But
signature tests do not cover other security technologies used to detect a threat,
such as if a new program starts communicating with a remote server over the
Internet.

"A very big part of the real world is missing," Marx said. "Most
products are tested against a set of outdated viruses. In most cases, most AV
products will pass these tests."

The cooperation between security vendors is notable, especially in the highly
competitive security software industry. Vendors Panda, F-Secure, Sunbelt Software
and Symantec are participating as well as AV-Test.org and Virus Bulletin, another
testing organization based in Abingdon, England.

Marx has written a draft of a behavioral testing scheme. Early next year, the
Anti-Malware Testing Working Group will refine those guidelines for use by groups
such as AV-Test.org and Virus Bulletin. Use of the guidelines, however, will
be voluntary.

Most vendors feel the new behavioral tests will more fully evaluate the different
ways their products can provide protection to a PC. Security companies have
often publicly argued over signature-based tests, with disputes centering around
the age of the virus samples used.

Companies that fail a signature test often argue that the particular sample
that caused them to fail was too old and not even commonly found on the Internet.
Some security vendors will remove signatures in their products for older malicious
software so PCs are not burdened with large signature databases.

The Anti-Malware Testing Working Group will also provide an unbiased forum
for those disputes. Today, "there is little recourse," if a vendor
fails a test and has an issue with the test's parameters, said Mark Kennedy,
an antivirus engineer with Symantec.

There is concern, however, that the behavioral tests may put too much a strain
on testing groups, Marx said. Setting up real-world malicious software scenarios
takes a lot more time, Marx said. Usually, a behavioral test is limited to around
50 current malware samples, he said.

However, testing organizations are in early discussions that could result in
some cooperation in order to reduce that burden, he said.

Marx said AV-Test.org recently conducted a first
round of behavioral tests for PC World, a magazine owned by IDG.

The results shows that none of the eight products tested performed very well
in behavioral tests due to the increasing sophistication of malware. Security
companies have said their labs are having a hard time keeping up with the startling
increase in the quantity of malware circulating on the Internet.

Marx said his lab alone receives between 2,000 and 2,500 different samples
of malicious software per hour.