Felstead based his argument on the warning that appears when links suspected of harboring malware appear within Bing's results and those links are clicked by the user.
"By using the API instead of the user interface, AV-TEST bypassed our warning system designed to keep customers from being harmed by malware," said Felstead. "Bing actually does prevent customers from clicking on malware infected sites."
Felstead said that users see the warning only once in every 10,000 searches, or 0.01% of the time, a number close to AV-TEST's 0.012%. "In any case, the overall scale of the problem is very small," Felstead asserted.
AV-TEST confirmed today that it relied on a Bing API (application programming interface) to collect search results from Microsoft's engine.
"No links were clicked/followed through the search engine," Andreas Marx, CEO of AV-TEST, said in a Monday email reply to questions. "We simply grabbed the URLs and downloaded them on our own systems for further analysis. We didn't want to test the warnings from the search engine but simply how many potentially malicious websites are returned by the search engine."
Microsoft cited vacationhotline.net as an example of how Bing warns users. But the site's owners denied the infection allegation. Computerworld confirmed that Bing shows this warning. (Image: Microsoft.)
Marx acknowledged that some search engines ward off users from suspicious links with warnings, but of those, not all are as clear as Bing's, and could be easily ignored or dismissed by users.
He also cited Felstead's claim that "Our data shows that these warnings block 94% of clicks to malicious sites" to defend AV-TEST's approach.
"Microsoft argues that their warning is 94% effective, so 'only' 6% of the people will click on the malicious link anyway," Marx countered. "Still, that's a lot of people."
Additional protective measures, including Bing's in-results warnings, those displayed by browsers -- all the major Web browsers have mechanisms for warning users of potential danger when they click on some links -- and others generated by security and antivirus software, were "out of scope for this study," Marx said.
Marx stood by the study, and said it would not be revised to take Microsoft's complaints into account. "The report was NOT designed to be a 'safety comparison' for search engines," he said.
However, he confirmed that AV-TEST was considering revamping its methodology for future tests. "In the next report, we might be able to report that search engine A warns [of] 30% of the malicious links, or if it's just 1%, or more like 70-80%, plus how many false positives we've seen," Marx said.
As to why Bing indexes suspicious links and shows them in its results, Felstead contended that "most are legitimate sites that normally don't host malware but have been hacked."