Microsoft 'impartial' browser test fails the grade

Browsers given same scores despite OS, settings

By  

The geniuses at Microsoft's marketing department have come out with a new tool designed to assist users in determining the safest browser for them to use while surfing the web. The conclusion? Only Internet Explorer 9 is suitable for most users' needs.

The site, found at yourbrowsermatters.org, seems like a parody of everything someone would do to mock a blatant attempt by Microsoft to self-promote its own products. I especially liked the ".org" URL to lend the "testing" site an air of grassroots respectability. But this is not a parody, because Microsoft seems to be taking itself seriously with this one.

The site, which features a suspiciously Metro-like interface, detects your browser agent and then provides a score based on what the site believes is the browser's security features, on a four-point scale.

The "test"--and I use that word very loosely--rates browsers based on these four categories:

  • "Dangerous Downloads
  • "Phishing Websites
  • "Attacks on your Browser and its Plugins
  • "Attacks on Websites"

Based on these criteria, Your Browser Matters seems to rate your browser regardless of how your settings are configured, or what operating system you're using. I used the site's test (you just have to open the home page to activate the test itself) on a variety of browsers with operating systems that are relatively impervious to malware thanks to security settings, and got consistent results across the board for the browser itself.

To show what I mean, here is a list of the browser scores I receive when going to Your Browser Matters on a 64-bit Windows 7 machine:

  • Firefox 7.0.1. Score: 2/4
  • Chrome 14.0.835.202. Score: 2.5/4
  • Opera 11.51. Score: No score available
  • Internet Explorer 9.0.8112.16421. 4/4

I'll get to Opera in a minute, but before that, check out these scores from an Ubuntu 11.04 (x86_64) box:

  • Firefox 7.0.1. Score: 2.5/4
  • Chrome 14.0.835.202. Score: 2.5/4
  • Opera 11.51. Score: No score available

Same scores, despite the fact that there is very little chance based on my configuration settings in Ubuntu and the browsers that any malware would be able to execute on Linux.

But wait, it gets more interesting. I hadn't run Chrome for a while on this machine, and it wanted to do an upgrade to 14, which I noted after I'd already visited the Microsoft test site. What you see above is the result I got after upgrading Chome to v. 14. But the original test run was for Chrome 13 (13.0.782.220, to be exact) and wouldn't you know, Chrome 13 had the same score of 2.5 out of 4. And this test site seems to have a problem with Chrome. When looking at the detailed Score page on the site after the test, the Chrome 13 browser was detected as "Chrome 14." (Chrome 14 was properly detected as such on the Ubuntu machine.)

I wanted to be thorough, so I ran some OS X Lion browsers through the same test. Here, Your Browser Matters seemed to choke out more.

  • Safari 5.1 (7534.48.3). No score available
  • Aurora 9.0a2 (2011-10-10). No score avialable
  • Chrome 14.0.835.202. 2.5/4

Again, Chrome scores a 2.5 out of 4, regardless of what platform it's on. But once again, the test got the version wrong: the detailed Score page actually listed the browser as Chrome 13. Clearly, Your Browser Matters has issues with figuring out what version of Chrome is actually running, at least on Linux and OS X machines.

And then there's these unscored browsers. What's with the lack of scores for Opera and Safari? I will be fair and not question Aurora's lack of score--the cutting edge alpha build is likely not on anyone's radar yet. But not Safari and Opera? Here, Microsoft's testing methodology FAQ provides an answer:

"Initially, we decided to limit our focus on the browsers with a majority of the market share on Windows. (These three combined represent 93% of all web browsing on Windows today.) Our focus is to provide a browser score for majority of the population browsing the Internet using a PC."

Hrm, right. And explicitly ignore any other operating system along the way. Gotcha. But this basically proves that Microsoft is merely reading the browser agent, regardless of the operating system or the individual browser's settings, and spitting out a canned result. When it sees a browser agent it doesn't recognize, it chokes out and doesn't give a score.

Microsoft claims the test is impartial, too. From the FAQ:

"We've also vetted the test with independent partners such as the Anti-Phishing League, Identity Theft Council, and Online Trust Alliance and security first groups such as WhiteHat Security and Cenzic to ensure that the score is well designed. Only the browsers which do not implement adequate security features receive a poor score. The case in point being Internet Explorer 6 and 7--both receive low scores because they doesn't [sic] have most of the features that help protect against modern threats."

First off, note that these impartial organizations only vetted the tests, not the test scores. Second, it may seem that Microsoft is being impartial, given that Internet Explorer 6 gets a 0 of 4 score and Internet Explorer 7 gets 1 out of 4, but given that Microsoft is trying very hard to get people to upgrade away from IE6 and IE7--and well they should--this is really no surprise.

In fact, while I am sure slapping Firefox and Chrome around is all fun for Microsoft, that may be the second goal here: get users to upgrade from earlier versions of IE to IE9. That's why Firefox and Chrome have such low "scores"--Microsoft doesn't want people to upgrade out of the IE family altogether.

The whole thing is pretty silly, though I will go on record as saying that any effort to get folks away from IE6 deserves the Nobel Prize as far as I'm concerned. But to actually call this test impartial is ridiculous. Anyone--including Mozilla, Apple, and Opera--could set up a test that makes their browser look good. Like political polls, it's all how you frame the questions.

Read more of Brian Proffitt's Open for Discussion blog and follow the latest IT news at ITworld. Drop Brian a line or follow Brian on Twitter at @TheTechScribe. For the latest IT news, analysis and how-tos, follow ITworld on Twitter and Facebook.

Join us:
Facebook

Twitter

Pinterest

Tumblr

LinkedIn

Google+

SecurityWhite Papers & Webcasts

See more White Papers | Webcasts

Answers - Powered by ITworld

Ask a Question