January 18, 2001, 12:00 AM — I just love statistics - especially when they support a point I want to
make. You can use them to prove anything. Hey, numbers don't lie! But
they don't always give the whole picture either.
Quite a few news stories recently revolved around Attrition.org's Web
defacement statistics. Almost 60 percent of the defaced Web servers are
running on the Microsoft Windows/NT platform; an interesting statistic
clearly demonstrating a problem with Windows/NT servers.
However, statistics merely measure and categorize data. Using
statistics alone as a research tool is like trying to build a house
with just a screwdriver. Yeah, you need it, but you need some other
tools as well.
An article on SecurityWatch managed to draw a bunch of erroneous
SecurityWatch states that Attrition endorsed MacOSX and Power BSD as
the "safest" platforms. Funny, I didn't see that statement from
Attrition. They also stated that more servers on the Internet are
running on NT platforms than all other Operating Systems combined.
Obviously they didn't check Netcraft, which lists the market share as
closer to 20 percent (http://www.netcraft.com/survey/).
One of the more absurd conclusions posted to the "ihateapple.com" site
(no bias here) (http://www.ihateapple.com/). In the posting "Web
Defacement Figures", "Russ" states, "Attrition.org has lists of
supposed website defacements and hacks for last year." He then jumps
to the conclusion that Windows 2000 must be the most secure OS, since
it only accounts for 9.96 percent of the defacements. Uh, Russ? W2K has
only been around for about a year. Incidentally, the "ihateapple.com"
site was defaced on 12/31/00:
link to Attrition's index is provided as proof that the site was
defaced - I don't recommend clicking on the defacement.)
I asked Attrition's statistics specialist, Matt Dickerson (aka "Munge")
what he thought of all the attention the Attrition defacement
statistics generated. Matt commented:
"It's interesting what people choose to take away from our stat pages.
I am beginning to think that it's more a Rorschach test than anything
else. For the past year, we've updated os.html daily and os-graphs.html
monthly. It's never got this sort of reaction before."
Attrition staff go through a lot of effort to explain how they compile
the data used in their statistics - fully expecting this to be weighted
into whatever research their readers are performing.
It's easier to look at the pretty pie chart than do real research. So,
what *do* the statistics indicate?