Hacked! How we got attacked by malware fiends and lived to tell the tale.
Yes, my Web site was hacked. What was almost as damaging, though, were the tools set up to warn people about it.
It's one thing to write about hackers, scammers, and malware, as I've done for various venues for the last 15 years. It's quite another to experience the nastiness first hand. Yesterday it was my turn. Here's my story.
Last night around 7 pm I Iogged onto my occasionally NSFW humor site, eSarcasm, to post something snarky. What I saw though, was something different and much scarier than usual -- A big fat warning sign from Google Chrome:
Uh oh, I thought. That couldn't be good. But surely it's some glitch. Why would hackers target us? (OK, I can think of a few reasons. Maybe it was one of those humor-impaired Craigslist strippers we'd been mocking lately. We know Apple fanboys don't like us much. Or possibly 4chan's legions had stumbled onto our site and found something displeasing.) So I checked with my partner in snark, JR Raphael. He was seeing it too.
[ See also: How to murder a Flash cookie zombie ]
We tried other browsers. Firefox displayed a magenta page that directed users to StopBadware.com, a site co-sponsored by Google and Mozilla designed to steer people away from malicious Web sites. My Firefox NoScript plug in confirmed that, sure enough, a script trying to redirect to brnighome.com was running on our home page, but only some of the time. And we could find no evidence of any links to brnighome.com anywhere on the site.
Now what do we do?
We sent a panicked note to our Web host, Doreo, hoping someone was manning the support lines after hours and would take pity on us. In the meantime, we took the site offline and searched for information about brnighome, which was scant.
Turns out this was not an uncommon problem with OpenX. For the past year attackers have routinely exploited vulnerabilities in OpenX to serve up 'malvertising' to unsuspecting users. To its credit, OpenX has patched these holes as quickly as it finds them. Unfortunately, we didn't know about these vulnerabilities, and we had not patched our software. We simply set up OpenX to rotate some banners and forgot about it. That was a mistake.
Getting rid of the malware was as easy as getting rid of OpenX. (Hasta la vista, baby. Don't let the virtual screen door hit you on the way out.)
A bigger problem? Getting our site off of Google's blacklist. Even after we'd gotten rid of the malicious code, visitors to our site were still seeing those scary red screens telling the world we were bad bad webmasters who must be shunned. And this was happening on a day that the Google gods had been smiling down upon us, sending us lots of traffic.
It's like getting that cute guy or girl down the hall to finally notice you, on the day you've got a big juicy canker sore on your lip.
In a word, oy. But it gets worse.
My personal site, dantynan.com, was also red flagged. Why? Because I'd installed a widget that served up a scrolling list of headlines from eSarcasm. This meant any other site that had installed that same widget (a few hundred at last count) would also display the Red Screen of Death. This was very bad.
Fortunately, StopBadware has a simple process for reviewing sites that have cleaned up the nasty bits from a hack attack. We submitted our site and waited.
Google's Webmaster tools dashboard is also supposed to offer an option for re-reviewing your site after it has detected nasty stuff on it. This option was invisible for us (and we are not the only ones). Fortunately I discovered a solution -- reloading the dashboard from scratch reloads the initial "Request review" button under the Diagnostics/Malware tab.
According to Google, it could take as long as 48 hours to get off the blacklist. That would have been a disaster for us. But we were lucky. Within 8 hours, the ban had been lifted.
Now all we had left to do was tell our users what happened, and hope that a) no one had gotten infected by visting our site, and b) people who'd encountered the Red Screen of Death would be willing to come back again. The jury's still out on both of those.
Lessons learned? Obviously keep all your site's plug ins and software up to date. (Though this would have been much easier if OpenX had plugged into the WordPress admin dashboard and not required its own, or if it had some mechanism for alerting users when security holes had been discovered besides its support forums.)
A more important lesson though, is that the price of Internet publishing is eternal vigilance. Being hacked can happen to anyone. Even you.