September 14, 2010, 7:52 AM — by Brent Huston, Microsolved, Inc. - I interact with a lot of people and organizations that have compromised machines. Often, my email or phone call is the first they have heard of the problem. Reactions vary from shock and denial to acceptance and occasionally rage. Even worse, when they hear that their machines are attacking others or being used in active attacks, many have no idea how to handle the situation.
Should you ever get a call like this from me or someone else, here are a few tips that you might find helpful for proceeding.
1. Be polite. I am calling to help you. Even though my message may mean more work and possibly some pain for you and your staff, knowing about a compromise is MUCH better than not knowing. Usually, the more polite and nice you are, the more information I will help you understand. I can usually point you in the right direction to begin to understand the issue, but if you act like a jerk, I will likely leave you to it.
2. Begin an investigation as soon as possible. Invoke your incident response process. If you don't have one, ask for help, or retain assistance. But, please, treat a caller who explains and demonstrates that you have a system compromise with immediate attention. I see hundreds of compromised systems a day and I don't have time to beg and plead with you to reduce your risk and the risk your systems present to others. I am happy to substantiate my claims, but after I notify you, TAKE ACTION. The majority of compromised systems involved in notification remain under attacker control for extended periods. Often, weeks and months pass by before any apparent action (such as mitigation or clean up) takes place.
3. Do a thorough job of mitigation. I would say that more than 25% of the time (I just started formally tracking this to gather better metrics.) when a site goes through "clean up", they end up compromised again and right back where they started from. Likely many of these machines are simply bot-infected and the bots just place their malware back on the system after "clean up" is done. Removing the basic tag files or malware, but not understanding how they got there in the first place and fixing that is pretty much meaningless. For example, I have been working with a site presently that has been used as a PHP RFI verification tag file host for weeks. They have "cleaned up" every day for several weeks to no avail. Every night, they get hit by another PHP RFI scanner and it exploits their system and drops a new tag or malware bot. I have tried explaining no less than 10 times how they need to identify the underlying PHP issue, harden the PHP environment (yeah, I sent them the settings) to no avail. This is an example of how to fail at risk, threat and vulnerability management. Don't do it. Fix the real problems. If you don't know how, ask and then follow the guidance provided. If you need more help, either retain it or get a scanner and start hardening.
4. Respect the law. Don't beg me not to turn this over to law enforcement. I have to. I want to, if you are critical infrastructure or some other member of the high threat club. Fix your stuff and manage security appropriately if you're a member of the club; or you deserve to explain to law enforcement why you declined. Either way, I am going to try and help you and everyone by making the report.
5. List a contact for security issues on your site. Please, when I do call, I need to know who to talk to. At the very least, let your reception folks know how to handle security calls. The last thing you want is for the attacker to continue to compromise your systems while I play in "Voicemail-Land" forever. Remember, help me help you.
Lastly, even if you don't get this call, do your due diligence. Make sure that your systems are secure and that you have security processes in place. Retain someone to help you manage risk and perform validation. Work with them to create effective risk management techniques for your organization. Hopefully, you won't be on the other end of the line tomorrow or the next day as I make my round of calls....
If you have any additional suggestions or comments on this approach, please feel free to drop a comment below. As always, thanks for reading and be careful out there.
Responding to a Compromised System Alert [State of Security]
Brent Huston is CEO and Security Evangelist for Microsolved, Inc., a leading provider of security assessments and penetration testing, as well as, the developer of the HoneyPoint line of security software products.