February 17, 2013, 4:35 PM — While reverse engineering malware may not be something you've ever even considered, after a three day class, I can say that, yes, it's difficult, but also that it's well within the grasp of ordinary mortals (at least those ordinary mortals who can wrap their brains around assembly language and system calls) -- especially if they take the right class, get the right tools, and practice.
What is it?
Reverse engineering malware is the process of taking a captured executable (a stand-alone executable or a library file, such as a DLL) and doing the computer equivalent of an MRI. You should always take some care in where and how you do this kind of work. Analyzing malware in a "safe" environment -- such as a virtual system on a machine that isn't connected to your network -- would be the option of choice. At least this would allow you to snapshot your OS and revert to it whenever you inadvertently kick the malware under your microscope into action.
Unix admins who have some experience using the strings command to identify strings defined within binaries or hex dumps provided by commands such as od will have some rudimentary knowledge of the contents of executables, but there is lot more that you can see if you examine the malware file using a debugger (a tool that allows you to step through a program, watching for the effects that each step makes) or a disassembler (a tool that turns the machine language of your malware into assembly language).
With a disassembler, you would perform what is called "static analysis". With some experience, you will come to see how the malware you are analyzing was intended to work. Maybe it inserts itself into another process, maybe it creates a file, maybe it changes user settings ...
With a debugger, you can break and change the code "on the fly". You can exploit test conditions that may have been designed to make the malware appear benign under certain conditions. You can jump over code that isn't worth your time to analyze. This process is often referred to as "dynamic analysis".
Some debuggers and disassemblers are free while others cost a bundle, but if you are going to be doing serious work with malware analysis, you should keep in mind that any tool that helps you get answers in hours instead of days should be considered a good investment (do the math!), especially if you will be using it often.
Some of the tools that you are likely to learn about if you take a class or do some online research are IDA Pro, Immunity Debugger and Olly Debugger. The IDA tools are available on Mac OS X as well as Linux. In the AoE class that I took, I had opportunities to work with all of these tools and much appreciated that I had someone walking around the classroom willing to answer all kinds of questions.