How to choose and use source code analysis tools

The high cost of finding and patching application flaws is well known. Wouldn't it be cheaper to write secure code in the first place?

One of the fastest growing areas in the software security industry is source code analysis tools, also known as static analysis tools. These tools review source code (or in Veracode's case, binary code) line by line to detect security vulnerabilities and provide advice on how to remediate problems they find - ideally before the code goes into production. (See How to Evaluate and Use Web Application Scanners for tools and services that search for vulnerabilities as an outside attacker might.)

The entire software security market was worth about US$300 million in 2007, according to Gary McGraw, CTO at Cigital, Inc., a software security and quality consulting firm in Dulles, VA. McGraw estimates that the tools portion of that market doubled from 2006 to 2007 to about $180 million. About half of that is attributable to static analysis tools, which amounted to about $91.9 million, he says.

And no wonder; according to Gartner, Inc., close to 90% of software attacks are aimed at the application layer. If security were integrated earlier in the software development lifecycle, flaws would be uncovered earlier, reducing costs and increasing efficiency compared with removing defects later through patches or never finding them at all, says Diane Kelley, founder of Security Curve, a security consultancy in Amherst, N.H. "Although there is no replacement for security-aware design and a methodical approach to creating more secure applications, code-scanning tools are a very useful addition to the process," she says.

Despite the high degree of awareness, many companies are behind the curve in their use of static analysis tools, Kelley says, possibly due to the big process changes that these tools entail.

Key decisions in source code analysis

1) Should you start with static tools or dynamic tools or use both?
In addition to static analysis, which reviews code before it goes live, there are also dynamic analysis tools, which conduct automated scans of production Web applications to unearth vulnerabilities. In other words, dynamic tools test from the outside in, while static tools test from the inside out, says Neil McDonald, an analyst at Gartner.

Many organizations start with dynamic testing, just to get a quick assessment of where their applications stand, McDonald says. In some cases, the groups that start this initiative are in security or audit compliance departments and don't have access to source code. The natural second step is to follow up with static analyzers, enabling developers to fix the problems found by dynamic analysis tools. Some companies continue using both, because each type yields different findings.

An important differentiator between the two types is that static analyzers give you the exact line of code causing the problem, while dynamic analyzers just identify the Web page or URL causing the issue. That's why some vendors offer integration between the two types of tools.

According to the chief scientist at a large software vendor, dynamic assessment tools "tend to be brute force," he says. "You have to hit every parameter to find the vulnerabilities, whereas static tools investigate the whole landscape of the application." He recently chose a code scanner from Ounce Labs, after outsourcing the work to Cigital since 2006. He became interested in application security when customers began requiring PCI DSS certification. He plans to add in dynamic testing in the future, but the static analysis tool is the cornerstone of his application security program.

2) Do you have the source code?
Most static analyzers scan source code, but what happens if you want to analyze third-party software or code written so long ago that you only have the executable? In that case, Veracode, Inc. offers binary code scanning through a software as a service platform. "A vendor may not be willing to give you source code, but they will give you executables or binary," Kelley says.

At the Federal Aviation Administration, Michael Brown, director of the Office of Information Systems Security, says he chose to use Veracode's services this year because of the amount of vendor-written code the FAA anticipated to use as a result of its modernization of the national airspace system. Brown says he wanted to ensure the code was not only functionally correct and of high quality but also secure. He wanted a service rather than a tool to reduce the need for training. So far, the results have been eye-opening, he says. "A lot of the code didn't really take security into account," he says. "There were cases of memory leaks, cross-site scripting and buffer overflows that could have been a cause for concern."

3) What do you currently use for software quality?
Some tool vendors, such as Coverity, Inc., Klocwork, Inc., Parasoft Corp. and Compuware Corp., originated in the quality-testing arena and have added security capabilities vs. vendors like Ounce and Fortify Software, Inc., which were solely designed for security. It's worthwhile to check into the quality tools you already use to see if you can leverage the existing relationship and tool familiarity. You should also consider whether it's important to your organization to have the two functions merged into one tool in the long term, McDonald says. (See Penetration Testing: Dead in 2009? for more on the interplay of quality assurance and vulnerability detection.)

Source code analysis tools: Evaluation criteria

-- Support for the programming languages you use. Some companies support mobile devices, while others concentrate on enterprise languages like Java, .Net, C, C++ and even Cobol.
-- Good bug-finding performance, using a proof of concept assessment. Hint: Use an older build of code you had issues with and see how well the product catches bugs you had to find manually. Look for both thoroughness and accuracy. Fewer false positives means less manual work.
-- Internal knowledge bases that provide descriptions of vulnerabilities and remediation information. Test for easy access and cross-referencing to discovered findings.
-- Tight integration with your development platforms. Long-term, you'll likely want developers to incorporate security analysis into their daily routines.
-- A robust finding-suppression mechanism to prevent false positives from reoccurring once you've verified them as a non-issue.
-- Ability to easily define additional rules so the tool can enforce internal coding policies.
-- A centralized reporting component if you have a large team of developers and managers who want access to findings, trending and overview reporting.

Do's and Don'ts of source code analysis

DON'T underestimate adoption time required. Most static analysis projects are initiated by security or compliance, not developers, who may not immediately embrace these tools. Before developers get involved, McDonald suggests doing the legwork on new processes; planning integration with other workflows like bug-tracking systems and development environments; and tuning the tool to your unique coding needs. "Don't deploy to every developer at once," he adds. "Ideally, you'll get someone who wants to take on a competency role for security testing."

The chief scientist at the large software vendor has developed an application security awareness program that includes training on common vulnerabilities, through podcasts and videocasts. Once he builds up awareness, he'll educate developers on secure coding standards. To complete the circle, he'll introduce Ounce's static code analysis tool to enforce the standards and catch vulnerabilities "so it's a feedback loop," he says. (See Rob Cheyne Pushes for Developer Security Awareness for a look at a similar agenda.

DO consider using more than one tool. Collin Park, senior engineer at NetApp, says the company uses two code analysis tools: Developers run Lint on their desktops, and the company uses Coverity each night to scan all completed code. "They catch different things," he explains. NetApp began using these tools when its customer base shifted to enterprise customers who had more stringent requirements. While Coverity is better at spotting vulnerabilities such as memory leaks, LINT catches careless coding errors that developers make and seems to run faster on developer desktops, Park says.

According to Kelley, organizations typically implement static analyzers at two stages of the development process: within the development environment, so developers can check their own code as they're writing, and within the code repository, so it can be analyzed at check-in time. The chief scientist uses this method. "In the first scan, if the engineer takes every finding and suppresses them, a milestone scan will catch those and generate a report," he says.

DO analyze pricing. Vendors have different pricing strategies, McDonald says. For instance, while all continuously add information to their libraries about the latest vulnerabilities, some charge extra for this, while others include it in the maintenance fee, he says. In addition, some vendors charge per seat, which can get expensive for large shops and may even seem wasteful for companies that don't intend to run the scanner every day, while others charge per enterprise license. Additionally, some vendors charge for additional languages, while others charge one price for any language they support, McDonald says.

DO plan to amend your processes. Tools are no replacement for strong processes that ensure application security from the beginning, starting with defining requirements, which should focus on security as much as functionality, according to Kelley. For instance, a tool won't tell you whether a piece of data should be encrypted to meet PCI compliance. "If a company just goes out and buys one of these tools and continues to do everything else the same, they won't get to the next level," she says.

The chief scientist says it's also important to determine what will happen when vulnerabilities are found, especially because the tools can generate thousands of findings. "Does the workflow allow them to effectively analyze, triage, prioritize or dispose of the findings?" he says. He is working with Ounce to integrate the system better with his current bug-tracking system, which is Quality Center. "It would be great to right-click on the finding to automatically inject it into the bug-tracking system," he says.

At NetApp, Park has reworked existing processes to ensure developers fix flagged vulnerabilities. As part of doing a code submit, developers do a test build, which must succeed or it can't be checked in. Then, when they check in code, an automated process starts an incremental build. If that build fails, a bug report is filed, complete with the names of developers who checked in code before the last build. "Developers are trained to treat a build failure as something they have to look at now,'" Park says.

NetApp also created a Web-based chart that's automatically updated each night, to track which managers have teams that were issued Lint or Coverity warnings and whether they were cleared.

DO retain the human element. While the tools will provide long lists of vulnerabilities, it takes a skilled professional to interpret and prioritize the results. "Companies don't have time to fix every problem, and they may not need to," Kelley says. "You have to have someone who understands what is and is not acceptable, especially in terms of a time vs. perfection tradeoff.'"

The chief scientist calls this "truly an art form" that requires a competent security engineer. "When the tool gives you 10,000 findings, you don't want someone trying to fix all those," he says. "In fact, 10,000 may turn out to just be 500 or 100 vulnerabilities in actual fact."

Park points out an instance where the Coverity tool once found what it called "a likely infinite loop." On first glance, the developer could see there was no loop, but after a few more minutes of review, he detected something else wrong with the code. "The fact that you get the tool to stop complaining is not an indication you've fixed anything," Park says.

DON'T anticipate a short scan. NetApp runs scans each night, and because it needs to cover thousands of files and millions of lines of code, it takes roughly 10 hours to complete a code review. The rule of thumb, according to Coverity, is for each hour of build time, allow for two hours for the analysis to be complete. Coverity also enables companies to do incremental runs so that you're not scanning the entire code base but just what you've changed in the nightly build.

DO consider reporting flexibility. At the FAA, Brown gets two reports: an executive summary that provides a high-level view of vulnerabilities detected and even provides a security "score," and a more detailed report that pinpoints which line of code looks troublesome and the vulnerability that was detected. In the future, Brown would like to build into vendor contracts the requirement that they meet a certain security score for all code they develop for the FAA.

DON'T forget the business case. When Brown first wanted to start reviewing code, he met with some pushback from managers who wanted a defined business need. "You've got program managers with schedules to meet, and they can view this as just another bump in the road that's going to keep them from making their milestones," he says.

Brown created the business case by looking to independent sources like Gartner and Burton Group for facts and figures about code vulnerability, and he also ran some reports on how much time the FAA was dedicating to patch management.

The chief scientist justified the cost of the Ounce tool by taking the total cost of the product and comparing that to the effort involved in a manual review. "With millions of lines of code, imagine how many engineers it would take to do that, and by the way, we want to do it every week," he says. "The engineers would fall down dead of boredom."