January 17, 2011, 10:48 AM — The fallout from last month's allegations that the Federal Bureau of Investigations attempted to deploy backdoors in the OpenBSD operating system are continuing to echo through developer circles, as more potential clues are unearthed. But if anything, these clues tend to muddy the answer to the key question: did the US government employ contractors to insert deliberate security holes into OpenBSD?
Here's what we know so far.
On Dec. 11, OpenBSD founder and lead developers Theo de Raadt received an email from Gregory Perry, CEO of GoVirtual Education, a Florida-based VMWare training firm, in which Perry told de Raadt he was "aware of the fact that the FBI implemented a number of backdoors and side channel key leaking mechanisms into the OCF, for the express purpose of monitoring the site to site VPN encryption system implemented by EOUSA [an acronym for the US Dept. of Justice], the parent organization to the FBI."
In his message to de Raadt, Perry stated that while Perry was the CTO at NETSEC, a now-defunct contracting group, "Jason Wright and several other developers were responsible for those backdoors." Perry said that he was now able to share this information with de Raadt because his non-disclosure agreement with the FBI had "recently expired."
Almost immediately, discrepancies began to appear in Perry's claims. Notably, Scott Lowe, the OpenBSD author and advocate whom Perry named as a paid front-man for the FBI to advocate OpenBSD (and presumably get more vulnerable OpenBSD systems deployed), vehemently denied the charges. (Actually, it was two Scott Lowes who fit that description, and both men denied the allegations.)
Upon further reflection, it seems a little unlikely that the FBI or any government agency would push just OpenBSD, since the Internet Protocol Security (IPsec) protocol suite that is alleged to have been broken is widely used by a number of other operating systems. If IPsec was indeed given a backdoor, then its wide adoption would have done all of the work for the government.
Jason Wright, the one developer Perry named as a participant in the government plan, has also publicly denied the accusations, and has demanded an apology from Perry.
Meanwhile, former FBI cyber-crime agent E.J. Hilbert added fuel to the fire due to a Dec. 14 tweet that stated, "I was one of the few FBI cyber agents when the coding supposedly happened. Experiment yes. Success No." However, Hilbert's original meaning of the quote was initially taken as confirmation that Perry's allegations had merit. In subsequent tweets, Hilbert clarified that he was referring to the FBI's own security audits of code they were planning to deploy, a point on which he elaborated within a Dec. 15 article on ThreatPost:
"Hilbert... said that what the bureau does do--as many federal agencies do--is to perform stringent testing of any new software package that it is considering for internal use. That testing can include vulnerability assessments and attempts to find exploitable flaws in the application, but that's not the same as deliberately inserting a flaw and then repackaging the app for public consumption."
Freelance software developer Marsh Ray has made a significant amount of headway in his search for what may have happened. On. Dec. 18, he posted a lengthy analysis of a bug he located in OpenBSD that could fit the bill for a deliberate security hole in the main OpenBSD tree--but in his opinion did not.
"This bug doesn't sufficiently meet the criteria for a malicious backdoor," Ray wrote in mailing list post to the OpenBSD community, copied in his blog.
"The bug does not leak key material or establish a covert channel, it would require an active attack to exploit and even then would probably need to be used in connection with some other defect in order to result in meaningful unauthorized access. Yeah sorta it maybe could be used as part of that, but not really its own," Ray added.
In Ray's analysis, there is an easier explanation for the security holes found in the encryption code around that time. In the 1990s and early 2000s, tools to devise encryption and security analysis were scarce, and US developers has a further obstacle. Due to US bans on the exporting of encryption technology, many developers on encryption software in OpenBSD had to explicitly document the non-US origins of the encryption code. Logistically, this was a sub-optimal state of affairs, as Ray details:
"Because of the need to document the non-US origin of this code, the code produced from the hackathons needed be committed to the OpenBSD source control system (CVS) before the developers returned to the US, whether it was fully-baked or not. As OpenBSD consistently adhered to a calendar-based release schedule (every six months), this effectively set the clock ticking to get it in release shape."
This hack-and-go procedure, Ray concludes, may have contributed to the creation of several security holes in the OpenBSD code.
At this point, evidence seems to point to a definite government project to test the vulnerability of OpenBSD for the government's own deployments--tests that included the insertion of backdoors into OpenBSD within test systems, but did not include committing those backdoors to the main OpenBSD tree. This theory would fit with Perry's assertions--though they were certainly overblown--and the denials by Wright and the two Lowes. In this theory, technically everyone was telling the truth: backdoors were not inserted into the main OpenBSD codebase, but backdoors were being inserted at the government's behest for their own security testing.
In his Dec. 18 blog post, Ray indicates that OpenBSD should have issued a statement of disclosure about the work Wright and another then-NETSEC developer, Angelos Keromytis were doing in this part of OpenBSD. But that would mean de Raadt would have had to have directly known what Keromytis and Wright were tasked to do by their employer NETSEC at the time.
de Raadt has denied knowing what was going on in this area. In a Dec. 21 post to the [openbsd-tech] mailing list, de Raadt confirmed that he was aware of the status of Wright's and Keromytis' employment, and that NETSEC were being contracted by the government to work on OpenBSD. This work directly benefited OpenBSD in a number of ways, and not just in the cryptography sections of the operating system. But, de Raadt states, he was not aware that NETSEC was doing the backdoor work at all.
"If [Wright] and [Keromytis] knew NETSEC was in that business, I wish they had told me. The project and I might have adjusted ourself to the situation in some way; don't know exactly how. With this view, I do not find Jason's mail to be fully transparent," de Raadt wrote. de Raadt further emphasized that he does not believe any vulnerabilities made it into the OpenBSD tree.
But there is still speculation that something odd was going on with OpenBSD a decade ago. A long blog post by OpenBSD developer Michael Shalayeff entitled "how i stopped worrying and loved the backdoor" seems to imply that the hostile political environment between OpenBSD and NetBSD developers in those days could have been a contributing factor in slipping a deliberate security hole in OpenBSD. (The "seems" qualifier is due to Shalayeff's unclear writing style and use of language, so caution must be used when using this article as a reference.)
Shalayeff holds forth the notion that due to the hostilities between the two BSD communities, de Raadt often would commit code to the OpenBSD tree in what Shalayeff terms a "stealth" manner.
"'[S]tealth' means that purpose of the diffs was not disclosed in the commit messages or the private openbsd development forums except with a few "trusted" developers," Shalayeff wrote. By choosing to not be clear on why code was committed to OpenBSD, possibly in an effort to block NetBSD progress, Shalayeff opines, de Raadt indirectly assisted anyone who would want to insert vulnerabilities into OpenBSD, particularly NETSEC.
Ray noted Shalayeff's article as well and summarized it in a blog post yesterday:
"Probably the most straightforward interpretation of Mickey's story simply confirms what we already knew: funny stuff was going on in the source tree at that time and people crossing international borders sometimes receive some heavy arm-twisting by the US government, even if they are American citizens such as Jacob Appelbaum and Moxie Marlinspike. It's not hard to imagine that pressure being applied to someone seeking to continue working or studying in the US," Ray wrote.
Ray was not planning to revisit the OpenBSD issue. Then he came across a New York Times article Sunday which describes the Stuxnet worm attacks as a joint US-Israeli cyber-operation to cripple Iran's nuclear enrichment program. One of the US participants, according to the Times article, is the Idaho National Laboratory (INL), which Ray notes is the current employer listed for Wright.
"So we know Jason Wright was hacking on OpenBSD IPsec crypto code at the time the backdoor was alleged to have been added, and that he was pentesting Siemens SCADA systems at the time Stuxnet was being constructed and at the very same national nuclear research lab identified by the New York Times," Ray concluded in his blog post. "This guy sure seems to have a talent for coincidences."
Ray's inference seems strong, but in reality it does not definitively prove that Wright was involved in any efforts to punch holes into the public OpenBSD code (nor involved in the Stuxnet project, since the INL or any US agency has neither confirmed or denied involvement in Stuxnet). All it demonstrates is that Wright is clearly very good in the security arena in which he works.
Though Perry may have thought he was finding a smoking gun, it seems that at worst he uncovered areas where OpenBSD procedures may have been lax. A lack of full disclosure from Wright and Keromytis could have helped OpenBSD avoid fallout from Perry's charges. The practice of committing undocumented patches to OpenBSD asserted by Shalayeff and confirmed by Ray has not helped, either, because such commits could make it possible (though still unlikely) that NETSEC could have gotten something into the OpenBSD tree.
History may show otherwise, but right now this incident seems to be a story of missteps, and not maliciousness.