August 13, 2009, 8:28 PM — I spent quite a bit of time researching Developer Tools You Don't Use — and Why You Don't Use Them. I heard from more than 30 people on every aspect of the subject, and as you might imagine, some of the comments hit the cutting room floor. Among them, alas, was the full-length response from Alice Kaerast: "I understand enough about security to feel comfortable that my code is written securely, and the beauty of open source means that there are many people actively testing and checking code, all of whom know about detecting vulnerabilities better than any software."
[ See also: Convincing the Boss to Accept FOSS ]
Her message-in-depth was too much of a tangent for the core part of my article, but it touched on two issues that were relevant to the essential "Why don't developers use these tools?" question I was burning to answer. (It says something about me that I really do get worked up about questions like this. I'm not sure if it's a good thing.) First is the underlying assumption by the open source community that FOSS code is inherently better because so many people have looked at it; I'm not going to address that here.
The second intersection with "I use open source development tools" is with developers' oft-stated reason not to use the QA tools because the software is so expensive. If open source has changed so much about the way programmers (and users!) work, from replacing Visual Studio with Eclipse to creating whole new application frameworks and content management system... shouldn't the answer to "I don't use these apps because they're so expensive" be, "So how 'bout an open source tool instead?"?
Naturally, that led me on a quest to find open source tools to reccommend (or at least identify) in the categories I'd enumerated as least used (data modeling tools, application modeling tools, load and stress test tools, refactoring tools, or performance tools, plus special-mention for security test tools, in case you weren't ready to blurt it out from memory). If nothing else, I thought, it could be a handy resource for developers who desperately want to use this kind of software but whose budget request from the corporate bean counters is answered, "You must be kidding."
At a minimum, I figured, a developer could poke at an open source testing tool — I decided to start with load- and stress-test tools, for no particular reason — to learn what the software could do for her, and to discover if indeed it did improve her code. That might not be a good primary motivation, though. I remember trying to figure out the difference between the $40 espresso machine and the $400 model (other than $360), and deciding that if I couldn't tell the difference I might just as well buy the cheap one. I did eventually purchase a $500 espresso machine as well as a home roasting setup requiring me to buy fair trade green coffee beans online... but then you know how obsessive I am. I don't think I'm typical. Most people would stop at the low-end espresso-maker and never learn what the quality difference is. With computing tools, the problem's far worse. Just ask any web designer who's been asked to develop a dynamic e-commerce site using Microsoft FrontPage because his client thought it was good-enough. (Oh no. I mentioned it, didn't I? Please. Get down off the window ledge. I didn't mean it. I'll never mention it again.)
It's easy enough to google for a list of open source tools (and there's even a helpful site devoted to open source testing), but that doesn't give me a lot of useful information about what makes one more valuable or helpful than another. So I asked several developers and professional QA people to tell me about the options. At that point my focus shifted entirely.
There are indeed open source load and stress testing tools, and it's probably not a bad idea to explore them. But according to the experts who advised me, you shouldn't imagine that they are equal to the proprietary tools.
For example, I'm told, Apachebench is easy enough to use and can generate large amounts of requests. The downside, said my friend Sean, is that it requests the same URL over and over, which may not exercise your application properly.
Similarly, Sean said,
But professional QA testers are generally dubious of the open source tools. Not because they aren't fans of open source (I've known some of these folks for years and trust their commitment) but because they believe the proprietary tools really are better. "I've used the commercial load testing tools (SilkPerformer, LoadRunner) and a cloud-based load testing tool (BrowserMob). I've spent a number of years looking for an open source tool that was as good as a commercial one and I've never found one I can recommend," said Ed Borasky (whom I also quoted in the original article). "The key differentiator is ease of script recording. If you're willing to use a conventional or custom programming language and write scripts and load test definitions by hand, there are plenty of open source tools. I guess the best I've encountered is JMeter. But easy capture of test scripts from users executing an application as far as I know can only be found in the commercial tools," Ed explained.
One contact dis-recommends all the open source load- and stress-test tools he's encountered because, he says, "Staff time is the 100-pound gorilla, not license cost (unless you have more people than you should)."
Also, these tools do rely on the developer's existing knowledge to interpret the results. It's important to understand what you're doing when you're stress testing, Sean said. "You want to find the breaking point of your application. You want to find out where your bottlenecks are in your code. Measuring response times isn't helpful here because a large component of that time is going to be the queuing delay, not the time the application processes information." Obviously, if you are going to turn to these tools, it behooves you to learn about the topic; there's plenty of options for a developer who wants to acquire that knowledge, but the tools won't give you a wizard-based "follow the bouncing ball" process that will assure you that yes, your e-commerce site can manage the holiday rush without breaking a digital sweat.
I suppose that dedicated open source developers should see this situation as an opportunity: a software domain in which the open source code quality cannot (yet) claim to be better. You up for the challenge?
