May 01, 2001, 6:38 PM — What programs can test graphical applications? As consultants, we recognize that a large part of our business is decoding what clients really want, but we are wary of this particular question, as it has a history of obscuring true goals. This two-part series will explore the quality assurance (QA) of programs with GUIs.
First, we need a brief theory of quality. There's an extensive amount of literature on quality, much of it applicable to computing. In fact, quality assurance is one of the few well-grounded domains of computing, in the sense that it has yielded replicable experimental data. Evidence about the effectiveness of object orientation or the correct philosophy for chip sets (CISC versus RISC?) remains ambiguous and generally open to interpretation. Quality researchers, though, proudly point to the principles they've established beyond reasonable doubt:
- Quality is built-in, not tested-in
- Tests can only prove a product wrong, not right
- It's far less expensive to detect faults earlier rather than later
- Individuals and organizations generate characteristic levels of errors that they reproduce in subsequent trials
What's the significance for the construction of a high-quality GUI application? Well, quality matters. In general, it matters more than any testing program bandaged on a project at the last minute. Does your project have the resources to manage a QA phase that only takes action when the product is almost ready? Then it has more than enough resources to document requirements explicitly, design the application carefully, and engineer the software throughout development. We've never seen a case where investing in end-phase testing and beta programs pays off more than it does with those kinds of up-front engineering.
Even when the above commandments are observed, there's generally more important work for a QA staff than the often disorganized grunt work traditionally assigned them. This month, we'll describe the activities a successful QA department undertakes as part of a development project. Next month, we'll look at specific testing frameworks for common GUI applications.
QA's contribution
Too many organizations try to paste QA onto the end of their development process. When they think they're nearly ready to go to market, they toss their newest employees, and sometimes others who don't fit in anywhere else, in a room, stir in a poorly defined beta program, and ask for a miracle. This is the intellectual equivalent of solving security for a rock concert by hiring a biker gang on the final weekend before the show. Both are recipes for spilt blood.
Real quality requires involvement from a QA engineer from the start. A good product has sound finances, a manufacturing plan, and so on, from the start, to focus on achievable objectives that meet customer needs. It also has a QA plan from the beginning.
From the first planning session, a good QA engineer asks, "What does it mean to have an 'adequate' response? Does the screen need to flash within three seconds? Thirty? Do you realize that writing custom widgets rather than using the native ones has a history of soaking up weeks of tinkering and just confuses end users? Will the documentation we're planning satisfy all three of the market segments we're targeting? Why have we budgeted for a smaller customer support staff, on a proportionate basis, than our competitors?"
The first job is to assess feasibility. A plan that calls for even elite coders to turn out hundreds of line of source daily, with all but one in a thousand flawless, is unrealistic.
In the best projects, documentation and testing protocols are written at the beginning and are constantly available to guide source development. While this is one of the precepts of extreme programming (XP), it's a principle that was known long before XP's current popularity. It's certainly standard procedure in more mature domains of engineering, including aerospace.
Perhaps most indispensably, QA engineers hunt for testability.
Testing only proves programs wrong, not right
Testing never proves a program right. That proposition is well-known dogma in QA circles, and neglected or not even understood outside them. Here's a stark example. Can you test your way to confidence that:
0.1 * (a + b) = (0.1 * a) + (0.1 * b)
How many values of a and b would you have to try to make you feel good about the formula?
