What to Look in a Test Automation Product: Features or Benefits

Almost all the same benefits apply to almost any automation, whether it is vulnerability testing by security experts, integration testing by large enterprises, load testing by tier-1 carriers, or acceptance tests of outsourced development. I am not sure if this is useful for you, but if it is: Please give a thumbs-up and I will know that at least some people have troubles with automation.

Test Automation

Manual testing and test scripts are often painful to develop and use, but basically still are the de-facto in the industry. A manual test, or self-developed test scripts doesn't require that much investments in tools, only in time and people. But when time, and people are exactly what you do not have, then the natural move is towards test automation.

When you start looking at test automation products, you can easily get distracted by the technological features in the collateral and related white-papers. Model-based! TTCN! XML syntax notation! Dynamic message sequences! 100% test coverage! I want that!

The leap from manual tests or simple scripts into fully automated test suites makes all self-respecting test engineers just simply excited. It is difficult to see the limits to the things where test automation could be used. But hold on! Are you thinking straight? Why was it that you were looking for test automation in the first place?

To remind you, there are three main things why people look for test automation:

• cheaper test execution,
• less resources in development and maintenance, and
• better test coverage.

Cheaper, Faster, More... Through Automation

Most test automation tools can be categorized in two groups:

• Capture-playback tools
• Test modeling tools 

A capture-playback tool lets you define your own test sequences, or if you look at it another way, it requires you to build the entire test setup just to produce a valid, conformant test sequence. It will then record that and let you re-run the test. Test automation people in this camp are known to say "If you need to do it more than twice, automate".

The challenge in this type of automation is that the tests are only as good as the capture you take to "seed" the tests. The test automation vendor takes no responsibility for the quality of tests. And you still need to build the "old fashion way" for the first test execution, and then the tool just automates the re-execution of the tests.

A test modeling tool lets you define a test using various test design technologies such as state-diagrams and context-free languages for message syntax. A test model can also be licensed from a third party. In that case, both the development of the test generation engine and the maintenance of the model is someone else's responsibility. You just run the tests and analyze the results. In short, the test modeling tools change how you build the first test, and from there will impact the maintenance and test execution.

Summary of test automation benefits:

• Test process efficiency is impacted by resources needed for test design, execution and maintenance. With record-playback tools you only get benefits if you need to run the tests more than once. Model-based tools are must if your tests change frequently due to better test maintenance capability.
• Tests need to be scalable based on the changing user requirements. Interfaces change, and load capacity changes constantly. Test equipment can become obsolete quite fast, whereas a software based solution keeps up with the development of the hardware and operating system components.
• The test process integration is best reached by walking through the test process and seeing that each step merges naturally with each other. Not all tools need to be Eclipse-integrated. Some (e.g. security test results) cannot go directly to bug reporting systems due to their confidential nature. But when needed, such integration needs to be feasible.
• Systematic and measurable tests give visibility into the quality assurance process. Adhoc tests with manual test design provide no visibility to the real test coverage, whereas model-based testing demonstrates full test coverage against the given test specification.
• Regression test usage is a critical feature, as all tests need to be re-run even after the launch of the product. Careful design of the test laboratory (not a cheap investment!) can result in reusable capture-playback tests, whereas model-based testing can usually be used as-is from unit test to regression test.