Test Automation Evolution

Software test automation has evolved through several generations of tools and techniques:

Capture/playback tools record the actions of a tester in a manual test execution, and allow tests to be run unattended, greatly increasing test productivity and eliminating the mind-numbing repetition of manual testing. However, even small changes to the software under test require that the test be recorded manually again. Therefore, this first generation of tools is not efficient or scalable.

Scripting, a form of programming in computer languages specifically developed for software test automation, alleviates many issues with capture/ playback tools. However, the developers of these scripts must be highly technical and specialized programmers who work in isolation from the testers actually performing the tests. In addition, scripts are best suited for GUI testing but don’t lend themselves to embedded, batch, or other forms of systems. Finally, as changes to the software under test require complex changes to the associated automation scripts, maintenance of ever-larger libraries of automation scripts becomes an overwhelming challenge.

Data-driven testing is often considered separately as an important development in test automation. This approach simply but powerfully separates the automation script from the data to be input and expected back from the software under test. Key benefits to this approach are that the data can be prepared by testers without relying on automation engineers, and the possible variations and amount of data used to test are vastly increased. This breaking down of the problem into two pieces is very powerful. While this approach greatly extends the usefulness of scripted test automation, the huge maintenance chores required of the automation programming staff remain.

Keyword-based test automation breaks work down even further in an advanced, structured and elegant approach. This reduces the cost and time of test design, automation, and execution by allowing all members of a testing team to focus on what they do best. Using this method, non-technical testers and business analysts can develop executable test automation using “keywords” that represent actions recognizable to end-users, such as “login”, while automation engineers devote their energy to coding the low-level steps that make up those actions, such as “click”, “find text box A in window B”, “enter UserName”, etc. Keyword-based test design can actually begin based on documents developed by business analysts or the marketing department, before the final details of the software to be tested are known. As the test automation process proceeds, bottlenecks are removed and the expensive time of highly-trained professionals is used effectively.

The cost benefits of the keyword method become even more apparent as the testing process continues. When the software under test is changed, revisions to the test and to the automation scripts are necessary. By using a keyword-based framework, organizations can greatly reduce the amount of maintenance needed, and avoid rewriting entire test scripts. Many changes do not require new automation at all, and can be completed by non-technical testers or business analysts. When required, changes to automated keywords can be completed by automation engineers without affecting the rest of the test, and can be swapped into other tests in the library as needed.

The keyword method has become dominant in Europe since its introduction in 1994, where it was incorporated into the TestFrame method and tool, and is now coming into its own in the USA. LogiGear’s Action Based Testing™ represents the continued evolution of this approach under the guidance of the original architect of the keyword method. This method is the foundation of LogiGear’s test automation toolset, TestArchitect™, which not only organizes test design and test automation around keywords, but also offers built-in actions that make it possible to automate many tests without scripting of any kind.

Hybrid testing tools merit a brief discussion due to the intense marketing efforts of several vendors that are helping to bring awareness of keyword-driven technologies to the industry. In these products, a keyword-like user interface is layered atop a traditional automated testing tool. Most of these tools simply provide a GUI window listing the library of low-level functions that automation engineers have produced in scripting language. Many attempt to offer “scriptless” automation through the use of GUI views, templates and business rules-based test design. One example emphasizes a graphical interface that enables non-technical users to create tests by specifying low-level actions. The tool then performs automatic code generation in a target scripting language. These canned keyword scripts must be regenerated each time a change is made to test, and direct editing is not recommended.

These hybrid solutions, in attempting to oversimplify test automation engineering, are unable to offer the power, flexibility and customization necessary to automate tests for complex systems. In addition, because these tools implement a keyword-based interface without an underlying testing method supported by the full keyword framework, the manual creation and maintenance of tests is rather labor intensive due to the use of only low-level keywords.