A Structured Evaluation for Selecting a Right Automated Testing Tool

One of the challenges in applying software test automation successfully in your projects is to select a most appropriate automated testing tool or framework.  It is a challenging decision as you have many tools to choose from and a number of requirements to satisfy, and automated testing tools may have hidden problems you don’t see or overlook at the time of making decision. Making the right tool choice is crucial to avoiding problems related to the tool that haunt your project execution.

In selecting a tool, you have a set of selection constraints, which can be classified into hard and soft constraints. Hard constraints are fixed and must be satisfied in any case. These constraints include platforms, executing environments, programming languages, and frameworks used to develop the target application under test (AUT).    

Soft constraints can be compromised or partially met. Such constraints include schedule, cost, maintainability, usability, level of programming and technical skills required. As these constraints are adjustable, you find it challenging in balancing among them in order to select the best tool given that most of the tools do not satisfy all of these constraints perfectly.

Tool Selection Criteria

To objectively evaluate a set of tools, it is recommended that you compare them based on the same set of criteria. You can define a set of criteria using your tool selection constraints and common characteristics of automation testing tools. Below are common characteristics that you may consider when weighting tools:

1. Platform: on which platforms does the tool being evaluated run? Platforms can be Windows, MacOS, Unix, Linux, Web, Mobile, etc. Some tools such as Selenium and Maveryx support multiple platforms, while others like Ranorex run mainly on Windows.
2. AUT programming languages: many tools support only a certain programming language used to write the AUT. For example, Abbot and Maveryx are used for testing Java applications. Many other tools like TestComplete and Ranorex can work with any languages used in the AUT provided that it can run on supported platforms.
3. Scripting languages: which languages does the tool support for writing test scripts? Many automation testing tools provide flexible scripting options, allowing testing teams to write test scripts in their most favorable languages.
4. Support: it is important to know whether the tool is actively maintained, upgraded, and supported by the original vendor and communities of users and developers. You most likely need help from the vendor or communities during your project. If the tool has not been released recently or the documentation is not up to date, it is possible that the vendor and the user community are no longer supporting the tool, or at least the interest in the tool has waned.
5. Usability: this criterion refers to how easy to learn and use the tool. It also indicates whether the tool is stable, robust and efficient. Most of software testing tools is easy to learn and use their basic features at first, but they require much more time to master advanced features, especially programming and maintaining scripts efficiently.
6. Script maintainability: one of the most challenging problems in test automation is to maintain test scripts to reflect changes in the requirements of AUTs. This is a time-consuming activity in projects where requirements change frequently, which is the nature of agile projects.  Some tools on the market such as Maveryx  and QTP have capabilities to automatically, albeit not fully, update scripts when software is changed. At the present, however, most of the script maintenance is done manually.
7. Required programming skills: one selling point for many automated testing tools is to support non-programmers, allowing testers who have limited programming skills to do automation test effectively. Tools like Selenium, Katalon Studio, Ranorex and TestComplete provide the record and playback features with scripts automatically generated. Still, some programming knowledge is needed to enhance and maintain test scripts, especially when you have complex test cases which have sophisticated scenarios and require many loops and checkpoints.
8. Automated testing approaches: various approaches are introduced in existing automated testing tools on the market. Available tools follow one or many of the following:
   1. Linear: procedural scripts executing step-by-step from the start to the end of a test case. The linear scripts are typically generated by the tool’s record feature
   2. Record/playback: tools have capabilities to record tests’ actions, generate test scripts, and playback test scripts automatically.
   3. Structured: unlike linear, this approach allows scripts to include control structures such as “if-else” and loop (“for”, “while”).
   4. Data-driven: the test execution flow is driven by data stored externally, in a database, spreadsheets or files.
   5. Keyword-driven: the test execution flow is driven by keywords typically stored in tables mapping keywords and input data needed for the execution. Objects captured from the record and playback capabilities are used as keywords in many tools such as Katalon Studio, Selenium, Ranorex, and TestComplete.
   6. Model-based: test procedures or scripts are automatically generated using requirements and behaviors model.
   7. Hybrid: supporting two or more of the above. Most of existing tools provide this hybrid approach.
9. Cost: of course, cost is an important factor in deciding which tool to use. You have to take into account both licensing and support costs. Some tools require no or nominal cost to acquire, but they incur a large sum of money spent to call for external support. Additional cost comes from training and spending extra effort for understanding and solving problems related to the tool.

Quantifying criteria’s value

It is sometimes difficult to determine a winner when facing a number of choices with many similar descriptive or qualitative characteristics. Some criteria are more precise to quantify using a numerical scale than using descriptions. For example, the level of programming and technical skills required can be easily quantify using a rating scale between 0 (none or no programming or technical skills required) to 10 (highest level or advanced programming and technical skills required).

Many of the criteria above can be quantified using the same range, such as, from 0 (none) to 10 (highest). For criteria with a definitive value “No” or “Yes”, using 0 for “No” and 10 for “Yes”.

Putting things together

The table below provides a brief summary of evaluation of 7 popular testing tools using the criteria discussed above. Much of the rating for Support, Usability, Script maintainability, Required programming skills is based on my subjective judgment. You may have a different assessment, resulting in other rating values. Due to space limitation, the information on the tools shown in the table is not complete, and the costs of commercial tools can vary dependent on types of license and maintenance needs. 

Dr. Vu Nguyen, Director of Software Engineering at KMS Technology.

Trigger moda