How do People Select Test Automation Tools?


A large majority of experienced testers and managers prioritize licensing and support costs and good test reports when deciding which test automation tools to use for their projects. In our survey conducted with 2,000+ test professionals, more than 60% of respondents with 3+ years of experience in test automation agreed that these criteria are the most important priorities in deciding a test automation tool. Moreover, respondents with less experience in test automation prioritize training and support materials over other criteria.

The top six priorities considered by more than 50% of respondents with 3+ years of experience are:

(1)   Licensing and support costs

(2)   Good test reports

(3)   Training, documentation, tutorials, guidelines

(4)   CI, DevOps support

(5)   Level of programming skills required

(6)   Level of skills and experience required

Automation-tool-selection-criteriaSuggested Tools

Open-source tools (coding experience required)

  • Selenium
  • Appium
  • Robot Framework
  • Cucumber

Free tools (limited community’s support)

Commercial tools (high support & license cost)

  • TestComplete
  • UFT
  • Ranorex

Software automation tools and frameworks are crucial for the success of test automation projects. But it is a difficult task to select a right set of test automation tools for a new software project. There are many tools with each having several unique good features. One single tool may not satisfy all project requirements and constraints, resulting in multiple automation tools to be used. Chosen tools seem to be working at the beginning, but troubles occur later as hidden problems are overlooked at the time of tool selection.

A tool vendor often provides a list of key features and characteristics when comparing their tool with others in the market. They tend to highlight their tool’s strengths while ignoring important features that may pinpoint its weaknesses. This practice confuses not only new test automation adopters but also experienced ones in making decision about which tools and frameworks for new test automation projects.

To select a suitable tool, one can base on a set of top priorities for which every interested tool is checked. Such checklist may come from experience of professionals as well as specific project requirements and constraints.

Understanding the priorities in determining automation tools is part of our survey which questioned software test professionals about their experience with test automation. The survey’s questionnaire was posted on several software testing websites and shared via several mailing lists including that of Katalon Studio’s.

We received answers from 2,291 respondents of which 1,877 (82%) have ever applied test automation on their projects. Of all respondents, 970 (or 42%) and 484 (22%) have at least one and more than three years of experience in test automation, respectively. The results reported here exclude the respondents who said to have yet to apply test automation.

Tool Selection Priorities by all Respondents

The chart below shows all attributes suggested as possible top priorities and the corresponding percentages of respondents who agreed to. One respondent could choose one or more attributes.

Tool Selection Priorities by all Respondents

The top four attributes chosen by at least 50% of respondents as their most important priorities in tool selection include:

(1)   Training, documentation, tutorials, guidelines: the highest percentage of respondents chose this attribute, which is concerned with training and materials for learning and using the tool, as a priority for their tool selection.

(2)   Good test reports: tool generated test documents and logs for status report, test execution analysis, and defect diagnosis.

(3)   Licensing and support costs: all costs associated with acquiring, maintenance, and support the use of tools.

(4)   Level of programming skills required: if testers on the team do not possess good programming skills, learning programming to use a tool is a real concern for them.

The highest priority is identified by 56% of all respondents. Many respondents, 58%, have less than 1 year of experience in test automation. So, it is reasonable that training and materials for learning and using tools is the most important attribute for choosing a tool.

The fifth highest priority chosen by almost 50% of respondents is the level of skills and experience required by tools. This attribute, which is related to the 4th highest priority or the level of programming skills required, refers to the need of skills and experience to use the tool effectively. For example, Selenium is a popular test automation framework but it requires adopters to possess a high level of technical skills and experience to get started.

It is quite surprising to find that a majority of respondents did not consider user interface/user experience (UI/UX), support services, long-term commitments, and application lifecycle management (ALM) tool integration as the priorities in selecting an automation tool. Noticeably, less than 20% of respondents chose the ability to integrate with the ALM tools as an important priority.

It is worth noting that although functionality/feature-rich of a tool is often cited as a strength, it is not appreciated by many respondents with just 46% of them using it as a priority.

Tool Selection Priorities Evaluated by Experienced Respondents

We considered two respondent groups, one with at least 3 years of experience in test automation and the other with 1 year or less. It is intuitive to believe that the feedback on the top priorities for selecting tools by experienced test automation professionals is more reasonable than that by less experienced ones.

As shown in the figure below, the most important priorities evaluated by highly experienced respondents differ significantly from the view of those with less than 1 year of experience in automation. The top six priorities considered by more than 50% of respondents with 3+ years of experience are:

(1)   Licensing and support costs

(2)   Good test reports

(3)   Training, documentation, tutorials, guidelines

(4)   CI, DevOps support

(5)   Level of programming skills required

(6)   Level of skills and experience required

Highly experienced respondents tend to be more consensual on the priorities than those with less experience, with six criteria being chosen by more than 50% of respondents. More than 60% of them agreed that licensing and support costs and good test reports are the top two tool selection priorities. These attributes are more important than training and support materials, which is not surprising as they already have experience in test automation.

Licensing and support costs is ranked highest by experienced respondents while it is ranked fifth by less experienced respondents with less than 50% of them identifying it to be a priority. Many tools are very affordable initially, but their maintenance, support, and related costs are extensive in the long run, a fact clearly witnessed by be experienced testers.

Continuous integration (CI) and DevOps support is considered a priority for 55% of experienced respondents, but it is not a priority for most (less than 20%) of respondents with less than 1 year of experience. Similarly, experienced respondents appreciate user community more than less experienced respondents. A strong and active user community of a tool for sharing knowledge and troubleshooting issues being faced is essential for professional testers.

The survey shows, surprisingly, that the ability to integrate with the ALM tools and UI/UX, which are often referred to as the key attributes of test automation tools, do not seem to be important to both experienced and inexperienced respondents with less than 30% of respondents choosing these tool characteristics as a priority.

Tool Selection Priorities Evaluated by Experienced Respondents


Instead of presenting expert opinions, this article reports several of the findings from our study surveying software professionals about their perspectives on automation tool selection priorities. These findings serve as a reference for checking expert opinions and making decision about which tools to be used in test automation projects. This survey suggests several important implications:

Cost versus technical expertise. This survey shows that licensing and support costs is the top tool selection priority identified by 64% of experienced respondents. This is a reason why most respondents have used open-source tools and frameworks for their projects such as Selenium, JMeter and Appium. However, most of these tools, unlike typical commercial alternatives, require technical expertise (programming skills and experience) to build, integrate and deploy before utilizing them effectively. Fortunately, recently emerging automation solutions like Katalon Studio provide an integrated and ready-to-use environment by leveraging popular open-source frameworks and libraries such as Selenium and Appium. These solutions are an attractive alternative to both open-source and commercial solutions as they are affordable and do not require technical expertise to deploy, programming skills and experience to apply.

Good test reports are important. The basic report of a test execution is whether a test case is passed, which is supported by most, if not all, automation tools. However, the automation community needs much more than that, as suggested in this survey. They need informative and insightful test reports to help diagnose and analyze defects and root causes, test coverage, test effectiveness, and other analysis. Such reports become necessary for managers to make informed decisions about the quality of their products.

It is essential for an automation tool to support CI and DevOps tool integration. As test automation is the cornerstone of CI and DevOps practices, a serious automation tool must be able to integrate with CI and DevOps toolchains. The lack of CI and DevOps support of an automation tool can significantly reduce its effectiveness and benefits in agile projects where speed is a priority. As the trend of increasing DevOps adoption continues, the integration capability becomes an essential feature for automation tools.

Learn more: Strategy to Select the Right Automation Testing Tool

About the Author:

Dr. Vu Nguyen, Advisor at Katalon, Inc.,  Director of Software Engineering – KMS Technology, Lecturer at the University of Science, Vietnam National University.

The Selenium alternative
that makes a difference.
Scroll up