As the term suggests, test failure analysis is the process of dissecting a failed test case and analyzing what went wrong. This practice helps teams get to the source of a failure in software testing, correct the causing problem and prevent its future recurrence.

A structured test failure analysis should typically take into account the following aspects.

1. Application vs. test failures

When a test goes unpassed, does the problem lies in the codebase of the software you’re testing or in the test itself? Detecting the right cause of failure helps navigate your next actions in the correct direction. For example, having defined that the defect is due to a testing error, we can review our test cases or metrics and fix them correspondingly, not waste time and effort scouting for errors in the code build or design documents.

2. Root cause analysis

What is the core cause of the issue? Stopping at how test failure displays on the surface is just like making a diagnosis only by observing the symptoms. Furthermore, a problem can be caused by multiple different factors and possibilities. It requires testers to take a deep dive and narrow down the root cause so that resources can be directed to resolving it promptly.

3. The severity of failure

How critical is the failure? It can vary differently in the range of severity from low to high. In several cases, what appears to be a minor issue can develop into a serious one. Testers, for that reason, should examine how seriously the issue can affect your development process, whether it needs to be solved before moving to the next steps or can be delayed for later consideration.

4. The scope of failure

Besides evaluating the test failure’s serious level, teams should also figure out to what extent the impact spans across other environments. Your test analysis should assess how many builds or configurations can experience the same problem.

Why test failure analysis is key to faster delivery?

Automated software testing powers faster releases and higher-quality user experiences. As developers make changes to the applications, automation engineers, test leads, and developers work together to resolve issues that occur on some of the automated tests.

Test failure analysis activities allow these teams to perform root cause analysis (RCA) for these failures. These activities can take increasing amounts of time as the scale of automated testing grows.

Katalon’s AI-powered test failure analysis streamlines and reduces the time spent in these activities by automatically analyzing exception logs of failed tests and categorizing failure reasons so that teams get summarized results and can target their RCA activities at groups of tests rather than one by one.
Why do tests fail?

A “failed” test is not always a test that ran correctly and identified a bug. There could be something that prevented the automated test from executing properly. A failed test is usually caused by issues in three common areas:

Automation bugs
Application bugs
Network, authentication, or other third-party issues

1. Automation bugs

Automation bugs are caused within the automated test. Some common symptoms are:

Inability to locate application elements to interact with
Object wait time issues
Inability to interact with elements or intercepted interactions

These types of issues often occur when either the test engineer does not create robust automation or the chosen testing suite isn’t smart enough to proactively address these common issues. Katalon greatly reduces these types of issues using intelligent features such as smart wait and self-healing; however, some automation issues still occur, and common bugs are typically reviewed during test failure analysis activities.

2. Application bugs

We expect developers to update applications. Unfortunately, not all changes are fully documented or communicated downstream, which can lead to automated test failures due to the following reasons:

The original assertions about application logic are no longer accurate
Expected elements no longer exist
New libraries change interactions or objects

While we want automated tests to find these differences when the changes are permanent and validated, then the automated test needs to be updated.

3. Network or third-party bugs

There are a broad array of issues that can be caused by either network connectivity issues, authentication system issues, or any number of transactions and/or service calls to third-party providers that the application under test interacts with. The most common resolution to these issues is to rerun the test. A readily available report that groups these types of failures together can be the difference between a quick and successful rerun of a test or dozens of hours spent identifying related issues across numerous tests.

Katalon offers the best AI-powered test failure analysis

Katalon’s modern and comprehensive software quality management platform now provides test failure analysis that uses AI to review test failures, inspect logs, and provide consolidated analysis across all tests executed.

Test engineers, quality leads, and developers using the Katalon Platform can streamline their root cause analysis (RCA) and failure analysis activities with readily available reports grouping and test failures categorizing across the three major failure groups described above: automation, application, and network/third-party issues.

The AI-powered analysis highlights the possible causes and also provides links to resources for potential solutions, and continuously learns from your testing activities. Below are examples of the new capabilities.

Exceptions, log, recording, screenshots: All you need to trace back what happened

Easily review log information captured for each failed test and pinpoint where and when the issue occurred and which assertion failed at which test step.

Similar failures analysis: Find all other failures with the same cause

Katalon AI analyzes the similarity in the exception logs of other failed test results to correlate failed tests with similar failures. This saves time in collaborating and maintaining tests across testing activities.

Test failure categorization: Categorize failure reasons and recommend troubleshooting tips

Failure categorization: Categorize failure reason and recommend troubleshooting tips

Katalon AI automatically categorizes failure reasons under Application Bug, Automation Bug, or Network/Third-party Issues. In many cases, the system can recommend a troubleshooting guideline. This feature, especially for novice automation testers, helps to save time in analyzing and finding a solution to fix a test script.

Top 10 exceptions: What errors caused the most failures?

Review test failures over a selected period of time to have a clear view of the main causes of failed test scripts:

Did the automated tests catch application bugs?
Did they fail due to network issues?
Or does a high number of automation errors imply a need to improve automation skills or standards?

Example of a Test Result in Katalon