Latest Innovations In The "Boring" Regression Testing Space

In today's dynamic software landscape, teams struggle with where to invest their time: new feature testing or regression testing.
 

Obviously, ensuring the reliability and stability of applications is paramount so that the end-user experience is not impacted.
 

However, teams must also ensure that new features and capabilities that will propel the company's business to the next level are also properly tested and ready to be put in the hands of the end users.

But, how to find the right balance?

Traditional regression testing approaches rely mostly on selecting tests that have been created for testing a new feature when it was first released and re-running those tests after release to identify any application regressions.
 

The challenge there is how to select the "right" tests. Which ones will be most effective in catching regressions and, more importantly, regressions that would impact the end-user experience in the application?
 

The common approaches used to do this have been subjective, which is not scalable or repeatable.
 

At the same time, regression tests are the usual targets for starting any test automation endeavor because those tests are expected to be executed repeatedly before each release.
 

Unfortunately, the next challenge is to keep those tests up to date as the application evolves.
 

This is where most test automation efforts die — after around two years as the effort to maintain those automated tests results in a negative ROI.
 

Leveraging AI and real-user insights

To prevent this situation, there are emerging innovative approaches to regression testing, such as observing real user behavior in production environments and leveraging the latest AI techniques to automatically create, maintain, and run automated regression tests.
 

This enables teams to focus their efforts in testing new features that will positively impact the business of the company in the market, while ensuring the existing business-critical capabilities continue to work release after release.

But why observe real user behavior to automatically generate automated regression tests?

It's simple: because requirements are inherently ambiguous.
 

For many decades the software industry has tried to come up with approaches to reduce the ambiguity in requirements with the goal of decreasing the number of defects introduced in the application code throughout the SDLC.
 

Unfortunately, that never really worked.
 

The main goal of regression testing is to prove that new code is not going to unintentionally break existing code that wasn't expected to be changed during a release.
 

And that can be achieved by testing through the UI, APIs, DB, etc., or a combination of them all.
 

Given that everyone struggles for time during a release, the most common approach teams opt to follow is to design end-to-end test cases from the user's perspective — i.e., through the UI.
 

The assumption is if the user flow is working, then everything else must be working too because the user is able to accomplish what he/she wants. Therefore, the user gets value and the company satisfies its business metrics (e.g., revenue increase, growing number of customers, etc.).
 

On the flip side, if the user flow is not working, the end users won't be able to complete the task they were trying to achieve (e.g., buy a product, transfer money, book a trip, etc.) and the company's business will be negatively impacted (e.g., loss of revenue, customer churn, etc.).
 

There's a catch to this approach though: automating tests through the UI is very time-consuming and the tests tend to be flaky. So the maintenance effort on automated tests only increases as more tests are automated. Eventually, it becomes unsustainable.
 

There is, however, a silver lining. These emerging approaches to regression testing promise to eliminate the need for a human to automate end-to-end tests from a user's perspective.
 

Revolutionizing test automation with TrueTest

Imagine if your team could focus on automating tests for new features in the release while knowing the business-critical user flows are working, without the team having to automate any of those regression tests.
 

That's what Katalon TrueTest is gearing up to deliver. It's a very tall order to deliver on that promise. And over the past year the TrueTest Beta program has received tremendous insights from our beta community, which continue to be used to enhance TrueTest and take us closer to delivering on that promise.

If you're interested in joining the TrueTest Beta Program, sign up here. We'll do our best to extend an invitation as our capacity allows.