There is no doubt about it: Artificial Intelligence (AI) and Machine Learning (ML) has changed the way we think about software testing. Ever since the introduction of the disruptive AI-powered language model ChatGPT, a wide range of AI-augmented technologies have also emerged, and the benefits they brought surely can’t be ignored. In this article, we will guide you to leverage AI/ML in software testing to bring your QA game to the next level.
AI/ML in software testing is the integration of Artificial Intelligence and Machine Learning technologies into many phases and aspects of software testing to enhance it. With the help of AI/ML, software testers can now supercharge their testing by leveraging the human-level decision-making capabilities of these technologies.
An intelligent system like AI/ML can bring tremendous benefits to QA teams that know how to incorporate it the right way. Although not a truly recent technology, AI/ML has improved so much in the past few years, and there are so many scenarios and possibilities of utilizing them.
According to the State of Software Quality Report 2024, test case generation is the most common application of AI for both manual testing and automation testing, followed by test data generation. You can download the report for the latest insights in the industry.
There are so many that we can use AI/ML to power-up our software testing, and the key to unlock those capabilities is knowing what these technologies can potentially do, then find creative ways to incorporate them into your day-to-day testing tasks. Note that there are 3 major approaches when choosing an AI/ML system to incorporate into your software testing, including:
The final decision on which approach to use depends on the vision of the entire organization and the team. Whichever one you choose, there are generally 5 areas an AI/ML system can contribute the most:
We all know how disruptive ChatGPT has been to the software engineering industry. Basic coding tasks have now been handed to ChatGPT, requiring freshers in the field to immediately upgrade their skills to meet the rising standards.
Similarly, in the software testing field, we can also request the ChatGPT to write some test cases for us. Keep in mind - ChatGPT is not the one-size-fit-all solution. Sometimes it works, but sometimes when the requirements get too complex, you need to provide extra context so it can better execute your prompt.
For example, I asked ChatGPT to “write a Selenium unit test to check if clicking on a button on https://exampleweb[dot]com will lead to the correct link “https://exampleweb[dot]com/example-destination”, and ChatGPT provides a fairly well-written code snippet with detailed explanation of each step as well as clear assertion to verify that the destination link is indeed the expected URL. It is actually impressive!
Traditionally, these automated test scripts have to be written by skilled testers with coding skills using test automation frameworks. Not just that, ongoing maintenance was necessary whenever source code changes occurred, or else the scripts won’t understand the updated code, resulting in wrong test results. This is a common challenge among automation testers since they don’t have enough resources to constantly update their test scripts in the ever-changing Agile testing environment.
Thankfully, when incorporating AI/ML in software testing, you can now use simple language prompts to guide the AI in crafting tests for specific scenarios and speed up your test maintenance work.
The power of Machine Learning lies in its ability to learn from data and make predictions on its own without having to be explicitly programmed for each specific scenario.
This means that as time goes on, the AI learns more about how users use the app that you're testing. It starts adjusting how it creates tests to match what your business needs. It figures out the usual ways you test things by looking at the rules, existing tests, and historical data to then suggest the best test cases for your situation.
Imagine you're running an eCommerce website with thousands of products, different user paths, and constant updates to the website. It is extremely challenging for testers to cover all possible scenarios even with certain areas automatically tested.
This is where Machine Learning comes into play. At first, you need to feed the AI with data about how users interact with your website. This data includes things like what products they view, what actions they take, and when they abandon their carts.
As the AI gathers and analyzes this data over time, it starts noticing patterns. It recognizes that certain products are frequently viewed together, and that users tend to abandon their carts during specific steps in the checkout process.
With this information, the AI no longer has to follow a predefined set of tests; instead, it tailors the testing to mimic how real users actually interact with the website. In this case, the AI will focus more on testing the checkout process and the interactions between related products.
The great thing about this technology is that the more you use it, the better it becomes. Talking about aging like fine wine! In a further time frame, AI/ML can even turn into a full-on predictive analytics engine, capable of detecting potential issues based on the patterns and defects it had learned. Having a leading indicator to guide the direction of QA activities empowers QA managers and Team Leaders to make more informed decisions.
In certain cases, to test comprehensively you would need a large volume of data, or data with a variety of combination inputs.
Let’s use the eCommerce website example above again. For global eCommerce websites that ship across borders, generally there are variations in the way shipping fee is calculated, such as package weight, geographic zone, distance, carrier selection, coupons, or even additional charges. Such complexity makes covering all shipping scenarios in the testing project quite difficult.
However, you can leverage AI/ML to generate a mock data set that you can use to execute the test case without having to collect real-life data. In the case mentioned above, you can totally command the AI to provide you with a list of addresses in regions that your business operates in. If you want to go into the minute details (shipping weight, time zones, additional fees, etc.), just tell the AI and save yourself hours of manual work.
Software and websites often undergo updates, especially in organizations following the Agile testing approach. When these updates happen, test scripts created to test specific parts of the software can stop working correctly.
For instance, if we wrote a test to click on a button identified by the label "login-button," any change in this label could cause the test to fail. Fixing numerous test cases each time code changes (which happens frequently) can be time-intensive. This is also a common bottleneck testers face during regression testing sessions.
AI can automatically adjust the test scripts whenever there's a code change. For example, if AI can't locate an object using its current method, it will try another method and proceed with the test. This feature is known as the "Self-Healing Mechanism." It reduces the burden on testers and ensures tests continue running smoothly even when there are changes in the code.
Visual testing is an entirely different arena to software testers. In the past, human testers had to rely on their eyes to find visual differences between how the user interface (UI) looked before it was launched and how it appeared after it was launched. The problem arises when we want to automate visual testing, which involves comparing the screenshots of the UI before and after it was launched:
This is a common problem with visual testing tools too, but with AI, it has been successfully addressed. The AI can learn if certain zones should be “ignored” even if there are changes there, and if the differences it notices between the 2 screenshots are reasonable. In a way, we give the AI the capability to look at visual testing from a more human perspective, instead of comparing every single pixel so rigidly.
Seeing that it is only the beginning phase of AI-based testing, and yet there have been so many bottlenecks and challenges commonly found in automation testing successfully addressed, we can only expect to see more and more exciting stuff coming along. For now, AI is the future, but in the future, AI will inevitably be the norm, and testers must fully leverage their benefits to catch up with the trend.
Here is how AI can help:
It is important to note that AI/ML does not replace testers. They only supercharge testers, which means that they should become an “extension” of their mind, helping them solve problems faster and smarter. It is similar to having an assistant that can perform amazing cognitive tasks, all the time.
Interested? Watch a recording of our webinar on how Generative AI can help you level up software testing:
Having AI/ML in software testing comes with added responsibilities that we should embrace. Some of the challenges that testers should be aware of include:
Testing using AI and testing for AI systems are 2 completely different topics.
Testing with AI is using AI models to supercharge your testing, while testing for AI systems is to test if those models themselves perform as expected. However, the non-deterministic nature of AIs makes testing for them quite challenging.
By non-deterministic, we mean that AI systems can exhibit completely different behaviors for the same request. In the context of generative AI, this means they provide answers to the same question differently.
This makes traditional software testing concepts like “test coverage” obsolete, since it is impractical to measure test coverage when there are theoretically infinite possibilities and scenarios that can happen within a large enough AI system.
Katalon is a comprehensive AI-augmented software quality management platform, orchestrating the entire software testing lifecycle from test creation, execution, maintenance, to reporting across web, API, desktop, and mobile applications.
A forerunner in the AI testing landscape, Katalon continuously enriches its product portfolio with these innovative AI-infused capabilities, empowering global QA teams with unprecedented accuracy and operational efficiency. Learn more about Katalon AI testing here.