.svg)
There is no doubt about it: Artificial Intelligence (AI) and Machine Learning (ML) has changed the way we think about software testing. Ever since the introduction of the disruptive AI-powered language model ChatGPT, a wide range of AI-augmented technologies have also emerged, and the benefits they brought surely can’t be ignored.
In this article, we will guide you to leverage AI/ML in software testing to bring your QA game to the next level.
What is AI and ML?
First we need to define the concept of AI/ML properly.
According to the definition from Google Cloud:
- Artificial intelligence (AI) is the use of technologies to build machines and computers that have the ability to mimic cognitive functions associated with human intelligence.
- Machine learning (ML) is a subset of AI that automatically enables a machine or system to learn and improve from experience. Instead of explicit programming, machine learning uses algorithms to analyze large amounts of data, learn from the insights, and then make informed decisions.
What is AI and ML in Software Testing?
With that in mind, AI/ML in software testing is the use of AI/ML technologies to assist software testing activities.
According to the State of Software Quality Report 2025, test case generation is the most common application of AI for both manual testing and automation testing, followed by test data generation. You can download the report for the latest insights in the industry.
There are so many that we can use AI/ML to power-up our software testing, and the key to unlock those capabilities is knowing what these technologies can potentially do, then find creative ways to incorporate them into your day-to-day testing tasks. Note that there are 3 major approaches when choosing an AI/ML system to incorporate into your software testing, including:
- Building your own AI tailored to the needs of the organization, which is a long-term and resource-intensive move
- Leveraging foundation models via APIs or open models
- Using an off-the-shelf tool with built-in AI features for testing
How to use AI/ML in Software Testing?
The final decision on which approach to use depends on the vision of the entire organization and the team. Whichever one you choose, there are generally 5 areas an AI/ML system can contribute the most:
1. Automated Smart Test Case Generation
We all know how disruptive ChatGPT has been to the software engineering industry. Basic coding tasks have now been handed to ChatGPT.
Similarly, in the software testing field, we can also request the ChatGPT to write some test cases for us. Keep in mind - ChatGPT is not the one-size-fit-all solution. Sometimes it works, but sometimes when the requirements get too complex, you need to provide extra context so it can better execute your prompt.
For example, I asked ChatGPT to “write a Selenium unit test to check if clicking on a button on https://exampleweb[dot]com will lead to the correct link “https://exampleweb[dot]com/example-destination”, and ChatGPT provides a fairly well-written code snippet with detailed explanation of each step as well as clear assertion to verify that the destination link is indeed the expected URL. It is actually impressive!
Traditionally, these automated test scripts have to be written by skilled testers with coding skills using test automation frameworks. With AI, this process is sped up, significantly.
Not just that, ongoing maintenance was necessary whenever source code changes occurred, or else the scripts won’t understand the updated code, resulting in wrong test results. This is a common challenge among automation testers since they don’t have enough resources to constantly update their test scripts in the ever-changing Agile testing environment.
Thankfully, when incorporating AI/ML in software testing, you can now use simple language prompts to guide the AI in crafting tests for specific scenarios and speed up your test maintenance work.
2. Test Case Recommendation
Imagine you're running an eCommerce website with thousands of products, different user paths, and constant updates to the website. It is a huge challenge to cover all possible scenarios, even with repetitive areas automatically tested.
This is where Machine Learning comes into play. At first, you need to feed the AI with data about how users interact with your website. This data includes things like what products they view, what actions they take, and when they abandon their carts.
As the AI gathers and analyzes this data over time, it starts noticing patterns. It recognizes that certain products are frequently viewed together, and that users tend to abandon their carts during specific steps in the checkout process.
And that's exactly what TrueTest from Katalon is doing. It leverages AI/ML technologies to map user journeys and identify important scenarios to be tested. After that, it auto-generates automation test cases for those scenarios. This drastically speeds up automation speed.
3. Test data generation
Personally I think this is one of the most helpful use cases of AI. To cover the most scenarios, you would need a gigantic volume of data with high variety.
Manually creating this amount of data is time-consuming, and you can't use consumer data either because of concerns about data privacy.
That's when AI comes in.
Let’s use the eCommerce website example. For global eCommerce websites that ship across borders, generally there are variations in the way shipping fee is calculated, such as:
- Package weight
- Geographic zone
- Distance
- Carrier selection
- Coupons
- Additional charges.
You can use AI to provide you with a list of addresses in regions that your business operates in. If you want to go into the minute details (shipping weight, time zones, additional fees, etc.), just tell the AI and save yourself hours of manual work.
4. Visual Testing
In the past, human testers had to rely on their eyes to find visual differences between how the UI looked before it was launched and how it appeared after it was launched.
The problem arises when we want to automate visual testing, which involves comparing the screenshots of the UI before and after it was launched:
- The computer recognizes extremely minor, pixel-level visual inconsistencies between those screenshots as visual bugs. These so-called “bugs” don’t affect User Experience that much.
- Different computers render images differently, so the images in comparison may technically be different but not visually noticeable
- Certain UI elements (such as the Cart icon on eCommerce websites) change all the time, and test scripts don’t understand that those changes happen for a good reason. They only understand that visual differences mean visual bugs.
AI can learn if certain zones should be “ignored” even if there are changes there, and if the differences it notices between the 2 screenshots are reasonable.
Benefits of using AI/ML in software testing
AI-powered testing is still in its early stages, yet it’s already solving many of the long-standing challenges in traditional automation testing.
As the technology matures, we can expect it to become an essential part of every testing process. Today, AI represents the future, but soon, it will be the standard. Testers who embrace it early will stay ahead of the curve.
Here’s how AI/ML makes a difference:
-
As AI accelerates development, developers write more code at higher speeds, creating more potential for defects. AI testing helps testers match that pace and maintain quality.
-
As applications integrate AI features, they introduce new quality concerns like bias, explainability, and adaptive behavior. AI-based testing provides the intelligence to address these evolving issues.
-
AI speeds up test creation, strengthens test maintenance, and enhances reliability across updates.
-
It offers smarter analytics and recommendations, helping teams make more informed decisions.
-
It streamlines the entire testing process, improving efficiency and reducing human error.
Most importantly, AI doesn’t replace testers; it even empowers them. Think of it as a cognitive extension: an intelligent assistant that works continuously, learns from every test cycle, and helps testers solve problems faster and smarter.
Challenges of AI/ML in Software Testing
Having AI/ML in software testing comes with added responsibilities that we should embrace. Some of the challenges that testers should be aware of include:
- Training Data Quality: AI/ML models require a large and diverse datasets for effective training, so during the early days of AI implementation, its recommendations may not yet be tailored to the organization’s specific needs. However, over time, they get more and more adaptable and familiar with the patterns in the system, leading to better insights.
- Unforeseen Test Cases: AI/ML models might miss certain scenarios if they have not been adequately trained, and we wouldn’t even know that those test cases are missing.
- Overfitting and Underfitting: Overfitting occurs when a model is too specific to the training data and doesn't generalize well, while underfitting occurs when the model is too simple to capture patterns. Ensure to balance these 2 extremes when working with AI by being explicit and detailed about what you want it to accomplish.
- Model Drift: there is an interesting phenomenon in AI development where the model becomes less effective due to changes in the software usage, known as model drift. This requires continuous data monitoring to combat.
Best practices when using AI/ML in software testing
1. Gain a foundational understanding of AI/ML systems as well as your workflows
AI seems like magic (in some ways it is) but the key to using AI is being practical in your approach. Lay a solid foundation of the AI/ML systems you’re working with, understand your own workflows, then find out ways that you can integrate two of them with each other. AI should streamline your workflow and simplify tasks which you previously spent a lot of time on, and you can only do that if you understand both of them correctly.
2. Be patient
AIs take time to develop and learn about the tasks you assign them.
Treat the AI as a blank canvas that you can gradually train to perform complex tasks. It is even better to have a dedicated plan for how you want to integrate AI into your workflow.
You don’t have to take big leaps. Even baby steps that are non-disruptive work also: move at your own pace.
3. Learn prompt engineering
When working with AI and especially generative AI, it is crucial to provide them with well-structured and precise input prompts to generate accurate and relevant outputs from the models. It gives you a certain sense of control over the probabilistic nature of the system.
Prompt engineering is all about bringing context, specifications, and boundaries to the table, and that is a real skill in itself.
4. Remember that it is just a tool
No matter what, AI is only a tool, and it is a truly powerful tool if used in concert with the testers.
Testers will not be replaced by AI, but rather, they should and will be empowered by the technology. The more skilled and experienced the tester is, the more benefits they can extract out of these tools. Bring your own creativity and originality to the table, and let AI propel those ideas to another level.
As Alex Martins perfect puts it:
Testing with AI vs testing for AI systems
Testing using AI and testing for AI systems are 2 completely different topics.
- Testing with AI is using AI models to supercharge your testing
- Testing for AI systems is to test if those models themselves perform as expected. However, the non-deterministic nature of AI means that traditional software testing concepts like “test coverage” obsolete, since it is impractical to measure test coverage when there are theoretically infinite possibilities and scenarios that can happen within a large enough AI system.
Challenges of testing AI systems
- AI/ML models can be highly complex and involve intricate mathematical algorithms.
- There are almost infinite possibilities of results generated by AI, and it is impractical to test all of them
- AI/ML applications are considered "black boxes," meaning their internal workings are complex and hard to interpret. This makes it challenging to determine what inputs are meaningful to test.
- AI/ML models can be susceptible to adversarial inputs—carefully crafted inputs that exploit vulnerabilities and cause unexpected behavior. These inputs might not be captured by traditional coverage criteria.
- AI/ML models can adapt and "learn" from new data, making their behavior evolve over time, and test cases for them must evolve accordingly.
Start Using AI/ML in Software Testing With Katalon
Katalon is a comprehensive AI-augmented software quality management platform, orchestrating the entire software testing lifecycle from test creation, execution, maintenance, to reporting across web, API, and mobile applications.
- TrueTest: Analyzes production behavior then recommends tests to generate. Once you approve, TrueTest autonomously generates tests that you can then execute across a wide range of environments.
- StudioAssist: Leverages ChatGPT to autonomously generate test scripts from a plain language input and quickly explains test scripts for all stakeholders to understand.
- Katalon GPT-powered manual test case generator: Integrates with JIRA, reads the ticket’s description, extracts relevant information about software testing requirements, and outputs a set of comprehensive manual test cases tailored to the described test scenario.
A forerunner in the AI testing landscape, Katalon continuously enriches its product portfolio with these innovative AI-infused capabilities, empowering global QA teams with unprecedented accuracy and operational efficiency. Learn more about Katalon AI testing here.
|
