Introducing Katalon AI Test Runner: Run Your Manual Test Cases With AI in Katalon TestOps

insights-ai-test-runner

Katalon AI Test Runner runs your existing manual test cases as-is in Katalon TestOps, verifies every step with a separate AI, and keeps evidence for each verdict.

Key takeaways
The 30-second version of this article
Katalon AI Test Runner, in Katalon TestOps, runs your manual test cases as-is, with no script to write and no framework to build first.
Every run follows five steps: choose tests, check readiness, run at scale, verify every step, and review results with proof.
A separate AI judges each step from captured test evidence such as screenshots and actions, so the AI doing the work never approves its own result.
Each step is marked passed or failed, with video, screenshots and an AI verdict kept as evidence.
Testers keep the judgment work: deciding what to test, reviewing flagged results, and making the release call.

Manual test cases hold a lot of what your team knows about the product: the flows that matter, the data that breaks things, the checks that catch regressions. Running them by hand every release is where the hours go, and it is the work that gets squeezed first when a release date moves.

Katalon AI Test Runner is now available in Katalon TestOps as Run with AI, bringing agentic test execution to your manual test library. It reads your natural-language manual test cases, runs them against your application, checks each step, and records results and supporting evidence. Your testers keep the work that requires experience, judgment and subject-matter expertise.

What Katalon AI Test Runner Does

Katalon AI Test Runner is an agent that executes natural-language test cases against your application. It follows each step in a cloud browser on Katalon TestCloud, compares what happens with the expected result, and records the outcome. It is literally running your existing manual test cases, in natural language, with no script to write and no framework to build first.

That is a different job from most AI for manual testing. Many AI tools for manual testing suggest scenarios or help draft test cases. Katalon AI Test Runner takes the test cases your team already wrote and runs them. For how agents fit into a QA team more broadly, see What Is Agentic QA?

What does not change is who owns the judgment. Deciding what to test, how serious a failure is, and whether a build is ready all stay with your testers. The runner takes over the repetitive part: following the same steps, release after release.

How a Run Works

Every run follows the same five steps.

  1. Choose the tests. Start Run with AI from a test case, a test suite or a manual test run. You can also schedule a run, or start one from Katalon AI Assistant or from an AI assistant connected through the Katalon MCP server.
  2. Check readiness. Before the run, the AI reviews each test case for clarity and completeness and flags gaps. A tester provides clarifications to close any gaps once, instead of working around them every cycle.
  3. Run at scale. Independent test cases run in parallel in the cloud, and a data-driven test case runs one session per dataset row. Repeat runs do not use AI; they are fully deterministic. If a step fails because a screen changed, the runner hands that step to the AI, which adapts the execution at runtime and updates the test case accordingly.
  4. Verify every step. A separate, read-only AI judges each step from screenshots and actions, and fixed rules decide the verdict. If a step is unclear, the runner retries it and then hands it to a person instead of passing it. The AI doing the work never approves its own result, which is the argument in The Agent Passed Its Own Test. That's the Problem.
  5. Review the results with proof. Every step leaves a verdict and evidence, so a reviewer can open the proof instead of trusting a status.

Pass or Fail, With Evidence for Every Step

Before it reports a pass or fail, the AI checks the evidence for each step. Each step is marked passed or failed. Failed steps carry an AI verdict: the AI’s assessment of what went wrong, the expected result next to the actual result, and a suggested next action. When the cause is a real product bug, the AI verdict says so, instead of leaving you to work out whether the test data, the environment or the application caused the failure.

Katalon AI Test Runner screenshot

A failed step in an AI Session. The AI Verdict panel shows the expected and actual result, a Product bug label, the root cause, a recommended action and a Create defect button.

Every result comes with evidence that testers, leaders and auditors can review: video, screenshots and the AI log for each step, kept as a recorded execution history. From a failed result you can create a defect in Jira or Azure DevOps with the steps to reproduce and a link back to the result.

For anything that matters to a release, review the evidence before you trust the result. A green status is a claim, and the evidence is the proof, which is the point of Katalon AI Test Execution and False Passes.

Where It Runs and What It Connects To

Runs execute on Katalon TestCloud, against web applications in Chrome or Edge on Linux, Windows or macOS, and against mobile applications. For an application on a private network, such as a staging environment behind a firewall, choose Katalon Tunnel or IP allowlisting when you start the run.

Credentials belong in Secrets & Variables in Katalon TestOps. Logins that use one-time passcodes work when the code is sent by email to a configured test inbox, and the codes are hidden from results and logs.

Results land in Katalon TestOps test runs and reports. From there you can push results to Azure DevOps Test Plans or Xray, and create defects in Jira or Azure DevOps with the evidence attached.

Who Katalon AI Test Runner Is For

Katalon AI Test Runner is built for teams that run regression by hand every release and have a manual QA team with deep business and application knowledge. It gives those teams the ability to run their existing manual tests in a fully automated fashion, without hiring automation engineers and without asking testers to learn a new framework or scripting language. It is manual testing automation that keeps the test cases where they are, as your team already wrote them.

Running the steps stops being the job, and reviewing the results the AI flagged becomes the job. Testers read the flagged results, investigate failures, and spend the freed time on exploratory testing, risk analysis and change impact analysis. For that shift, read Manual Testers Are Getting Promoted. For those who are tired of trying to automate tests and never reaching the test automation coverage they wanted, read Your Manual Tests Are Already Automation.

How to Get Started

You need three things: Katalon TestOps, manual test cases written as steps with expected results, and an application that Katalon TestCloud can reach. Start with one test case that a new teammate could follow without asking questions. Put the application URL in the steps, keep logins in Secrets & Variables, and start the run from the test case, a test suite or a manual test run.

Watch the first run. The moments when the runner pauses to ask for input are the most useful part, because each question points to something the test case does not say. Add the answer to the step, and the readiness check will have less to flag next time.

For a step-by-step walkthrough, watch the Katalon AI Test Runner how-to series on YouTube: creating test cases from requirements, running them with AI in parallel, and reading the results, reports and analytics.

Frequently asked questions

Alex Martins
Alex MartinsVice President, AI Transformation

Over the years, Alex has had the opportunity to learn from and work with highly talented people at some of the largest software and services companies in the world where, together, they transformed the way software was developed, tested and released for countless Fortune 500 companies.

Connect on LinkedIn →
True Platform has arrived

Test everything. Ship with confidence.

From generating test cases to spotting failures, Katalon's AI works across your entire pipeline, automatically.

The Next Evolution of Manual Testing: Introducing Katalon AI Test Runner

Introducing Katalon AI Test Runner. Watch it run manual test cases live, with evidence for every pass and fail.

Live webinar
9:00 to 10:00 AM EST · Live online
Alex MartinsVP of Strategy
Gia EspositoSenior Solutions Engineer
Save your seat