What is automation testing?

what is automation testing
Key takeaways
The 30-second version of this article
Automation testing uses software to run tests and compare actual results with expected ones, so a person doesn't have to click through each step.
A test is a short list of steps and checks: open the app, act on it, and verify what you expect to see.
Automate tests that run often, are stable, matter to the business, or need many data sets. Keep exploratory, usability, and one-off tests manual.
Start small: automate a handful of high-value tests, run them in CI/CD, measure the time saved, then scale.
Automation has real costs: upfront effort, ongoing maintenance, and flaky tests. It only checks what you tell it to check.

Automation testing is the practice of using software to run tests on an application and compare the actual results with the expected ones, without a person carrying out each step. A tester writes or records the test once, and it can then run on every build, in parallel, and on demand.

You'll also see it called automated testing, test automation, automated software testing, software testing automation, or QA automation. They're used to mean the same thing.

What is Automation Testing_ Automation testing defnition, best practices, and how-to guide

How automation testing works

Most automated tests do three things: they put the application in a known state, act on it the way a user would, and check the result against what you expect. If the check matches, the test passes. If it doesn't, the test fails and tells you which step broke.

Here's a real test that does that. It opens a to-do app, adds an item, and checks that the item is listed and the counter updates.

We use the TodoMVC demo app, which the Playwright team maintains for demos and examples. It needs no sign-in and no test account, so you can copy the test below and run it yourself in a couple of minutes.

The test in code (Playwright)

const { test, expect } = require('@playwright/test');

test('adds a to-do item to the list', async ({ page }) => {
  // 1. Open the app
  await page.goto('https://demo.playwright.dev/todomvc/');

  // 2. Type a new to-do and press Enter
  const newTodo = page.getByPlaceholder('What needs to be done?');
  await newTodo.fill('Write the regression suite');
  await newTodo.press('Enter');

  // 3. Check the item appears and the counter updates
  await expect(page.getByTestId('todo-title')).toHaveText(['Write the regression suite']);
  await expect(page.getByTestId('todo-count')).toHaveText('1 item left');
});

What each step does:

  1. Open the app. Playwright gives each test a fresh browser context, and page.goto loads the TodoMVC page in it, so every run starts from the same empty list.
  2. Act like a user. The test finds the input by its placeholder text, types "Write the regression suite", and presses Enter, just as a person would.
  3. Check the result. The two expect lines are the assertions. The first checks that the list holds exactly one item with that text. The second checks that the counter reads "1 item left". If either check fails, the test fails.

To run it, create an empty folder, save the file as tests/add-todo.spec.js, then run npm install -D @playwright/test, npx playwright install chromium, and npx playwright test.

The test passed in under 2 seconds:

Terminal output of npx playwright test showing 1 test passed: adds a to-do item to the list, in 1.8 seconds
The test passed in under 2 seconds:

Run npx playwright test --reporter=html and Playwright also writes an HTML report, which you open with npx playwright show-report. This is what you'd share with your team or attach to a pull request:

Playwright HTML report showing add-todo.spec.js with 1 passed, 0 failed, 0 flaky, and 0 skipped tests
...Playwright also writes an HTML report... This is what you’d share with your team or attach to a pull request:

This is what the app looks like at the end of the run, with the item added and the counter updated:

TodoMVC demo app with the to-do item "Write the regression suite" listed and the counter showing 1 item left
This is what the app looks like at the end of the run, with the item added and the counter updated:

The same test in Katalon Studio

Not every tester writes JavaScript, and not every team wants to. Here's the same test built in Katalon Studio. It opens the same app, adds the same item, and checks the same two things: the item's text and the counter.

In the Script view, the test is Groovy code:

import static com.kms.katalon.core.testobject.ObjectRepository.findTestObject

import org.openqa.selenium.Keys as Keys

import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI

// 1. Open the app
WebUI.openBrowser('')

WebUI.navigateToUrl('https://demo.playwright.dev/todomvc/')

// 2. Type a new to-do and press Enter
WebUI.setText(findTestObject('Page_TodoMVC/input_new-todo'), 'Write the regression suite')

WebUI.sendKeys(findTestObject('Page_TodoMVC/input_new-todo'), Keys.chord(Keys.ENTER))

// 3. Check the item appears and the counter updates
WebUI.verifyElementText(findTestObject('Page_TodoMVC/label_todo-title'), 'Write the regression suite')

WebUI.verifyElementText(findTestObject('Page_TodoMVC/span_todo-count'), '1 item left')

WebUI.closeBrowser()
Katalon Studio Script view of the Add a to-do item test case, with the Groovy steps and the passed run in the Log Viewer
In the Script view, the test is Groovy code:

The input, the to-do title, and the counter are saved as test objects in the Object Repository, and each step refers to them by name. To organize a full suite this way, see our guide to building a test automation framework.

Switch to the Manual view and the same test reads as 7 plain-language steps, with no code: Open Browser, Navigate To Url, Set Text, Send Keys, Verify Element Text (twice), and Close Browser. Each step shows the test object it acts on, where there is one, and its input value. A tester who doesn't code can read it, edit it, or add a step from a list of keywords. Both views show the same test, so testers who code and testers who don't can work on it in one place.

Katalon Studio Manual view of the Add a to-do item test case showing 7 steps, with the Log Viewer reporting 7 of 7 steps passed
Switch to the Manual view and the same test reads as 7 plain-language steps...

All 7 steps passed on Chrome.

Which one should you pick?

Both tests do the same job. The difference is who writes and maintains them.

  • Pick a code framework like Playwright if your testers are comfortable in JavaScript or TypeScript, you want tests to live in the same repository as the app code, and you have the time and skills to set up test management, dashboards, test data, and CI/CD yourself.
  • Pick Katalon Studio if your team has mixed skills. Studio goes from record-and-playback through to full scripting in the same tool, so testers who don't code and engineers who do can work on the same tests. It covers web, mobile, API, and desktop testing in one product, runs tests locally, and can hand them off to Katalon Runtime Engine (KRE) or Katalon Test Cloud when you need to run at scale. It also works alongside the Selenium and Playwright ecosystems, so it can sit next to tests your team already has.

To build this test yourself in Studio, Start free trial.

What to automate and what to keep manual

Automation isn't about testing everything. Every test you automate should save time, reduce risk, or catch bugs sooner than running it by hand. If it doesn't, it may cost more than it returns. Maintenance is a big part of that cost, which is why some teams use self-healing test automation to handle locator changes.

Good candidates for automation:

  • Tests you run often: regression, smoke, and sanity tests that run every sprint, every build, or every pull request.
  • Stable, predictable flows: features that don't change every week and give the same result every time.
  • Business-critical paths: login, checkout, payments, and API contracts you must know work before each release.
  • Data-heavy tests: the same steps run with many input sets, where you reuse the logic and change only the data.
  • Tests that take too long by hand: tedious flows that eat hours each release cycle.
  • Tests across many environments: the same check on several browsers, devices, or operating systems.

Keep these manual, at least for now:

  • One-time or rarely run tests, where building the automation costs more than running them by hand.
  • Exploratory testing, usability, and UX checks that need a person's judgment.
  • Features or UI elements that are still changing quickly.
  • Tests that need physical devices or hardware you can't control from a script.

Most teams end up with a mix: automation for the repetitive checks, and testers' time for the exploratory work that needs a person's judgment. For a side-by-side comparison, see manual vs automation testing. For a scoring method, see how to select test cases for automation.

Types of automation testing

Automation is an approach, not a test type, so most kinds of tests can be automated. These are the ones teams automate most often:

  • Unit tests: check individual functions or components in isolation. They're fast and make up most of a healthy suite.
  • API tests: send requests straight to an endpoint and check the response, with no UI in between.
  • Smoke tests: quick checks that the core features work before deeper testing starts.
  • Regression tests: rerun after every code change to make sure existing features still work.
  • Data-driven tests: run the same steps with many data sets.
  • Cross-browser tests: check that the app works on the browsers, devices, and operating systems your users have.
  • UI tests: drive the interface the way a user does, like the to-do test above.

The test pyramid is a useful guide to the balance: many unit tests, fewer API tests, and a small set of end-to-end UI tests for the critical journeys. For a fuller breakdown of each, see types of automation testing.

Code frameworks vs ready-to-use software

A framework is a code library that gives you the structure for writing and running tests: how you find elements, make assertions, organize test files, and report results. Playwright, used in the example above, is a framework. You get full control, but your team writes and maintains the code, and builds whatever the framework doesn't include, such as test management, dashboards, or a way for non-coders to contribute.

A tool is ready-to-use software that wraps a framework and adds features around it, such as recorders, keyword-based editors, built-in reports, and CI/CD integrations. Katalon Studio, shown above, is a tool. The right choice depends on your team's coding skills, the platforms you test, and how much setup and maintenance you can take on. For a comparison, see our guide to automation testing tools.

How to start automation testing, step by step

Step 1. Pick a few tests to automate first

Start with 5 to 10 tests that run often and break things when they fail: a smoke test of the login flow, a checkout path, a key API call. Starting small lets you measure the effort against the time saved before you scale up. Lists like test cases for login page and test cases for API testing are a good place to look for ideas.

Step 2. Choose your approach

Decide whether you'll build on a code framework or use a tool, based on who will write and maintain the tests. If most of your team doesn't code, look at low-code testing tools. If you test Windows or macOS apps, look at desktop testing tools.

Step 3. Write or record your first tests

Keep each test short and focused on one behavior, with clear assertions, like the to-do test above. Each test should set up its own data and run on its own, without relying on another test. Independent tests are easier to debug and safe to run in parallel.

For more on structuring each one, see our guide to writing automation test cases.

Step 4. Prepare the test environment

List what the tests need to run on: browsers, devices, operating systems, test data, and any services the app depends on. You can run tests on your own machine, on remote machines or a grid you manage, or on a cloud service. Use predictable test data, and mock third-party services you don't control, so tests fail only for reasons that matter.

Step 5. Run the tests and read the results

Run the automated test suite and review every failure. Each one is either a real bug, a test that needs updating, or an environment problem. A good report tells you which, quickly.

Katalon vs Selenium smart test reporting built-in

For what a useful report contains, see how to build a good test report.

Step 6. Add the tests to your CI/CD pipeline

Run the suite automatically on every pull request or build, so the team gets feedback within minutes of a change instead of at the end of the sprint. This is where much of the time saving comes from, because the checks run without anyone starting them.

Step 7. Measure results and optimize

Running the tests isn't the last step. Track a few numbers to see whether automation is paying off:

  • Execution time: how long a full run takes, and whether it's creeping up.
  • Maintenance effort: how many hours each sprint go into fixing broken tests.
  • Defect leakage: bugs that reach production even though the tests passed.
  • Coverage of critical flows: how much of what matters your suite actually exercises.

Review these regularly, retire tests that no longer earn their place, and fix the slow or brittle ones first. To put a number on the return, see how to calculate test automation ROI. As the suite grows, these test automation best practices help keep it fast and trustworthy.

banner5.png

Benefits and limits of automation testing

What automation gives you:

  • Speed: automated tests run in parallel and overnight. A regression cycle that takes a tester hours or days by hand can run as a pipeline job while the team works on something else.
  • Consistency: a test runs the same steps the same way every time, so results are easier to compare and reproduce.
  • Faster feedback: tests that run on every change catch regressions while the change is still fresh in the developer's mind.
  • Wider coverage: the same test can run across many browsers, devices, and data sets without extra manual effort.
  • Better use of testers' time: repetitive checks move to the machine, and testers spend more time on exploratory work and harder scenarios.

For more on each, see benefits of automation testing.

What it costs, and what it can't do:

  • Upfront effort: someone has to design, write, and review the tests, and set up the environment and pipeline. The return comes later, from every run after that.
  • Maintenance: tests break when the application changes, and someone has to update them every release. A suite that grows without care becomes a second product to maintain.
  • Flaky tests: a test that passes and fails at random teaches the team to ignore failures. Fix or quarantine flaky tests as soon as they appear.
  • It only checks what you tell it to: a test that checks the counter won't notice a broken layout, confusing copy, or a slow page. Usability, accessibility judgment, and the bugs nobody thought to look for still need a person.
  • Skills: writing maintainable tests is its own skill, and teams need time to build it.

That last point shows up in our own research. In Katalon's State of Software Quality Report 2025, a survey of 1,500+ QA professionals, 45% named the team's lack of skills and experience in test automation as one of the biggest challenges to automation testing, 38% said requirements change too often, and 26% said test maintenance is costly.

challenges of applying automation testing

None of these is a reason not to automate. They're reasons to start small, automate the right tests, and budget time for maintenance from day one.

Writing these choices into a test strategy keeps the whole team working from the same plan.

Frequently asked questions

Henrik (Minh Huy) Dao

Minh Huy Dao (Henrik) is a Senior Quality Engineer at Katalon, where he has worked since 2023. He has nearly 10 years of experience in QA and test automation, including roles at TMA Solutions, Axon Active and Doctor Anywhere. His work spans automation with Java, JavaScript and WebdriverIO, as well as CI/CD with Jenkins and leading Scrum teams.

Connect on LinkedIn →
Get started

Test everything. Ship with confidence.

From generating test cases to spotting failures, Katalon's AI works across your entire pipeline, automatically.

The Next Evolution of Manual Testing: Introducing Katalon AI Test Runner

Introducing Katalon AI Test Runner. Watch it run manual test cases live, with evidence for every pass and fail.

Live webinar
9:00 to 10:00 AM EST · Live online
Alex MartinsVP of Strategy
Gia EspositoSenior Solutions Engineer
Save your seat