Test Execution: A Simple (But Complete) Guide

To execute locally, remote, or on-cloud? That is the question.

Test execution is more than just “hitting Enter and seeing if it passed or not". A lot of questions must be asked before test execution begins:

• What is the purpose of this test?
• What are the entry criteria?
• What test data will be used?
• Which tests should run and in what order?
• What are the expected outcomes?
• How will bugs be logged and tracked?
• How will test execution be monitored and reported?
• What are the exit criteria?
• Is automation involved?

In this article, I'm going to help you answer all of those questions. This article is a simple but complete guide on all things you need to run tests efficiently, cost-effectively, and smartly.

So, let's dive in!

What is test execution?

Test execution is the process of systematically running predefined test scripts to validate that a specific feature’s code behaves as expected. It is the core of testing itself, since without test execution on a certain feature, how can you know if that feature works or not?

If you look at the traditional software testing life cycle, test execution is the 5th step, following test case development and environment setup.

Methods to execute a test case

There are two primary ways in which a test case can be triggered and executed:

1. Manual execution: As the name suggests, you simply execute the test cases step by step as outlined in the test plan. This approach is suitable for exploratory testing, ad hoc cases, or scenarios where automation testing is not feasible due to its complexity. But, if the test case is a repetitive one (meaning that it is run again and again in each cycle) you should consider automation testing.
2. Automated execution: With this method automation scripts or tools are leveraged to execute test cases automatically. All you have to do is press the “Run” button and let the script do the rest.

Manual test execution

Manual test execution is all about perform the test steps exactly how an end user would.

For example, if you're testing a Login page, here are the steps that you are going to (manually) execute:

1. Navigate to the URL of the Login page
2. Click on Username field
3. Type the username into the Username field
4. Click on Password field
5. Type the password into the Password field
6. Click Login
7. Observe the state of the Login page to see if the Login is successful or not

Once done, you record the test result (pass/fail) in a test management system (usually a simple Google Sheet). If the test fails, you notify the dev team about it for them to start troubleshooting.

Automated test execution techniques

If you choose the automation approach, that means you are using a test automation framework or a test automation tool to help you automate all of the actions that you would have otherwise done manually.

Automated testing opens up a wide range of possibilities for test execution that can make your life a lot easier. There are generally three approaches to automated test execution:

1. Batch execution: With this approach, you can group multiple test cases together and execute simultaneously, which can save a lot of time and resources. These groups of test cases are called “test suites.” Test cases under a test suite usually have some similarities.
2. Scheduled execution: This can be done once you have a full test case management system in place. You can schedule specific tests to run at a predefined time. This is particularly useful for regression testing.
3. CI execution: Test cases can also be configured to automatically run as a part of the CI/CD pipeline. As soon as a new build is generated, it triggers the test execution to check the new build for any new bugs immediately, minimizing human intervention.
   
   

📚 Read More: How to build an automation test suite?

Where to execute a test case?

You can execute test cases locally, remotely, or on-cloud. Each comes with their own advantages and disadvantages:

1. Local execution

Local execution is when test cases are run directly on a tester's or developer's machine. Let's say that you have a Selenium script on your machine to automate the Login page test, so you open the Terminal and run it using Chrome (installed on your laptop).

That means all processing happens on your machine. The test was executed on your CPU, RAM, and browser.

Advantages:

• Fast feedback for small-scale tests.
• No additional infrastructure setup is required.
• Useful for debugging and quick validation.

Challenges:

• Limited to the tester’s system capabilities (hardware, OS, etc.).
• May not reflect real-world scenarios (different environments, network conditions).

Best for:

• Unit tests or small test suites.
• Quick verification during development.

2. Remote execution

Remote execution runs test cases on a dedicated server or machine that may have different configurations from the tester's local system.

A good example of remote execution is that when you connect to a QA server in your company network via SSH:

ssh qauser@qa-server.company.com

Then, you're going to run the automation script on that server. This server has a configured browser instance (or headless browser like Chrome Headless) and test framework installed, so that you have a test environment closer to production.

Advantages:

• Simulates different environments (OS, browser, devices).
• Useful for testing across multiple environments.

Challenges:

• Requires infrastructure setup and maintenance.
• Can be slower due to network latency.

Best for:

• Integration tests.
• Cross-browser and cross-platform testing.

3. Cloud-based execution

Cloud execution involves running test cases on cloud platforms provided by third-party vendors. For example, you can go with Katalon TestCloud, a cloud-based test automation solution, where you can create tests and execute them on the cloud environment of your choice.

In this image, I am running Katalon tests on Chrome 124 in Windows:

Advantages:

• Access to a wide variety of environments, devices, and browsers without setup.
• Scalable for large test suites.
• Continuous availability with minimal maintenance.

Challenges:

• Subscription costs.
• Security concerns for sensitive data.

Best for:

• Large-scale automated test suites.
• Cross-device compatibility testing.
• Continuous Integration/Continuous Deployment (CI/CD) pipelines.

--> Learn more about on-cloud test execution with Katalon TestCloud

Test execution states

An important concept related to test execution is “assertion.”
 

Simply put, an assertion is the condition for the expected behavior of the application under test. It is a checkpoint to compare if the actual behavior matches the expected behavior.
 

There are several types of assertion:

1. Equality assertion (check if actual_value equals/matches expected_value)
2. Boolean assertion (check if the value is true or false)
3. Existence assertion (check if the element exists)
4. Comparison assertion (check if the actual_value is higher/lower/equal to a certain value)

These assertions are usually embedded within the test script and define the pass/fail conditions of a test.

A test case goes through the following status markers in an execution process:

1. Untested: At this stage, the test has not been executed yet.
2. Blocked/on hold: This status is for test cases that can’t be executed due to dependencies (unresolved defects, unavailable test data, system unavailability, necessary components are not yet developed, etc.).
3. Failed: This status is when the actual outcome does not match the expected outcome. In other words, the assertion conditions were not met. This is when the team launches further investigation to identify the root cause.
4. Passed: This is when the test case is executed successfully, and the actual outcome matches the expected outcome. Testers love to have a lot of passed test cases, since it is a signal for software quality.
5. Skipped: Sometimes a test case can be skipped if it is not relevant to the current test scenario being tested. The reason to skip the test is usually documented.
6. Deprecated: This occurs when the test case is no longer valid due to updates in the application. The test case can be totally removed or archived.
   
   

📚 Read More: The Life Cycle of a Bug in Software Testing

Activities of test execution

There are 3 stages of test execution:

1. Preparation
2. Execution
3. Evaluation

Let’s see what happens in each stage:

1. Preparation

At this stage, QA teams set up the necessary hardware-software configuration for later execution. Ask yourself:

• Have all test cases been developed and reviewed?
• Is the test environment set up correctly?
• Is all the test data identified and prepared
• What is the timeline for testing activities?
• How will the test results be monitored?

When it comes to setting up testing environment, there are 2 options:

1. Invest in real physical devices. This provides the highest level of realism, and is especially helpful for device-specific testing, such as battery testing or performance testing. However, this approach can be extremely costly, considering the number of devices QA teams have to take into account to ensure test coverage.
2. Start testing on cloud. There is a wide range of cloud-hosted testing environments that is available on-demand for QA teams. These solutions allow QA teams to test on any browser, device, OS, and scale along the way, without the need to continuously maintain the infrastructure.
   
   

Read More: Emulator vs Simulator vs Real Device Testing: Which is Better For Mobile Testing?

On a similar note, preparing the right test data is also important to ensure that test execution runs smoothly. Test data preparation goes hand-in-hand with test data management (TDM). The basis of TDM is about generating the right type of data to facilitate test execution, while managing and anonymizing them to comply with data privacy regulations.

2. Execution of Test Cases

After that comes the execution phase.
 

You can execute a test case, test suite, dynamic test suite, and test suite collection in Katalon Studio. For test suite collections, you can choose to execute associated test suites in sequential mode or in parallel mode.

To get started, download Katalon Studio and start executing tests with ease.

In Katalon Studio, you have access to a rich array of test environments to carry out your tests:
 

Once you have downloaded and installed Katalon Studio, start by creating a test case:

You have up to 3 modes to write a test:

1. No code: turn on the Record-and-Playback feature and start testing manually. Katalon turns all of your manual actions into an automated test script that can be re-executed across environments.
2. Low code: you also have access to a wide range of keywords in the keyword library. These are essentially code snippets for automating actions. Combine these keywords together for the steps you want to automate and you should have a full test script in no time. The only code you have to do is setting the parameters for these keywords. 
3. Full code: want a lot of flexibility? Switch to the Scripting mode and code away. Katalon supports coding in Groovy (Java). Switch back to the No-code and Low-code mode whenever you want and get the best of both worlds.

Once you’ve created a test case, simply click the “Run” button, choose your desired environment, and the test cases are executed.

3. Post-execution evaluation

Finally comes the evaluation part. This is the final stage of the testing project, and QA teams usually create a test report to summarize what they found.
 

Test results are then automatically pushed to Katalon TestOps where they can later be turned into rich reports with detailed analytics to help QA teams make informed decisions on their testing activities. 

Generally there are 4 important items to cover in a test report:

1. Visualizations (charts, graphs, diagrams, etc.) to demonstrate patterns in testing activities
2. Monitoring metrics
3. Performance metrics
4. Comparative analysis (compare results across versions/time periods)
5. Recommendations (overall insights from a tester’s perspective as to what area(s) should be focused on during the debugging process)
    

Read more: How to Build a Good Test Report?

How to prepare test data for execution?

The goal of test data preparation is to provide the inputs your test cases need so they can be executed realistically, consistently, and in a way that actually validates the software's behavior.

Without the right data, even perfectly written test scripts will either fail incorrectly or give meaningless results.

Good test data starts from understanding what tests you are going to execute. For example, if you're going to test a Login page, here is the test data you'll need to prepare:

• Valid login: username=testuser, password=Password123
• Invalid password: username=testuser, password=wrongpass
• Unregistered email: username=ghost@domain.com, password=anything

Best practices for test execution

1. Before testing, make sure to develop a comprehensive test plan that outlines the objectives and resources for test execution.
2. Test data goes hand in hand with smooth execution. Prepare the test data meticulously, covering all relevant scenarios. Also make sure to have effective test data management in place.
3. Have prioritization criteria and assign execution order for your test cases. For example, if test case B is dependent on test case A, it is a good idea to place test case A in higher priority.
4. Record test results in a spreadsheet or a dedicated test case management system. Document the activity thoroughly. Include test logs or screenshots if necessary to provide the dev team with as much information as possible to effectively troubleshoot.
5. Apply regression testing to check if new changes introduce new bugs to existing features or not.

Upgrade your automation today.