Test Execution: How To Do It Properly

To execute locally, remote, or on-cloud? That is the question. But test execution is more than just “hitting Enter and seeing if it passed or not.” If you are doing it manually, test execution is about performing every single test step as outlined in the test case. For automation testing, test execution takes on another layer of complexity: In which environment should you run the test? Which test cases should you prioritize for execution? 
 

In this article, we’ll explore the best practices of test execution. We will also review automation testing tools with features that support better execution of test cases.

What is Test Execution?

Test execution is the process of executing the test scripts written to check the code of a specific feature. It is the fifth step in the software testing life cycle, following test case development and environment setup.
 

To be more specific, during the third step of test case development, testers (or sometimes the developers themselves) write a script in their preferred language and test automation framework. After that, they need to configure the test environment in a way that most realistically simulates the production environment. This environment is where test execution happens.
 

Mock data is also prepared during these steps. Certain test scenarios require mock test data to comply with data privacy/security requirements. For example, if you are testing the login page, you should not use real user credentials to run the tests. Instead, QA teams usually have a synthetic dataset that can be fed into the test cases. Once everything is prepared, testers can simply run those scripts and see the results.

Methods To Execute a Test Case

There are two primary ways in which a test case can be triggered and executed:
 

1. Manual execution: As the name suggests, you simply execute the test cases step by step as outlined in the test plan. This approach is suitable for exploratory testing, ad hoc cases, or scenarios where automation testing is not feasible due to its complexity. If it is a repetitive test case (meaning that it is run again and again in each cycle) you should consider automation testing.
    
2. Automated execution: With this method automation scripts or tools are leveraged to execute test cases automatically. All you have to do is press the “Run” button and let the script do the rest. Check out this list of the top automation testing tools on the market.

Read More: How To Transition From Manual Testing To Automation Testing (The Easy Way)?

If you choose the automation approach, there are quite a lot of things you can do to make life easier:

1. Batch execution: With this approach, you can group multiple test cases together and execute simultaneously, which saves a lot of time and resources. Groups of test cases are called “test suites.” Test cases under a test suite usually have some similarities.
2. Scheduled execution: This can be done once you have a full test case management system in place. You can schedule specific tests to run at a predefined time. This is particularly useful for regression testing.
3. CI execution: Test cases can also be configured to automatically run as a part of the CI/CD pipeline. As soon as a new build is generated, it triggers the test execution to check the new build for any new bugs immediately, minimizing human intervention.

Test Execution States

An important concept related to test execution is “assertion.”
 

Simply put, an assertion is the condition for the expected behavior of the application under test. It is a checkpoint to compare if the actual behavior matches the expected behavior.
 

There are several types of assertion:

1. Equality Assertion (check if actual_value equals/matches expected_value)
2. Boolean Assertion (check if the value is true or false)
3. Existence Assertion (check if the element exists)
4. Comparison Assertion (check if the actual_value is higher/lower/equal to a certain value)

These assertions are usually embedded within the test script and define the pass/fail conditions of a test.
 

A test case goes through the following status markers in an execution process:

1. Untested: At this stage, the test has not been executed yet.
2. Blocked/On hold: This status is for test cases that can’t be executed due to dependencies (unresolved defects, unavailable test data, system unavailability, necessary components are not yet developed, etc.).
3. Failed: This status is when the actual outcome does not match the expected outcome. In other words, the assertion conditions were not met. This is when the team launches further investigation to identify the root cause.
4. Passed: This is when the test case is executed successfully, and the actual outcome matches the expected outcome. Testers love to have a lot of passed test cases, since it is a signal for software quality.
5. Skipped: Sometimes a test case can be skipped if it is not relevant to the current test scenario being tested. The reason to skip the test is usually documented.
6. Deprecated: This occurs when the test case is no longer valid due to updates in the application. The test case can be totally removed or archived.

Activities of Test Execution

There are 3 stages of test execution:

1. Preparation
2. Execution
3. Evaluation 

Let’s see what happens in each stage:

1. Preparation

At this stage, QA teams set up the necessary hardware-software configuration for later execution. Ask yourself:

• Have all test cases been developed and reviewed?
• Is the test environment set up correctly?
• Is all the test data identified and prepared
• What is the timeline for testing activities?
• How will the test results be monitored?
   

When it comes to setting up testing environment, there are 2 options:

1. Invest in real physical devices. This provides the highest level of realism, and is especially helpful for device-specific testing, such as battery testing or performance testing. However, this approach can be extremely costly, considering the number of devices QA teams have to take into account to ensure test coverage.
2. Start testing on cloud. There is a wide range of cloud-hosted testing environments that is available on-demand for QA teams. These solutions allow QA teams to test on any browser, device, OS, and scale along the way, without the need to continuously maintain the infrastructure.

Read More: Emulator vs Simulator vs Real Device Testing: Which is Better For Mobile Testing?
 

On a similar note, preparing the right test data is also important to ensure that test execution runs smoothly. Test data preparation goes hand-in-hand with test data management (TDM). The basis of TDM is about generating the right type of data to facilitate test execution, while managing and anonymizing them to comply with data privacy regulations.
 

Read More: A Comprehensive Guide To Doing Test Data Management
 

2. Execution Test Cases With Katalon Studio

After that comes the execution phase.
 

You can execute a test case, test suite, dynamic test suite, and test suite collection in Katalon Studio. For test suite collections, you can choose to execute associated test suites in sequential mode or in parallel mode.

Let's start by downloading Katalon:

Download Katalon and Start Executing Tests With Ease

 

In Katalon Studio, you have access to a rich array of test environments to carry out your tests:
 

Once you have downloaded and installed Katalon Studio, start by creating a test case:

You have up to 3 modes to write a test:

1. No-code: turn on the Record-and-Playback feature and start testing manually. Katalon turns all of your manual actions into an automated test script that can be re-executed across environments.
2. Low-code: you also have access to a wide range of keywords in the keyword library. These are essentially code snippets for automating actions. Combine these keywords together for the steps you want to automate and you should have a full test script in no time. The only code you have to do is setting the parameters for these keywords. 
3. Full-code: want a lot of flexibility? Switch to the Scripting mode and code away. Katalon supports coding in Groovy (Java). Switch back to the No-code and Low-code mode whenever you want and get the best of both worlds.

Once you’ve created a test case, simply click the “Run” button, choose your desired environment, and the test cases are executed.
 

Interested? Here's a demo for you:

3. Evaluation

Finally comes the evaluation part. This is the final stage of the testing project, and QA teams usually create a test report to summarize what they found.
 

Test results are then automatically pushed to Katalon TestOps where they can later be turned into rich reports with detailed analytics to help QA teams make informed decisions on their testing activities. 

Generally there are 4 important items to cover in a test report:

1. Visualizations (charts, graphs, diagrams, etc.) to demonstrate patterns in testing activities
2. Monitoring metrics
3. Performance metrics
4. Comparative analysis (compare results across versions/time periods)
5. Recommendations (overall insights from a tester’s perspective as to what area(s) should be focused on during the debugging process)
    

Read More: How To Build A Good Test Report?

Best Practices for Test Execution

1. Before testing, make sure to develop a comprehensive test plan that outlines the objectives and resources for test execution.
2. Test data goes hand-in-hand with smooth execution. Prepare the test data meticulously, covering all relevant scenarios. Also make sure to have effective test data management in place.
3. Have prioritization criteria and assign execution order for your test cases. For example, if test case B is dependent on test case A, it is a good idea to place test case A in higher priority.
4. Record test results in a spreadsheet or a dedicated test case management system. Document the activity thoroughly. Include test logs or screenshots if necessary to provide the dev team with as much information as possible to effectively troubleshoot.
5. Apply regression testing to check if new changes introduce new bugs to existing features or not.
    

Upgrade Your Automation Testing, Today, Right Now!