Sanity testing, smoke testing, and regression testing comparison

While ‘Smoke Testing’, ‘Sanity Testing’, and ‘Regression Testing’ are widely practiced in the world of QA, there still exist some misinterpretations or misconceptions around these concepts. For your team to make the most optimal use of the 3 methods, this article will help you to understand and differentiate them from one another. 

What is a software build?

Before diving into the detailed comparison between the 3 testing methods, one should have a firm grasp of what a software build is. 
 

In software development, a build is a set of executable code ready either for testing or deployment. A software can contain thousands of source code files depending on its sophistication. When the project reaches a certain stage of development, these files are compiled into standalone applications that can be tested and shipped.

In case there are problems with the software build, teams will return to the source code to examine and modify them. Not every build can make it to the public release. Problems such as improperly installed environments, flaky networks, build scripts, and multiple other factors can render a build unqualified for deployment or even a complete failure. Thus, software build needs to go through rigorous testing to make sure it’s fully functional. 

Smoke testing

What is smoke testing?

Smoke testing, also known as build verification testing or build acceptance testing, is a non-exhaustive testing type that ensures the proper function of basic and critical components of a single build. For example, smoke tests on Gmail might include functions such as composing and sending emails.

Smoke testing is performed in the initial phase of the software development life cycle (SDLC) on attaining a fresh software build from developers. Its intent is to detect abnormalities of the core functionalities before teams move on to further exhaustive testing. This leaves little room for major errors to ripple through and become harder to fix down the line.

Smoke testing examples

The main focus of smoke testing is identifying and testing the important features and not the entire application, as in the example below.

Assume a project for a mobile banking application, which includes features such as:

• Login/Verify Pin
• Transfer Money
• Change Pin
• Display Customer Name
• Provide Bill
• Show Balance

Once Build 1 is sent to the QA team, they first need to pick out the most important features and implement tests upon them. In this case, they should be Login, as no further actions can be taken if the user cannot access the main interface, and Withdraw money - the basic functionality of an ATM application. 

Due to their heavy dependence on the other features, it would make more sense to validate these builds first to early detect any show-stopping bugs and save time for later development. This is called smoke testing.   

When do we do smoke testing?

Smoke testing can be done either by developers or testers, whenever new functionalities are developed and integrated with an existing build. 

Depending on the roles in software teams, either developers or QEs will carry out smoke testing. Once test results come back with green ticks, end-to-end testing or system testing will be followed through. 

Smoke testing is also performed to validate the core functionalities from the viewpoint of UI/UX. For example, the login feature should ensure users are allowed to log in by entering the correct username and password.

How does smoke testing work?

Commonly takes place after code reviews, if the initial build passes the smoke tests, it is qualified for further functional testing such as unit or integration testing. If the smoke tests fail, the build is rejected and handed back to the development team. 
 

The standard process is illustrated below.

To achieve the best results when executing smoke testing, it is essential to follow these common practices:

• Schedule smoke test suite on CI pipelines to automatically run when a new build is added
• Test early and regularly to assure the stability of the code build in every sprint
• Choosing the fitted testing method based on the requirements and resources of your project. If the budget is limited, a hybrid or an automated approach might be more optimal compared to conducting it all manually

Advantages of smoke testing

• Early bug troubleshooting: It quickly evaluates the essential features at the initial stages of the project for earlier corrections
• Time and resources efficiency: Smoke test verifies the stability of a delivered build to avoid any QA efforts that may otherwise be wasted 
• Reduction of regression issues: Newly introduced build is first tested to reduce the major flaws that can break the integration with the original material
• Automation potential: Automated smoke testing can significantly reduce testing time. Applying automation with an AI-based platform facilitates faster feedback and continuous testing

Disadvantages of smoke testing

If the team can only afford manual testing, it will be time-consuming, much like regression testing. It is therefore recommended to use an automation tool to increase testing capacity and efficiency. A quality management platform like Katalon enables teams to test more scenarios in a short span of time. 

Sanity testing

What is sanity testing?

Stemmed from regression testing, sanity testing is a type of testing performed to ascertain no issues arise in the new functionalities, code changes, and bug fixes. Instead of examining the entire system, it focuses on narrower areas of the added functionalities. 
 

Why do we do sanity testing?

Sanity testing is executed before the production deployment and after the build is delivered with minor code changes, to fulfill the below objectives:

• Validate critical functionalities and evaluate newly added ones
• Ensure the introduced changes don’t clash with the current functionalities
• Test rational and logical implementations of developers

Sanity testing examples

Take the mobile banking project above again. Assume in Build 2 of 3 features, defects are found in the Login and Show Balance modules. Developers will then resolve these bugs and send them back for clearance while still in the designing of new features. When Build 3 is released with 5 features, sanity testing in software testing is performed to ensure the new changes brought in by developers are working properly.

Features of sanity testing

Some major features of sanity tests are:

• Quick and simple: They are easily designed and performed with the aim to receive quick feedback
• Narrow and deep: One or few functionalities are covered in depth
• Undocumented and unscripted: In general, sanity testing doesn’t require test scripts or test cases
• Performed by testers: Usually, testers are the ones performing sanity testing

Advantages of sanity testing

• Speedy evaluation: It identifies defects in lesser time due to the unplanned and intuitive approach on limited functionalities
• Save time and effort: It acts as a gatekeeper by determining if the application should be further tested or not, which saves time of rigorous regression tests in case the release is in poor condition
• Less cost-intensive: Compared to other types of testing, sanity testing is usually cost-effective
• Identify deployment issues: Sanity tests can early detect compilation problems that may arise if the basic functionality is not working fine, or the previous bugs still exist but done from the developer’s end

Disadvantages of sanity testing

• Limited coverage: It only covers few functionalities, which leaves room for error in the unchecked ones
• Lack of future reference: As it is often carried out without script or documentation, future references are not available
• Time inefficiency in small projects: It takes more time to verify specific components than to check the whole application at once

Differences Between Smoke Testing vs Sanity Testing vs Regression Testing

Regression testing vs sanity testing

• Sanity is considered surface-level testing with a shallow and broad approach. Meanwhile, regression testing is extensive and in-depth.
• Sanity testing doesn’t use scripts while regression testing asks for scripts and documentation.
• Sanity testing is usually carried out manually but regression testing is preferred to be automated.

Smoke testing vs regression testing

• As smoke testing only focuses on certain modules and is performed on unverified builds, it doesn’t consume too much time and resources. On the other hand, regression testing requires a lot of time, resources, and effort.
• If the build doesn’t pass smoke testing, the application is prevented from further testing. In the case of regression testing, any issues found will not hinder the testing process

Sanity testing vs smoke testing

• Both smoke and sanity testing aim to provide quick feedback on the overall quality of the software, allowing developers to make necessary adjustments before proceeding to more extensive testing.
• The main objective of smoke testing is to test the stability of the build, while that of sanity testing is to verify the rationality of application components.
• Smoke testing is usually well-planned with scripts and documentation. In contrast, sanity testing is not a planned test and is done when time falls short.

FAQs about sanity testing

Why is sanity testing a subset of regression testing?

Regression testing is a software testing practice rather than a testing in itself. It contains multiple types of tests, with sanity tests being a checkpoint in the process to decide if the build can proceed with the next level of testing. 
 

Basically, sanity testing works just as regression testing but deals with a smaller test suite. This subset contains critical test cases that are run first before the examination of the whole package of regression tests.
 

The relationship between sanity testing, smoke testing, and regression testing can be seen in the graph below.

Is sanity testing done before regression testing?

Yes, regression testing is performed after executing sanity tests of any modified or added functionality. It ensures an application still works as expected after any code changes, updates, or improvements. It determines whether or not the software is eligible for further functional validation.

Which comes first, sanity or smoke testing?

Smoke testing is executed first in the early stages of the SDLC to set out a foundation of bug-free and reliable core functionalities. Once passed smoke testing, the build moves onto sanity testing to ensure its stability and flawless integration with the existing features.  

It is the best practice that the testing team start with smoke testing, followed by sanity and then regression tests, based on the project’s timeline and requirements.
 

Conclusion

Many QA teams have now practiced the above methods for the substantial benefit of early bug detection. However, as the number of tests increases over time, maintaining them can become a challenge. Manually executing these tests can be repetitive, low in efficiency and accuracy.
 

Therefore, automation has become a critical element in software development practices. With an automated regression testing process, product teams can receive informative feedback and respond more promptly. Finding the right automation solution can take your regression testing to higher levels.

Katalon - Comprehensive software quality management platform

Katalon is an end-to-end automation solution that supports automated functional testing, turning these processes into easy and simple tasks for testers. 

View automated test results

You can review test results with the comprehensive and customizable test reports in LOG, HTML, CSV, and PDF formats, and forward them as email attachments.

Root cause analysis

Changes introduced to the application code or environment configurations can affect multiple areas of the system. Katalon allows teams to review all failed test cases at once and group them into smaller groups based on their similarities, which will help the user to identify the common root cause of the related issues.

Ready-to-use framework

Regression testing helps developers to focus their efforts on building new functionalities for the application rather than keep on returning to check for defects in the old features. Those hours you’d have to spend on building your own automation engine can be used for the testing itself. As a hybrid testing framework built upon Selenium and Appium, Katalon supports different testing methodologies, including

• Data-driven testing (DDT)
• Behavior-driven development testing (BDD)
• Keyword-driven testing

Cross-platform regression testing 

More coverage usually means investing in a never-ending list of physical machines. Katalon provides an all-in-one regression testing tool for your website, web services, and mobile application. The tool also supports running scripts on multiple devices, browsers, and environments.

Reduce test maintenance effort

Following the page object design model, Katalon ends the maintenance nightmare by storing locators across tests in an object repository. When your UI changes, clicks are all it takes to get your automation scripts up and running again.