How Do Product Teams Test Software Quality?

Modern digital users have the patience level of a typical 5-year-old. Apps that incorporate software testing tools have one core objective to meet: not to make the first encounter with a glitchy UI and unusable functionalities.

Software quality in software engineering is a general expectation business stakeholders have. Unfortunately, testing is also a “sacrifice” for time to market, postponing the release date and rejecting builds last minute in the development process.

This is when testing, automation, and software quality tools are the must-have additions to achieve a consistent level of release confidence and speed. This article covers several different testing types, software quality testing tools, and how they are used together to help teams launch exceptional digital experiences.

Read More: What Is Software Testing?

Software Testing Type 1: Functional Testing

Delivering fully working software isn’t as easy as it sounds. 

A “Start for Free” button can be one bug away from returning a 404 error page or performing no actions upon click. But “code fast, ship fast,” while still maintaining quality, is an endeavor every client wants. 

So, we test.

Functional testing types range from manual to automated methods. Using the testing pyramid strategy, teams assess software reliability based on the software quality attributes of code quality and maintainability, app functionality, and overall performance.

Source: Symbio

You might like: Top 8 Automated Functional Testing Tools

I. Code quality and maintainability

1. Static code analysis

Checking the code without even running it? That’s what static analysis is all about. 

Static analysis tools add another layer of quality checks to find things like syntax errors, inconsistent adherence to coding styles or security loopholes in a non-runtime environment. Of course, manually reviewing or peer-reviewing code is an option, but automation is often incorporated for instant and extensive feedback.

Source: ComputingLearner

• Real-life example: 

As seen from the code snippet above, there’s a red underscore for the fourth line of code. Team A immediately sees this syntax error (int = 5, not 5.5) and seeks additional support for slip-throughs like this. Acting as the “Grammarly” for IDEs, they’ve applied static code analysis to scan through codes from the early stages of coding.

2. Unit testing

From small to big changes, unit testing is the fastest way to make sure the code is clean and safe to merge. Quick code-level automated tests are run to examine an isolated code section, class, or function from its roughest form. Unit testing software tools come particularly handy when re-testing impacted areas from new commits or refactoring. 

• Real-life example: 

Team A has coded a logic that if a valid combination of username and password is input into a login form, users will be directed to a “Create your first project” page. 

A login form is a single unit. To quickly test if its underlying code will successfully drive users to the right place, a unit test would report back what needs to be reviewed sooner. If there’s something wrong, say passing non-existent emails, fixing it now would be much easier compared to when newer code is added.

3. Integration testing

Testing individual code in isolation might not uncover issues relating to dependencies between various parts of an app.

• Real-life example: Team A is testing the Checkout and Payment page of a flight ticket booking web app. The goal is to find any mismatches in the total amount of money displayed. This means the Checkout page says $200, the Payment page should be no different. As these are two individual webpages, an integration test is used to determine if the data are in sync.

II. App functionality

Code-level checks are done – but how will they function when the end-users interact with the app? Application-wide functional testing helps to stop the dreadful worry of broken features.

1. API testing

Quality engineers and developers don’t wait until the UI layer takes shape to start software testing. As the layer that holds all the business logic together, API testing pinpoints faults much earlier between API calls and endpoints. 

• Real-life example: 

Team A is testing a sign-up/login form, ensuring that only valid usernames and passwords are recorded in the database. Through an API automated testing tool, the quality engineer:  

• Selects the POST method 
• Inputs the endpoint
• Enters a non-existent email address
• Sends the request
• Receives the result as TRUE

Soon after, the team finds out that the API/backend logic does not match the original requirements, leading to the exception of passing invalid inputs.

2. UI testing

The UI stays true to the saying “dress to impress.” 

Visual or UI testing is all about making a good impression on the very first touchpoint. No one wants to keep scrolling through a 2006-looking website where:  

• Images can’t be loaded.
• Buttons are missing.
• Texts are blended into the background.

Plus, don’t fall for the trap of doing the scan manually. Humans can only do so much.

Just three webpages? Spot-the-difference is pretty fun.

300 web and mobile pages? Time to give your eyes a break.

• Real-life example:

Team A has finished building the front end for an e-commerce website. 

Functional tests have passed and gone green. Yet when trying to access it on a web and mobile browser, it’s a different story. 

On the web version, everything is fine. But on the mobile browser, the price tag or cost of the item is nowhere to be found. 

Upon further inspection, the team realizes that the error is due to how the different devices render UI elements. 

3. Regression testing

New code can break old code.

In other words, that latest payment option you just added might cause the Checkout button to become unclickable. Automating regression testing is actually a common practice due to the fact that code changes happen so often. To prevent both new and old bugs from swirling up the system upon going live, some of the types of regression testing include: 

• Smoke testing
• Sanity testing
• Complete regression
• Partial regression

Real-life example:

Team A has just finished coding an enhancement for the submit comment section on a blog page. To eliminate false emails, the email verification feature was added. 

A code change like this could potentially break nearby functionalities – the “Submit” button for instance. For greater confidence that existing features will remain unaffected, previously run regression tests that have passed are wired up to re-run. The reasoning behind this is that these tests were already written for previous builds, making them the perfect fit to catch regressions in existing areas of the codebase.

4. End-to-end testing: Mimicking the user journey

Functional requirements always tie to a specific customer journey or a set of actions an end-user would perform via an app’s UI.

 Automated end-to-end tests cover a top-to-bottom scenario like: 

• Add item to cart
• Checkout 
• Add payment details
• Purchase item

Tip: Automated end-to-end tests are innately heavy in test steps, making them the longest in runtime. Katalon’s automated testing IDE allows you to play/run from the most critical test step, skipping through actions like opening browsers or going to the URL to keep your testing time less of a burden.

5. Exploratory testing

Replying on user stories to write tests keeps you focused on meeting the expected outcomes. 

But what about the unexpected, less conventional cases that might be unearthed?

By removing the formal structure of automated tests, exploratory testing allows quality engineers to creatively test the system-under-test unscripted and find quality issues on-the-fly. That said, exploratory testing is not just randomly clicking here and there. Software testers still have to quickly sketch out what it is that they want to test and discover.

• Real-life example: 

Team A has assigned Joey – a quality engineer – to test a client’s cooking instruction website. Without any pre-written test steps to follow, he has to test a feature that allows users to upload a picture of their final dish onto a shared album. Joey has tested plenty of apps before, so even though no desired outcomes were listed, he has the experience to test for things like: 

• Maximum image size allowed
• Supported image file type(s)
• Maximum number of images to upload at once

6. User acceptance testing

User acceptance testing differs from exploratory testing by its target audience – the stakeholders and the client. 

After the development and testing team have agreed that everything is good to go, it’s now about finding out if stakeholders think the same. 

In software projects, a week or less will be fully dedicated to the people that wanted the product in the first place. As these users do not have a quality engineering background, they will go through the app according to instinct. Moreover, user acceptance testing isn’t for finding defects like earlier quality stages. It instead gives a high-level view of the app and helps to determine if everything makes sense as a whole.

Software Testing Type 2: Performance Testing

User experience can easily be impacted by an app that crashes or a page that takes a lifetime to load. At any given level of load or pressure, performance testing serves the purpose of sustaining maximum functionality and operability of an app. 

The following are the common types of performance testing.

1. Load testing

Load testing sees if an app takes seconds or an eternity to respond back to user requests. Quality engineers use software load testing tools to simulate a specific workload that mimics the normal and peak number of concurrent users, then measures how much the response time is affected. 

• Real-life example:

Suppose Team A wants to see how their app behaves under traffic spikes for a holiday sale of an e-commerce website. The goal is to determine if 100,000 users rushing to find coupons would result in frustrated users staring at a spinning loading icon for over 3 minutes to make a payment.

2. Stress testing

Stress testing pushes a software or app beyond its limits. 

Whatever the average load of a system is, stress testing takes it a step further. Beyond the reliable state, software must be stress-tested for its breaking point(s) and corresponding remedies. 

• Real-life example: 

Team A is managing a web-based application for college course enrollment. Usually, when courses open for selection every semester, there are roughly 150 concurrent users standing by when the time hits. In case the number of sessions reaches 170, the system would remain stable or recover quickly if it crashes. 

3. Soak testing

In everyday English, “soak” means submerging something into a matter, such as dipping stained t-shirts into a pool of detergent. In the spheres of software quality, soak testing lets the system-under-test endure a specific workload, for a specific period of time. 

• Real-life example: 

Whether it’s hours or days, Team A now inspects the issues that go on after a continuous and extended runtime. To not let test suites get in the way of development activities, the quality engineering team schedules soak tests to run overnight or during the weekends. The results? A memory leak was spotted.

Software Testing Tools

Manual vs. automation testing

While the investment in automated software testing tools has proven clear-cut ROI, identifying the right candidates, scenarios and strategies to automate require teams to know the difference between manual and automated testing.

Manual testing tools (Test management tools)

Manual testing is done by writing out test steps on some type of text editor, making manual testing tools and test management tools similar. The actions and procedures outlined are performed through interacting with an app's UI using one of these tools below:

1. Google Sheets/Microsoft Excel: Free and widely available, Google Sheets or Microsoft Excel is pretty much the classic option for software quality engineers. With plenty of test case templates, testers can quickly jot down fields like the test case ID, test priority, or creator/assignee.
2. Zephyr Scale: Zephyr offers a more integrated manual testing workflow by improving bi-directional requirements traceability. Manual, automated or BDD tests link up to one another and defects logged for developers. 
3. TestRail: Purpose-built for creating and managing manual test cases, this web app allows users to author, filter and export/import manual tests.
4. Azure Test Plans: Azure Test Plans is a browser-based tool for test management. As part of the Azure DevOps ecosystem developed by Microsoft, teams using this tool suite can also integrate testing and review results on build and release pipelines.

Automated testing tools

Automated testing can be done in one or two ways: using a testing framework, or vendor testing solutions.

A testing framework is built using open-source libraries. It provides users with the ability to code and customize their own quality management engine to test applications. 

Testing solutions are pre-built by a vendor and equipped with the testing essentials to plan, author, organize, execute and analyze. Automated software testing tool vendors commonly offer individual tools, a complete platform, or a mix. If a team is only testing a specific system-under-test or area, say web APIs, then single tools are preferred. A testing or software quality management platform is used more in instances where applications are interconnected and demand cross-compatibility between a multitude of application types.

• Katalon Platform: Supporting web, mobile, and desktop apps and UI and API automated testing, Katalon is a comprehensive software quality management platform that quality engineers and developers can both harness. Without additional third-party tools or framework maintenance, teams can have a unified testing workspace to author in low-code or full-code, execute on preconfigured cloud environments and maintain automated tests stress-free with the page object model design.
• Selenium: You can’t go into testing without knowing Selenium. As one of the first open-source libraries dedicated to web automated testing, Selenium is great for teams with a powerful force of code-savvy developers and quality engineers.
• JMeter: Specialized for load testing, JMeter is a Java-based open source software to measure the response rate and performance of web apps. Conveniently, JMeter also works with Selenium to run functional and load tests in parallel.
• Postman: Postman is solely built to test the functionality, health, and performance of APIs. Users can define and save environment variables, and export tests in JSON settings for another member of the team to reuse.
• JUnit: Test-driven development or Java-based software, the annotations, assertions, and test runners in JUnit are best used for developer-side testing. Compatible with IDEs like Eclipse and NetBeans, JUnit enables users to define reusable automated tests through the conditions of errors and expected/actual results. 

Bug/Defect tracking tools (Test management tools)

The best bug tracking tools are those that let teams trace back defects to their original user story, requirements, and tests. Throughout the development process, developers and the entire software team can track the remaining defects, prioritize/de-prioritize them and assign them to a designated member to resolve.

• Jira: Jira enables software teams to track defects, issues, requirements, and testing activities all within one place. Data for the latest status of pull requests or tested builds are available in real-time. In terms of integrations, Jira has built up a well-versed list of manual/automation, continuous integration, and many other indispensable tools in software projects.
• Monday.com: This up-and-coming project management system serves both business and engineering departments. From roadmap planning and workflow automation to bug tracking, Monday.com makes managing development and testing work items easier.

CI/CD

Automated testing is the key enabler for continuous integration and continuous deployment (CI/CD). For developers, software quality engineers and the rest of the team to have the confidence to release iterations frequently, testing needs to constantly happen. Given that automated testing sits at the core of CI/CD, integration with testing frameworks and tools is a must.

• Jenkins: Jenkins infuses automation workflows throughout the building, testing and deployment stages of software development. This CI orchestration system comes with an abundant set of plugins to integrate with other tools, run tests and aggregate results for easy failure investigation. 
• GitHub Actions: Powered by a massive open source community, GitHub Actions expands further to configuring an automated CI/CD pipeline. To enable a reliable engine before merging anything in, Github Actions lets testing be the instant decision-maker to keep untested code at bay. 
• Bitbucket: Built by Jira, Bitbucket provides automated security scans, private repositories, code encryption and enterprise-geared capabilities for better security. 

Communication

Messaging platforms in Agile development needs to be more than a day-to-day chat room. The entire team needs communicate and stay in sync at all stages of the software development process. Especially during times where hybrid working modes have become prevalent, communication across the engineering department needs to be solid. Tool suites including CI tools, testing platforms and the rest of the engineering techstack needs to be connected and provide instant quality feedback.

• Slack: Enable real-time alerts on builds, progress of bug tickets or automated test results. Standardize processes for pull requests, deployment status and shorten time-to-resolution through group channels. Some other nifty feature sets that Slack offers are huddles, message scheduling, reminder settings and more. 
• Microsoft Teams: Sharing similar functionalities to Slack, Teams brings easy access and linkage between other services in the Microsoft ecosystem. Names such as OneDrive, OneNote, Outlook and more.

Katalon Software Quality Management Platform | Quality Software, Quality Digital Experiences

Katalon is a software quality management platform used by +30K Product teams and +1M quality professionals. Providing an end-to-end automated testing solution for APIs, web, mobile and desktop apps, the Katalon Platform enables software teams to launch the best digital experiences with the best quality tools.

• No more DIY test frameworks: Why start building when you can start testing? Katalon reallocates the time spent on maintaining a separate framework to the actual testing itself. From built-in record-and-playback, cross-environment execution to easy test maintenance with object repositories, every critical functionality for testing is ready to use without extra development required.
• Easy to learn, fast time-to-value: Whether you’re switching from an old tool or kick-starting your automated testing journey, Katalon promises simplicity. Courses, documentation, and a community of +1M practitioners will get you onboarded in no time.
• Lower developer dependency: Even for teams with mixed levels of automation expertise, getting out a release candidate only takes hours instead of weeks using Katalon. Here’s a success story from System Automation, a licensing and board management software firm that saw a 96% speed increase in testing cycles.
• Low-code/No-code, but also full code: Reducing coding efforts doesn’t limit your ability to customize more advanced actions and test scripts. Once you’re comfortable with record-and-playback and drag-and-drop, jump over to scripting with Java and Groovy.
• Full visibility on software quality: Get a high-level picture of software quality with data on manual/automated test results, unresolved defects and build quality drawn across the Katalon Platform and your DevOps ecosystem.
• SaaS, on-premise, or the private cloud: Skip the infrastructure maintenance hassle to run tests in private networks.
• AI-augmentation to optimize testing cycles: AI capabilities like identifying similar test failures, self-healed broken locators or smart UI comparison methods require no setup with third-party tools. Plus, all the latest AI/ML trends in testing you’ve read about are probably already in Katalon’s product roadmap.