What is Exploratory Testing?

Exploratory Testing is an unscripted, manual software testing type where testers examine the system with no pre-established test cases and no previous exposure to the system. Instead of following a strict test plan, they jump straight to testing and make spontaneous decisions about what to test on the fly. 
 

Prior to the testing phase, they might list down some concepts or areas to explore, but the essence of exploratory testing still emphasizes the tester’s personal freedom and responsibility in simultaneous knowledge acquisition and quality checks. In a way, exploratory testing is similar to an exciting adventure where testers don’t really know what lies ahead of them, and they must utilize certain techniques to uncover those mysteries.
 

During exploratory testing, testers explore the software by interacting with it, trying out different features, then observing its behavior. They may even intentionally break the software, input unexpected data, or explore edge cases to uncover potential issues. As long as testers get to understand the system’s workings and can suggest strategies to test the system more methodically, they have accomplished the task.

Why Exploratory Testing?

Exploratory testing plays a crucial role in software testing, especially Agile projects, where automation testing has taken precedence. Automation testing admittedly brings a wide range of benefits to QA teams, but it is impossible to ignore exploratory testing and more broadly manual testing. 

1. Exploratory testing finds bugs we aren’t yet aware of

A recent research compares exploratory testing (ET) with test case based testing (TCT) in terms of their effectiveness for catching software defects. Results show that test case based testing excels in catching immediately visible defects (Mode 0) or defects that require only 1 interaction (Mode 1) to cause failure, while exploratory testing does a better job in catching complex defects requiring 2 and 3+ user inputs (Mode 2 and Mode 3). 

In other words, exploratory testing helps testers catch bugs that automated testing would have missed. Automated testing only catches bugs that we know may happen (we can only create test scripts for something we know about), while exploratory testing catches bugs that we don’t even know to be existing in the first place, tapping in the Unknown Unknown region of our understanding. 

Exploratory testing and manual testing in general expands the test coverage to blind zones of automation testing, pushing product quality to a new level.

2. Exploratory testing encourages innovation and knowledge sharing

Exploratory testing is free-style and not as rule-based as automation testing, and this characteristic opens up a lot of room for creativity and innovation. If a tester individually performs exploratory testing, they need to apply their domain knowledge, intuition, critical thinking, and even user-centric thinking to interact with the system and uncover potential issues.
 

An even better approach to exploratory testing is applying the pair programming principles, which involves 2 people - 1 driver and 1 navigator. In a time-boxed period, the driver performs the actual testing while the navigator observes, provides guidance, and takes notes where necessary. Sometimes called Crowd Exploratory Testing, this approach maximizes the level of creativity and subject expertise of both testers while encouraging knowledge sharing and collaboration between team members. 

3. Exploratory testing encourages continuous learning 

In the words of Cem Karner who coined the term for this testing type: “[exploratory testing treats] test-related learning, test design, test execution and test result interpretation as mutually supportive activities that run in parallel throughout the project.” As exploratory testing requires minimal to no planning, it can be conducted almost whenever we want, giving us the autonomy we need to learn about the product while also performing quality checks, saving time and resources.

4. Exploratory testing complements continuous testing 

In this era, the entire Quality Engineering industry is moving towards shift-left testing and continuous testing where testing is performed in synchronicity with development. Shift-left testing allows testing to happen earlier, leaving ample time for troubleshooting while improving the project’s agility.
 

However, driven solely by automation, there is always a critical gap in this approach as we have mentioned above: the lack of investigation into the Unknown Unknown zone. When testers become too fixated on what can be automated to fit in the Continuous Testing approach, they get the cognitive bias called “tunnel vision”, ignoring alternative solutions. At the end of the day, automation testing only checks if a feature works and does not find bugs. 
 

Exploratory testing requires every tester to acknowledge this gap, and encourages them to explore beyond scripted scenarios, uncovering new and unexpected behaviors. A highly recommended approach is to combine manual testing with automated testing to utilize the benefits of both.
 

Read more: Agile Testing Methodology: A Complete Guide For Agile Testers

Exploratory Testing Example

Consider yourself in the position of testing an eCommerce website with features like online shopping, payment processing, and order management. If you adopt an Exploratory Testing approach, your objectives would be to navigate through the website, identifying possible usability issues, interface defects, and any potential risks.
 

You would interact with various functionalities of the website, such as browsing product catalogs, adding items to the shopping cart, and completing the checkout process. After that, you can explore different menus, screens, and buttons to ensure smooth navigation and intuitive user experience. You would also input values and scenarios into certain field forms, such as using different payment methods or applying discount codes, to assess the accuracy of calculations and the transaction processes. There is no rule - your goal is to explore and familiarize yourself with the website.
 

In the process, you may find critical issues such payment gateway failures and security vulnerabilities, or minor problems like broken links and inconsistent product descriptions. Even if you did not find any bugs, you still learn a lot about how the system works. This knowledge will come in handy when you start developing automation test scripts in the future.
 

There are many website or application-specific features that a tester needs to pay attention to when doing exploratory testing. For example, exploring a personal finance application would require them to apply the mindset of a customer needing security and accuracy, compared to exploring a website that places stronger emphasis on interactivity.

Pros and Cons To Exploratory Testing

Pros:

• Encourages creative and innovative thinking
• Identifies defects that are usually missed by formal testing methods
• Simultaneously provides a comprehensive understanding of software functionality and quality
• Requires less preparation time compared to formal testing
• Efficient in quickly identifying major defects

Cons:

• Difficult to measure and control without a formal test script
• Requires skilled and experienced testers for effective defect identification
• Replication and automation of results can be challenging
• Inconsistencies may arise from different testers using different approaches
• Time-consuming if we can only uncover minor issues with no impact on software performance

When To Do Exploratory Testing?

Exploratory testing can be applied throughout various stages of the Software Development Life Cycle (SDLC) thanks to its versatile and spontaneous nature. For example, at the Requirement Analysis stage where you need to gather information on the system for test planning, exploratory testing can provide a lot of valuable insights.
 

You can even apply it for Unit Testing, Requirements, Functional, Load Testing, Sanity Testing, etc. However, its effectiveness is most pronounced during end-to-end testing as an extra verification step prior to production, or even thereafter.
 

One of the remarkable aspects of exploratory testing is its flexibility in terms of timing, as it can be conducted at any moment, even if there are only 5 minutes available, by virtually anyone with software access.

Types Of Exploratory Testing

There are 3 styles of exploratory testing:

1. Free-style Exploratory Testing: as the name suggests, this approach does not require any rules or specifications, and the tester has great control over what they want to do with the application. However, it does not mean that testers approach it with randomness. In fact, free-style testers must themselves build an exploration strategy in their head based on their experience and intuition before performing the actual testing.
2. Scenario-based Exploratory Testing: this approach is performed on real-user scenarios where the QA team isolates and tests 1 scenario at a time from all angles. This is a more structured approach than free-style testing.
3. Strategy-based Exploratory Testing: there is an overcompassing strategy that guides the exploratory testing activity, requiring the testers to apply various testing techniques (boundary value analysis, equivalence, risk-based, error guessing, etc.)

How To Do Exploratory Testing?

Although it does not require initial planning, findings from exploratory testing sessions must still be carefully documented for future references. Therefore, the process of exploratory testing revolves around the documentation and interpretation of testing activities, including 4 major stages:

1. Create a bug taxonomy
2. Test Charter
3. Time Box
4. Review and Interpret Results

1. Create a Bug Taxonomy

Bug taxonomy is the categorization of software bugs based on their attributes, characteristics, and criteria. Any bugs with similar attributes can be grouped into predefined classes to provide a structured framework for better understanding, analysis, and management. Here is a list of basic bug categories that you can consider:

1. Severity (High - Medium - Low impact to system performance/security)
2. Priority (High - Medium - Low urgency)
3. Reproducibility (Reproducible, Intermittent, Non-Reproducible, or Cannot Reproduce)
4. Root Cause (Coding Error, Design Flaw, Configuration Issue, or User Error, etc.)
5. Bug Type (Functional Bugs, Performance Issues, Usability Problems, Security Vulnerabilities, Compatibility Errors, etc.)
6. Areas of Impact
7. Frequency of Occurrence

These taxonomies can be organized in a simple Google Sheet, but when the project scales, QA teams may need a dedicated bug management and tracking tool for this part.
 

Read more: Agile testing methodology: A complete guide for agile testers

2. Create a Test Charter

A test charter is a high-level document that outlines the goals, scope, and approach of a specific testing session. It is similar to the test plan in terms of purpose but more narrow in terms of scope. A test charter specifically created for exploratory testing should include the following basic items:

• Objectives: Define what the testing team aims to achieve.
• Approach: Provide a high-level overview of how the testing will be conducted.
• Scope: Determine the extent and boundaries of the testing activities.
• Test Deliverables: Outline the documentation and artifacts to be created during testing.
• Dependencies: Identify any dependencies or prerequisites for testing.
• Test Environment: Plan the hardware, software, and network configuration for testing.

A free-style exploratory testing session would not need a test charter, but scenario-based and strategy-based sessions would definitely benefit from one to improve collaboration between members. This test charter also includes the taxonomies defined in the previous step for better management of newly discovered defects.

3. Define The Time Box

The “Time Box” sets a specific time limit for exploratory testing activities. It helps them stay focused and prevents them from spending excessive time on a single aspect of the system. This maximizes their testing efforts within that time frame to identify the critical bugs and pinpoint major areas for improvement. 
 

In this time box, if a bug is found, testers will record and describe it in great detail using the taxonomies, then submit it to a bug management system where the status will be set to “New”. The developers will then investigate and try to reproduce the bugs locally before fixing it.

4. Review and Interpret Results

Testers can then evaluate the test results by comparing them against testing metrics, including customer feedback, usage statistics, and system performance data. This evaluation enables them to gain a general understanding of the test effectiveness, identify system defects, and gain valuable insights into the strengths and weaknesses of the product. It also serves several other purposes:

1. Recognition of gaps in the testing process
2. Guide the creation of new requirements and user stories
3. Sometimes, it may lead to modifications in the scope of the system to be developed or tested.
4. Help managers and stakeholders make informed decisions
5. Help testers to recognize patterns and trends in the testing outcomes, which can assist in predicting and addressing potential issues in the future.

Tools For Exploratory Testing

Technically there are no “exploratory testing tools”, since exploratory testing is simply a mindset and approach to testing, and therefore there is no tool dedicated to it. However, there are testing tools that improve the exploratory testing process. These include:

• Bug tracking tools: these tools allow testers to track and manage defects found during exploratory testing so that both them and the dev team can be updated on the current bug status
• Test management tools: Although exploratory testing does not use test cases, testers would still benefit tremendously from test management tools helping them plan, organize, and track the test they did. Without test management tools, many testers won’t be able to replicate the bug in the next release because they forgot everything about it.
• Session Recording Tools: these tools allow testers to record their interactions with the software while exploring. It is a simple and easily accessible way of documenting exploratory tests in real time and sharing findings with the team. It also works as reference material when developers need to reproduce the bug locally

Other than these tools, testers may also need cross-browser testing tools if they want to explore the application behavior on different combinations of browsers - devices - OS without having to invest into physical machines supporting those environments.

Improve Exploratory Testing And Automation Testing With Katalon

Exploratory testing is just a small part of the larger testing process, which is why many leading testing tools are built to support several testing types and testing activities, not exploratory testing solely, to ensure that QA teams are supported throughout the entire STLC. 

Katalon is a comprehensive quality management platform supporting test creation, management, execution, maintenance, and reporting for web, API, desktop, and even mobile applications across a wide variety of environments, all in one place, with minimal engineering and programming skill requirements. 
 

For exploratory testing specifically, Katalon can:

• Record-and-Playback feature to record screen and turn your sequence of actions into test scripts 
• Centralize test management
• Provide robust automation testing features for future regression test runs
• Generate smart reporting and analytics for data-driven decision making
   

The Record-and-Playback feature of Katalon essentially records all of the actions happening on your screen and stores them as automated test scripts that you can re-execute across a wide range of browsers and devices. In a way, this feature helps you simultaneously document the test, manage it in a centralized repository, and re-test it in the future.
 

All of the elements (buttons, form fields, etc.) captured during these Record-and-Playback sessions are stored as Keywords (essentially code snippets) in the Object Repository. Testers can easily drag-and-drop them to construct a new test script without having to code anything. It allows testing teams to quickly move from manual testing to automated testing at a low learning curve. In addition to all that, the Smart Test Reporting features give teams advanced test analytics for more informed decision-making. 
 

Download Katalon now

Differences Between Ad hoc Testing And Exploratory Testing

Ad hoc testing and exploratory testing have many similarities, but they are not at all interchangeable terms. Exploratory testing involves a certain degree of spontaneous planning, while ad hoc testing involves absolutely no planning beforehand.
 

Exploratory Testing vs Scripted Testing

It is important to note that scripted testing is not just automation testing. As long as the testing approach is systematic, structured, predefined, and test cases are documented in detail beforehand, then it is scripted testing. Even manual testing with proper documentation can also be categorized as scripted testing. 
 

Exploratory Testing FAQs

1. Is Exploratory Testing white box testing?

No, Exploratory Testing is black box testing, not white box testing. White box testing is a testing technique where the tester has knowledge of the internal structure, design, and implementation of the software being tested. Exploratory Testing does not rely on knowledge of the system’s internal workings.

2. Can exploratory testing be automated?

Exploratory Testing is primarily a manual testing technique. Some aspects of exploratory testing can be supported by automated tools, but the core essence of exploratory testing relies on human exploration and judgment. Test scripts for bugs found by exploratory testing should still be created for regression test runs to verify that they have been fixed. You can leverage the GPT-powered test case generation feature in Katalon to quickly achieve this.

3. Is exploratory testing functional testing?

Since it is just an approach, exploratory Testing can include both functional testing and non-functional testing.

4.Why exploratory testing is required for Agile projects?

Exploratory Testing is well-suited for Agile projects due to its flexible and adaptive nature. Agile projects often require fast feedback and continuous testing to accommodate frequent changes and iterations, and exploratory testing is perfect for that.

5. What are some exploratory testing best practices?

• Defining clear objectives and charters for each exploratory testing session.
• Adapting the testing approach based on the software's context and risk areas.
• Combining domain knowledge, user perspective, and intuition to uncover issues.
• Capturing detailed notes, screenshots, and videos to document findings.
• Collaborating with developers, business analysts, and other stakeholders during testing.
• Balancing exploration with structured testing techniques to achieve comprehensive coverage.
• Sharing insights and lessons learned from exploratory testing with the team.

Start Testing Now