We’ve been hosting a series of webinars on Katalon’s TrueTest, and the audience brought a wide range of practical, hands-on questions. Here’s a detailed recap of those frequently asked questions and the answers, so you can get a clear understanding of what TrueTest is and how it works.
Q: How do you capture the test steps and flows in TrueTest?
A: TrueTest automatically captures every user interaction. These interactions are organized into a user journey map, where each unique user session becomes a flow. From there, flows can be converted into automated test cases, which are exported into Katalon Studio projects.
Q: Are the flows created manually?
A: No. Flows are automatically generated from real user sessions. Testers don’t need to press a record or stop button.
Q: Do we need to install a plugin or something in our web server to make TrueTest work? How does it track user actions? What about security in production?
A: No server plugin is required. TrueTest works via a JavaScript code snippet, much like Google Analytics or FullStory. It tracks changes in the DOM structure when users interact with the application, without impacting performance. Importantly, sensitive data (PII/PHI) never leaves the browser. TrueTest includes safeguards to prevent any such data from being stored.
Q: Looking ahead, our production environment contains PHI data. Is TrueTest cleared to handle sensitive data?
A: Yes. PHI and PII are automatically detected and blocked from leaving the browser. On top of that, TrueTest applies additional cleanup before storing any data.
Q: How secure is TrueTest to use for application automation?
A: TrueTest relies on the same well-established mechanisms as common analytics tools. Combined with its extra safeguards for PII/PHI, this ensures high levels of security.
Q: Since TrueTest observes user behavior, how do we ensure that critical edge cases (that users rarely trigger) are still covered?
A: Edge cases and negative scenarios are usually not found in production user data, but are tested in staging or test environments by testers. TrueTest combines requirements-based testing (to cover edge and negative cases) with production-usage testing (to cover real-world flows). The result is stronger overall coverage.
Q: What safeguards exist so AI-generated tests don’t break with small UI changes, and how does TrueTest handle false positives/negatives?
A: TrueTest automatically updates object locators when minor UI changes occur, such as a button moving slightly on the screen. This prevents test failures caused by small changes and helps reduce false positives and negatives.
Q: Can TrueTest perform data-driven testing?
A: Yes. TrueTest supports data-driven testing by binding test case variables to Excel spreadsheets, enabling execution of the same tests with multiple data sets.
Q: Does TrueTest mainly do browser/UI interactions or API testing as well?
A: TrueTest currently focuses on browser and UI interactions. API support is planned and will be added next.
Q: How does TrueTest handle A/B testing?
A: Development teams can configure user session attributes to indicate which variant of an application a user sees. This allows separate user journey maps and test cases to be generated for each A/B variant.
Q: In enterprise SaaS platforms where customers log in to manage data, how does TrueTest ensure critical multi-role workflows (like Admin, Manager, Employee) are captured?
A: TrueTest allows instrumenting across multiple applications to treat them as one, or set up login configurations as reusable scripts in Katalon Studio. This allows teams to configure session attributes to represent user roles (such as Admin or Employee). These attributes can then be applied to filter user journeys and generate test cases specific to each role, ensuring that multi-role workflows are captured and tested.
Q: Can TrueTest also do accessibility testing?
A: Accessibility testing is available in the Katalon platform, but not directly within TrueTest today. However, TrueTest can help assess AI agent accessibility, showing how AI agents interact with your application.
Q: How can AI be used for performance testing in Katalon?
A: TrueTest is building capabilities to highlight where applications experience performance drops. It does this by overlaying a performance heat map on top of the user journey map. This lets teams see which parts of the app slow down under real usage conditions. In addition, TrueTest can monitor performance in production. While the tool can generate test cases for performance testing, how and where those tests are executed at load scale (e.g., simulating thousands of users) is up to the QA team.
Q: Is TrueTest separate from Katalon Studio?
A: TrueTest is part of the Katalon Platform. It requires Katalon Studio to review and execute test cases. When purchasing TrueTest, licenses for Studio are included.
Q: Is TrueTest a cloud-based product? Is there an on-premise option?
A: TrueTest is cloud-based. A private cloud option will be available later, but there is no on-premise version. Privacy is maintained by blocking PII/PHI capture, and test flows are automatically captured without manual recording.
Q: How effective is automation testing compared to manual testing? Is manual testing still necessary for mobile and web-based applications?
A: Automation should handle most testing, saving time and improving speed. Manual testing, however, remains essential, especially for exploratory testing, edge cases, negative scenarios, and ensuring user experience. The two approaches complement each other.
Wrapping up
Coverage gaps, test resilience, and multi-role workflows are some of the real-world challenges QA professionals are trying to solve. TrueTest aims to address these by automatically generating and maintaining tests from real user behavior while still supporting traditional requirements-based testing.
This blog should give you a solid understanding of what TrueTest does and how it fits into modern testing workflows.