Dear Katalon Community,
I'm thrilled to share with you the latest updates on TrueTest, Katalon's innovative product designed to revolutionize the way you approach regression testing. As the Product Manager for TrueTest, I've seen firsthand the evolution of this tool, and I'm excited to share our journey and the major improvements we've incorporated in our new release.
TrueTest is not merely a tool; it is a paradigm shift in regression testing. By capturing and analyzing real user interactions within applications, TrueTest identifies and prioritizes common user flows. These insights then translate into actionable test cases and relevant test data, ensuring that regression testing is a true reflection of user behavior in production environments.
From July to December 2023, we embarked on a six-month Beta Program journey with 400+ participants, including 100+ new organizations. Your feedback has been invaluable, and I want to extend my heartfelt thanks to all early adopters. Your engagement has been crucial in shaping TrueTest’s direction.
1. Lengthy Onboarding Process
Our journey wasn't without its challenges. We realized the onboarding process was longer than expected, ranging from two days to six months. Factors like lengthy security approvals, dependency on development bandwidth to install TrueTest and the need for top-level buy-in to try innovative solutions were the most common hurdles.
Acknowledging these challenges, we have made several key enhancements to accelerate the TrueTest onboarding process, including:
2. The Evolution of TrueTest's Automated Test Generation Process
TrueTest initially focused on tracking user interactions in the production environment to create detailed user journey maps, identifying 'User Flows' - patterns reflecting real user behavior. TrueTest would use those user flows to automatically generate the corresponding test cases. Lastly, the TrueTest user would simply be able to run those tests against any environment, usually pre-production, as part of the release testing process.
However, we soon discovered that due to differences between the environment setup in production and pre-production, the tests generated wouldn't run as-is.
Despite our aspirations, the most common challenges preventing tests from running across environments revolved around diverse technology stacks, complex authentication mechanisms (e.g. SSO, OTP, CAPTCHA, etc.), and intricate business logic that was set up in production, but not in pre-production environments (or vice-versa).
To work around those challenges, we added validation steps to the test generation process with the goal of ensuring tests would run across environments. However, what we learned from our beta users was that our validations were causing the test generation process to achieve suboptimal user flow coverage - users in production were going through a lot more flows than TrueTest would show and generate tests for. That valuable beta user feedback highlighted the need for improvement.
3. Introducing Our Refined Strategy
With a renewed focus on maximizing user flow coverage, TrueTest now automatically generates tests post-user journey map creation. This empowers testers to deeply understand user interactions while having all the corresponding user flow-based tests on hand. At a minimum, the TrueTest user can now perform manual testing where environmental differences exist, or they can edit the automated tests to account for those differences, effectively bridging the gap between existing tests and actual system usage.
4. Data-Driven Test Cases Are a Hit
Our TrueTest beta users raved about how TrueTest generated fully parameterized automated tests. That enabled them to simply provide their existing test data files to the newly generated tests and start execution, saving them valuable time and effort if they were to do that entire parameterization process manually.
In addition, our beta users were also very appreciative of the visual assertions that TrueTest automatically adds to every generated test. That ensures each test will perform at least one assertion, flagging potential functional issues during a regression test execution without requiring the tester to edit the test. If testers would like, they can add their own assertions to validate specific functionalities and business logic.
5. Engaging Human Expertise
Recognizing the current limits of AI and automation, we value human collaboration with AI which is not yet fully mature and hasn't achieved its full potential. This collaborative approach ensures comprehensive and ready-to-use test cases. As AI continues to evolve, it will unlock even more use cases, which will continue to be incorporated into TrueTest.
Our enhanced approach with TrueTest is a commitment to ensuring that your regression testing is as relevant and effective as possible. By combining AI-driven insights with human expertise, TrueTest aims to eliminate guesswork and streamline the development and maintenance of test cases.
We invite you to experience the enhanced TrueTest – a tool designed to illuminate real-world application usage and elevate your testing strategies.
With gratitude and excitement,
Mai Le - Product Manager, TrueTest, Katalon