Highlights from QA Leaders Live Episode 2: how insurers should test AI agents, split work between agents and experts, and make test evidence a byproduct.
When a claim is settled at 4 a.m. with no adjuster in the room, passing a test suite is no longer the bar. Regulators in the US, Europe, and Australia now ask a simpler question: show me the evidence, and show me who is accountable.
That was the starting point for Episode 2 of QA Leaders Live, held on November 10, 2026. Host Alex Martins, VP of Strategy at Katalon, sat down with two people who see insurance quality from very different seats:
- Yuvraj Singh, AVP of Insurance Delivery and Practice at TestingXperts, who works with carriers, MGAs, and underwriters across North America, Europe, the Middle East, and Asia Pacific
- Richie Yu, Managing Director at RYU Solutions Inc., a QA leader with 20+ years in regulated industries who now helps teams adopt agentic testing
The question was not whether insurers will use AI agents. They already do. The question was what quality looks like when they do: what agents should handle, what people must own, and how to govern it all across hundreds of releases a year. Here is what we learned about test automation, end to end testing, test management, and AI governance in insurance.
From Pilot to Production Is the Hard Part
Most teams have finished the experimentation phase. They know what the tools can do, and both the models and the skills of the people using them keep improving. Richie's point: generating test cases or scripts is no longer the hard part. Doing it at scale, in a controlled way, is.
Two things stand in the way, according to Richie:
- Change management. Agentic work has to be consistent across the testing lifecycle, without dropping traceability or evidence. A team must be able to show that the execution record is solid and that each test case tests what the design says.
- Cost structure. Adopting the tools brings a new LLM bill that did not exist before. Used carelessly, it gets expensive fast.
Alex summed it up: pilots and build it yourself projects are a necessary phase, but moving to production in a way that delivers real value is the biggest challenge teams face right now.
The Decision Is the Product
Yuvraj gave the line of the episode: "For us, the decision is the product." Quality engineering in insurance used to be a checkpoint. It is now tied to business outcomes, and the pressure arrives all at once instead of in sequence:
- Business: the head of claims wants straight through processing, with more claims settled without human intervention.
- Regulators: they care about outcomes. Can you defend a decision an agent made, with a solid audit trail behind it?
- Technology: a rating engine gives the same output for the same input. Agents and models change, and they can surprise you.
Yuvraj named three shifts that raise the stakes for whoever owns quality:
- The blast radius. A wrong answer in a demo stays in the demo. A wrong answer to a policyholder asking whether a flood is covered, multiplied across thousands of policies, is a disaster.
- The question. Testing used to ask whether the claim button, the API, or the premium calculation worked. Now end to end testing has to ask whether the decision itself was right.
- The clock. A release used to stay tested until the next version. Now model drift happens every few days, so testing has to be continuous and backed by evidence you can show to customers, ombudsmen, and regulators.
Alex added context from the National Association of Insurance Commissioners survey he cited on the episode: more than half of health insurers use third party components in their AI systems, and some rely on them entirely for certain workflows. When much of the decision logic is built outside the company, the risk surface grows.
Agents Handle the Volume, Experts Own Correctness
Richie began with a reality check: agents work only with the context they are given. Give them a vague prompt or the wrong context and they will do their best with what they have, which is rarely what you meant. Write the process down, walk it through, and make sure the agents follow your intent.
Yuvraj's rule of thumb: let agents take work that is high in volume, repeatable, and low in risk. Bring in subject matter experts to own correctness. In insurance, he said, that split looks like this:
Agents do well | Subject Matter Expert (SME) must own |
|---|---|
AI test case generation from policy documents and past claims | Business rules that live in people's heads, not in code |
Writing and maintaining test automation scripts | Edge cases shaped by past disputes |
Triaging claims so adjusters do not start Monday with a wall of red | Distinctions such as sudden water damage versus long term seepage |
Test data generation | Regulatory nuances, such as UK motor and home renewals not exceeding new business pricing |
Selecting which regression testing suites to run | Final sign off on the evidence |
His example: given a water damage user story, an agent produced 300 to 400 test cases. An experienced SME would write 30 to 40, because they know which differences the policy treats as material and which disputes came up before.
Richie explained why experts matter even more in established enterprises. Agentic testing runs smoothly on a brand new app with clean user stories. A five year old production app is a patchwork of components, patches, and documentation that does not reconcile, so the agent lacks the context. His framing: the agent is an assistant that speeds up the first draft, and the QA lead fills the context gap.
Yuvraj added a warning. Many carriers still run legacy systems where business rules sit partly in code and partly in the head of an SME who has been there for 25 years. When that person leaves, the test oracle leaves with them. Every time an SME corrects an agent, that correction should become an executable test, so the knowledge lives in the pipeline and not in a slide deck.
Automate the Yes, Escalate the No
Should agents settle low value claims end to end with no human review? Alex put the debate to Yuvraj, who deals with it "almost every other day." His rule: automate the yes, escalate the no.
A cracked windshield on the way to the airport, a dropped phone, a missed flight, lost baggage: each is clear, repeatable, and low in dollar value. There is no reason for an adjuster to spend time on a $400 to $500 claim when that expertise could go to harder work. Agents can handle these, provided the guardrails are set: what amount an agent may settle, and what disputes come in after it has settled. Anything suspicious, such as possible fraud, goes to a human every time.
Yuvraj also warned against assuming a human reviewer is always the safer option. He recalled a reported case in the US where claim denials were approved so fast that adjusters could not have had the time or information to review them. A tired adjuster at 5 p.m. on a Friday is not real oversight. A rubber stamp is not accountability.
Alex took it one step further: in a space this complex, people should not just be in the loop. They should be driving the AI.
Test Evidence and AI Governance: A Byproduct, Not a Project
An audience member asked what examiners now ask insurers for. Yuvraj listed what governance teams need to have ready:
- Inventory. Everything that influences an agent's decision: models, scorecards, vendor tools, and the agents themselves.
- Behavior. What each of those can do, and where the line is drawn on what they must not do.
- Testing evidence. Positive, negative, end to end, and edge cases, plus proof that agents do not act in a discriminatory way. Accuracy alone is not enough.
- Monitoring. Who notices when a model changes, who owns it, and who answers for it.
Alex noted that many governance programs still run in spreadsheets, which cannot keep up with agent volume. Richie drew a clear boundary: QA proves the system works as designed at the moment a change is introduced. Production needs its own guardrails, because a model provider can change or throttle a model's behavior even when you did not. If 90% of transactions normally pass straight through and that rate shifts, the business needs to look closer.
Yuvraj's main point: when someone asks for evidence and the team scrambles to rebuild traceability, find execution records, and confirm environments, evidence has become a mini project. In a well run test management process, it assembles itself as testing happens. That means planning from day one: a requirements traceability matrix that maps requirements to test cases, regulatory and state level checks covered, evidence captured on every failure, and a named person for who wrote, ran, and signed off each test.
What to Set Up in the First 30 Days
Another audience member asked how to spend the first 90 days of a QA program for a new or modernized application. Yuvraj's advice was to put the planning into the first 30 days so nothing surprises you later. Questions to answer early:
- What will each agent do, and what is its core responsibility?
- What knowledge, repository, or SME input are the underlying models trained on, and do they return consistent results?
- Is there a concrete reference for every scorecard, such as rating, telematics, credit risk, and external factors?
- Are adjusters using public LLMs, and is that accounted for?
- Who is the single accountable owner?
On accountability, Yuvraj was direct. Put names on paper: the business owner (for claims, the head of claims) and the lead testing engineer who approved the evidence.
How Do You Test Nondeterministic AI Output?
Richie's rapid fire answer was a layered check. Teach agents the established test design techniques. Use a second LLM to check whether those techniques were applied correctly. Then have the QA lead review whether the result makes sense. Checks and balances, not blind trust.
What to Take Back to Your Team
- Test the decision, not just the interface. In insurance, the outcome is what regulators and policyholders judge.
- Split the work on purpose. Agents take volume and repetition. SMEs own correctness, edge cases, and sign off.
- Capture expert knowledge as executable tests. When an SME corrects an agent, that fix belongs in the pipeline.
- Automate the yes, escalate the no, within clear guardrails on amounts and disputes.
- Make evidence a byproduct. Build requirements traceability and QA audit trails in the first 30 days, not when the examiner calls.
- Name the owners. A business owner and a lead testing engineer, on paper.
Richie closed with advice for anyone in testing right now: take the time to learn these tools and approaches. "You don't want to sit this one out."
Watch the Full Episode and Join the Finale
Watch the Episode 2 recording for the full conversation, including the audience Q&A.
The Season 1 finale, How to Prioritize Software Testing When AI Increases Code Velocity, is live on December 2, 2026, at 10:00 AM EST. Alex Martins is joined by Suneet Malhotra (Senior Manager of Test Engineering, ex Amazon and ex Tinder) and Navyatha Vijayakumar (Software Delivery and Quality Advisor, ex ABB and ex Honeywell) to talk about risk based testing, continuous testing, and test prioritization when development moves faster than ever.

Test everything. Ship with confidence.
From generating test cases to spotting failures, Katalon's AI works across your entire pipeline, automatically.

