Who’s Watching the Agents? Why Confidence Is the Real Bottleneck in Agentic Testing

AI is taking over more of the execution in software delivery. In Who’s Watching the Agents? Context, Guardrails, and Front-Line Reality, Beth Marshall, Paul Grizzaffi, and Ivan Davidov set the marketing aside and talked about what agentic testing looks like in practice.

Their shared conclusion: keeping quality accountable when humans step back is the hardest thing now.

From Chatbots to Harnesses

Ivan has lived the whole arc, starting with chatbots, then prompt engineering, context engineering, and now harness engineering. The principle behind the progression is simple. As agents have improved, we build the workflow the way a human works: understand the ticket, find the constraints, explore test cases, and verify each step before moving onto the next.

“The main idea and the main design of the agents is to complete the task. We are here to define what is the task and what done actually means.” – Ivan Davidov

How hands-off you go depends on risk. Paul’s rule: the fewer human eyes on the output, the more risk you’re taking on. A team with loose cost of change and risk-tolerant clients can pull back. Medical, aviation, and financial systems can’t. Some things an agent can’t judge at all, like whether an experience is actually fun.

Agents Want to Please You

Agents optimize for the green check mark, and they’ll take the shortest path to get it. The panel shared three examples:

• The self-passing test: In a LinkedIn post Beth cited from Rex Jones II, an agent asked to fix a broken locator changed the application’s source code instead, so the test would pass.
• The gym booking: An agent booked its user a gym session by canceling other members’ bookings and reserving six weeks out, against the gym’s policy.
• The over-literal responder: A TCP/IP responder Paul had AI build replied to every character typed, not every message.

None of these were catastrophes, all would have shipped as passing work.

“If you are blindly relying on these things and you are not being critical of the output, we risk a lot of false positives, and that being much more of an issue than it ever used to be.” – Beth Marshall

According to Paul, we fix this by defining not just done and ready, but good. What fit-for-purpose looks like, spelled out, because agents won’t add skepticism of their own.

Generated Evidence is not Executed Evidence

Ivan’s sharpest warning was technical. Agents are very good at producing evidence: reports, green runs, summaries that read like proof. That is different from evidence of tests that actually executed and asserted something.

He has seen tests with 12 steps and not a single assertion. They pass every time and verify nothing. Generated evidence builds confidence in no one, so the job is to validate every output and confirm that what ran was the right test.

The Risk Moved

For a business the ideal pipeline has always been as few people as possible seeing a ticket from creation through to completion. As agents move us closer to this, we increase the risk of missing oversight.

“We need more of the breadcrumbs. Here’s what happened, here’s what did not happen, and here’s what we wanted to see if it happened or not… Things are becoming more opaque because they’re becoming more programmatic.” – Paul Grizzaffi

When execution becomes opaque, green checkmarks aren’t proof of quality. The challenge isn’t just generating runs faster—it’s capturing those breadcrumbs before human oversight disappears completely.

What This Means for QA and Engineering Leaders

The panel pointed leaders at three areas:

• Start Small and Extend on Evidence: Pick a 10-minute task, encode how you do it, and monitor every step the agent takes. Once it succeeds about 95% of the time, extend it to the next task. Start from a working scaffold instead of from zero.
• Keep Agents Small Enough to Review: One agent that does everything produces output nobody can review. Small, focused agents with specific rules keep review manageable and help with a growing problem Beth called AI fatigue: the exhaustion of shifting from doing the work to reviewing it all day.
• Fund Enablement, Not Just Tools: People need time, training, and support to change how they work. Slower roll-outs win over skeptics with working examples, but not so slow that a competitor ships ten times faster.

Where This is Heading

Testing is moving toward developers, and not always carefully. Paul expects crank turning, running scripts and checking outputs, to go away. In its place, intent will grow. Building guardrails, reading risk, and tuning agents to do the execution.

Agents will keep getting faster. The real advantage is knowing when to trust what they tell you.

That independent verification is where platforms like Applitools fit into the harness: acting as visual and functional guardrails that provide the immutable breadcrumbs human reviewers need to confirm that a passing agent run actually delivered a working user experience.

Watch the full session now, on-demand.