Why LLMs Break in Test Execution and How to Fix It

Execution-by-intent sounds compelling: describe a test in English, and let AI drive the browser. In practice, probabilistic reasoning produces inconsistent flows, diluted assertions, and unpredictable results—especially across dynamic applications.

In this technical deep dive, Adam Carmi, Applitools Co-Founder and the inventor of visual testing, breaks down why LLM-driven execution fails at scale and what architecture replaces it. We’ll analyze LLM-driven test execution, highlight where it introduces inconsistency, and show how English-based tests can be executed deterministically to maintain coverage and repeatability.

Key takeaways:

• Why identical intent can produce inconsistent execution flows in LLM-driven systems
• Where assertions, dynamic page state, and probabilistic reasoning collide
• Why orchestration and tool-calling do not remove randomness from execution
• What a deterministic execution stack requires to scale across complex applications

play video

Please Register for Access to This Content. Return registrants will have instant access.

Related Session

AI You Can Trust: Where Determinism Belongs in Modern Testing

Watch Now On-Demand