Why LLMs Break in Test Execution and How to Fix It
Execution-by-intent sounds compelling: describe a test in English, and let AI drive the browser. In practice, probabilistic reasoning produces inconsistent flows, diluted assertions, and unpredictable results—especially across dynamic applications.
In this technical deep dive, Adam Carmi, Applitools Co-Founder and the inventor of visual testing, breaks down why LLM-driven execution fails at scale and what architecture replaces it. We’ll analyze LLM-driven test execution, highlight where it introduces inconsistency, and show how English-based tests can be executed deterministically to maintain coverage and repeatability.
Key takeaways:
- Why identical intent can produce inconsistent execution flows in LLM-driven systems
- Where assertions, dynamic page state, and probabilistic reasoning collide
- Why orchestration and tool-calling do not remove randomness from execution
- What a deterministic execution stack requires to scale across complex applications
Execution-by-intent sounds compelling: describe a test in English, and let AI drive the browser. In practice, probabilistic reasoning produces inconsistent flows, diluted assertions, and unpredictable results—especially across dynamic applications.
In this technical deep dive, Adam Carmi, Applitools Co-Founder and the inventor of visual testing, breaks down why LLM-driven execution fails at scale and what architecture replaces it. We’ll analyze LLM-driven test execution, highlight where it introduces inconsistency, and show how English-based tests can be executed deterministically to maintain coverage and repeatability.
Key takeaways:
- Why identical intent can produce inconsistent execution flows in LLM-driven systems
- Where assertions, dynamic page state, and probabilistic reasoning collide
- Why orchestration and tool-calling do not remove randomness from execution
- What a deterministic execution stack requires to scale across complex applications
Applitools is the world’s most intelligent test automation platform. See it in action with a personalized demo.
Expert Speakers
Applitools Co-Founder & CTO
Adam Carmi is the Co-Founder and CTO of Applitools, and the original inventor of Visual Testing—a breakthrough that helped redefine how teams validate modern UIs. With more than two decades of leadership in software engineering and innovation, Adam has been at the forefront of test automation’s evolution, building AI-powered solutions used by leading enterprises worldwide. He holds B.Sc. and M.Sc. degrees in Computer Science from the Technion and is a respected voice in the testing community, regularly speaking at conferences, webinars, podcasts, and meetups around the globe.