Give Your Optimizely Devs (and Agents) Visual Guardrails to Prevent Bugs. Save your spot for the Oct. 29th webinar
Give Your Optimizely Devs (and Agents) Visual Guardrails to Prevent Bugs. Save your spot for the Oct. 29th webinar
I’m taking Raja Rao’s course, Modern Functional Testing with Visual AI, on Test Automation University. This course challenged my perspective on software testing. I plan to summarize a chapter of Raja’s course each week to share what I have learned.
The first chapter compares modern functional testing with Visual AI against legacy functional testing with coded assertions of application output. Raja states that Visual AI allows for modern functional testing while using an existing functional test tool that relies on coded assertions results in lost productivity.
To back up his claim, Raja shows us a typical web app driven to a specific behavior. In this case, it’s the log-in page of a web app, where the user hasn’t entered a username and password, and yet selected the “Enter” button, and the app responds:
“Please enter username and password”
From here, an enterprising test engineer will look at the page above and say:
“Okay, what can I test?”
There’s the obvious functional response. There are other things to check, like:
You can write code to validate that all these exist. And, if you’re a test automation engineer focused on functional test, you probably do this today.
Raja shows you what this code looks like:
There are 18 total lines of code. There is a single line of navigation code to go to this page. One line of directs the browser to click the “Sign In” button. The remaining 16 lines of code assert that all the elements exist on the page. Those 16 lines of code include 14 identifiers (IDs, Name, Xpath, etc.) that can change, and 7 references to hard-coded text.
You might be aware that the test code can vary from app version to app version, as each release can change some or all of these identifiers.
Why do identifiers change? Let’s describe several reasons:
So, in some cases, existing test code misses test cases. In others, existing test code generates an error even when no error exists.
If you’re a test engineer, you know that you have to maintain your tests every release. And that means you maintain all these assertions.
Let’s walk through all the tests for just this page. We covered the negative condition for hitting “Sign In” without entering a username or password. Next, we must verify the error messages and other elements if one, or the other, field has data (the response messages may differ). Also, we need to handle the error condition where the username or password is incorrect.
We also have to worry about all the cases where a valid login moves to a correct next page.
Okay – lots of tests that need to be manually created and validated. And, then, there are a bunch of different browsers and operating systems. And, the apps can run on mobile browsers and mobile devices. Each of these different instances can introduce unexpected behavior.
What are the implications of all these tests?
Raja points out the key implication. Workload. For every team besides QA, it matters to make the app better – which means changes. A/B testing, new ideas – applications need to change. And every version of the app means potential changes to identifiers – meaning that tests change and need to be validated. As a result, QA ends up with tons of work to validate apps. Or, QA becomes the bottleneck to all the change that everyone else wants to add to the app.
In fact, every other team can design and spec their change effectively. Given the different platforms that QA must validate, the QA team really wants to hold off changes. And, that’s a business problem.
What would happen if QA had a better way of validating the output – not just line by line and assertion by assertion? What if QA could take a snapshot of the entire page after an action and compare that with the previous instance?
QA Engineers have desired visual validation since browser-based applications could run easily on multiple platforms. And, using Applitools, Raja demonstrates why visual validation tools appeal so much to the engineering team.
In this screen, Raja shows that the 18 lines of code are down to five, and the sixteen lines of validation code are down to three. The three lines of validation code read:
https://gist.github.com/batmi02/b5174f538e13e3226dba7fac61fc2afc
So, we have a call to the capture session to start, a capture, and the close of the capture session. None of this code refers to a locator on the page.
Crazy as it may seem, visual validation code requires no identifiers, no Xpath, no names. Two pages with different build structures but identical visual behavior are the same to Applitools.
From a coding perspective, test code becomes simple to write:
You can open a capture session and capture multiple images. Each will be treated as unique images for validation purposes within that capture session.
Once you have an image of a page, it becomes the baseline. The Applitools service compares each subsequently captured image to the baseline. Any differences get highlighted, and you, as the tester, identify the differences that matter as either bugs or new features.
The big benefit of visual validation with Visual AI comes from comparing new code with new features to old code.
When Raja takes us back to his example page, he now shows a new version of the login page which has several differences – real improvements you might look to validate with your test code.
And, here, there are bugs with the new code. But, does your existing test code capture all the changes?
Let’s go through them all:
So, to summarize, your existing tests catch two bugs (different locators for the username and password field), miss two bugs (the broken logo and the text and alert overlap), and don’t have anything to say about the new feature. You have to modify or write three new tests.
But wait! There’s more!
With Visual AI in Applitools, you actually capture all the important changes and correctly identify visual elements that remain unchanged, even if the underlying page structure is different.
Visual AI captures:
So, note that Visual AI captures changes in the visual elements. All changes get captured and identified. There is no need to look at the test code afterward and ask, “what did we miss?” There is no need to look at the test code and say, “Hey, that was a false positive, we need to change that test.”
With Visual AI, you no longer have to look at the screen and determine which code changes to make. You are asked to either accept the change as part of the new baseline or reject the change as an error.
How powerful is that capability?
Well, Raja makes a comparison of the work an engineer puts in to do validation using legacy functional testing and functional testing with Visual AI.
With legacy functional testing:
In summary, you could spend 60+ minutes, or 3,600+ seconds, for all this work.
In contrast, automated visual validation with Visual AI does the following:
So, 10 seconds for Visual AI. Versus 3,600 for traditional functional testing. 360X faster.
I would think that a productivity gain of 360X might appear unreasonable. So did Raja. When he went through the real-world examples for writing and maintaining tests, he came up with a more reasonable-looking table.
For just a single page, in development and just a single update, Raja concluded that the maintenance costs with Visual AI remain about 1000x better, and the overall test development and maintenance would be about 18x faster. Every subsequent update of the page would be that much faster with Visual AI.
In addition, Visual AI catches all the visual differences without test changes and ignores underlying changes to locators that would cause functional tests to generate false positives. So, the accuracy of Visual AI ends up making Visual AI users much more productive.
Finally, because Visual AI does not depend on locators for visual validation, Visual AI ends up depending only on the action locators – which would need to be maintained for any functional test. So, Visual AI becomes much more stable – again leading to Visual AI users being much more productive.
Raja then looks at a more realistic app page to have you imagine the kind of development and test work you might need to ensure the functional and visual behavior of just this page.
For a given page, with calculators, data sorting, user information… this is a large amount of real estate that involves display and action. How would you ensure proper behavior, error handling, and manage update?
Extend this across your entire app and think about how much more stable and productive you can be with Visual AI.
Raja summarizes the chapter by pointing out that visual validation with Visual AI requires only the following:
That’s it. Visual AI catches all the relevant differences without manual intervention. Using Visual AI avoids both the test development tasks for ensuring that app responses match expectations, and it eliminates the more difficult steps of maintaining tests as part of app enhancement.
Next Up – Chapter 2.
