How Visual UI Testing can speed up DevOps flow [Revisited]

At its core, DevOps means applying best practices to automate the process between development and IT teams. In practice, DevOps requires automating the app builds, running tests to verify source code integrity, and automating the release process until the app is ready for deployment.

Continuous Integration / Continuous Delivery (CI/CD) makes up the heart of the DevOps workflow that supports the processes, developers, testers, and clients.

With the advent of automated visual UI testing, people went about discovering the best way to integrate Automated Visual Testing in the DevOps workflow to speed up the process.

In this article, I will share the workflow I use to integrate my source code with a CI/CD service to build the code and run the automated visual UI tests. And, I will show the end result – a full integration between the source code management system, and CI/CD, to report any discrepancies or failures in either.

The other half of this article will demonstrate a practical example of implementing a recommended Visual Testing Workflow.

While doing so, we will be covering topics like CI/CD, Source Code Management systems, and much more.

Let’s start!

Source Code Repository Management System

Where do you store your source code? Many of the external source code repository management (SCRM) systems make your code available to your team wherever they are and whenever they need access. Popular SCRM systems include GitHub, BitBucket, GitLab and Azure Repos. If you don’t already use one or more SCRM systems, I highly recommend you start

SCRM underpins DevOps source management. For example, I can add or modify code and push the changes to the centralized repository. My colleagues elsewhere would do the same thing. Developers joining the team can clone the centralized up-to-date repository to begin coding instantaneously.

SCRM systems have developed best practices and workflows. For instance, the diagram below depicts the workflow of implementing a new feature in your application.

Start by cloning or branching to the dev branch.

1. Create a new branch name relevant to the feature you are adding.
   1. Add your code and commit the changes locally.
   2. Push the branch to an SCRM system- GitHub for instance.
2. Create a Pull Request (PR) based on the new branch.
3. Invite your team members to review and comment.
4. Incorporate any changes. If need be, approve and merge the PR.
5. The branch dev has the new feature and ready to deploy.

I use this typical SCRM Workflow on a daily basis. It’s quite simple and straight to the point, yet it organizes the development process with all team members without losing code or overlapping.

Continuous Integration / Continuous Delivery (CI/CD)

Continuous Integration, or CI for short, is a fully-automated development methodology that seamlessly integrates your daily code changes into a centralized source code management system. CI ensures a high level of code integrity by enforcing your own software development process for any code changes.

When employing CI in your development workflow, creating a new PR in your SCRM system triggers the CI workflow via a webhook more often than not.

The CI workflow is depicted in this diagram:

• Start by pushing your branch to the SCRM.
• Create a Pull Request.
• Allow your colleagues to review and comment.
• At the same time, creating a  PR triggers a Build Pipeline inside the CI Provider.
• You can configure the Build Pipeline to build your code and run the tests. In addition, you can add more tasks to the workflow as you see fit.
• Finally, the CI Provider reports the results of building the code and running the tests back to the SCRM.
• The results will be displayed on the PR page to signal success or failure.

The CI results will be either a success or failure. In case of a failure, you need to review the PR and fix any issues reported. Otherwise, you can safely merge the PR into your code.

On the other hand, the Continuous Delivery, or CD, automates the process of generating a releasable package from your code. The success results of the CI process triggers a Release Pipeline inside the CD Provider. I won’t be covering CD in this article. To find out more about CD, you can read this detailed guide on CD/CI: What’s Continuous Integration and Continuous Delivery.

Visual Testing Workflow

Automated visual UI testing and CI/CD integrate well together! They complement each other and facilitate the work of DevOps.

Once you have an automated integration and delivery solution, you can easily integrate visual UI tests into your CI/CD workflow. Simply add an additional task in the workflow to run the visual UI tests. Typically, the CI Provider runs the automated tests and sends the results to the Visual Testing Provider. The Visual Testing Provider analyzes the testing results and reports them back to the SCRM on that specific PR.

The diagram below summarizes this whole workflow:

• Start by pushing your branch to the SCRM system.
• Create a Pull Request.
• Let your colleagues review and comment.
• At the same time, creating a PR triggers a Build Pipeline inside the CI Provider.
• The Build Pipeline builds your code and runs the tests (unit, integration, function and visual UI tests).
• The CI Provider reports back the results to the SCRM system.
• Meanwhile, the Visual Testing Provider collects the test run results, analyzes them and reports back success or failure to the SCRM system.

Every Visual Testing Provider supports an SCRM system integration to ease communication when running visual UI tests. For instance, Applitools offers GitHub, BitBucket, and other integrations with SCRM systems.

Let’s see this workflow in action in the demo section.

Demo

Now that we know where automated visual UI testing fits in the DevOps workflow, let’s work through an example by trying to automate the entire Visual Testing Workflow.

In this step-by-step guide, we’ll tackle the following:

1. Cloning the example code.
2. Configuring Applitools API Key.
3. Connecting the GitHub repo with CircleCI.
4. Configuring the GitHub repo.
5. Connecting the GitHub repo with Applitools.
6. Running a visual UI test locally.
7. Running the Visual UI testing workflow.

For this article, I opt to use the source code that I developed for my previous article Cross-browser Testing With Cypress.io. The code contains a single visual UI test that runs against one of the Applitools customer’s website- the https://www.salesforce.com– and fills the contact form located at https://www.salesforce.com/uk/form/contact/contactme/.

1. Cloning the example code

Start by cloning the source code that we’re going to use for this demonstration by running this command:

git clone git@github.com:bhaidar/applitools-github-circle-ci.git

The command clones the GitHub repo to your local computer. Once the code is cloned, enter the applitools-github-circle-ci directory, and run the following command to install the packages:

npm install

This command installs all the necessary NPM packages used by the source code. The app is now ready!

2. Configuring Applitools API Key

Before you can locally run Applitools Eyes Cypress SDK, make sure you get an Applitools API Key, and store it on your machine as an environment variable.

To set the APPLITOOLS_API_KEY environment variable, you can use the export command on Mac or set command on Windows as follows:

3. Connecting the GitHub repo with CircleCI

For this tutorial, we’re going to integrate with our GitHib repo with CircleCI. If you’re not familiar with CircleCI, check out their documentation and sign up for a free account with GitHub or Bitbucket.

Once you’ve created an account and logged in to the CircleCI website, click the Add Projects menu item.

Search and locate your GitHub repo and click the Set up Project button as shown in the diagram.

You are then redirected to the Project Settings page to complete the set up of your project.

On the Project Settings page, select the Linux operating system option and Node as a language. Scroll down the page until you see:

The CircleCI team offers you a step by step guide on how to integrate CircleCI into your application. Make sure you follow the steps listed as shown here:

1. Create a folder called .circleci and add a file called config.yml
2. Populate the config.yml file with the sample on the page
3. Update the sample .yml to reflect your project’s configuration
4. Push the change to GitHub.
5. Start building! This will launch your project on CircleCI and its webhooks will listen for updates to your work.

Switch back to your code editor and let’s add the CircleCI requirements.

3a. Build in GitHub

Start by creating the dev git branch by running the commands below:

git checkout -b dev
git push --set-upstream origin dev

Then create a new folder named .circle at the root folder of the app. Then create a new file inside the .circleci folder named config.yml.

The CircleCI provides a sample config.yml file that you can use to start working with CircleCI. For now, paste the following inside the config.yml file:

The “config.yml” defines a single build job. The build job defines some general configurations and some steps.

This job makes uses of the cypress/base:10 Docker image. It also defines the working directory to be the name of the app itself.

The build job steps are:

1. Checks out the source code to the working directory inside the Docker container.
2. Restores the cached NPM packages if any.
3. Installs all NPM package dependencies located in the package.json file.
4. Stores the packages inside the cache to retrieve and use them later.
5. Runs a multi-line command as follows:
   1. Exports the APPLITOOLS_BATCH_ID to the CIRCLE_SHA1 auto-generated value at runtime.
   2. Runs the single visual UI test with Cypress.io

It’s very important to include both commands, exporting the APPLITOOLS_BATCH_ID  environment variable, and running the visual UI test, in the same command inside the config.yml file.

3b. Integrate with Applitools

In addition, we need to amend the applitools.config.js file to include the batchId. The applitools.config.js file should now look as follows:

module.exports = {
    showLogs: false,
    batchName: 'Circle CI batch',
    batchId: process.env.APPLITOOLS_BATCH_ID
}

One task remains – adding the APPLITOOLS_API_KEY environment variable on the CircleCI project level, so the visual UI tests can run and connect with Applitools servers. You have the choice to either place this environment variable inside the config.yml file or add the environment variable on the CircleCI website. The latter is best and the recommended option so you don’t share your API on GitHub. You simply embed it at the CircleCI project level.

1. Locate and click the Jobs menu item on CircleCI.
2. Near the project name, click the gear icon.
   
3. Click the Environment Variables menu item.
4. Click on the Add Variable button.
5. On the popup modal window, fill in the following fields:
   1. Name: The name of the environment variable in this case APPLITOOLS_API_KEY.
   2. Value: Copy the API Key from the Applitools website and paste it here.
6. Click the Add Variable button to add this environment variable.

3c. Push Changes

Now let’s push these changes to our local copy of source code to GitHub. Automatically, CircleCI will detect a change in the repo and will run the test job. Consequently, the Applitools visual UI test will run.

Run this commands to push your changes to GitHub:

git add .
git commit -m "Add CircleCI support to app"

Switch back to CircleCI and check the status of the job.

The diagram lists all the steps as defined in the config.yml file. CircleCI runs each step in the same order it appears in the config file.

The job fails! Let’s expand the last step to find out why.

If you read through the logs you will notice the following:

CypressError: Timed out retrying: Expected to find element: 'input[id="reg_form_1-UserFirstName"]', but never found it.

Cypress.io has generated an error as it cannot find an input with ID reg_form_1-UserFirstName on the contact form.

The SalesForce team has changed the IDs of the contact form since I wrote the visual test script in my previous article!

Don’t worry, we will fix this in the coming sections. More importantly, we have linked our GitHub repo with CircleCI.

Let’s move on and tackle other tasks.

4. Configuring the GitHub repo

Before we proceed, let’s configure our GitHub repo with the best practices.

1. Locate the GitHub repo and go to the Settings page on GitHub.
2. Click on the Branches menu section.
3. Click the Add rule button to define the approval on new code pushed to this repo.
4. Enter the name master into the Branch Name Pattern field. The rules we are going to add target only the master branch in this repo. You are free to decide on the branch you want.
5. Usually, I opt to check the Require pull request reviews before merging option. For the sake of this demonstration, and since I am the only developer working on this code, I will keep it unchecked.
6. Make sure to check the Require status checks to pass before merging option. This ensures CircleCl allows you to merge a PR into your master branch after a success status outcome.
7. Check the Require branches to be up to date before merging option, and then select the ci/circleci: build checkbox. In other words, you are specifying the CircleCI job to listen to for status checks.
8. Locate and click the Create button to add this rule.

That’s it for GitHub!

Let switch to CircleCI and configure one option there.

1. Locate and click the Jobs menu item on CircleCI.
2. Click the gear icon on the top-right side of the page.

1. Locate and click the Advanced Settings menu item.
2. Scroll down the page until you hit the option Only build pull requests, and make sure to select On. This way, CircleCI build process will be executed only when a new PR is created, and not on every push to the master branch.

That’s it for CircleCI!

5. Connecting the GitHub repo with Applitools

So far, we have managed to integrate CircleCI and GitHub successfully. Any status update on running the build job on CircleCI will be reflected on the GitHub repo.

Let’s now integrate GitHub with Applitools so that our GitHub repo will receive any status updates on whether the visual UI tests ran successfully or failed.

Go to the Applitools Test Manager website and log in with your GitHub credentials.

1. Navigate to the Admin page.
2. Locate and click the Teams tile.
3. In my case, I am the only member of the team. I click the row that represents my own record.
4. Click the Integrations tab.
5. Locate and click the GitHub option.
6. Click the Manage repositories button.
7. Sign in to your GitHub account again as requested. Click the Sign button and authorize Applitools to access your GitHub account.
8. Locate your GitHub repository and click Apply.

That’s all! Now any status updates on visual UI tests on the GitHub repo you’ve selected will appear right away.

Now, if you navigate back to the GitHub repo Settings page and click the Webhooks menu item, you will see two Webhooks were added to your repo. One for CircleCI, and another for Applitools.

6. Running a visual UI test locally

In this section, I will be fixing the single visual UI test script that I have by changing the ID names for all the fields on the SalesForce contact form so that the test runs successfully.

As mentioned previously in my articles, the recommended approach for doing visual testing with Cypress.io is to assign data-* attributes to fields. Such data attributes never change, and provide a consistent way to keep your visual tests running.

Since we don’t have control over the source code behind the contact form, I will simply create a new Git branch and fix the visual test code to reflect the new IDs being used.

Run the following commands to create a new branch in Git:

git checkout -b amend-visual-test-to-use-new-ids
git push --set-upstream origin amend-visual-test-to-use-new-ids

I’ll go through the script code and change the way I find input fields on the form:

// Fill First Name
cy.get('input[id="reg_form_1-UserFirstName"]')
.type('John')
      .should('have.value', 'John');

To something like this:

// Fill First Name
cy.get('input[id^="UserFirstName"]')
.type('John')
.should('have.value', 'John');

The new script code now looks like this:

Let’s give it a shot and run the single test, included in the source code, locally, and make sure it runs.

To run the test, we need first to open the Cypress Test Runner app by running this command:

npx cypress open

Click the salesforce-contactus.spec.js file and Cypress will run the test for you. The result is shown below:

The test runs successfully! Let’s make sure the test results are displayed on the Applitools Test Manager website.

Let’s visit the Applitools Test Manager, and verify the results.

Great! Our test runs fine.

7. Running the Visual UI testing workflow

Now that our test runs successfully locally, what we need to do is push our new branch to GitHub, create a new PR and watch how CircleCI and Applitools will be triggered. Their status updates will appear on the PR itself on GitHub.

Run the following commands to push the new branch to Github:

git add .
git commit -m "Fix visual test script"
git push

Navigate to the GitHub repo, and create a new PR.

Notice how the CircleCI build pipeline has started:

ci/circleci:build Pending -- CircleCI is running your tests

Let’s wait for CircleCI to finish running the job and for Applitools to report the results of running the tests on this PR:

Both CircleCI and Applitools report success on running the single visual UI test!

Congratulations!

• The ci/circleci:build job ran successfully
• The scm/applitools job that compares the git branch to the baseline git branch ran successfully
• The tests/applitools job that runs all the visual UI tests ran successfully

Only now you can go ahead and merge the PR into the source code.

Conclusion

This article is just the tip of the iceberg on how Applitools automated visual UI tests can seamlessly integrate with a CI engine to support and facilitate the DevOps workflow.

Now that you have a starting point, I will leave the floor to you. Explore,play and configure! Be brave and try more advanced workflows.

Other Related Content

• Know The Answers, Get The Job
• How Cross-Browser Testing Is Evolving
• Angular and Cypress UI Testing with Applitools
• How to visually test VueJS apps using Cypress.io and Applitools [step-by-step tutorial]
• React, Cypress and Applitools for Functional and Visual Testing
• Learn Cypress Testing on Test Automation University
• More blog posts by Bilal Haidar
• Request an Applitools demo
• Sign up for a free Applitools account
• Check out the Applitools tutorials.