Test Automation Framework: Build vs. Use vs. Buy

A guide to help you get started with test automation and understand how to choose the best test automation solution for your team.

Automated testing is a solution to produce high-quality, robust, and reliable Web, Mobile, or Desktop applications or APIs amid the ever-growing complexity of technology, massive competitive pressures, and the need for cost optimization.

Writing Web, Mobile App, API, or Desktop tests is not just nice to have anymore. In different companies, the responsibility of test automation is shared between developers and test engineers to increase productivity and collaboration, especially if they are working in an Agile environment.

And also if you are planning to implement CI/CD pipelines in your team, you need to know that Test Automation now is a vital part of this process and without it, you can’t release your products frequently.

Because of that, in this article, I’d like to share my thoughts from my previous experience about getting started with a Test Automation Framework. Shall we build our own frameworks? Use existing open-source projects/frameworks/engines? Buy a commercial tool or framework?

Before you start thinking about Test Automation Framework, you need to be ready as a team with different inputs/requirements in your Test Strategy.

These include:

• What are our needs as a testing team from this framework/tool?
• What are our goals or targets, maybe including the ROI calculation or how we can measure our success with it?
• What are the technical skills of your team?
• Who will write the test scripts (Test Engineers, Developer, or both)?
• Do we have specific business cases you need to consider while selecting a framework or tool?

If we know the answers to the above questions you will decide easily which option is the suitable one for your team.

Sounds interesting! Let’s get started.

Let’s assume that we have 3 teams, each one of them will pick one solution and we will discuss the advantages and disadvantages of each solution.

I usually imagine building a test automation framework like a LEGO game. You have different tools, languages, design patterns, layers, configuration, and dependencies like LEGO pieces and you need to decide which pieces you need to build your own framework.

Image source – https://unsplash.com

With Lego you can build any shapes or blocks, you can architect anything you need and you will have different results.

Team 1: Build Framework

Let’s first clarify: what is the meaning of build?

Build means to build a customized framework in-house based on the open-source testing libraries or SDKs such as Selenium WebDriver, Cypress, Espresso, XCUITest, EarlGrey, Rest Assured, K6, etc.

The idea behind building a test framework from scratch is based on your requirements, your needs, or your target from it to be able to use these pieces correctly.

Image Source – https://www.pngitem.com

From my perspective, there is no template or must-have features for the test framework (ex: you must use BDD) but you always should implement what matches your needs, requirements, and your goal.

So the goal here keep it simple, reusable, reliable, and scalable:

An example of a custom TAF features

The Advantages:

• Customized framework, without extra features and implemented to cover our functional, edge, and critical test cases.
• Built from scratch and we know every single piece in the framework
• Integrated with our tech stack.
• If there is a problem or something in our app that needs custom development, test engineers can ask the developers for help.
• The maintenance responsibility and the ownership credits are owned by the team.
• If it will be a native framework built on top of native testing libraries such as Espresso, XCUITest, or EarlGrey for mobile test automation, the mobile engineers can help and write tests scripts as well because it’s the same context and tech stack.
• If the team or the company has code standards we can use them when building the framework to avoid any future risk or audit issues.

The Disadvantages:

• Need a good level of experience from the testing team to be able to implement a reusable and scalable framework including mobile and web development skills.
• Need constant maintenance from the team which requires time and effort.
• Implement good and up-to-date documentation with the framework structure, architecture, and roadmap to be easy to onboard any new joiner in the team.
• It will take time till you have a good version you can rely on but for the long-term, it’s a great investment in the team.

Team 2: Use Open-Source Framework/Project

“Don’t reinvent the wheel” – maybe you hear this sentence when you think about building your own framework. In other words, don’t build it but use an existing open-source framework, project, or engine.

Image source – https://www.reviewgeek.com/

But the question here is, is this beneficial?

Open-Source frameworks or engines are great ideas. It saves the team time and effort, you can easily integrate this framework inside your app or your testing project and start writing the automated tests immediately.

The Advantages:

• Free and ready to use immediately.
• Open-sourced which means you can fork (make your own copy) and implement new features or make changes in the existing features.
• Flexibility in supporting multiple programming languages.
• Saving team’s time and effort and you can calculate the ROI after a short time.
• Not required a lot of resources (test engineers), because it includes all the features your team needs, so the other test engineers can work on different tasks or tickets.
• If the framework/engine is popular and a lot of companies are using it, you will find support or answers for your questions in case you face issues from the community.

The Disadvantages:

One of the biggest disadvantages I can see in the open-source projects is the maintenance. For example you can open an open-source project on GitHub today and you can find the following notifications

So this is a big issue in my opinion to be honest, because maybe you and your team are relying on this project/framework and suddenly the owner and the maintainers become busy or decide to stop supporting or investing time in this repository.

So if you are using this project you should be able to fork it and be ready to fix any issues or implement new features because there is no one who will do this for you. This requires high technical knowledge and skills from your team to be able to tackle this issue.

Also, remember that most open-source tools are created by people for a specific reason and when that reason is fulfilled, then the support for that tool can wane.

And sometimes if we are working on a banking app or system maybe it will be a difficult decision to select an open-source framework because of the security or audit process in the company.

Other questions you should consider as a team:

• How long does it take to fix the opening issues from the owner or the maintainers?
• How long does it take when you request new features?
• Is the framework scalable, reliable, and easy to use?

How to select the right test automation framework that matches your business and technical needs?

Actually, this is always a common question when we start thinking about applying test automation frameworks/tools in our company or team. Which criteria should we consider? Which functions or features? etc.

And the best answer, in this case, is always: “It depends”.

Because that I made a list of options to help you to select the right tool. Based on the key criteria you can decide which tool will help you in test automation (for sure you can add or remove any criteria because not all of them are required):

• Ease of developing and maintaining the scripts
• Ease of test execution
• Intuitive Test Report generation
• Cross-browser Testing (web)
• Emulators, simulators, or physical devices (mobile)
• Support DDT, BDD, and maybe keyword testing
• Ease of support from the community and the maintainers.
• Ease of integration with CI/CD tools (Self-hosted like Jenkins or cloud-based such as Bitrise)
• Support for visual testing tools such as Applitools
• Support cloud Testing tools
• Support Docker and/or Kubernetes

In the end, you will check which framework/tool will fit your business needs and requirements then you can start with a POC (proof of concept) project to apply this tool to the most important business test cases and verify your selection and the ROI.

Team 3: Buy a Commercial Tool

From the first impression, you feel like it’s the easiest choice, but actually, it’s harder than the previous options because it’s related to the team budget, money, and cost.

Image source – https://zusammengebaut.com/

When the team decides to buy a framework or tool to use in test automation, this is because of different reasons such as:

• There is a good budget for test automation and they need to use it.
• The team doesn’t have the required technical skills to build their own framework or use an open-source project, so this tool will help them to write codeless test scripts easily and quickly.
• They need to implement or write automated test scripts as soon as possible because there is a kick-off for a new project and they should automate the business test cases.
• There are not enough test engineers for the time being in the company and this tool will not require big effort.
• Sometimes these tools/frameworks contain exclusive features needed in their business and it will take a long time to implement them in-house.

Advantages:

• Don’t need an in-house skill set to get up and running.
• Good documentation, tutorials, webinars, blogs, and training.
• There is a support team that will help you in case there are issues or questions from the team.
• Write and run automated test scripts quickly and easily.
• Unified platform for all types of testing (mobile, web, API, and Desktop).
• Seamless CI/CD Integration with different platforms.
• Support parallel execution to speed up as you grow.
• When we buy it, we will have a license which is a good point if we are using an audit in our company to avoid any future security or data issues.

Disadvantages:

• The biggest disadvantage of commercial tools has to be the cost
• Sometimes it’s expensive and there is no flexibility in the pricing packages.
• Sometimes not suitable for startups or small companies.
• Not all of them are scalable for the large engineering teams.
• Not suitable for specific business for example not supporting Salesforce, Blockchain, or ERP applications.
• Sometimes they are relying on record and playback which is hard to maintain in the future.
• Sometimes because it’s not required technical skills from the testing team, maybe this will affect their learning curve in the future

The Decision

Always remember “there is no silver bullet” or “magic wand” – you always need to find the suitable solution for your needs. There is no best automation tool or framework, each one has advantages and disadvantages and it always depends on what you expect from this framework. It’s differentiated from one business to another and from one team to another.

In the end, maybe you find that you can have a balanced approach by utilizing in-house, open-source and commercial all together for your Automation Framework, or you might just select one of them.

Conclusion

Test Automation is a journey, not a destination. Continuous learning, improvement, and practicing is required, and maybe the current suitable solution does not fit for your future plans.

Thank You!

Good luck and happy testing!