How to Execute Effective Automated Tests

Automated tests can help to speed up delivery, reduce errors, and save testing engineers time to run comprehensive and complex tests. However, before automating all test cases, it is necessary to make a rigorous assessment of what can be automated, when, how, and what cost and effort is required to achieve the goal. In this article, we will discuss and clarify what testers need to consider when they are choosing what should be automated or not.

What is Automated Testing?

Automated testing is a tool used to take care of repetitive tasks, receive quick feedback, and accomplish other difficult or time consuming testing tasks. Automated testing is used to improve the quality of the final product by reducing the number of errors displayed to the user. End-to-end or UI testing is one of the many approaches for testing a web application.

Why Do We Automate Tests?

As we mentioned above, we automate to avoid human error by executing repeated manual work, get faster feedback, and ensure that our tests are consistent. However, you cannot automate all tests right off the bat. Automating all test cases requires a lot of effort, results in increased costs, and this takes much more time than unit or integration tests. Generally you would automate only once your platform is stable and the user flow is very consistent. Automating every scenario and behaviour is not feasible. The main reason that we automate a test is repeatability. Before automating your next test case, step back and estimate how often you will run this test.

Another thing that is very important in Agile development practices is exploratory testing. Its purpose is not only to find bugs, but also to generate ideas on improving testing and the user experience. Exploratory testing helps to identify functional defects quickly and increase involvement and collaboration. So, you need to identify which test cases are worthy of automation.

Here are some points to consider before starting to write automated tests:

Tests that should be automated:

High risk – business critical test cases – features which may cause considerable damage to the company if they deviate from the business rules.
Tests that check fixed bugs won’t reoccur.
Regression or repetitive testing point of view.
Tests that need to be run on multiple browsers or environments.
Performance testing like load and stress tests.
Tests that utilize the same workflow but different data for its inputs for each test run, such as data-driven and boundary tests.
Functional tests that take a long time to perform and may need to be run during breaks or overnight.

Tests that should not be automated:

Test cases that are newly designed and not executed manually at least one time, except when you have TDD or BDD styles.
Test cases for requirements that change frequently.
Test cases which are executed on an ad-hoc basis, such as exploratory testing.
Test cases that require visual confirmation such as face ID.

In the Agile Testing book, Lisa Crispin and Janet Gregory mention some reasons about the importance of automated testing such as:

Manual testing is very slow;
Reduce Error-Prone testing tasks;
Save time to perform other types of test;
Provide some security, if we make changes in the code, we have tests that they can tell us what we broke;
Provide early and frequent feedback;
The tests provide a good documentation;
ROI, over time the automation effort pays off;
Automation myths – e.g. always return good quality. Basically it is based on many other factors such as skills, training, etc.

What is Continuous Integration and Continuous Delivery?

In software engineering, Continuous Integration (CI) is a practice where developers and testers frequently integrate code into a shared repository. Each commit can be verified by automated tests and an automated build. One of the key benefits of Continuous Integration is that you find errors more quickly and can investigate more easily.
It is not possible to talk about CI without mentioning Continuous Delivery (CD). CD is the practice that ensures the product is always ready to be deployed. So, continuous delivery actually requires continuous integration. Finally, reducing complexity and costs is a key benefit of applying a CI/CD pipeline.

When Do I Need to Run My Automated Tests?

You should run automated tests all the time or as many times the team deems necessary. In fact, CI servers like Jenkins, GitLab CI or CircleCI keep checking their version control at regular intervals and can run the entire suite of tests when they detect any file was changed. You can also decide if the slowest tests (i.e. end-to-end tests) will be done one time per day such as nightly builds.

The login feature test cases are a good candidate to be automated – it is not a page with frequent changes, and it is easy to start your automation project. Nearly all automated tests that are covering login functionality could be run before merging new code into code base.

Most CI servers have plugins that help to improve code coverage. There are some tools like SonarQube that not only provide code coverage but also evaluate clean code practices, technical debt, etc. Having nothing automated is bad, but having 100% of automation may not be a good idea. For example, if you are using Java language, what is the value of automating all gets and sets of the application?

Conclusion

Test automation is a medium and long-term investment and effort. For example, if you are generating assets manually that can take days. Using CI/CD that time can be reduced to minutes. Also, to ensure you are saving money you can use ROI to help calculate the efficiency of any investment. It’s a powerful way to test the efficiency of test automation. As a matter of fact, it helps to evaluate the overall quality of the product.

Test automation, when used correctly, can bring several benefits to an organization. These are obvious. Automation provides the ability to verify and react to ever-changing business requirements and generate new tests continuously.

However, it is important to consider the cases in which automation should be used. It is necessary to analyze each project to define which ones are really worth being automated and consider that, even when choosing to use a test automation tool, it is still necessary to use manual tests for several functionalities. In simple terms, the best technical approach implies knowing which tests should be automated by the team and which should remain as manual. Activities that are repeated most often and are the most intensive are the best candidates for automation. I hope this article has helped you get an idea about how to start the automation journey.

This post was published under the Quality Assurance Community of Experts. Communities of Experts are specialized groups at Modus that consolidate knowledge, document standards, reduce delivery times for clients, and open up growth opportunities for team members. Learn more about the Modus Community of Experts program in the article Building a Community of Experts.

References

Posted in Agile, Quality Assurance

Pedro Hyvo

Pedro is a QA Automation Engineer at Modus Create with over six years of experience in testing web-based applications using Java, JavaScript, and Ruby languages. He has an MBA in Project Management with international certifications in QA. During his free time, Pedro loves to travel, run, watch soccer games, and hang out with family and friends.