Test Recorders for Web Automated Tests – Part 1

When you already have a team of manual QAs or you do not want to hire test automation engineers, you may be tempted to use test recorders for automated web testing. Tread carefully, as this kind of decision will shape your QA department for a long time.

In the first part of this two part series we will focus on the capabilities of test recorders. In the second part we will jump into their compatibility with existing technology stacks.

What are Test Recorders

A test recorder is a piece of software that allows quality assurance teams to record steps that they have executed manually with their mouse and keyboard in a browser. Then, it replays them to check if your app works properly. This is why the name for this technique is called “Record and Reply”. Some commonly used tools are:

• Snaptest.io
• Ranorex
• Telerik Test Studio
• TestCafe Studio
• Ghost Inspector
• Katalon
• Crossbrowsertesting

Potential Benefits from Test Recorders

Test recorders are aimed at small teams working with small products. In theory they promise the following advantages:

• People without coding skills can create and execute automated tests
• As you do not need developers in tests, you pay less for their salaries
• You record the steps, so the test creation process should be faster compared to writing code
• You do not need to setup all parts of the test pipeline, as test recorders come with extra features, like schedulers, parallelization of test execution, and a rich presentation of the results.

Talk to the Team

First you should find a clear answer to the question “why do we need test automation?”. With such an answer in mind, go talk to your team. You will get a lot of valuable feedback from people who will work with the chosen solution everyday. Questions worth asking:

• What do they think?
• In which direction do they want to go in the long run?
• What experience do they have with test recorders?
• What is their willingness to learn new things?
• How do they imagine the implementation of test automation in your company?

Are Test Recorders Really Codeless?

There are some situations that require code. Some notable mentions:

• Check if an email has arrived in the inbox
• Send a fake email to check if it will appear in the UI
• Setup test users in the test database
• Use a SSH tunnel to connect to a database
• Download a single file from a given URL to check if the download functionality works
• Clear the Download folder of the browser
• Setup the browser to run with changed parameters
• Generate a dummy 100MB file for an upload test
• Do a cleanup routine after finishing the tests

Test recorders do not exclude the need to write code. They have some programming language under the hood. So ask questions about that language:

• Can you access, modify, and add custom code to allow non-standard operations as described above?
• Is it popular and used widely by QAs? Otherwise it may be difficult to find people who will work with it.
• Is the language already used in your company? If not, that means expanding the technology stack with another language.
• Can you use any library you want? The ability to use custom libraries and packages is a must have if you do not want to reinvent the wheel.
• Do you have people who can work with it? If not, you will need to either upgrade one of QAs to Automation QA. You will pay for license and for better skilled employees at the same time, or you’ll need to borrow developers to help the QA team, since they will always be involved in the solution’s maintenance.

Building Trust in Test Automation

Manual test cases should cover functionality as much as possible. Otherwise, things which are missed during the writing of the tests phase will not be tested for. In the long run your application will start to collect bugs in untested areas. The same rule applies even stronger to automated tests.

During manual testing, QA can be creative: derive from a test case path, or try something unexpected.

Automatic tests only follow the code. For example, imagine you have 10 automated smoke tests, testing a happy path. In the first month the tests found two critical bugs, and everybody likes automatic tests. In next month, nothing is found, and people start to feel overconfident. In the 3rd month, you get a Friday evening call from the client support department, saying in a passive aggressive manner that “when a user will enter email without @, the whole web page crashes“. You are forced to refine automatic tests, to include such user scenarios. But if these missed bugs are seen frequently, people will start to lose confidence in once beloved automated tests, and push for more manual testing, and your test automation initiative just failed.

With time, you will want your tests to be more and more detailed and cover as many cases as possible. This is where test automation should show it’s true power – its ability to test multiple scenarios in a short time.

Data Driven Testing

Test recorders test multiple scenarios with data driven testing, separating data to be used in tests from tests itself. This allows you to change data without needing to change the test. This way the tests can be easily executed with different sets of data, which increases coverage and speeds up testing multiple scenarios. What is important is how a given test recording tool achieves that. Questions to ask your team include:

• How is data for tests stored and edited?
• Can you generate data on the fly?
• How easy will it be to implement any faker library?
• Can you use this generated data in all parts of the system?
• URLs
• Object identificators like CSS class names, element IDs or Xpaths
• Contents of API requests like GET, POST (including headers)
• Assertions verifying your actions bring correct result

Believe me, you may not need to use it all today, but you may need them in the future.

Reusability & Maintainability

If you decide to reuse one element on the page (i.e. a calendar), do you need to record the test from the start? Or can you reuse parts of the existing test? It may not be troublesome if you carefully design the structure, and find common parts that can be reused.

Complexity of your Application

It is easy to record the steps for a simple process. Login usually has two steps: fill two fields form, and submit. But what if your page has a big complexity with many dependencies? Like dropdown values, dependent on a selection of certain checkboxes? You can multiply such dependencies. At some point the number of possibilities will be too big to record all of them. Even if you do, changing to a single form would require recording all tests again. Generally the more complex your app is, the harder it will be to utilize recorded tests.

Hardcoding / Refactoring Initiatives

Hardcoding values in automated tests is a generally bad practice. A test that passes today may fail tomorrow because:

• Dates valid today will not be valid tomorrow
• Test users may change
• Content of your web page will evolve

Unfortunately hardcoding is the default recording mode for all the tools. You need to do some extra work, to make them variables, using a provided solution for data driven testing.

If your tests will collect many hardcoded values, converting them into variables will take more time than if you were to use a coded solution. With a coded solution you just search in text files and replace them in your editor.

Another example of refactoring is putting commonly repeated actions into universal methods. This every day practice for solutions uses pure code, and may require extra work for recorded tests.

Level of Support for Test Recorder

Open source coding solutions usually have a decent level of community support, and lower the time to master the tool. The older a project is the more mature it is. It will have less bugs and more features. What level of support will you get from the company producing a test recorder? You can check:

• How many issues are in their bug tracker / forum?
• How long does it take them to reply to support requests?
• What does a typical conversation between user and support look like?
• What are their policies for implementing new features in the product?

Source

After considering most of the above, you should know what the capabilities and limitations of a test recorder are. In a second post in this series we will dive more into compatibility with your existing technology stack, processes and HR strategies your company has in place.