Knowing how to deal with unstable tests is critical in testing because floating autotests slow down the speed of all development.
If you have not encountered unstable tests, be sure to read this article, as it attempts to systematize the causes of instability in tests. If you’ve encountered unstable tests, see how many fall into the listed areas.
This article aims to tell you how to deal with each of the reasons.
Over the years, I have come across unstable tests more than once, but instead of looking at specific cases, let’s try to group the causes of instability by the components involved in performing autotests:
The tests themselves;
A framework for running tests;
Services and libraries on which the system under test and the test framework depend;
The operating system and device with which the autotesting framework interacts.
Figure 1 shows the hardware / software stack that supports the application or system under test. At the lowest level is the hardware. The next level is the operating system, followed by the libraries that provide the interface to the system. At the highest level is middleware, a layer that provides interfaces to specific applications.
However, in a distributed system, each application service and the services it depends on can reside on a different hardware / software stack, just like the service running the test. This is illustrated in Figure 2 as a complete test runtime.
As discussed above, each of these components is a potential unstable test area.
The tests themselves
The tests themselves can cause instability. Typical reasons:
Improper initialization or cleaning;
Incorrectly selected test data;
Wrong assumption about the state of the system. An example is system time;
Dependency on asynchronous actions;
Depends on the order in which tests are run.
Framework for running tests
An unreliable framework for running tests can lead to instability. Typical reasons:
Failure to allocate enough resources for the system under test, which leads to its failure;
Wrong planning of tests, so they “contradict” and cause each other to fail;
Insufficient system resources to meet test requirements.
Services and libraries that the system under test and test framework depend on
The application (or the system under test) can be a source of instability
An application can also have many dependencies on other services, and each of these services can have their own dependencies.
In this chain, each service can cause unstable tests.
Slow response or no response when requested from the test;
Over-subscription to resources;
Application and test changes occur at different rates.
Testing environments are called sealedif they contain everything that is needed to run the tests (i.e. there are no external dependencies such as servers running in a production environment).
A sealed environment is less prone to instability.
Operating system and device with which the autotesting framework interacts
Finally, the hardware and operating system can be a source of test instability. Typical reasons include:
Network failures or instability;
Resources consumed by other tasks / services unrelated to the tests being run.
As you can see from the wide variety of failures, reducing instability in automated testing can be quite challenging. This article describes the areas and types of instability that can occur in these areas, so it can serve as a cheat sheet in locating the cause of unstable tests.
In the following articles, we will look at ways to solve these problems.
Links to sources
Where do unstable tests come from? (original/ translation of an article on Habré)
Unstable tests on Google and how we fix them (original)
My Selenium tests are not stable! (original)
Avoid unstable tests (original)