Unstable tests are one of the main problems in automated testing (Part 2)

This is the continuation of a series of articles on unstable tests.

In the first article (original/ translation into Habré) it was said about 4 components in which unstable tests can occur.

In this article, we will give tips on how to avoid unstable tests in each of the 4 components.

Components

So there are 4 components in which unstable tests can occur:

  • The tests themselves;

  • A framework for running tests;

  • Services and libraries on which the system under test and the test framework depend;

  • The operating system and device with which the autotesting framework interacts.

This is depicted in Figure 1.

The reasons, options for localizing the problem and options for resolving instability are discussed below.

The tests themselves

The tests themselves may be unstable.

The reasons can be in test data, autotest scripts, autotest preconditions, and initial states of other dependencies.

Table 1 – Reasons, options for localizing the problem and options for solving instability in the tests themselves.

Reasons for unstable tests

Problem localization options

Solution options

Incorrect initialization or clearing.

Look for compiler warnings about uninitialized variables. Check your initialization and cleanup code. Check that the environment is set up and cleaned up correctly. Make sure the test data is correct.

Explicitly initialize all variables with correct values ​​before using them. Configure and clean up the test environment correctly. Make sure the first test does not harm the state of the test environment.

Incorrectly selected test data.

Restart the tests yourself.

Make tests independent of any state from other tests and previous runs.

Wrong assumption about the state of the system. An example is the system time.

Check application dependencies.

Remove or isolate your application’s dependencies from aspects of the environment that you have no control over.

Runtime dependencies, waiting for asynchronous events to appear in a specific order, waiting without timeouts, or race conditions between tests and the application.

Log the time when the application was accessed. As part of debugging, add delays to check the results of the Autotest run.

Add sync items to your tests so that they wait for specific application states. Disable unnecessary caching to have a predictable timeline for application responses. DO NOT ADD clear expectations, this may lead to test instability in the future.

Dependence on the order in which the tests are run (The solution is similar to the second reason).

Restart the tests yourself.

Make tests independent of any state from other tests and previous runs.

A framework for running tests

An unreliable framework for running tests can lead to instability

Table 2 – Reasons, options for localizing the problem, and options for solving instability in the framework for running tests

Reasons for unstable tests

Problem localization options

Solution options

Failure to allocate enough resources for the system under test, resulting in a failure.

Check the logs to see if the application has appeared.

Allocate enough resources.

Wrong planning of tests, so they “contradict” and cause each other to fail.

Run tests in a different order.

Make tests independent of each other.

Insufficient system resources to meet test requirements (Similar to the first case, but here resources are consumed during the execution of the workflow).

Check your system logs to make sure you haven’t run out of resources.

Fix memory leaks or other resource leaks. Allocate enough resources to run your tests.

Services and libraries on which the system under test and test framework depends

The application (or the system under test) can be a source of instability.

An application can also have many dependencies on other services, and each of these services can have their own dependencies.

In this chain, each service can cause unstable tests.

Table 3 – Reasons, options for localizing the problem, and options for solving instability in the application or the system under test

Reasons for unstable tests

Problem localization options

Solution options

Race condition.

Log access to shared resources.

Add sync items to your tests so that they wait for specific application states. DO NOT ADD clear expectations, this may lead to instability of tests in the future.

Uninitialized variables.

Look for compiler warnings about uninitialized variables.

Explicitly initialize all variables with correct values ​​before using them.

Slow or no response when requested from a test.

Log the times when requests and responses are made.

Check and correct all causes of delays.

Memory leaks.

Take a look at the memory consumption while running the benchmarks. The Valgrind tool will help you find the problem.

Correct the program error causing the memory leak. In this article on wikipedia there is a great description of these error types.

Over-subscription to resources.

Check the logs to see if the resources have run out.

Allocate enough resources to run your tests.

Changes to the application and tests occur at different rates.

Examine the history of changes.

Enter the rule when you change the code, write tests for it.

Operating system and device with which the autotesting framework interacts

Finally, the hardware and operating system can be a source of test instability.

Table 4 – Reasons, options for localizing the problem, and options for solving instability in the OS and the device with which the autotesting framework interacts

Reasons for unstable tests

Problem localization options

Solution options

Network failures or instability.

Check for errors in the system logs.

Fix hardware errors or run tests on different hardware.

Disk errors.

Check for errors in the system logs.

Fix hardware errors or run tests on different hardware.

Resources consumed by other tasks / services unrelated to the tests being run.

Examine the activity of the system process.

Reduce the activity of non-test run processes.

Conclusion

As you can see from the wide variety of failures, reducing instability in automated testing can be quite challenging. This article describes the areas and types of instability that can occur in these areas, so it can serve as a cheat sheet in locating the cause of unstable tests.

Links to sources

  • Where do unstable tests come from? (original/ translation of an article on Habré)

  • Unstable tests on Google and how we fix them (original)

  • My Selenium tests are not stable! (original)

  • Avoid unstable tests (original)

  • Unstable tests are one of the main problems of automated testing (original/ translation of an article on Habré)

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *