The Three Deadly Sins of Autotests
In recent years, of all the irons, they only say that without autotests, nowhere. Any self-respecting developer should write tests. Any tester who aspires not to be on the sidelines of life should write autotests. What distinguishes a seasoned autotester from a beginner is the understanding that not all tests are equally useful. And sometimes it’s better not to write anything. As my mother taught us – “do not write, you will pass for a smart one.” My article will not be about how to write tests, but about how not to write them.
Disclaimer: there will be C# code examples in the article, but I tried to make them as clear as possible for people familiar with any programming language. We will write tests for the function of feeding the cat:
Check it out, I don’t know what
How well do you understand what your tests are testing? Pfft, what a question, because you wrote them for a specific scenario. In practice, it often turns out that tests do not check what you intended. They fall in the wrong places. They pass when there is an error in the code. For example, you decide to write a positive test for our function, the script “If you pass a valid value for the number of servings, the cat will be fed.” The test might look something like this:
Is this a good test? And let’s check. Let’s change the condition for the minimum number of servings in our function:
Big change, right? And our test, as it was green, remained. And if the change was accidental, we won’t catch this bug.
Conclusion number one: make sure your test fails in situations where it should fail. Doing this for all tests, of course, is expensive, but I recommend this exercise for beginner test writers, as well as when writing complex integration tests.
Conclusion number two: don’t forget the boundary value technique.
Seconds matter
How do you know if your tests are passing quickly? Set values that are reasonable for your system and command. How to determine if your tests can run faster? Experiment. Often tests are artificially slowed down for one of the following reasons:
contain redundant steps
do not use mocks where it is justified
unnecessarily accessing the file system
Let me show you the second reason. We want to check that our system will not allow to feed the code if the CatProvider says that the cat is not ours. I emphasize that this is not an integration test for checking the connection with CatProvider, but a unit for the operation of our system. Here’s what happened:
The code of the method under test shows that the test will actually call the CatProvider. If it’s a third-party loaded service, the test might wait a few seconds before getting a response. Also, the service can be stopped at the moment of running our tests, and the test will fail. In this unit test, it is appropriate to add a call to the stub instead of the real service. It will look like this:
For the magic to happen, you need to change the product code, in one way or another, adding work with the ICatProvider interface and determining which implementation to use. This is a normal situation – sometimes you have to make changes to the production code to improve testability without affecting the main functionality.
Remember that one of the main purposes of autotests is to provide quick feedback on the impact of changes. Therefore, do not let the speed of your tests run out of control.
Say no to addiction
Most often for tests (especially end-to-end ones) you need a lot of lots of data. Some teams come to the decision to use ready-made ones – created by hand on a test environment or copied from the sale. What is the danger? The fact that this data can change at any time, your tests do not control them. I’ve encountered test instabilities many times for this very reason. Let’s see what it looks like:
What to do with it? Create data directly in tests. Don’t forget to clear the data at the end. Don’t forget about test independence.
It is worth noting that ready-made data is often used for the sake of speed, because. it takes N seconds to create data, and this seems to contradict the previous point. But a) stability is more important than speed, b) creating data doesn’t have to be a long process. For example, our developers were able to teach the application to run in InMemory mode, which allows you to quickly create data on a clean database and stop wasting time analyzing false negative tests.
Instead of an afterword
All of the above has been gained through years of work, but is not a guide to action. The author acknowledges that in your situation, the described changes may lead to a deterioration of the test system. Think with your head, but do not forget to check with reality. All the beaver and green autotests!
