coverage percentage, mutants and working with mocks

What's the catch with coverage percentage?

When developers write unit tests, it is often tempting to focus on achieving a high percentage of code coverage. Many CI/CD systems impose certain coverage percentage requirements in order to accept changes.

And so, you sit in front of the monitor and you just want to shoot, write at least something in order to see the coveted green tick. And there is even such a possibility…

Coverage percentage shows how many lines of your code are executed during testing. That is, a simple function call in the test is already suitable to “cover” this function. Everything is ready! You don’t have to stress, you’ve got the tick.

Stories like this lead to tests not validating the application's behavior as expected. In our practice, we encountered two main problems:

• Tests don't check output
  For example, a check was written for two cases: ok and error (“processing was successful” and “method returned an error”). Sometimes there is no verification of output data at all. As a result, the coverage rate is high and the quality of verification is low.

• No checking of edge cases
  It's good if the test checks a pair of values ​​from the so-called equivalence classes. But what about the border ones? Very often they are simply ignored. For example, checking the handling of a null value or an empty array may be critical to the quality of the product, but will not greatly affect the coverage percentage.

You could say that the coverage percentage is useful for making sure that the code has been completed, but it does not indicate that your code is 100% reliable and tested. There are other approaches for this, for example, mutation testing.

Mutation tests: what are they and why are they needed?

Mutation testing is an approach in which small changes called “mutations” are intentionally made to your code. The purpose of such changes is to see if your tests can notice them and fail if the program's behavior changes. For example, a mutation may involve replacing the operator > on < or changing the value of the returned constant.

The idea is that if your tests don't notice these changes, then they're too shallow and aren't testing the logic of your code deeply enough.

Mutation testing metric

One of the key metrics in mutation testing is the “percentage of mutants detected” (Mutation Score). This is the percentage of mutations that the tests were able to catch. The higher this indicator, the higher the quality of the tests.

Let's look at a simple example. Let's say we have a function isPositive that checks if the number passed is positive:

func isPositive(n int) bool {
	if n > 0 {
    	return true
	}

	return false
}

Let's write several tests to check the function:

func Test_IsPositive(t *testing.T) {
	assert.True(t, isPositive(5))   // Тест с положительным числом
	assert.False(t, isPositive(-3)) // Тест с отрицательным числом
}

These tests check basic scenarios: when the number is positive and when it is negative. There are no corner cases here, everything is simple.

Now imagine that in the process of mutation testing the condition n > 0 was changed to n >= 0 :

func isPositive(n int) bool {
	if n >= 0 {  // Мутация: изменено условие с > на >=
    	return true
	}

	return false
}

The function will now return true not only for positive numbers, but also for zero, and our tests will pass successfully and will not notice this.

To catch such a mutant, you just need to add a check for the edge case:

assert.False(t, isPositive(0))

Let’s imagine that in a similar situation, in the process of writing new functionality, a tired developer Vasya accidentally changes the code. At first glance, this is a small thing, but sometimes such details can disrupt the operation of important business scenarios. The tests will not notice the changes, and instead of spending the weekend in peace, Vasya urgently fixes the bugs that managed to get into production.

Thus, regular mutation testing and monitoring Mutation Score will help track potential vulnerabilities and proactively close them with tests.

What's next?

Now imagine that we have written tests, added datasets, carefully checked all the output values, and the mutation testing report still hints to us that we are doing something wrong.

One common pitfall is values ​​that are passed as arguments to an external service call. Such calls are closed in tests stubs and mocks. The problem is that stubs don't check arguments at all, and mocks provide a tempting option not to do so. As a result, your test can easily miss the situation when something unexpected is sent to production instead of correct data.

Dangers of Using Mocks

What are stubs and mocks?

Both are objects that simulate the behavior of your code's real dependencies. They allow you to test system components in isolation, which is especially useful when working with external services or databases.

Stubs are simple stubs that return predefined data. Mocks, in turn, control the passing of arguments, method calls, and check the expected behavior of the object under test.

There is no point in considering stubs in this case; their task is simply to replace a call that is unnecessary for verification with a fictitious response. But, often, it is very important to check the call, so let’s move on to the specifics of working with mocks.

What problems can arise when working with mocks?

Using mocks such as gomock.Any in Go, can reduce the severity of argument checks. When a method accepts any argument, it can hide potential problems associated with changing the data structure.

mockService.EXPECT().SendRequest(gomock.Any()).Return(nil)

This code allows the method SendRequest accept any arguments, which is convenient if the specific value is not important for the test. However, this also reduces the accuracy of the tests.

Let's return to Vasya.

Let's say the SendRequest method takes as input a structure with the ID, Name and Content fields. Vasya is working on a new task where he needs to add new information to Content depending on the conditions. He makes a mistake when writing the condition.

A test using gomock.Any succeeds because it only cares that the method was called. As a result, the problem went into production, and Vasya again spent his day off fixing bugs, which could have been avoided with a more rigorous test.

Fixing the test:

Instead of using gomock.Any, you can add a stricter check with specific values:

// Определяем структуру ожидаемого запроса
type Request struct {
	ID  	int	   `json:"id"`
	Name	string `json:"name"`
	Content string `json:"content"`
}

// Создаем ожидаемый объект
expectedRequest := Request{
	ID:  	1,
	Name:	"Test",
	Content: "This is a test request",
}

// Указываем, что метод должен вызываться с ожидаемым объектом
mockService.EXPECT().SendRequest(expectedRequest).Return(nil)

In this example, we define a Request structure that contains the ID, Name, and Content fields. We then instantiate expectedRequest with specific values. This allows the test to check incoming arguments and ensures that problems associated with changes to the data structure are detected.

If we have conditions depending on which the field values ​​change, we need to add tests for each of them.

So, now we began to work with mocks more carefully, we check all arguments and calls, and use gomock.Any only for repeated checks. What's left?

Ignoring side effects

Sometimes the check may not be as obvious if the method must change the state of an object or do additional things. For example, a method could update the cache or initiate an asynchronous process, but if the tests ignore this aspect, then the error in setting up the asynchronous process will not be detected.

Let's say we have a system that manages users. The method we are testing not only returns information about the user, but also updates the system state by recording the ID of the last active user.

// Структура для хранения данных о пользователе
type User struct {
	ID   int
	Name string
}

// Структура для хранения состояния системы
type SystemState struct {
	LastActiveUserID int
}

// Интерфейс для работы с пользователями
type UserDatabase interface {
	GetUser(id int) (User, error)
}

// Метод, который возвращает пользователя и обновляет стейт
func GetUserAndUpdateState(db UserDatabase, state *SystemState, userID int) (User, error) {
	user, err := db.GetUser(userID)
	if err != nil {
    	return User{}, err
	}

	state.LastActiveUserID = user.ID
	return user, nil
}

Now let's look at what a test for this method might look like:

func TestGetUserAndUpdateState(t *testing.T) {
	mockDatabase := NewMockUserDatabase
	state := &SystemState{}

	// Создаем тестового пользователя
	expectedUser := User{ID: 1, Name: "Test User"}

	// Настраиваем мок так, чтобы он возвращал ожидаемого пользователя
	mockDatabase.EXPECT().GetUser(1).Return(expectedUser, nil)

	// Вызываем тестируемый метод
	user, err := GetUserAndUpdateState(mockDatabase, state, 1)

	// Проверяем, что не возникло ошибки
	assert.NoError(t, err, "expected no error")

	// Проверяем, что возвращаемый пользователь совпадает с ожидаемым
	assert.Equal(t, expectedUser, user, "expected user to match")
}

Everything looks good, there is error checking and output value, but it does not take into account an important detail – updating the state. Such a test will not track if we stop updating the state at all. To fix this we need to add a check:

// Проверяем, что состояние изменилось правильно
assert.Equal(t, expectedUser.ID, state.LastActiveUserID, "Expected LastActiveUserID to match the retrieved user ID") 

Ready! Now we have enough techniques to catch mutants and make our tests really useful and high quality 🙂

Conclusion

In this article, I tried to highlight the key points that I encountered in practice when writing unit tests. Using these recommendations will already allow you to notice how the quality of your tests will increase, and with it the confidence that your code really works as intended.

Share your thoughts and experiences in the comments! What testing approaches do you use?