Testing Exceptions

Reflections on testing beyond the happy path.

Test-driven development (TDD) is a great method for getting quick feedback on design and implementation ideas and getting to a working solution faster.

However, the emphasis on the “ideal scenario” can make you forget about all the possible errors and unexpected situations. Sooner or later, you will realize that the implementation can fail for a number of reasons, and, in an effort to make it more reliable, you will want to test the error handling code using automated tests.

This doesn't have to be complicated, but it can raise some interesting questions. I'll try to address some of them in this article.

Throwing Exceptions with Dynamic Mock Object

IN question to another article AmirB asks how to use Fake Object for testing exceptions. In particular, since Fake is a test double with a consistent contract, it would be inappropriate to allow it to throw exceptions that are specific to different implementations.

This is quite abstract, so let's look at a concrete example.

The article where AmirB asked his question used the following interface as an example:

public interface IUserRepository
{
    User Read(int userId);
 
    void Create(int userId);
}

Sure, it's a bit weird, but it should be enough for the current task. As AmirB wrote:

“In scenarios where mock objects are used (like Moq), we can mock a method so that it throws an exception, which allows us to test the expected behavior of the System Under Test (SUT).”

Specifically, it might look like this when used Moq:

[Fact]
public void CreateThrows()
{
    var td = new Mock<IUserRepository>();
    td.Setup(r => r.Read(1234)).Returns(new User { Id = 0 });
    td.Setup(r => r.Create(It.IsAny<int>())).Throws(MakeSqlException());
    var sut = new SomeController(td.Object);
 
    var actual = sut.GetUser(1234);
 
    Assert.NotNull(actual);
}

This is just an example, but the point is that since you can make a dynamic mock object do anything you can define in code, you can also use it to simulate database exceptions. This test simulates a situation where IUserRepository throws out SqlException from the method Create.

Possibly implementation GetUser now looks like this:

public User GetUser(int userId)
{
    var u = this.userRepository.Read(userId);
    if (u.Id == 0)
        try
        {
            this.userRepository.Create(userId);
        }
        catch (SqlException)
        {
        }
    return u;
}

I wouldn't be surprised if you thought this example was far-fetched. Interface IUserRepositoryClass User and method GetUserwhich unites them, are primitive in their own way. I originally created this small code example to discuss data flow validation, and now I'm using it for other purposes. Hopefully you can overlook this. The main point I'm trying to make is more general and doesn't depend on the details.

Fake

In the article also suggested FakeUserRepositorywhich is small enough that I can repeat it here.

public sealed class FakeUserRepository : Collection<User>, IUserRepository
{
    public void Create(int userId)
    {
        Add(new User { Id = userId });
    }
 
    public User Read(int userId)
    {
        var user = this.SingleOrDefault(u => u.Id == userId);
        if (user == null)
            return new User { Id = 0 };
        return user;
    }
}

The question is, how do you use something like this when you want to test for exceptions? It's possible that this little class might throw errors that I couldn't predict, but it certainly doesn't throw any SqlExceptions!

Should we complicate things? FakeUserRepositoryadding the ability to throw certain exceptions?

Throwing Exceptions from Test Doubles

I understand why AmirB asks this question, because it really does seem wrong. For starters, it goes against the single responsibility principle (Single Responsibility Principle). FakeUserRepository then there would be more than one reason to change: you would have to change it if the interface changed IUserRepositorybut you would also have to change it if you wanted to simulate a different error situation.

Good coding practices apply to test code, too. Tests are code that needs to be read and maintained, so all the good practices that keep production code quality apply to tests. This may include SOLID principles, unless you think SOLID is an outdated concept.

If you really need to throw exceptions from your test double, perhaps a dynamic mock object like the one above is a better option. No one is saying that if you use Fake Object for most of your tests, you can't use a dynamic mock library for one-off test cases. Or perhaps it makes sense to create a one-off test double that throws the exception you want.

However, I would consider it a sign of “bad code” if it happens too often. Not “bad test”, but “bad code”.

Is the exclusion part of the contract?

You may wonder if a certain type of exception is part of the object's contract. As I always do when I use the word “contract”I mean a set of invariants, preconditions and postconditions, taking the example of Object-Oriented Software Construction (construction of object-oriented software).

When you have static typing, you can assume a lot about the contract, but there are always rules that can't be expressed that way. Parts of the contract are implied or conveyed in other ways. Code comments, docstrings (docstrings) and the like are good options for conveying these details.

What conclusions can be drawn from the interface? IUserRepository? And what kind? should not?

I would expect that method Read will return the object User. The code example was created in 2013, before C# had nullable reference types (nullable reference types) At that time I started using Maybeto signal that the return value may be missing. This conventionso the reader must be aware of it in order to properly understand this part of the contract. Since the method Read does not return Maybe<User>I would assume that it is guaranteed to return a non-null object User; this is a postcondition.

These days I also use asynchronous APIs to indicate that I/O is involved, but again, the example is so old and simplistic that it's not the case here. However, regardless of how it's conveyed to the reader, if an interface (or base class) is designed for I/O, we can expect it to fail sometimes. In most languages, such failures manifest themselves as exceptions.

As a result of such reflections, at least two questions arise:

Should I SqlException be part of the contract at all? Isn't that an implementation detail?

Class FakeUserRepository does not use SQL Server and does not throw away SqlExceptions. One can imagine other implementations that use a document database, or even just another relational database other than SQL Server (Oracle, MySQL, PostgreSQL, etc.). These implementations would not throw away SqlExceptionbut perhaps other types of exceptions.

According to dependency inversion principle,

“Abstractions should not depend on details. Details should depend on abstractions.” — Robert K. Martin, Agile Principles, Patterns, and Practices in C#

If we do SqlException part of the contract, then the implementation detail will become part of the contract. Moreover, in the case of implementation, as in the above method, GetUserwhich intercepts SqlExceptionwe also violate Liskov Substitution PrincipleIf you implement a different implementation that throws a different type of exception, the code will no longer work as intended.

Loosely coupled code shouldn't look like this.

Many specific exceptions are of a type that you won't be able to handle anyway. On the other hand, if you decide to handle certain error scenarios, make it part of the contract, or as Michael Feathers says, extend the domain.

Integration testing

How do we unit test specific exceptions? Actually, we shouldn't.

“Personally, I avoid using try-catch blocks in repositories or controllers and prefer to handle exceptions at the middleware level (e.g. via ErrorHandler). In such cases, I write separate unit tests for the middleware. I think this is the most appropriate approach.” — AmirB

I think this is a great approach for exceptions that you decide not to handle explicitly. Such middleware will typically log or otherwise notify operators of the problem. You could also write generic middleware that retries or implements the Circuit Breaker (circuit breaker), but there are already libraries that do this. Consider using them.

However, you may decide to implement a specific feature using the capabilities of a specific technology, and the code you are going to write may be complex or important enough to require good test coverage. How do you do that?

I would suggest integration testing.

I don't have a ready example related to throwing specific exceptions, but something similar might be useful. The example codebase that accompanies my book Code That Fits in Your Head simulates an online restaurant reservation system. Two customers may compete for the last table on a given date — a typical race situation.

There are several ways to solve this problem. You could decide to redesign the entire application to handle edge cases like these reliably. However, for the purposes of the example in the book, I chose architecture without locks inappropriate. Instead, I looked at solving this problem by using the lightweight transaction capabilities of .NET and SQL Server via TransactionScope. While this is a convenient solution, it is entirely dependent on the technology stack being used. This is a good example of an implementation detail that I would not want to include in unit tests.

Instead, I wrote an integration test that runs on a real SQL Server instance (automatically deployed and configured on demand). It tests behavior systems, not implementation details:

[Fact]
public async Task NoOverbookingRace()
{
    var start = DateTimeOffset.UtcNow;
    var timeOut = TimeSpan.FromSeconds(30);
    var i = 0;
    while (DateTimeOffset.UtcNow - start < timeOut)
        await PostTwoConcurrentLiminalReservations(start.DateTime.AddDays(++i));
}
 
private static async Task PostTwoConcurrentLiminalReservations(DateTime date)
{
    date = date.Date.AddHours(18.5);
    using var service = new RestaurantService();
 
    var task1 = service.PostReservation(new ReservationDtoBuilder()
        .WithDate(date)
        .WithQuantity(10)
        .Build());
    var task2 = service.PostReservation(new ReservationDtoBuilder()
        .WithDate(date)
        .WithQuantity(10)
        .Build());
    var actual = await Task.WhenAll(task1, task2);
 
    Assert.Single(actual, msg => msg.IsSuccessStatusCode);
    Assert.Single(
        actual,
        msg => msg.StatusCode == HttpStatusCode.InternalServerError);
}

This test tries to make two parallel reservations for ten people. This is also the maximum capacity of the restaurant: it is impossible to accommodate twenty people. We want one of the requests to “win this race”, while the server should reject the loser's request.

The test focuses solely on the observed behavior from the client side. Since the codebase contains hundreds of other tests that check HTTP responses, this test focuses only on status codes.

The implementation handles a potential overbooking situation as follows:

private async Task<ActionResult> TryCreate(Restaurant restaurant, Reservation reservation)
{
    using var scope = new TransactionScope(TransactionScopeAsyncFlowOption.Enabled);
 
    var reservations = await Repository.ReadReservations(restaurant.Id, reservation.At);
    var now = Clock.GetCurrentDateTime();
    if (!restaurant.MaitreD.WillAccept(now, reservations, reservation))
        return NoTables500InternalServerError();
 
    await Repository.Create(restaurant.Id, reservation);
 
    scope.Complete();
 
    return Reservation201Created(restaurant.Id, reservation);
}

Please note the usage TransactionScope.

I have an illusion that I can radically change this implementation detail without breaking this test. However, I have not yet tested this hypothesis in practice.

Conclusion

How do you automatically test error-handling code paths? Most unit testing frameworks provide APIs that make it easy to verify that a specific exception was thrown, so this isn't the hard part. If a specific exception is part of the contract of the system under test, just test it that way.

On the other hand, when it comes to objects that are composed of other objects, implementation details can easily “leak out” in the form of specific exception types. I would think twice before writing a test that checks whether client code (like the one mentioned above) handles SomeController) a certain type of exception (for example, SqlException).

If such a test is difficult to write because you only have a Fake Object (e.g. FakeUserRepository), that's a good sign. The rapid feedback that test-driven development provides has worked again. Listen to your tests.

You probably shouldn't write this test at all, as there seem to be problems with the intended code structure. Better decide this one problem.

If you want to deepen your knowledge in testing automation, you may find the course “QA Automation Engineer” useful. In it, you will learn how to write automated tests in Java, master a pool of tools (Postman, SoapUI, Selenium, IntelliJ IDEA, JUnit, Cucumber and others), and also gain practical experience in writing automated tests on real cases. You can view the program and recordings of open lessons on the course page.