We introduce snapshot testing, or five stages of accepting the inevitable

1. Negation

2. Anger

3. Bargain

4. Depression

5. Acceptance

So let's begin.

Denial is the starting point, here we only have unit tests and the desire to improve reliability. At this stage, we explore what tools are available, who uses what, and decide whether we need snapshot testing at all.

There are many other technical tasks that can be taken on. But this does not negate the benefits they can bring. So my goal was to show the team the benefits, develop an implementation plan, and conduct a demonstration.

First, you need to understand the goals of snapshot testing, what it is and what are the main practices for its use.

What is snapshot testing?

Many are already familiar with this term and the testing pyramid. In it, snapshot tests take a position above unit tests, which means a longer launch process and a more labor-intensive support process. But the number of such tests should be significantly less than unit tests and will allow the formation of a new layer of testing in the application.

Testing pyramid

The testing pyramid is one of the ways to ensure software quality, a visualization that helps group tests by type of purpose. It also allows you to agree on the rules for writing tests, dividing them into types, and identifying the main focus of testing in each group.

Purpose of snapshot testing

The main goals are to improve the quality of development, reduce the number of layout errors when creating and refactoring components, and validate the appearance for the user.

When writing tests, the required state of the UI component is compared with a reference image obtained externally (from the design) or generated by the developers themselves. It should be borne in mind that the development of components often does not follow the pixel perfect approach, since there are many factors that influence the final result. And this leads to high labor costs for development.

Therefore, we chose the second path, mainly by reducing the working time for preparing images by the designer (format, dimensions, states), as well as fitting models for tests to these images.

After comparison, you can immediately verify the correct result of component development or promptly identify an error and correct it. Also, when refactoring or implicitly changing a UI component, it will be clear that the changes resulted in an error if they differ from the reference image.

Basic Tools

There are two libraries for snapshot testing:

You can also create your own native implementation without using third-party libraries.

Because iOSSnapshotTestCase is updated rarely and does not support SPM, and the native implementation requires a lot of effort, we used the library SnapshotTesting.

Comparison table of snapshot testing solutions

Some numbers

To understand the performance and numbers that should be expected in theory, we tested and generated snapshots in different quantities. It is based on a view with a size of 375x90pt, which corresponds to the full width of the screen and the height of the two middle cells. The run itself was done on Xcode version 13. Also for the test we added an option with generation HIEC images in case you need to save the size of snapshots.

Snapshot generation speed 375x90pt

• 5000 images ~ 150 MB and 30 minutes

• 2000 images ~ 65 MB and 6 minutes

• 1000 images ~ 30 MB and 2 minutes

• 1000 HIEC images ~ 24 MB and 6 minutes

Snapshot testing speed 375x90pt

• 5000 images ~ 6 minutes with success / 10 minutes with complete failure

• 2000 images ~ 2-3 minutes for both passage options

• 1000 images ~ 2 minutes for both options

• 1000 HIEC images ~ 2 minutes pass / 6 minutes total failure

Having demonstrated the process of writing snapshot tests to colleagues and listed all their advantages, we can begin to implement them in our project.

We have all the cards at our disposal: a project with components, a library for testing and free hours for implementation. And here are the first difficulties!

It immediately becomes clear that we need to understand where we will store the reference images.

Depending on the testing purposes, there are several storage options:

• Project folder – All reference images are in the same place as the tests. The downside of this solution is the constantly growing volume of images, so the repository may reach limits if you have a large application or a very dynamic design system. But it is quite easy to integrate the storage of reference images and the tests themselves, and you can also simplify the process of merging a branch into master.

• Dedicated repository – the entire volume can be left for storing images and have a clean and visual history, but, again, there is a risk of overflow in the case of a large number of tests and constant changes in the design.

• GitLab LFS or an external database is the most suitable option at first glance, since the storage capacity can be unlimited, but the complexity of integration can be higher and introduces its own limitations.

Having searched for information about who stores snapshots where, it became clear that everything depends on the project and the number of tests, there is no single universal solution, and you need to decide for yourself. Having considered all the pros and cons, we decided to store it in a dedicated repository with the possibility of later moving to a bank vault.

We implemented a separate repository with an SPM package in which reference images are stored in package resources to optimize image files.

Next we proceed to implementation.

From component creation to tests

After creating a component in a design system project and writing unit tests, you need to write snapshot tests to generate reference images with different states of the component.

For example, we created a component 'ChatDateTableHeaderView', it has three states: day, month and year / day and month / today. We add this component to the screen we need in the demo project with the necessary states and the ability to view them. Now we can write snapshot tests for state data.

Creating a snapshot test for a component

Snapshot tests have been added as a new test target to the design system project to separate it from unit tests and a separate run on CI.

You also need to set up a test device. We have adopted the iPhone 11 Pro as a standard, and at CI it is tested on the same device. If you set another one, they will fall when running tests, so we leave the iPhone 11 Pro.

Add a folder for the snapshot test and create a file with the SnapshotTests prefix, for the 'ChatDateTableHeaderView' component there will be 'ChatDateTableHeaderViewSnapshotTests'. All we have to do is import the design system project and UIKit to access the components.

import UIKit

final class ChatDateTableHeaderViewSnapshotTests: SnapshotTestCase {

    private var sut: ChatDateTableHeaderView!
    private var viewModel: ChatDateTableHeaderViewModelMock!

    override func setUp() {
        super.setUp()

        sut = ChatDateTableHeaderView()
        viewModel = ChatDateTableHeaderViewModelMock()
    }

    override func tearDown() {
        super.tearDown()

        sut = nil
        viewModel = nil
    }
}

// MARK: - Mocks
private extension ChatDateTableHeaderViewSnapshotTests {

    struct ChatDateTableHeaderViewModelMock: TextViewModelProtocol {
        var title: String? = "1 июля"
    }
}

Here you can notice 'SnapshotTestCase', which is simply a descendant of XCTestCase and is needed for convenience. Further in the text I will tell you in more detail why.

We create a testable view and mock data, if necessary. Writing the basis of snapshot tests is similar to writing unit tests, which allows you to quickly understand the process and navigate other people's tests.

The main difference between the tests is the purpose of testing: in unit tests we check individual properties of models and the execution of events, while in snapshot tests we only check a specific pixel-by-pixel state in the form of an image.

Next, you need to generate a reference image—the state of the component—in order to later compare it with that obtained during the test.

Writing the first test

It is worth considering that by default the name of the snapshot is taken from the name of the test, so you should immediately follow the rules for naming tests for readability and quickly finding a test that failed.

func testDayAndMonthTitle() {
    // when
    sut.viewModel = viewModel

    // then
    assertImageSnapshot(matching: sut)
}

The matching parameter accepts the view that needs to be compared with the reference image. If automatic calculation of the minimum required size is not suitable, you can explicitly specify the size of the component to generate.

func testDayAndMonthTitle() {
    // when
    sut.viewModel = viewModel

    // then
    assertImageSnapshot(matching: sut, size: defaultSize)
}

If we run such a test, it will crash because it does not find a reference image, and then it will automatically generate it.

Click on the eye icon and it will open a diff image, which shows exactly how the reference image differs from the received one. You may notice that the numbers are different. It helps a lot when there is a difference that is invisible to the eye, such as indents or shades of colors.

func testDayAndMonthTitle() { // when viewModel.title = "11 марта 2024" sut.viewModel = viewModel // then assertImageSnapshot(matching: sut, named: "2") }

By default, version '1' is set during generation; this can be seen in the image title “testDayAndMonthTitle.1.png”. Next, we follow the same steps as when writing the tests themselves. You can also specify your own values ​​that will be used to identify the version, or not specify it at all.

Difficulties and nuances

That seems to be it! Tests are written quickly, CI runs tests correctly, but that’s not the case.

Along the way, certain errors pop up – both locally and during the run itself on CI.

Let's look at the identified problems and options for solving them, as well as possible nuances that may arise in the future:

• One of the most obvious mistakes was, oddly enough, simple carelessness when setting up the device for generating snapshots. Due to different sizes and ppi on the local machine and CI, the snapshots did not match, and the tests failed. Since we currently have one device for testing tests, snapshots and design reviews, we need to stick to it. Or implement the appropriate check and run on several devices with different snapshot sizes.

• We also had to re-upload all the snapshots 2-3 times, as changes were made that affected almost all components, in particular the text. Fortunately, this is easy to do by simply setting the 'isRecording = true' property in the test file. Such cases should be kept to a minimum, followed by improvement and identification of major errors.

• Next comes the difference in the Xcode version. CI machines have newer versions of Xcode, and as a result, there are differences in the rendering of components. A specific example is text rendering: everything is the same, but the text is shifted literally by a pixel, the problem was in the 'baselineOffset' property. We solved it with minor modifications and checking the iOS version when setting the property, as well as rewriting all snapshots in the resources.

• There may also be differences in shadow rendering or minor changes in system components with different versions. They are often solved by lowering the precision value in the 'assertImageSnapshot' method, but you should not lower it below 0.98, otherwise you can lose the main feature of snapshot testing – comparison by pixel; in addition, this is a range that is not visible to the eye.

• Another point worth considering is the rounding of corners: when generating snapshots, the 'maskedCorners' property is not taken into account. To do this, we draw the corners using 'UIBezierPath'.

• Since the process is new, it was necessary to write a document according to which the team could easily integrate into the process and understand the intricacies of the work. The information is simple, but relatively voluminous, so we additionally made a short check of the steps for a full circle of writing tests.

It's time for the next stage.

At this stage, we are already writing snapshot tests for new components or for those that have already been modified. Due to the fact that we already have quite a lot of ready-made components, it is necessary to cover them with snapshot tests, but this is not so easy.

You need to coordinate the relevance of the component with the design, write the tests themselves and make sure that everything works. And also find time for this, which is only available during technical days.

This is one of the obvious reasons why snapshot testing is ranked above unit tests in the testing pyramid.

But don't be discouraged! Everything is gradual: component by component, test by test. Considering the advantages and the additional layer of protection for the application from visual bugs, it’s worth it!

Here we are in the final stages of accepting snapshot tests as a new form of validation for our application.

Tests are written quickly, are easy to read, the entry threshold is low, and it has become easier to find layout errors.

To summarize, I would like to share the output numbers:

• 29 seconds to run all snapshot tests;

• about 400 reference images with a total weight of 14.6 MB;

• several satisfied developers who have secured their components;

• quarter for study, implementation and improvements.

Future plans

There is always something to improve. Here are a few options that can make working with snapshot tests easier:

• Implement tests with the ability to generate snapshots for different device models and interconnect them with CI, as well as send them for design review to improve the quality of component validation on different screens.

• Make a template for quickly creating a testing file.

• Explore the possibility of storing snapshots in a separate storage. It is worth checking the feasibility of using it in comparison with a separate repository in Gitlab: accesses, number of steps for implementation and adding snapshots, verification speed, etc.

• Simplify the process of generating and adding snapshots to the resources of a separate repository or external storage.

• Prepare for the transition to SwiftUI or partial use.

Conclusion

Snapshot tests turned out to be an excellent addition to our team, since it is important for us to have a stable and responsive interface without broken layout and visual bugs.

I will be glad to receive suggestions and feedback, as well as your experience in implementing snapshot tests.

Thank you for your attention!

For information, I used these links, as well as video reports on YouTube: