Find differences, or implement Snapshot tests for the web. Sound Experience

Hi all! In touch Natalia Danilina And Chechikov Ivan from Sound. In this article we want to share our experience of implementing snapshot tests for a web application – we will tell you what it is and what tasks it is used for.

Details are under the cut.

It all started with a search for answers to questions: how to reduce testing time, how not to burn out on regression, how to test a feature without a design review, and how to reduce the number of bugs in the product after release?

What's the point? We have a large number of manual test cases, which are covered by e2e and component autotests. Our automated tests run every time a regression starts (and it takes from 3 to 4 days). And there are situations when, after completing the process, we find bugs in the product related to the UI – they are difficult to catch at the regression stage due to the human factor.

Therefore, we found an approach that could solve such problems in the future – this is how snapshot tests came into play.

Snapshot tests are automated tests that are used to test web pages. They allow you to compare the current state of a web page with the previous (or expected) state and detect any changes or differences. We also use snapshot tests to compare individual HTML elements (popups, modals, frames, etc.). The current state of the component in the test environment and the reference are compared. The standard can be a sales component, but it is better if it is an actual design component.

Snapshot tests use pixel-by-pixel comparison. Each reference pixel is compared to a corresponding pixel in the test environment object image to detect differences. This allows tests to be very accurate and detect even small changes to a web page or component. However, it is necessary to take into account the error in comparison, since there may be idle errors with a small percentage of difference (up to 5%).

Working with snapshot tests, we identified a number of their advantages:

Fast feedback. Snapshot Testing allows you to quickly check whether the test result has changed after making changes to the code. This provides quick feedback to developers and testers, which helps them respond faster to problems that arise.

Automation of interface testing. Snapshot Testing automates the interface testing process, which allows you to solve related problems faster and more efficiently.

Simplifying the testing process. Snapshot Testing simplifies the process of interface testing, as it does not require a large amount of code to write tests. This reduces the time spent writing and maintaining tests, freeing up resources for other tasks.

Improving code quality. Thanks to snapshot tests, you can easily track changes in the interface and quickly detect errors. Therefore, the quality of the code increases and the user experience improves. In the future, we plan to use snapshot testing in the development process.

Reducing time for regression testing. Implementation of Snapshot Testing allows you to reduce the time spent on regression testing. Therefore, interface testing is automated and problems are identified faster.

We implemented snapshot tests in our UI framework written in Kotlin using Playwright. We created a separate directory for them in the framework and allocated a job for running tests in GitLab CI. Our TMS is Allure TestOps.

The tests themselves are not logically complicated. Let's look at an example of a cookie comparison test in a dark theme:

.....
class CompareCookiesBlackSnapshotsTest : WebTest() {

    @Test
    @TestOpsId(44908)
    @DisplayName("Сравнение попапов кук (темная тема)")
    @Owner(CHECHIKOV_IVAN)
    fun testCase() {
        step(
            """
            |*Precondition*:
            |Пользователь не авторизирован и находится на главной странице"
            """.trimMargin()
        ) {
            assertThat(page).containsURL(baseUrl)
            page.waitForLoadState()
        }

        step("Проверяем, что включена темная тема") {
            SideMenuBar(page).checkedSwitcherDarkTheme()
        }

        step("Проверяем что попап куки есть на странице") {
            assertTrue(CookiesPage(page).cookiesBody.isVisible)
        }

        step("Сравниваем с эталоном") {
            screenStageBytes = CookiesPage(page).cookiesBody.screenshot()
            assertScreenShots(screenStageBytes, imageName = "cookieBlack")
        }
    }
}
.....

The steps are pretty self-explanatory: we execute the precondition by checking the homepage URL. The next step is to make sure that the dark theme is enabled, and then that there is a cookie on the page. Finally, we take a screenshot of the current state of the cookie and call the method for comparing the test image with the reference.

.....
    fun assertScreenShots(stageBytesArray: ByteArray, imageName: String) {
       val prodImage = ImageIO.read(File("src/test/resources/screens/expectedImages/$imageName.png"))
        val stageImage = ImageIO.read(ByteArrayInputStream(screenStageBytes))
.....

The standards are located in the /screens/expectedImages directory under git. In tests, we know in advance what a particular standard will be, so we put the name of our image into the $imageName variable. After taking a screenshot of the test component, we transfer it to a BufferedImage object, just like the reference.

.....
val prodImageResized = resizeImage(prodImage, stageImage.width, stageImage.height)
.....

.....
fun resizeImage(image: BufferedImage, width: Int, height: Int): BufferedImage {
        val scaledImage = image.getScaledInstance(width, height, Image.SCALE_SMOOTH)
        val resizedImage = BufferedImage(width, height, image.type)
        resizedImage.getGraphics().drawImage(scaledImage, 0, 0, null)
        return resizedImage
    }
.....

We bring the comparison objects to the same size:

.....
val imageComparison = ImageComparison(prodImageResized, stageImage).setAllowingPercentOfDifferentPixels(5.0)
val diff = imageComparison.compareImages()
.....

The library performs the magic of comparison image comparison. We pass our BufferdImage objects for comparison to the ImageComparison object and set the error percentage to 5.0. We make a comparison and get a discrepancy object diff.

.....
        val diffImageStream = ByteArrayOutputStream()
        val prodImageStream = ByteArrayOutputStream()
        val diffImg = diff.result
        ImageIO.write(diffImg, "png", diffImageStream)
        val diffBytesArray = diffImageStream.toByteArray()
        ImageIO.write(prodImageResized, "png", prodImageStream)
        val prodBytesArray = prodImageStream.toByteArray()
        saveScreenshot("actual", stageBytesArray)
        saveScreenshot("expected", prodBytesArray)
        saveScreenshot("difference", diffBytesArray)
        assertEquals(ImageComparisonState.MATCH, diff.imageComparisonState,
            "Изображение в тестовой среде отличается от эталона $imageName")
    }
.....

We want to see images of the compared test component, the reference and the difference image in the test run results, so we translate all our comparison objects into a byte array and save them as screenshots.

What it looks like in Allure TestOps:

The text, cross icons and cookies in the sample differ from the screenshot from the test environment. In the difference image, we see the changes marked with red rectangles. Below is a message about the differences between the images.

So, we have given an example of one of our snapshot tests. We are now continuing to expand our coverage because in the long term we hope that Snapshot Testing will help significantly improve the efficiency of the QA team, reduce the time spent on routine tasks, improve the quality of testing and speed up the detection of errors in the application interface.

Thanks for reading! If you have questions, we will be happy to answer them in the comments.