How to refuse screenshots in testing

Have you often encountered a situation where tests failed due to a small difference between screenshots? For example, the spinner did not have time to disappear, the scrollbar did not spin, the notification disappeared a little faster than necessary, and so on.

Have you often encountered a situation where tests failed due to the fact that a third-party element appeared in the screenshot, which was released by another team? Have you often encountered… a variety of reasons why you have to sit and analyze screenshot test failures?

My name is Alexander Gonchar, I am a quality assurance engineer at T-Bank. I want to share my experience on how to get rid of screenshot tests.

What do screenshot tests depend on?

Screenshot testing is very popular: what could be simpler than comparing a current screenshot with a reference one.

Having worked with screenshot tests for many years, I have encountered a wide variety of problems that cause such tests to fail. I tried to summarize the factors that screenshot tests depend on:

• Storage: read or write access to CI locally and with colleagues; amount of allocated space, network access, monetary costs.

• Differences between launch environments – locally and in the CI image: OS, browser version and settings, regional settings, time zone, screen resolution, fonts, and so on.

• Code changes in your and third-party teams: styles, new common components, new functionality affecting several products or pages at once, refactoring, design;

• The need to update standards when there are valid changes in functionality, design, browser, and so on.

These problems led to a lot of time being spent on analysis, restarting tests, and updating standards. Something had to be done about this.

To decide whether we need to continue working with screenshot tests, we decided to find out:

• advantages of not taking screenshot tests;

• what is checked by screenshot tests;

• how compilation works in Angular and rendering in the browser.

Research results

Pros of not using screenshot testing:

1. Test execution speed. A separate test runs faster, since there are no actions with screenshots: saving, proofreading, comparing, updating.

For example, several passage time measurements a simple test in the same environment showed that tests without screenshots run faster:

2. CI pass rate: Reducing test execution times has a positive effect on pass times, such as MR.

3. Speed ​​of feature delivery to the customer: lead time improves, the customer and the business are happy.

4. There is no need to waste time analyzing failed tests – for example, if the test failed due to a small difference between the screenshots.

5. No need to update the standard.

6. No need to waste resources restarting failed tests.

7. Testers can devote more time to testing, quality assurance processes, and advanced training.

What we check using screenshot testing. If you look at the screenshot test, we will see that it checks the identity of the current screenshot to the reference one. The test does not check whether the logic worked correctly, for example, calculating cashback. The test does not check whether the user sees the required data, such as the account amount or notification.

An analogy can be made that it’s like using a UI login in an e2e test and being dependent on it and on the team that develops the authorization mechanism. Why test a login if the test is supposed to test something else?

The quality assurance process assumes that during testing we evaluate how the software will be tested, what checks will be carried out at what level or stage.

We need to understand that we need to test our product, and not the logic, design elements or components of, for example, another team.

A logical question: what about checking the design, layout, that it didn’t work, that icons and pictures are displayed correctly, that the data is displayed on the page, rendering in the browser, after all?

Let's consider this question using the example of an Angular application.

Compilation in Angular. Historical background: since Angular 9, used AOT compilation. It is more reliable than JIT compilation, which was used before. AOT compilation consists of three stages: analysis, code generation and validation, which allows you to avoid errors in the properties and methods of components and services in templates.

In AOT mode, compilation occurs when the application is built before loading in the browser, with HTML and CSS files included in JavaScript files as a string, and there is build error detection, reducing the chance of them getting into the application.

The chance that layout, static text, styles and other elements will have errors tends to zero, although they may arise due to errors in Angular itself or development errors. These are errors that passed the code review, but were not noticed on the feature branch, test bench, pre-production, and so on.

Rendering in the browser looks like that:

• The browser downloads, reads and parses the sources. At the same time, DOM from the HTML document and CSSOM styles are loaded.

• It is being formed render tree is a collection of rendering objects or frames. They contain their corresponding DOM objects and styles.

• The layout and paint processes are performed, when the position on the page is calculated for each object, and then rendered.

Thanks to the way Angular prepares the necessary data for rendering in the browser, the likelihood of rendering problems is also close to zero.

Level of checks. Have you ever wondered why this or that test is covered by the e2e test? Where is the test written if a bug was found in the production: do you add an e2e test, an integration test, or go to the developers so that they write a unit test? SLT and testing pyramid come to the rescue to answer these questions.

As an example, let's take a couple of screenshot tests that check that the user sees a page with one or more managers, with the title “Manager ”, which displays the name of the manager and the corresponding icon, depending on the status, for example, verification.

Questions may arise: how is the data displayed on the page obtained? Is this a back or front function? Are unit tests written for this function and its logical paths? What unit tests are there for the front-end component, which draws the required text and icon depending on the received data?

Answering these questions one by one, we see that we not only don’t need a screenshot test, we don’t even need an e2e test: all the necessary checks can be implemented at lower levels. The logic for obtaining manager data can be covered with unit tests both at the back and at the front. The display of text and icons can be covered with unit tests on the front.

As a result, thanks to modern browsers and compilers, there is no need to:

• test static text, static icons and other static elements with screenshots, since they will almost always be on the page;

• test with screenshots elements that are displayed depending on certain conditions: response from the API, execution of a function on the front, fulfillment of a condition, for example ngIf.

Moving checks into integration and unit tests greatly reduces the likelihood of regression errors, since testing in the early stages catches potential problems much faster, which are very cheap to fix. The same unit tests run quickly, on an almost constant basis and react to code changes.

Here are a few examples of front-end unit tests that test the display of various data, where at the beginning of the test a method is used in which one or more managers are passed to the component.

Sole leader:

it('Должен вернуть тексты для единственного руководителя', () => {
    	defineDirectorsAndInitializeFixture(singleDirector);
 
    	const headerElem: HTMLElement = fixture.nativeElement.querySelector('[automation-id=director-header]');
    	const headerTextElem: HTMLElement = fixture.nativeElement.querySelector('[automation-id=director-text]');
 
    	expect(headerElem.textContent?.trim()).toBe('Руководитель');
    	expect(headerTextElem.textContent?.trim()).toBe(
        	'Укажите актуальные данные руководителя для продолжения работы с заявкой',
    	);
	});

Several leaders:

it('Должен вернуть тексты для нескольких руководителей', () => {
    	defineDirectorsAndInitializeFixture(severalDirectors);
 
    	const headerElem: HTMLElement = fixture.nativeElement.querySelector('[automation-id=director-header]');
    	const headerTextElem: HTMLElement = fixture.nativeElement.querySelector('[automation-id=director-text]');
 
    	expect(headerElem.textContent?.trim()).toBe('Руководители');
    	expect(headerTextElem.textContent?.trim()).toBe(
        	'Укажите актуальные данные руководителей для продолжения работы с заявкой',
    	);
});

Leader Icons:

it('Должен вернуть все типы иконок руководителя', () => {
    	defineDirectorsAndInitializeFixture(severalDirectors);
 
    	const connStoreDebugElem: DebugElement = fixture.debugElement;
    	const btnDebugElems: DebugElement[] = connStoreDebugElem.queryAll(By.css('[automation-id=director-icon]'));
 
    	const iconProperty = 'iconState';
    	expect(btnDebugElems[0].properties[iconProperty]).toBe('normal');
    	expect(btnDebugElems[1].properties[iconProperty]).toBe('error');
	});

Screenshot Test Opt-Out Process

To refuse screenshots, you need to:

• Collect defect metrics related to the front. Divide them into logic defects and design or layout defects to understand how often they arise.

• Select a set of critical tests that you can leave alone at first, for peace of mind.

• Select checks that are performed with screenshots and replace them with checks without screenshots. For example, accessing an element in the DOM, reading and checking its value and attributes

• Select tests, including tests from the point above, the verification of which can be implemented in integration or unit tests.

• Implement the checks from the point above in integration and unit tests.

• Avoid e2e-level tests whose checks are implemented in integration or unit tests.

• Return to the set of critical tests and perform steps 4-7 for them.

• Additionally, if you find a bug in the test environment or production, when fixing it, be sure to write a unit or integration test.

Instead of a conclusion

We got rid of screenshot tests, revised approaches to testing and the level of checks in general, because one does not work without the other. Testers became more involved in the processes and learned to write unit tests at the front.

One might ask: why should testers delve into unit tests, much less be able to write them, since traditionally this is the task of developers? We strive to ensure that testers are involved at all stages of the software life cycle, including unit testing. A tester can come to the developers with checks that need to be covered. Or implement them yourself.

Knowing how to write front-end unit tests is a good skill and it's very interesting.

Thank you for your time. Finally, here are some useful links: