automate custom scripts in web interfaces

Hello! My name is Evgeniy Konstantinov. I am developing automatic testing tools in Yandex. In this article I want to tell you about the history of the creation of the Testplane tool, which is designed for testing user scripts in the front end and is now available in open source.

Testplane will help you:

• test different scenarios in any browser;

• check the layout using screenshot testing;

• test your React components;

• run client unit tests in the browser context.

How it all began

It all started back in 2014, when a library of common components called “Lego” was developed inside Yandex, and outside – a library of components BEM. In both cases, ready-made blocks with layout were implemented, which somehow needed to be tested. The most suitable way for this is to take a screenshot of the component and compare it in the “was/is now” format. But after a long search on the Internet, we did not find a ready-made solution with which we could easily and quickly write tests in JavaScript, run them in the browser and see the results in the report.

At that moment it appeared Gemini — a utility for regression testing of the appearance of web pages. The tool had the simplest (but at the same time limited) API, its own implementation of screenshot comparison (including a patent for the testing method) and a report for analyzing the results. This was enough to cover all the main cases when testing interfaces.

But time passed, and testing requirements changed. Screenshots alone were not enough – I wanted more. Namely, testing real user scenarios. Yes, most scenarios can be checked using screenshots (for example, that the required data was inserted into the fields), but it is obvious that such testing is redundant and not always convenient. That's why in 2016 we published a new tool, Hermione.

This tool was a composition of three tools: Gemini, Webdriverio and Mocha. In this bundle, we received screenshot testing with all our developments from Gemini, a rich API for interacting with the browser from Webdriverio, and a test writing interface familiar to users from Mocha.

Why are we going open source?

For a long time, the main users of Hermione were internal services, and we have never been particularly involved in promoting it outside, even though the tool has been open source since 2016. For new developers coming from outside, it was a completely new tool. In addition, a search for the word Hermione most often reveals only the story of the heroine of JK Rowling’s books.

In general, it turns out that to go open source it is not enough to simply push the source code to GitHub (surprise, surprise). Therefore, this time we took the choice of a new name seriously and analyzed it for the presence of a trademark, domain for the site, style and other details.

I won’t describe in detail how to go open source correctly. Instead, I simply invite everyone who is interested to watch the broadcast. Yandex Open Source Jam, where you can find out our history. However, I will say that as a result of the rebranding, a new product called Testplane appeared.

Who is this Testplane of yours and what is its feature?

At the moment, our tool allows you to solve several testing tasks at once – checking user scripts in the browser, finding bugs in the layout using screenshot testing, writing tests for components (for example, in React) or running regular unit tests in the browser.

The tests themselves can be written in a syntax that is familiar to many. For example, this is what a simple test looks like:

describe("test", () => {
    it("example", async ({browser}) => {
        await browser.url("https://github.com/gemini-testing/testplane"); // Открываем тестируемый урл
        const aboutSection = await browser.$(".f4.my-3"); // Ищем нужный элемент на странице
        await expect(aboutSection).toHaveText("Browser test runner based on mocha and wdio"); // Библиотека ассертов доступна из коробки
        await aboutSection.assertView('plain-view'); // С помощью скриншота проверяем вёрстку
    });
});

Plugin system

For our tool, it’s quite easy to write your own plugin, in which you can implement your custom logic. This could be your own reporter, raising the dev server before starting a test, or sending statistics on test results. You can read more about available events in documentation. In addition, with the help of plugins you can even add custom CLI commands and options. For example, this is what a mini-plugin will look like, which will disable unstable tests in runtime if they fail:

export default (testplane: Testplane) => {
    const someStatStorage = require('your-stat-storage-client');

    // Перехватываем событие, чтобы отключить нестабильный тест
    testplane.intercept(testplane.events.TEST_FAIL, ({event, data: test}) => {
  // Проверяем, является ли тест нестабильным
        const isFlaky = someStatStorage.getTestStat(test.fullTitle());        if (isFlaky) {
		// Отключаем тест и указываем причину отключения
            test.skip({reason: 'This test was disabled automatically'});             

            // Вместо события FAIL бросаем событие PENDING (), на которое могут подписаться другие плагины 
            return { event: testplane.events.TEST_PENDING, data: test };
        }

  // Если перехватывать событие не нужно, достаточно вернуть любое falsey-значение
        return; 
    });
};

Scalability

Currently, using our tool, internal services in Yandex run tens of thousands of tests, which are completed in less than 10 minutes. This is achieved both due to the parallel execution of tests within one run, and due to the fact that the test pack can be divided into any number of chunks, which can be launched on different machines in CI. At the same time, at the output the user analyzes a single HTML report, which will display tests from all chunks.

Screenshot testing + UI for working with screenshots

Screenshot testing is an integral part of testing the service layout. Accordingly, we pay special attention to this type of testing: we have implemented our own comparison library looks-same with wide capabilities, convenient UI for working with screenshots (comparison, search for differences, updating screenshots) and automatic testing Storybook components (tests are generated automatically at runtime, and you only need to take reference screenshots in the UI):

Step-by-step execution of tests using REPL mode

To make it easier to write tests (as well as debug them), our tool has a REPL mode. With its help, the user can execute the test code step by step (line by line) to check its performance or find a problem area. In this mode, the user in runtime sees the process of executing his test in a locally launched browser or even in a remote browser via VNC. The main feature is that this testing option is available for absolutely any browser: Chrome, and even a remotely raised iOS device.

How to try

To configure Testplane in your project or simply build a test project, just run the command in the terminal:

npm init testplane some-project 
// Если не указать директорию, то проект будет развёрнут в текущей директории

After its execution, all the necessary npm dependencies will be installed in the specified directory, a config will be generated for running tests with pre-installed plugins, and a trial test will be added.

You can run tests using the CLI command npx testplane or through the UI, which can be opened with the command npx testplane gui.

Try Testplane, bring it feedback in issues and your PR. Also watch the broadcast Yandex Open Source Jamto learn more about open source – we are waiting for you!