We optimize automation: how we accelerated autotests by 3-4 times, preserving the old developments
A few words about the project
Although we cannot disclose the details of the project due to the NDA, in general terms, the task was as follows. We joined the development of the fintech API service, which interacted with the database, returning the necessary financial objects (prices, tariffs, etc.). Our task was to test mobile clients for this service – both web and native mobile applications.
Test automation on this project developed gradually, along with the complexity of the service. This is probably how long end-to-end tests appeared at one time, which we found on the project. Most of them did not work by that time, because the service had changed, and there was no one to support the tests – the only automation engineer left the project long before we arrived.
Even those tests that seem to correspond to the functionality sometimes fell due to confusion with versions or inaccessibility of external resources. For testing, a separate infrastructure was used – test benches, where the necessary versions were deployed for experiments. Different working groups had access to it, and they did not always act in concert. As a result of the actions of one group, some important API used by our service could fall off, because of which even a working test stopped passing. Those. the test no longer showed the serviceability of the service itself, but rather related to the test infrastructure as a whole.
How we came to chaos
It would seem that in this situation it is necessary to abandon all the old achievements and build the testing anew. But we acted more “humanely”. The test structure itself was preserved, focusing on solving specific problems – slow passage of tests, their instability and insufficient coverage of test cases. For each of them there was a solution.
Refactoring
First of all, we partially reworked the code of old tests, relying on more modern design patterns.
Part of the legacy code had to be removed – it was too difficult to maintain. In another part, we caught all the weaknesses – we replaced default sleep with normal wait-ers, made preparations for all tests in the global setup via annotations of test runners, etc. Many small steps reduced the average end-to-end test pass from 3-4 to 1-2 minutes.
Atomic approach
To speed up the creation of new tests and simplify the support of old ones, we have gone from cumbersome end-to-end cases.
Personally, I have nothing fundamentally against end-to-end testing, however, in the case when you need to check one specific screen (or even part of the information on it), going through all the steps, starting with user authorization, is too expensive. Imagine that we are testing an online store and we only need to check the check that will be sent to the buyer after purchasing a certain product. Instead of fetching only one screen from the system, we would go by login and password, select the tomar, confirm the purchase, etc. – would perform many steps that are not related to a specific testing task. But every step takes time. Even with all the optimization carried out, the launch of the end-to-end test took up to 2 minutes, while the verification of a specific screen took only 10 seconds. Therefore, where it was possible, we moved on to such “atomic” checks, referring only to the screen that interests us as part of the test case.
Along the way, just for screen comparison, we implemented snapshot testing, which allows you to check the lion’s share of the UI. Having tests and application code in one repository, we can use the methods of this application in tests, i.e. raise any screens that are needed in this process. So we can find errors in comparing test screenshots with reference ones.
We now have about 300 snapshot tests, and their number is gradually growing, since this approach can significantly reduce the time it takes to check the finished version before sending it to production. This whole set of tests starts automatically when pull request is opened and runs in 40 minutes – so developers quickly receive feedback about problems in the current branch.
Of course, a number of end-to-end tests have been preserved. You can’t do without them where you need to verify large business scenarios, but it makes sense to run them when all the details have already been verified.
Mocking
To exclude the influence of an unstable test bench on the result of the launch of our tests, we launched a mock server. About what decisions we then considered and why chose Okhttpmockwebserver, I already wrote on Habré.
As a result, the share of episodically falling ones due to external causes of the tests decreased significantly.
Kotlin DSL
In parallel, we made the tests more readable.
Those involved in UI testing know how difficult it is to find the truth among a bunch of locators in the long “footcloth” of the test (especially at the stage when it was still end-to-end tests). It’s easy to navigate in them when you’ve been on a project for two years and even in the middle of the night can remember what is what. But if you just came, moving into what is happening is a separate big task. So that new people do not have to deal with it every time, we decided to switch to Kotlin DSL. It is implemented quite simply and has a simple and understandable structure. Previously, tests consisted of a set of identical low-level calls – clicks, text input, scrolls, but now all this has turned into something more “business” – something like a BDD approach. Everything is visible and understandable.
In my opinion, this made us a certain reserve for the future. This project has already once faced the departure of a single automation engineer. For the tests, this did not end in the best way – they simply stopped supporting them, because the entry threshold turned out to be too high. Understanding such a dry code required a lot of time and a certain qualification. We redesigned the tests in such a way that it will be possible to quickly transfer people from other projects or from manual testing to automation at any time. Almost anyone can write the simplest tests on Kotlin DSL. So the automation can leave the low-level implementation, and to quickly write new simple tests to connect people from the functional team. They have enough knowledge of business logic, and the project will benefit from the fact that they will be more involved in the process of writing autotests. Kotlin DSL allows you to describe test cases exactly as they would like to see all the checks, leaving the low-level implementation of methods outside the scope of their work.
In general, all this made it possible to increase the coverage of autotests faster. If earlier it took 16-20 hours to implement the new test suite, then with the new approach, depending on the complexity of the tests, it takes from 4 to 12 hours (and the labor costs for support were reduced from 16-24 to 8-12 hours per week )
Article author: Ruslan Abdulin.
P.S. We publish our articles on several Runet sites. Subscribe to our pages in VK, Fb, Instagram or Telegram channelto learn about all of our publications and other Maxilect news.
P.P.S. Help us make blog articles more interesting: docs.google.com/forms/d/e/1FAIpQLSeqnPceNuK-JopYVxgF15gNWLIi5oM_AZesioCDGXhvr7Y7tw/viewform.
