coverage analysis and autotests

Writing automated tests is one of the most effective practices that allows you to check the functionality of all service components and promptly detect any failures. But writing a lot of automated tests is not always the right approach: there should not be many tests, but enough. Otherwise, the time and effort of testing and automation engineers will be wasted. But how do you know that there are enough automated tests, and, most importantly, how can you be sure that updating or adding new methods within the project will not create “untraceable gaps”? Obviously, this requires an automated test coverage analysis system.

We continue the series of articles about autotests in OK (you can read materials on this topic Here, Here And Here). And today we will tell you how the OK team copes with these tasks using the example of analyzing the coverage of API autotests.

The article is based on the joint report by the head of the testing automation team Emilia Kutsareva and the testing automation engineer at Odnoklassniki Elizaveta Andreeva “API: coverage analysis and autotests” at the IT conference “Stachka”.

About autotests in OK

OK is a large social network with a huge backend that includes thousands of methods and services. Constant availability (uptime) and fault tolerance of the product are critically important to us, so to track the performance of all its components, we try to cover everything necessary with test scenarios, and replace manual tests with automated ones whenever possible.

We currently have about 10 thousand automated tests across all major OK platforms:

• web — about 3150;

• API – about 2500 (+1500 auto-generated);

• Android – about 1400;

• mobile web — about 1200;

• iOS – about 1000.

In terms of number of calls and, accordingly, priority, API is the number one platform. This is due to the fact that API is used by both other platforms and external developers.

At the same time, OK is not only a large, but also a “live” product – we are constantly improving and improving the social network on all platforms to meet user demands. The approach where you can write autotests once and successfully use them throughout the entire product life cycle is not our case. Therefore, we allocate a lot of resources to work with autotests.

The nuance is that our resources are not unlimited, that is, it is important for us not only to control that at any given time all priority methods are covered by tests, but also to understand when there are enough tests. And this task is impossible without coverage analysis.

This will be discussed in more detail using the example of API coverage analysis.

Methods of assessing coverage

To understand the required API coverage and assess its relevance, we take information from various sources.

• Test Management System. When working with test cases in TMS, we can add marks about what is covered by automated tests and what is not. The approach is good because it allows us to take into account all the business logic, but to get the most objective picture, additional methods of analysis are needed. In addition, most often, test case marking is done manually, which takes a lot of time.

• Analysis of real user actions. We can analyze user actions, such as which parts of a site or application people interact with most often, to identify the most used API methods. Based on this information, we can prioritize test coverage.

• Automated API and UI tests. We can see which API methods are called by automated tests, and thus check what exactly we already have covered by automated tests.

To keep track of new methods and coverage tasks, we have added the following options.

• Jobs in TeamCity. Using jobs in TeamCity we monitor for new and changed methods.

• Jira-tasksWhen a new method appears, a task is automatically created in Jira to cover it with automated tests.

• Manual analysis. We also monitor method coverage using various configured filters, installed tags, collected dashboards in Jira, and even regular tables.

But such monitoring was not enough to process a huge backlog of methods and understand what needs to be covered with automated tests first. Accordingly, such an implementation lacks the final goal: it is unclear when to stop and focus on working only with new features.

A little background: the old approach to prioritization

It is obvious that in such a large project as OK, there is a huge number of API methods, and we cannot cover them with automated tests in any order – this is an initially wrong way. Therefore, we had prioritization, but it was quite simple.

First of all, we looked at the availability of tests for each individual method. For methods for which there are no tests, we obtained the number of calls over the last month. As a result, a list of methods requiring coverage was formed, and we ranked them by the number of calls. The logic is as follows:

Most calls = higher coverage priority

But this approach has its shortcomings. Let's highlight the main ones.

• Old tests = tests that provide sufficient verification. The current approach assumes that existing tests provide sufficient coverage of methods. Therefore, only methods that do not have tests are included in the ranking. It turns out that if the signature of a method has changed, for example, new parameters have been added, this will not be taken into account in any way and writing new tests for these parameters will not be a priority.

• We consider all services to be equally important. Ranking does not take into account the criticality of the service. Methods that work with sensitive user data are not highlighted in the list for coverage. For example, one method can work with messages, and another with user money, but be called much less often than the first, and therefore be lower in the queue for coverage, which is incorrect from the product's point of view.

• There are still a lot of tasks to doThe list of methods to be covered by automated tests can be long and still does not provide an understanding of priorities. In addition, there is no answer to the questions: “When can the coverage of tests for a method be considered sufficient? Is it possible to stop and do other tasks?”

Coverage Priorities: A New Metric

Over time, we decided to revise our approach to determining coverage priority and eliminate the aforementioned shortcomings.

Now we evaluate a set of parameters for each method at once and assign different coefficients for them (from 0 to 3).

Let's take a closer look at the added parameters.

• Availability of tests for the method. If there are no tests yet, the coefficient is higher; if there already are, it is lower.

• Number of method calls. The more method calls, the more critical its coverage and, accordingly, the higher the coefficient.

• Criticality of the service. Now the method for working with user money from the example in the previous section will be higher in the top.

• Number of existing tests per method. In our case, this value is related to the number of parameters in the method (with rare exceptions). This was added so as not to forget about changes in method signatures.

• Method availability to external user. Naturally, such methods need to be covered first. Previously, this was taken into account, but was not recorded anywhere (except in the head of the testing engineer).

• Possibility to work from different sessions. We tried to reconsider how we write tests now, and realized that we were looking at the impossibility of calling a method from different sessions and paid less attention to additional checks of the operation of those methods that, on the contrary, allow work from them.

• The presence of privacy restrictions. Here, too, the coefficient is assigned depending on the presence of restrictions.

Expanding the list of parameters taken into account and adding coefficients to them allowed us to change the logic of prioritization. So, now for each method:

Coverage priority = sum of coefficients

With this approach, we minimize the influence of human factors and unjustified randomization.

Note: The approach we have developed is universal. Therefore, if necessary, you can apply a similar priority assessment scheme to each project – all you need to do is rework it taking into account the metrics that are relevant specifically for you.

Project technology stack

Our project with automated tests is written in Java using JUnit5, Spring, Maven, Hamcrest.

The key tool in our implementation is a custom API client. This is a clear framework for developers and QA engineers, with which you can easily add a new method or change an existing one.

Many actions in it are automated. For example, when creating a new method in a special annotation, with which we mark the parameters, you can indicate the importance of the new parameter. After that, you do not need to write checks for its presence in the request – this will be done at the framework level.

In addition, the client used in autotests is automatically generated for each API project build. If the method was added to several main files, it will appear in the assembled client, and it will be possible to write autotests for it by connecting a new version of the client to the project.

Our documentation apiok.ru is also created automatically using the Reflection API. Templates are created based on certain annotations. The only thing left for a person to fill in is a verbal description. Return value types, the need for parameters and sessions are generated automatically.

I would like to separately note the non-obvious advantage of a custom client: we can use it not only for writing API tests, but also for automation on other platforms: Web, Mobile Web, Android and iOS.

Coverage measurement: Inspector

In pursuit of automation of “everything that is rational and justifiable to automate,” we could not help but automate the coverage prioritization assessment.

For these tasks, we use the Inspector service we developed, which analyzes API coverage with autotests and shows which methods are not covered.

Inspector is also written in Java, like the test project, using Spring, Gradle, Angular. At the moment it works with our old prioritization formula.

One of the main “trump cards” of Inspector is its convenient, visual interface.

Note: Our tool already has a user-friendly UI, but When developing your solution, first of all, it makes sense to make a working prototype with MVP, which will already allow you to reduce manual work and receive the first results of coverage analysis.

On the main page we can see:

• list of reports on monitored test runs on three environments: production, pre-production and test environment;

• number of test methods launched in the run;

• number of methods (called in a test run/run and total number).

You can open a more detailed report with tests and see what API requests were made in them. In addition, meta information is collected, so even the tags set in the test are displayed. There are links to other services, for example, from here you can go directly to the IDE for the desired test and view the code or open a report with the test execution steps.

Inspector provides request/response data from API tests in a convenient and structured table format.

In Inspector we also have access to reports on the coverage of API methods by tests in rounds. Moreover, we can filter all the information by the presence of tests and categories, as well as customize the display of the number of method calls for the required platforms over the past month.

In addition, reports can be exported to Excel for convenient work with the results obtained. When transferring, the installed filters are saved.

Technical implementation

“Under the hood” Inspector is integrated with our Data Warehouse. In OK, DWH is responsible for storing, processing and analyzing data on user activity on the portal, as well as generating KPIs for the analytics department and project managers. But as part of the integration with Inspector, DWH is used to obtain data on the number of method calls.

Inspector integration with our DWH is implemented via Druid, a system built to provide fast (real-time) access to large sets of rarely changing data. Druid is accessed via a JSON request using the POST method by application ID and time range.

The algorithm for obtaining and processing data now consists of several steps.

1. In the test project, a listener is initialized that controls the writing of data to the Inspector.

The InspectorTestExecutionListener interface has been implemented, which implements TestExecutionListener from JUnit5. TestExecutionListener allows you to integrate into the desired location for executing test suites and even collect metadata. When starting a test suite, we pass the name of the test run to the service using the testPlanExecutionStarted override.

2. During the autotest run, all API requests are intercepted and saved.

When a single test starts, we add it and the list of tags to InspectorRequestHolder, which serves as a container for storing intercepted HTTP requests.

In the startTest method, we simply assign the name and tags of the test.

Upon completion of the test, we save the results in a list and clear the variables that relate to the execution of the current test.

3. After the run is completed, information about tests and methods is loaded into the Inspector.

A report is generated and sent in JSON format, which includes the name of the test run, the current date, test information that includes meta information and HTTP requests made within the test, and the version number of our custom API client.

4. In the service, data is parsed and laid out in the database: tests separately, methods separately, round with coverage calculation separately.

5. A list of public/private methods is taken from the documentation repository.

6. All methods present in the API are added to the methods obtained from the run. The coverage will be calculated based on them.

7. The number of calls by platform is uploaded to Druid once a day with the call data for the last month.

8. From these tables, rounds/tests/methods are drawn on the UI.

The Future of Inspector

Now Inspector copes well with all current tasks: it significantly simplifies our work with autotests and coverage analysis. But we plan to make the tool even more convenient and functional. Thus, we plan to:

• Add a job with a new formula for choosing priorities. To be able to automatically calculate the coverage priority based on the sum of the coefficients.

• Improve the UIIn addition to the changes related to the new formula, we want to add the ability for QA engineers to create filters by commands and display only the necessary categories in the interface.

• Connect CI/CD. So that coverage assessment occurs automatically when writing automated tests.

Plans for the development of automation of the API platform

We have many methods, and their number is constantly growing. That's why we recently implemented automatic generation of API autotests. This helped us reduce the workload of specialists and significantly increase the coverage rate. At the same time, autogeneration does not affect methods related to business logic – these tasks will remain the responsibility of our QA engineers.

Now let's take a closer look at the direction in which we plan to develop the automation of our API platform.

• Automating Coverage Analysis Iterations. We want to make an automatic coverage analysis that will be updated at a certain frequency. Currently, this task is performed by a core QA engineer, who does part of the work manually. The improvement of Inspector will partly make the work of a core QA engineer easier, but we also want any QA engineer and test team leader to be able to use the service to track the current state of their project.

• Monitoring: metrics and alerts. Implementing monitoring will allow us to track business and technical metrics in real time, which can be useful for both, for example, the technical director and the developer. In turn, connecting alerts will give us the opportunity to immediately notify QA engineers about the emergence of new priority methods that require coverage by automated tests – this will make tracking controllable and understandable.

• Implementation of SLA. We want to reach a situation where we can guarantee certain fixed deadlines for covering API methods with automated tests. We also set deadlines now, but we do not yet clearly fix them. We want to regulate this with the help of automation tools.

• Quality Gate. In the future, we want to implement Quality Gates. This way, we will be able, for example, to block pull requests to the main branch if the code is not sufficiently covered by automated tests.

• Automation of API and Mobile Client Communication Testing. Android users make up 53% of OK's monthly audience, while iOS accounts for 9%. Both Android and iOS platforms rely on API calls. That's why it's important for us to check how new releases for mobile clients work with the API. We already have an option to check Android releases by running API autotests on a test environment, but this launch is not yet automatic.

Thus, our automation platform will undergo significant optimization in the future, which will make working with autotests even more convenient and efficient.

Conclusion

To work correctly with automated tests, it is important not only to think about writing the tests themselves, but also to take care of sufficient coverage – without this, it is impossible to guarantee the quality of the product, especially with its dynamic refinement and regular updating of methods.

Along with this, we must also think about the correct assignment of priorities – otherwise we may encounter a situation in which methods that are critical are “in eternal standby mode”.

Moreover, based on the experience of working with thousands of automated tests, we came to the concept that everything possible and economically feasible should be automated, and the coverage priority should be determined taking into account the criticality and urgency. Following this approach makes it possible to exclude the human factor, reduce the workload of specialists, and make all processes transparent and predictable.