TaaS and the Five Gates of Quality

Hello everyone! My name is Andrey Petukhov, I am a tech lead of the Testing Experience AvitoTech team, I am engaged in development testing systems V Avito. In this article, I'll tell you how we organized our testing process in a microservice architecture. Below, you'll learn how to use Testing as a Service (TaaS), why quality gateways are needed, and how all this helped testers focus.

How We Came to Testing as a Service

Initially, Avito was a monolith. Then testing was organized like this:

• one repository and one test suite;

• each change goes through unit, integration and e2e tests;

• tests catch bugs, sometimes they fail, the responsible guys monitor this during releases and everything is fine.

The business grew, and the number of people increased. The monolith began to slow down the development and delivery of features, so we began to move to a microservice architecture.

Units and integration tests went to microservices, but end2end tests, for example, remained in the monolith. Moreover, the monolith was released less and less often, and microservices grew exponentially, 200 were added per month. As a result, we lost a single entry point to the overall testing process.

We started noticing the problem before it became critical. And we came up with a solution — Testing as a Service (TaaS).

Testing as a Service in Avito

Our Testing as a Service is based on the Quality Gates concept. This is a software quality control mechanism that checks the code for compliance with pre-set criteria at different levels – gates. Checks can range from code to the operability of critical scenarios without being tied to a team. If the code passes all the gates, it is considered acceptable for further use or release. If not, it requires revision.

Gates are applicable to any entity within the pipeline: monolith, microservice, large service – it doesn't matter.

We wanted to make a system that would implement the Quality Gates concept so that each team could fine-tune the gates to their needs. And we did it, I'll tell you how now.

We have five Quality Gates. In the diagram below there are four, because the fifth one happens after the release and is not so interesting for us now.

1. Quality Gate 1. Intro – happens on the developer's machine or in CI.

2. Quality Gate 2. Units and integration tests.

3. Quality Gate 3. End-to-end tests of the service.

4. Quality Gate 4. End-to-end tests of Avito as a product.

Here are the main requirements we were aiming for:

• maintaining autonomywhich is provided by the microservice architecture. Teams are free to develop and release in their own cycles;

• different services – different needs. Since services vary in criticality, not everyone needs a set of all checks: units, integration tests, local and global end2end. For example, if it is a small service that lays out badges in a profile, there is no need to check it with end2end tests;

• testing is the responsibility of the team that owns it a specific service or piece of functionality.

What TaaS can do in Avito

Below is a list of features that Testing as a Service already supports in Avito:

• different mechanics for running tests. Manually, on PR, on schedule;

• metrics collection. Time for testing, flakiness for e2e tests, success rate of assembly with tests;

• centralized management of critical aspects. Coverage percentage, insufficient testing, etc. For example, if there is insufficient testing coverage of an important service, we simply won't let it be deployed.

We will not go into details of the technical implementation, because it is very specific for us and may differ from what you have.

How TaaS works in Avito

There are three entry points to TaaS:

• service dashboard. An internal UI where a developer or tester can go and run a test, for example;

• pipelines (in the diagram below – Deploy);

• Pull Requests.

The core principle of TaaS is single responsibility. TaaS doesn't know how it's being used. It doesn't know from which entry point the request comes: service, CLI, user — it doesn't matter. All it has is a request “give me a set of tests for this target”.

For different scenarios, we use different flows internally. If we understand that we additionally need, for example, a synchronous way of interaction, we can simply screw it on, based on the current interface.

Such simplicity is the key to flexibility. For example, we didn't have cron launches, but the guys needed nightly builds. We quickly added this mechanic practically for free.

What TaaS looks like in Avito

The service dashboard for the service owner looks like this:

Each gate can be opened and…

• view gate test reports;

• run gate tests;

• edit gate;

• remove gate.

The compromises we made

We had to make a number of compromises to get TaaS up and running. Here are the assumptions we're making for now.

• Only backend services are supported. It takes much more time to bring the frontend and mobile services into the common pipeline;

• It is not advisable to add a gate layer for all stages. For example, unit tests do not necessarily need to be dragged into the general pipeline, since they are always located next to the service. We know about them and pull information to the dashboard, but we do not allow this process to be managed;

• many different gates can accumulate on public utilities. We haven't completely moved away from services where a bunch of people contribute. Each team sets up their own gate, each of which can have 50 tests. When all the gates are launched, everything lags.

Benefits from TaaS in Avito

Despite the compromises, we have managed to develop a system that so far meets all our requirements in terms of scalability and autonomy.

• self-government. Creation, configuration and management of automated testing stages are the responsibility of the service owners;

• autonomy. Removed the bottleneck for “under the hood” processes. For example, for creating builds in CI and configuring them. Previously, you had to go to special people and wait a week for a TeamCity build. Now it's enough to click a couple of buttons in the UI and fill in a few lines;

• decentralization of testing. Each service is tested when it needs to be tested, regardless of external factors. There is no single entry point into automation;

• intuitive launch settings. The tester does not need anything to run the test: now there is no need to dive into the CI/CD settings and script in bash to run tests on the agent. The tester writes tests, and TaaS does the rest.

Product Metrics

Benefits are benefits, but we at Avito really love specific numbers. And we have them! The metrics show that the product is used and that it brings benefits to Avito's engineering teams.

• 732 configs with e2e tests for 308 services. Even more services with only unit tests;

• 2867 times in April 2024 users visited the dashboard and performed a target action;

• The use of key features has increased 4 times since launch to the present day.

Why do you need this?

Companies are growing, and in such conditions you may find yourself in a situation where the development process has moved ahead and adjusted to growth, but the testing process has not. Update your processes and tools to accommodate this growth.

Simplifying the testers' job = more focus on quality, not routine. A tester should test and find bugs, not work part-time as a DevOps. One option to organize this is Testing as a Service.

Thank you for your time! How do you conduct tests in a microservice architecture? What tools do you use and what solutions do you consider the best? Share your experience in the comments!

Read about how to relieve a QA engineer of daily routine using the example of a single Avito team at this link. From the text, you will learn about our experiment on writing 5000 tests and assembling a generator for testing.

And also subscribe to AvitoTech channel in Telegramwhere we talk more about the professional experience of our engineers, projects and work at Avito, and also announce meetups and articles.