Automated microservice testing in Docker for continuous integration

In projects related to the development of microservice architecture, CI / CD moves from the category of pleasant opportunity to the category of urgent need. Automatic testing is an integral part of continuous integration, a competent approach to which can give the team a lot of pleasant evenings with family and friends. Otherwise, the project runs the risk of never being completed.

You can cover the entire microservice code with unit tests with mock objects, but this only partially solves the problem and leaves a lot of questions and difficulties, especially when testing work with data. As always, the most acute ones are testing the consistency of data in a relational database, testing working with cloud services, and incorrect assumptions when writing mock objects.

All this and a little more is solved by testing the whole microservice in the Docker container. An undoubted advantage for ensuring the validity of tests is that the same Docker images that go into production are tested.

Automation of this approach presents a number of problems, the solution of which will be described below:

• conflicts of parallel tasks in one docker host;
• conflicts of identifiers in the database during test iterations;
• waiting for the availability of microservices;
• Association and output of logs to external systems;
• testing outgoing HTTP requests;
• testing web sockets (using SignalR);
• OAuth authentication and authorization testing.

This article is based on my speech at SECR 2019. So for those who are too lazy to read, here is a recording of the performance.

In this article, I will tell you how to use the script to run the tested service, database and Amazon AWS services in Docker, then the tests on Postman and, after their completion, stop and delete the created containers. Tests are performed every time the code changes. Thus, we make sure that each version works correctly with the AWS database and services.

The same script is run both by the developers themselves on their Windows desktops and by the Gitlab CI server for Linux.

For the introduction of new tests to be justified, it should not require the installation of additional tools either on the developer’s computer or on the server where the tests are run when committing. Docker solves this problem.

The test should run on the local server for the following reasons:

• A network is never completely reliable. Out of a thousand requests, one may fail;
  The automatic test in this case will not pass, the work will stop, you will have to look for the reason in the logs;
• Too frequent requests are not allowed by some third-party services.

In addition, it is undesirable to use the stand, because:

• Not only can bad code running on it break a stand, but also data that the correct code cannot process;
• No matter how hard we try to bring back all the changes made by the test during the test itself, something may go wrong (otherwise, why the test?).

About project and process organization

Our company has developed a microservice web application running on Docker in the Amazon AWS cloud. Unit tests have already been used on the project, however, errors often occurred that unit tests did not detect. It was required to test the whole microservice along with the database and Amazon services.

The project uses a standard process of continuous integration, which includes testing the microservice with each commit. After assigning a task, the developer makes changes to the microservice, he manually tests it and runs all available automatic tests. If necessary, the developer modifies the tests. If no problems are found, a commit is made to the branch of this task. After each commit, tests are automatically run on the server. Merge into a common branch and run automatic tests on it after a successful review. If the tests on the common branch pass, the service is automatically updated in the test environment on the Amazon Elastic Container Service (stand). The stand is necessary for all developers and testers, and breaking it is undesirable. Testers in this environment test a fix or a new feature by performing manual tests.

Project architecture

The application consists of more than ten services. Some of them are written in .NET Core, and some in NodeJs. Each service runs in a Docker container in the Amazon Elastic Container Service. Each has its own Postgres database, and some also have Redis. There are no common bases. If several services need the same data, then this data is transferred to each of these services through SNS (Simple Notification Service) and SQS (Amazon Simple Queue Service) at the time of change, and the services store them in their separate databases.

SQS and SNS

SQS allows HTTPS to put messages in a queue and read messages from a queue.

If several services read the same queue, then each message comes to only one of them. This is useful when running multiple instances of the same service to distribute the load between them.

If you want each message to be delivered to several services, each recipient must have a queue, and to duplicate messages in several queues, you need SNS.

In SNS, you create a topic and subscribe to it, for example, an SQS queue. You can send messages to topic. In this case, a message is sent to each queue subscribed to this topic. There is no method for reading messages in SNS. If during debugging or testing you need to find out what is sent to SNS, then you can create an SQS queue, sign it on the desired topic and read the queue.

API Gateway

Most services are not available directly from the Internet. Access is via the Gateway API, which checks access rights. This is also our service, and there are tests for it too.

Real time notifications

Application uses SignalRto show the user real-time notifications. This is implemented in the notification service. It is accessible directly from the Internet and works with OAuth itself, because it turned out to be impractical to integrate Web socket support into Gateway, compared to the integration of OAuth and the notification service.

Well-known Testing Approach

Unit tests replace things like a database with mock objects. If a microservice, for example, tries to create a record in a table with a foreign key, and the record referenced by this key does not exist, then the request cannot be executed. Unit tests cannot detect this.

AT Microsoft article It is proposed to use the in-memory base and implement mock objects.

In-memory database – this is one of the DBMS that supports the Entity Framework. It is designed specifically for tests. Data in such a database is stored only until the completion of the process using it. It does not need to create tables, and data integrity is not checked.

Mock objects model the replaced class only as far as the test developer understands its work.

How to get Postgres to start automatically and perform migration when the test starts, the article from Microsoft did not indicate. My solution does this and, in addition, no code specifically for tests is added to the microservice itself.

Move on to the solution

During the development process, it became clear that unit tests are not enough to find all the problems in a timely manner, so it was decided to approach this issue from a different perspective.

Test environment setup

The first task is to deploy a test environment. The steps that are required to start the microservice:

• Configure the service under test for a local environment, in the environment variables the details for connecting to the database and AWS are indicated;
• Launch Postgres and migrate by running Liquibase.
  In relational DBMS, before writing data to the database, you need to create a data scheme, in other words, tables. When updating the application, the tables must be brought to the form used by the new version, and, preferably, without data loss. This is called migration. Creating tables in an initially empty database is a special case of migration. Migration can be integrated into the application itself. Both .NET and NodeJS have migration frameworks. In our case, for security reasons, microservices are deprived of the right to change the data scheme, and the migration is performed using Liquibase.
• Launch Amazon LocalStack. This is an implementation of AWS services to run at home. For LocalStack there is a ready-made image in the Docker Hub.
• Run the script to create the necessary entities in LocalStack. Shell scripts use AWS CLI.

For testing on the project it is used Postman. It was before, but it was launched manually and tested an application already deployed at the stand. This tool allows you to make arbitrary HTTP (S) requests and verify that responses respond to expectations. Queries are combined into a collection, and you can run the entire collection.

How the automatic test works

During the test, everything works in Docker: the service under test, Postgres, the migration tool, and Postman, or rather, its console version – Newman.

Docker solves a number of problems:

• Independence from host configuration;
• Dependency Installation: Docker downloads images from the Docker Hub;
• Returning the system to its original state: just delete the containers.

Docker-compose unites containers in a virtual network isolated from the Internet, in which containers find each other by domain names.

The test is controlled by a shell script. To run the test under Windows we use git-bash. Thus, one script is sufficient for both Windows and Linux. Git and Docker are installed by all developers on the project. When installing Git on Windows, git-bash is installed, so everyone has it too.

The script performs the following steps:

• Building docker images

  docker-compose build

• Running DB and LocalStack

  docker-compose up -d <контейнер>

• Database Migration and LocalStack Preparation

  docker-compose run <контейнер>

• Launching the service under test

  docker-compose up -d <сервис>

• Run test (Newman)
• Stop all containers

  docker-compose down

• Posting Results in Slack
  We have a chat where messages with a green tick or a red cross and a link to the log get to.

The following Docker images are involved in these steps:

• The service under test is the same image as for production. The configuration for the test is through environment variables.
• Postgres, Redis, and LocalStack use off-the-shelf images from the Docker Hub. For Liquibase and Newman, there are also ready-made images. We build our own on their skeleton, adding our files there.
• To prepare LocalStack, a ready-made AWS CLI image is used, and an image containing a script is created on its basis.

Using volumes, you can not build a Docker image just to add files to the container. However, volumes are not suitable for our environment, because Gitlab CI tasks themselves work in containers. You can control the docker from such a container, but volumes mount folders only from the host system, and not from another container.

Problems you may encounter

Waiting for readiness

When the service container is launched, this does not mean that it is ready to accept connections. You have to wait for the connection to continue.

This task is sometimes solved using a script. wait-for-it.sh, which is waiting for the opportunity to establish a TCP connection. However, LocalStack may throw a 502 Bad Gateway error. In addition, it consists of many services, and if one of them is ready, it does not say anything about the rest.

Decision: LocalStack preparation scripts that wait 200 for both SQS and SNS.

Conflicts of Parallel Tasks

Several tests can run simultaneously in the same Docker host, so container and network names must be unique. Moreover, tests from different branches of the same service can also work simultaneously, so it’s not enough to register your names in each compose-file.

Decision: The script sets the unique value of the COMPOSE_PROJECT_NAME variable.

Windows Features

When using Docker on Windows, there are a number of things that I want to draw your attention to, since this experience is important for understanding the causes of errors.

1. Shell scripts in the container must have Linux line endings.
   The CR character for a shell is a syntax error. According to the error message, it is difficult to understand that this is the case. When editing such scripts on Windows, you need the right text editor. In addition, the version control system must be properly configured.

This is how git is configured:

git config core.autocrlf input

1. Git-bash emulates standard Linux folders and when calling an exe file (including docker.exe) replaces the absolute Linux paths on the Windows paths. However, this does not make sense for paths not on the local machine (or paths in the container). This behavior is not disabled.

Decision: append an additional slash to the beginning of the path: // bin instead of / bin. Linux understands such paths; for it, several slashes are the same as one. But git-bash does not recognize such paths and does not try to convert.

Log output

When performing tests, I would like to see logs from both Newman and the service being tested. Since the events of these logs are interconnected, combining them in one console is much more convenient than two separate files. Newman runs through docker-compose run, and therefore its output gets to the console. It remains to make sure that the output of the service also gets there.

The initial decision was to make docker-compose up without flag -d, but, using the capabilities of the shell, send this process to the background:

docker-compose up  &

This worked until it was required to send logs from the docker to a third-party service. docker-compose up stopped displaying logs to the console. However the team worked docker attach.

Decision:

docker attach --no-stdin ${COMPOSE_PROJECT_NAME}_<сервис>_1 &

Identity conflict during test iterations

Tests are run in several iterations. The base is not cleared. Entries in the database have unique IDs. If we write down specific IDs in the requests, we get a conflict at the second iteration.

To avoid it, either IDs must be unique, or all objects created by the test must be deleted. Some objects cannot be deleted, in accordance with the requirements.

Decision: Generate GUIDs by scripts in Postman.

var uuid = require('uuid');
var myid = uuid.v4();
pm.environment.set('myUUID', myid);

Then use the character in the request {{myUUID}}to be replaced by the value of the variable.

Interaction via LocalStack

If the service under test reads an SQS queue or writes to it, then to test this, the test itself must also work with this queue.

Decision: requests from Postman to LocalStack.

The AWS Services API is documented, which allows you to make requests without the SDK.

If the service writes to the queue, then we read it and check the contents of the message.

If the service sends messages to SNS, at the preparation stage LocalStack also creates a queue and subscribes to this SNS topic. Then it all comes down to the one described above.

If the service should read the message from the queue, then in the previous step of the test we write this message to the queue.

Testing HTTP requests coming from the tested microservice

Some services work over HTTP with something other than AWS, and some AWS features are not implemented in LocalStack.

Decision: in these cases can help MockServerthat has a ready-made image in Docker hub. Expected requests and responses to them are configured by an HTTP request. The API is documented, so we make requests from Postman.

OAuth Authentication and Authorization Testing

We use OAuth and JSON Web Tokens (JWT). For the test we need an OAuth provider, which we can run locally.

All interaction of the service with the OAuth provider comes down to two requests: first, the configuration is requested /.well-known/openid-configuration, and then the public key (JWKS) is requested at the address from the configuration. All this is static content.

Decision: our test OAuth provider is a static content server and two files on it. The token is generated once and commited to Git.

SignalR Test Features

Postman does not work with web sockets. For testing SignalR a special tool was created.

SignalR client can be not only a browser. There is a client library for it under .NET Core. A client written in .NET Core establishes a connection, passes authentication, and expects a certain sequence of messages. If an unexpected message is received or the connection is disconnected, the client terminates with code 1. Upon receipt of the last expected message, it terminates with code 0.

At the same time, Newman is working with the client. Several clients are launched to verify that messages are delivered to all who need it.

To run multiple clients, use the option –scale at the docker-compose command prompt.

Before starting Postman, the script waits for all clients to establish a connection.
We have already encountered the problem of waiting for a connection. But there were servers, and here is the client. A different approach is needed.

Decision: the client in the container uses a mechanism Healthcheckto tell the script on the host about its status. The client creates the file in a specific path, say / healthcheck, as soon as the connection is established. The HealthCheck script in the docker file looks like this:

HEALTHCHECK --interval=3s CMD if [ ! -e /healthcheck ]; then false; fi

Team docker inspect shows the normal status, health status, and completion code for the container.

After the completion of Newman, the script checks that all containers with the client have completed, moreover, with code 0.

Happinnes exists

After we overcame the difficulties described above, we got a set of stable working tests. In tests, each service works as a whole, interacts with the database and with Amazon LocalStack.

These tests protect a team of 30+ developers from application errors with a complex interaction of 10+ microservices with frequent deployments.