Matomo v4. How to get 7k+ rps? Building a high-performance cluster

Introduction

Now many people will probably think that this is a clickbait article, but this is not so, then I will share my experience on how to get results in the headline.

The essence of the problem

Engineers who have encountered Matomo support know that it is almost impossible to squeeze more than 400-500 rps out of it without the QueuedTracking plugin. But even with the plugin, the performance ceiling does not rise very high, somewhere up to 1100-1200 rps. Even if you see only 200th responses and large values in the php-fpm monitoring, the real rps will be in the Redis monitoring, namely the rpush commands, the rest of the php-fpm processes simply die due to timeout. Accordingly, this tracking data is simply lost.

My peak values were about 2000 rps and as a result, about 18% of the traffic was lost. This is a lot, a lot. I will not describe my entire thorny path to this decision, I’ll just go straight to it.

My solution to the problem of processing high traffic

I decided to exclude the QueuedTracking plugin from the process of delivering analytics to Redis. But the plugin uploads data from Redis to the database, so I needed to write my own api that would receive traffic, parse it and save it in Redis, in the format in which the plugin does it. This solves one of the main problems: all active php-fpm processes maintain a connection to the database and at the same time there is traffic from them, they check the availability of the database and receive settings, and this results in 1000+ connections to the database. This is very bad for the database, maintaining such a number of connections requires a lot of CPU resources, in my case it would be “too many connections”.

Since I have some knowledge and experience in writing applications on .net, I chose it. During the brainstorming process, I decided to divide the delivery into 2 stages:

The first api + producer with a minimum amount of code, takes analytics (url path and necessary headers), serializes it into json and sends it to a queue on RabbitMQ. As a result, 1 instance of the application already produces 7k+ rps
In the screenshot rps below, because my MacBook overheats quickly, but in the first 10-15 seconds rps is above 7k))
The second one is consumer from the RabbitMQ queue and producer in the Redis queue. Here we parse the url path and generate json that is understandable to the QueuedTracking plugin, and we get 3k rps and compensate for the difference in rps by the number of consumers, I did 3.

Yes, there are nuances: the settings need to be changed in 3 components, and not in one, but the result is worth it. And, of course, with updates additional tests are required.

As a bonus we get a small number of connections to the database (number of plugin workers + connections from the front)

I think many people immediately have a question: how can all this data be sent to the database within a reasonable timeframe? Considering that the upload speed of 16 workers (and this is the maximum number) of the plugin is about 3k rps, and this is if the database performance allows it! I will be happy to talk about this in the next article if the community is interested in this topic.

Cheat sheet on how to view content in a Redis queue

You can of course use the command on the Matomo node:

./console queuedtracking:print-queued-requests –queue-id={Queue number}

But this will be a php object, and we need json

So let’s connect to Redis and get started:

select {Db number} – select the desired database
lrange trackingQueueV1_{Queue number} 0 1 – receive 1 message
ltrim trackingQueueV1_{Queue number} 0 -1 – delete all messages in the queue

And we will see unreadable gobbledygook, or rather a compressed message. To fix this, go to /{dir_matomo}/plugins/QueuedTracking/Queue/Backend/Redis.php, we need line 103: