ClickHouse as a backend for Prometheus

Hello! My name is Mikhail Kazhemsky, I am Lead DevOps at IT integrator Hilbert Team. Lately, we have been frequently approached by customers who need help organizing the storage of long-term metrics in Russian clouds. Since this task is now relevant for many, in this article my colleague Denis Babichev and I decided to tell you how we use the powerful capabilities of ClickHouse for effective long-term storage of Prometheus metrics.

In the article you will find recommendations for using the tool and a description of alternative solutions such as Thanos, Grafana Mimir and Victoria Metrics.

Prometheus and problems with LTS

Prometheus is an open source monitoring system for collecting and storing metrics in time series databases (TSDB). The service uses a fairly simple query language, PromQL, which is convenient for both developers and DevOps/SRE engineers. Prometheus has already become an industry standard and has a huge community.

Prometheus is accessed via PromQL by services such as Grafana or various APIs via the http interface. Most often, the monitoring system contains Alertmanager – a service for sending alerts to PagerDuty, Email, Telegram and other channels.

Despite all the advantages, a single Prometheus instance will only work reliably for small businesses that do not deal with huge volumes of data. Under increased loads, Prometheus may experience crashes—for example, Grafana may stop showing metrics or alerts may be lost.

This happens for the following reasons:

• Prometheus TSDB is not optimized for long-term metrics storage.

• No horizontal scaling of Prometheus components.

• Although PromQL is a fairly simple query language, it has limited capabilities compared to other query languages ​​such as SQL.

However, LTS in Prometheus can be optimized. For example, make several independent Prometheus for each logical group of targets. This will relieve Prometheus and Grafana… however, Grafana will already show several metrics, gluing them together will be a whole task. But after this, the metrics will be able to accumulate for a longer period and not be deleted.

But this is only a temporary solution to the problem. Since Prometheus is not designed for long-term data storage, the service may again begin to crash or consume too many resources.

Unfortunately, all large enterprises can face a similar problem. Therefore, to solve this problem, you can use special products that can easily store an unlimited number of Prometheus metrics – these include Thanos, Grafana Mimir, Victoria Metrics etc. Let's briefly look at how the schemes for working with these solutions look like.

Thanos

Thanos is an open source project from Improbable. It is designed to seamlessly transform existing Prometheus clusters into a single monitoring system with unlimited historical data storage.

Let's look at how Thanos works.

1. In this scheme Prometheus is in Kubernetes.

2. Underneath Prometheus connects Sidecar-контейнер With Thanos.

3. Sidecar writes to the component Storewhich can be scalable.

4. Store writes to Object Storage via S3 protocol.

5. Compact near Storage allows for optimal storage of metrics.

6. In addition, there are components Query to get metrics and Ruler to evaluate alerts.

Grafana Mimir

Grafana Mimir is a fairly powerful and increasingly popular tool from Grafana Labs. This solution is a little more complex than Thanos. It is worth noting that in Grafana Mimir almost every element is horizontally scalable.

Let's consider the scheme along the way of recording metrics:

1. Requests from Prometheus come to distributors.

2. Distributors distribute metrics among ingesters.

3. Further ingesters write metrics to Long-Term storage in Оbject Storage.

4. compactor optimizes storage.

Reading occurs through another channel:

1. Requests come through query frontend and go to querier. These services allow you to combine metrics from different sources.

2. The requests then go to store-gatewaythrough which data is read from LT storage.

It's important to note that queriers also appeal to ingesters, because they send their data to LT only after a couple of hours.

Victoria Metrics

Victoria Metrics is perhaps the most popular tool at the moment. Its architecture is simpler than Mimir, but it is not inferior in performance and superior in resource consumption.

Victoria Metrics consists of 3 main components:

1. Scalable vminsert to record metrics.

2. Scalable vmstorage – optimized TSDB storage for long-term storage.

3. Scalable vmselect to retrieve data from storage.

In this case, components similar to Prometheus are used, but they are more scalable and optimized for long-term storage.

Each of the tools described is Thanos, Grafana Mimir, Victoria Metrics – performs its functions well in its own way. But in some cases, we use ClickHouse to store long-term Prometheus metrics. For example, if the customer already has a data platform created on the basis of ClickHouse, and he wants to enrich his data with monitoring metrics.

Let's take a closer look at how to use ClickHouse for this task.

ClickHouse as a backend

ClickHouse is a popular open source analytical database management system. It works in real time and has high performance. This is a good solution for applications that use large volumes of structured data sets. As a scalable system, it supports distributed query processing, data partitioning and replication, and sharding, making it capable of processing terabytes of information.

Initially, ClickHouse was created for Yandex Metrica tasks – mainly for online reports on non-aggregated data. But thanks to its scalability, speed and access to SQL, the service also copes well with the tasks of long-term storage of metrics.

It has the following advantages:

• ClickHouse “doesn’t slow down”.

• It itself can be ML Data Storage and is created for large analytical queries.

• Has high performance and native horizontal scaling.

• Available in a managed version from many cloud providers.

To solve the problem of long-term storage of monitoring metrics, Prometheus ClickHouse can offer the following options:

• Sharding. This is a horizontal cluster scaling strategy in which parts of a single ClickHouse database are hosted on different groups of hosts.

• Distribution tables. To distribute data across different shards, you need to create a distributed table using the Distributed engine.

Integration/Configuration

By default, ClickHouse does not integrate well with Prometheus, so you can use external tools such as QRYN or Carbon-Graphite.

QRYN

Works great with a huge number of metrics and other data, and also successfully transfers them to ClickhHouse.

QRYN offers the following features:

● Accepts data in Prometheus, Loki, Tempo, InfluxDB, Elastic, etc. formats.

● Compatible with PromQL, LogQL, Tempo APIs.

● Uses ClickHouse as a storage backend.

QRYN is configured quite simply – we select the path for it to ClickHouse, and it independently creates a database there with the necessary tables. After this, we configure Prometheus to record metrics.

# Настройка сервера QRYN
spec:
  containers:
    - env:
      - name: CLICKHOUSE_AUTH
        value:”demo:demo”
      - name: CLICKHOUSE_PORT
        value: ”8123"
      - name: CLICKHOUSE_SERVER 
        value: ”clickhouse-svc”

# Настройка сервера Prometheus
remote_write:
  - url: "http://qryn:3100/api/v1/remote/write"

The scheme for reading and writing metrics is as follows: Prometheus And Grafana go to a horizontally scalable service QRYNA QRYN – V ClickHouse.

Carbon-Graphite

Carbon-Graphite is a separate monitoring system from Prometheus – metrics in Graphite format are in a different format. Graphite is used for reading and Carbon is used for writing metrics.

To integrate with ClickHouse:

Both tools are written in Go and support inserting and reading using the Prometheus format.

At the same time, to work with Carbon, you must first create 3 tables necessary for its operation:

• The first uses the metrics engine from Graphite, which is already available in Clickhouse.

• The second and third are used for the index and tags (speeds up the search).

As a result, the following structure emerges:

1. Prometheus addresses Carbon.

2. Carbon writes to ClickHouse Managed.

3. Grafana reads from Graphite by PromQL and also addresses ClickHouse behind the metrics.

Carbon-Graphite-ClickHouse Test Results

As a testing tool we used prometheus-benchmark from Victoria Metrics and directly managed ClickHouse cluster, divided into front and back.

The specifications are as follows:

• Cluster1 (data): 2 shards, 2 hosts each, for Replicated MergeTree tables.

• Cluster2 (front proxy): 1 host for Distributed MergeTree tables.

• Each host has 2 RAM and 4 GB RAM.

The benchmark parameters are defined as:

• TargetCount — the number of “virtual” sites from which data is sent by node exporters,

• WriteReplicas — the number of instances sending data from “virtual” sites. Horizontal scaling TargetCount,

• WriteConcurrency — the number of simultaneous TCP connections for sending metrics.

Test results for different parameters.

Test 1

Ingest Rate — 116k wr/s

Test 2

Ingest Rate — 400k wr/s

Test conclusions:

• High Ingest Rate with low resources,

• Sampling time is not the fastest,

• Graphite-Carbon components are quite resource-intensive.

You can familiarize yourself with similar tests for Victoria Metrics and Grafana Mimir link.

Thus, ClickHouse may be a good option for storing long-term Prometheus metrics. ClickHouse is available in managed solutions in clouds, supports SQL and distribution of metrics across different shards, which ensures convenient work with Prometheus. All this makes ClickHouse a good solution for companies that already use it as the basis of their analytics infrastructure and want to enrich their analytics with Prometheus metrics.