the art of certificate monitoring

Greetings technoenthusiasts! Today we’ll tackle a topic that you either adore or successfully ignore until something breaks. We will talk about certificate monitoring.

It would seem that what could be difficult about monitoring certificates? A certificate has been issued – monitor its validity period. Many people use calendars, reminders, and sometimes even physical notes the old fashioned way. But manual methods are not ideal, because people make mistakes, forget and lose information.

And if you don’t automate such moments, then one day you may wake up and find out that the service is unavailable due to a rotten certificate. Let’s figure out what’s what.

Deeper into technology: Grafana, Prometheus and our exporters

First, let’s launch Grafana and Prometheus – these are our main monitoring tools. Then we’ll chat about exporters and connect one of them, add a cool panel with graphs. In order not to miss this whole thing, let’s teach Alertmanager to send us notifications in Telegram.

1. Let’s start with the basics: Grafana + Prometheus and Docker Compose

For those who are in the know and already a thousand times heard about awesome-compose/prometheus-grafana, you can skip this step. For the rest – you will need compose.yml file:

services:
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    command:
      - "--config.file=/etc/prometheus/prometheus_config.yml"
    ports:
      - 9090:9090
    restart: unless-stopped
    volumes:
      - ./prometheus:/etc/prometheus
      - prom_data:/prometheus
  grafana:
    image: grafana/grafana
    container_name: grafana
    ports:
      - 3000:3000
    restart: unless-stopped
    env_file: .env
    environment:
      - GF_SECURITY_ADMIN_USER=${GF_SECURITY_ADMIN_USER}
      - GF_SECURITY_ADMIN_PASSWORD=${GF_SECURITY_ADMIN_PASSWORD}
    volumes:
      - ./grafana:/etc/grafana/provisioning/datasources
volumes:
  prom_data:

Now create a directory prometheus with file prometheus_config.yml:

global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 15s

And the same for the directory grafana and file datasource.yml:

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    url: http://prometheus:9090
    isDefault: true
    access: proxy
    editable: true

In the example, we will store secrets in a file: .env. But it’s better not to do this, because the file can easily end up in the public domain. Use special tools to store secrets, encrypt files and everything will be fine.

Create a file .env and add ИМЯ_ПОЛЬЗОВАТЕЛЯ_GRAFANA And ПАРОЛЬ_ПОЛЬЗОВАТЕЛЯ_GRAFANA:

GF_SECURITY_ADMIN_USER=<ИМЯ_ПОЛЬЗОВАТЕЛЯ_GRAFANA>
GF_SECURITY_ADMIN_PASSWORD=<ПАРОЛЬ_ПОЛЬЗОВАТЕЛЯ_GRAFANA>

Let’s pick up all this beauty:

docker-compose up -d

So, the Docker magic has worked and we have Grafana on port 3000 And Prometheus on 9090. Just? Just!

2. Diving into the world of exporters: choosing a tool

Each exporter has its pros and cons. After all, as they say, every brush is good
on your canvas. Let’s figure out which one is best for you:

• ssl_exporter: its task is certificates and only. Provides rich information about certificates: expiration dates, algorithms, signatures and much more. Supports various modes for verification: TCP, HTTPS, SMTP and so on. But if you need more than just SSL/TLS monitoring, you’ll have to look for additional tools.

• blackbox_exporter: can do a lot of things, he will check everything, including certificates. Compared to ssl_exporter, data granularity and capabilities may seem weaker. Overkill if you only need certificate monitoring, but perfect for complex monitoring.

• node-cert-exporter: if you need to track certificates on disk, This is your choice. Very easy to use and does not require much knowledge.

• x509-certificate-exporter: Gives detailed information about certificates. Flexible, can extract Kubernetes file or secret data. More difficult to set up and operate compared to some other tools.

• cert_exporter: supports multiple certificate sources, including file system, Kubernetes and others. Very versatile for different use cases.

So which tool should you choose? Choosing an instrument is an art, and, like any art, it has its own subtleties and nuances. Choose what suits your needs and don’t be afraid to experiment.

3. Working with ssl_exporter

We have already discussed what ssl_exporter is. Keeps track of certificates well
and in different ways. Let’s add it to compose.yml:

ssl_exporter:
  image: ribbybibby/ssl-exporter
  ports:
    - "9219:9219"

Today our goal will be prometheus.io. Add scrape_configs V prometheus_config.yml:

scrape_configs:
  - job_name: "ssl-exporter"
    metrics_path: /probe
    static_configs:
      - targets:
          - prometheus.io:443
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: ssl_exporter:9219

So, let’s restart our compose.yml:

docker-compose restart

IN Prometheus running PromQL query – ssl_probe_success and if we see the following, then we have configured everything correctly:

ssl_probe_success{instance="prometheus.io:443", job="ssl-exporter"}

By default, the exporter creates a TCP connection to the target. In most cases this is enough, but if you want to proxy via http, use the module https. Additionally, you can use file keep track of certificates on disk:

scrape_configs:
  - job_name: "ssl-files"
    metrics_path: /probe
    params:
      module: ["file"]
      target: ["/etc/ssl/cert.pem"]
    kubernetes_sd_configs:
      - role: node
    relabel_configs:
      - source_labels: [__address__]
        regex: ^(.*):(.*)$
        target_label: __address__
        replacement: ${1}:9219

If there are several certificates, you can use glob search, which allows you to capture several files at once:

scrape_configs:
  - job_name: "ssl-files"
    metrics_path: /probe
    params:
      module: ["file"]
      target: ["/etc/ssl/**/*.pem"]
    kubernetes_sd_configs:
      - role: node
    relabel_configs:
      - source_labels: [__address__]
        regex: ^(.*):(.*)$
        target_label: __address__
        replacement: ${1}:9219

Can also be used in conjunction with Kubernetes, in documentation more information.

4. Add a panel to Grafana

We won’t reinvent the wheel, we’ll just log in and import ready-made panel from the ssl_exporter repository:

How not to sleep through problems

What are our plans? Let’s create an incredibly simple but very useful service in Flask, which will catch notifications from Alertmanager and forward them to Telegram. Let’s create a bot and configure Alertmanager to send notifications.

1. Taking off with Flask

Okay, let’s not waste time on trifles, let’s create a directory telegram and throw the file there app.py with elegant, minimalistic and true pythonista code:

import os
import requests

from flask import Flask, request, jsonify

app = Flask(__name__)

TELEGRAM_TOKEN = os.environ["TELEGRAM_TOKEN"]
CHAT_ID = os.environ["TELEGRAM_CHAT_ID"]
TELEGRAM_URL = f"https://api.telegram.org/bot{TELEGRAM_TOKEN}/sendMessage"


@app.route("/alert", methods=["POST"])
def webhook():
    data = request.json
    alertname = data["alerts"][0]["labels"]["alertname"]
    instance = data["alerts"][0]["labels"]["instance"]
    summary = data["alerts"][0]["annotations"]["summary"]
    text = f"⚠️ Alarm! {alertname} for {instance}. \n{summary}"
    payload = {"chat_id": CHAT_ID, "text": text}
    response = requests.post(TELEGRAM_URL, data=payload)
    return jsonify(status="success"), 200


if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8000)

And it will also come in handy Dockerfile:

FROM python:3.10-alpine
WORKDIR /app
COPY . /app
RUN pip3 install -r requirements.txt
ENTRYPOINT ["python3"]
CMD ["app.py"]

And, of course, irreplaceable requirements.txt:

flask
requests

2. Craft a bot:

1. Let’s launch @BotFather on Telegram.

2. /newbot, give it a name and voila! In your hands: ТОКЕН_БОТА.

3. Find out your chat_id by sending /start to your bot and looking at https://api.telegram.org/bot<ТОКЕН_БОТА>/getUpdates. Find result[0]message.chat.id and remember, it will be yours ВАШ_ЧАТ_ID.

Now add all this to compose.yml:

telegram:
  build:
    context: telegram
  stop_signal: SIGINT
  env_file: .env
  environment:
    - TELEGRAM_TOKEN=${TELEGRAM_TOKEN}
    - TELEGRAM_CHAT_ID=${TELEGRAM_CHAT_ID}
  ports:
    - "8000:8000"

And the secrets ТОКЕН_БОТА And ВАШ_ЧАТ_ID V .env :

TELEGRAM_TOKEN=<ТОКЕН_БОТА>
TELEGRAM_CHAT_ID=<ВАШ_ЧАТ_ID>

3. Let’s make friends between Alertmanager and Telegram

It’s time to figure out exactly where we will send notifications. Theoretically Alertmanager can send messages anywherebut we know that today our choice is Telegram.

Let’s start small. Let’s write a rule that will let us know when the certificate has less than 90 days of life left. For those who want to go deeper – into documentation collected various requests. IN prometheus create a file alert.rules.yml and write there:

groups:
  - name: ssl-alerts
    rules:
      - alert: Expiry Soon
        expr: ssl_cert_not_after - time() < 86400 * 90
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "SSL certificate is expiring soon (less than 90 days)"

Let’s add a couple of lines to prometheus_config.yml:

rule_files:
  - "/etc/prometheus/alert.rules.yml"
alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - "alertmanager:9093"

Now let’s do the following: create a directory called alertmanager and put the file in it alertmanager_config.yml:

route:
  group_by: ["instance"]
  repeat_interval: 3h
  receiver: "telegram-webhook"

receivers:
  - name: "telegram-webhook"
    webhook_configs:
      - url: "http://telegram:8000/alert"

Well, ours compose.yml also wants attention, add a container alertmanager:

alertmanager:
  image: quay.io/prometheus/alertmanager
  container_name: alertmanager
  command:
    - "--config.file=/etc/alertmanager/alertmanager_config.yml"
  ports:
    - "9093:9093"
  volumes:
    - ./alertmanager:/etc/alertmanager

And now – the magic:

docker-compose down; docker-compose up -d

So let’s see what our Alertmanager feels comfortable:

and the rule in Prometheus has taken its working position:

After about a minute, Prometheus will realize that there are some problems:

And after about 30 seconds, Alertmanager will send a notification to our service:

Which will forward it to you in Telegram:

That’s all! Now you have a bot in your arsenal that keeps its finger on the pulse of your services and, if something happens, instantly sounds the alarm! Well, you also feel this scent of opportunity, right?

Conclusion

Certificate monitoring is not an obsession, it is a concern. Enjoy comfort, technology, don’t forget to water your plants and monitor certificates.