Notifications in telegram about application crash, via grafana

This is the unannounced third part. The first two are here:

1. Preparing a Django application for local development and deployment

2. Django application in docker. Logging and monitoring (also in docker)

Not announced, because this method is suitable for any application with metrics via Prometheus, but how to configure them for a specific Django application can be seen in the previous parts.

First, you need to create a telegram bot, find the BotFather bot and create a new bot in it using the /newbot command, get your bot's token.

Now you need to get the ChatId. To do this, write a message to your new bot, and then open the following address in your browser (preferably incognito, or even better with curl)

https://api.telegram.org/bot/getUpdates

Replacing with your bot token. In response you will receive json with information about the latest messages sent to your bot. In this JSON find the chat field, which will contain your id.

The value of the id field inside the chat object is your Chat ID.

Well, the main thing is behind us.

Next, go to Grafana Home -> Alerting -> Contact points

There you have, if you haven’t had Contact points yet, edit the only existing one.

Enter any name and select Telegram in the Integration field.

There we indicate the BOT API Token and Chat ID received earlier, save.

Go to Alert rules in the same Alerting and create a new rule – New alert rule.

Here we assume that you already have Prometheus configured for your application, at least from the previous steps.

So, in the first field, enter any rule name. Below in A, select Prometheus, in Metric, enter up, and in the labels, select the application. Next in B, select input A, Function — Min, Mode — Strict. In C (Threshold), select Input — B, below IS BELOW — 1.

You can click Preview and make sure that the rule works.

Next, in point 3, you can select the folder where to put the rule and the waiting period.

Item 4 is also filled in at will.

In point 5 we specify arbitrary labels for the rule. They are essentially needed for the correct selection of the notifier.

Well, the last step in the article is choosing notification policies: go to Notification policies there in Alerting, click the three dots opposite Default policy — Edit and change Default contact point to the one we created at the corresponding step. Save. That's it!

Now we have notifications set up from the bot in Telegram if the up metric in Prometheus falls below 1.