Making Grafana Notifications Readable Again
What can you say about notifications about problems in grafana? Well, except for what they are. I can’t say that I really liked them, but they weren’t bad enough to be customizable.
But after updating grafana to the ninth version, everything changed. I found that the messages turned into unreadable sheets (in the picture on the left):
After spending some time searching Google and reading the documentation, I found out that grafana now uses the new unified alerting subsystem by default (it’s ngalert, it’s just alerting recently). For each notification channel, you can set your own message template. Used to describe patterns. Go Templatethe documentation describes available from the template data and functionsthe sources have standard pattern. In general, there is enough information to create your own template.
After some experimentation, this is what I came up with:
{{ define "tgshortbody" }}{{ range . }} <b>{{ or .Annotations.message .Labels.alertname }}</b>
{{ range .Annotations.SortedPairs }}{{ if ne .Name "message" }}{{ .Name }}: {{ .Value }}
{{ end }}{{ end }}
{{ with .ValueString }}{{ reReplaceAll "[[][^]]*metric="{?([^}"]*)}?'[^]]*value=([0-9]*([.][0-9]{,3})?)[^]]*](, )?" "$1: <b>$2</b>\n" . }}
{{ end }}{{ with .PanelURL }}<a href="https://habr.com/ru/post/675548/{{ . }}">Chart</a> {{ end }}{{ with .DashboardURL }}@<a href="https://habr.com/ru/post/675548/{{ . }}">Dashboard</a> {{ end }}{{ with .GeneratorURL }} <a href="https://habr.com/ru/post/675548/{{ . }}">Edit</a> {{ end }}{{ with .SilenceURL }} <a href="https://habr.com/ru/post/675548/{{ . }}">Mute</a> {{ end }}
{{ end }}{{ end }}
{{ define "tgshort" }}{{ with .Alerts.Firing }}⚠️{{ template "tgshortbody" . }}{{ end }}{{ with .Alerts.Resolved }}✅{{ template "tgshortbody" . }}{{ end }}{{ end }}As you can see, just a few lines. The most difficult thing was to remake here ValueString from unreadable something like[[var="B0" metric="Zombie" labels={} value=10.85 ] into something similar to that of an old grafana. Since this is a string, I had to use the black magic of regular expressions. To be honest, I don’t like this solution, it’s too unreliable, and unreadable, but there wasn’t much choice. FR on adding a non-textual representation created.
A few clarifications on the possible adaptation of the template: the template uses html tags understood by the telegram, they may have to be thrown out or corrected for other notification channels. Texts of messages after updating grafana we ended up in the fieldmessageannotations, since laziness is the most important quality of a system administrator, it was decided not to change this.
Templates are written in the graphana web interface in the alerting / notifications section: click “New template” and enter a new template.
Further on the same page, you need to indicate for the used notification channels (contact points) that our template will be used: go to the settings of the desired channel and write it in the Message field:
{{ template "tgshort" . }}OK it’s all over Now. You can see the new version of the message on the KPDV on the right. The emoticon, alas, does not come with the template.
P. S. The idea of using Unicode icons to display the status of a message was spied on @HetznerStatusChannel. It is very convenient, even without opening telegrams you see a message about a solution to a problem or about a new problem.
