Making a dashboard for logs using Promtail Loki Grafana

We will use Grafana version 10.2, Promtail and Loki version 2.9.2. If someone knows nothing at all about the stack used, look at the short description here. Everything will work under Windows; for Linux, the changes should be purely cosmetic. The article was written for a quick start, so performance, setting access rights, the variety of Grafana visualizations, etc. will not be discussed here.

Preparation

Download Grafana binaries here Here. If an Enterpise license is selected, you can get an error instead of binaries Sorry, our service is currently not available in your region. In this case, choose OSS.

The Loki and Promtail binaries are here Here

If someone doesn’t like binaries, here is detailed installation documentation Promtail And Loki. Default config version 2.9.2 for Promtail Hereand for Loki here. If you need another version, change in the links v2.9.2 to the desired one.

The logs will be in this format

[dateTime] [tenantId] [severity] [module] [message]

Example of a log entry

[01.11.2023 17:07:59] [node001] [INFO] [Export.Csv] [rsvp_flow_stateMachine: entering]

Please pay special Pay attention to your date and time format. The time zone can be explicitly specified in the logs; you can configure the time zone through the promtail config. Here it will be in the config. It’s easy to get confused in different time zones, especially if the logs are collected on different servers, and it’s doubly easy to get confused if everything happens in the cloud.

For testing, I made myself a log generator. It produces more or less meaningful content in the format I need and the quantity I need. For those who do not have ready-made logs for tests, it is advisable to make a file in advance that will contain at least a hundred entries. And to configure Promtail, just one line is enough.

Promtail setup

Promtail has two parameters that will greatly help in setting up

  • –dry-run no data is sent to Loki, and the logfile will be read from the same place every time it starts. You don’t have to add new entries every time you change the config and start Promtail; one line is enough.

  • –inspect Debugging information is output to the console

To configure Promtail, add one entry to the log file and launch Promtail. We change the config with each launch to test the changes.

It looks like this

promtail-windows-amd64 --dry-run --inspect --config.file=promtail-local-config.yaml
[01.11.2023 17:07:59] [node001] [INFO] [Export.Csv] [rsvp_flow_stateMachine: entering state SESSIONED]
[inspect: regex stage]:
{stages.Entry}.Extracted["module"]:
        +: Export.Csv
{stages.Entry}.Extracted["msg"]:
        +: rsvp_flow_stateMachine: entering state SESSIONED
{stages.Entry}.Extracted["severity"]:
        +: INFO
{stages.Entry}.Extracted["tenantid"]:
        +: node001
{stages.Entry}.Extracted["time"]:
        +: 01.11.2023 17:07:59
[inspect: labels stage]:
{stages.Entry}.Entry.Labels:
        -: {filename="c:\\LogManagement\\logs\\test.log", job="logs"}
        +: {filename="c:\\LogManagement\\logs\\test.log", job="logs", module="Export.Csv", severity="INFO", tenantid="node001"}
[inspect: timestamp stage]:
{stages.Entry}.Entry.Entry.Timestamp:
        -: 2023-11-03 16:32:11.2903454 +0700 +07
        +: 2023-11-01 17:07:59 +0700 +07
[inspect: labeldrop stage]:
{stages.Entry}.Entry.Labels:
        -: {filename="c:\\LogManagement\\logs\\test.log", job="logs", module="Export.Csv", severity="INFO", tenantid="node001"}
        +: {job="logs", module="Export.Csv", severity="INFO", tenantid="node001"}
[inspect: template stage]:
{stages.Entry}.Extracted["msgData"]:
        +: rsvp_flow_stateMachine: entering state SESSIONED
[inspect: output stage]:
{stages.Entry}.Entry.Entry.Line:
        -: [01.11.2023 17:07:59] [node001] [INFO] [Export.Csv] [rsvp_flow_stateMachine: entering state SESSIONED]
        +: rsvp_flow_stateMachine: entering state SESSIONED
2023-11-01T17:07:59+0700{job="logs", module="Export.Csv", severity="INFO", tenantid="node001"}  rsvp_flow_stateMachine: entering state SESSIONED

Now the config will be configured in parts. If it’s more convenient to look at everything at once, then then the entire config.
To configure, take the default config, leave this part as is

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://localhost:3100/loki/api/v1/push

Next we change path And job
job logs label for the configuration name (there can be several configurations)
path path to the log file (or files). If you have log file rotation configured, then check the template thoroughly to ensure that promtail accurately finds new files.

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: logs
      __path__: c:\LogManagement\logs\test.log

Next, we set up the parsing of logs directly. The analysis takes place according to certain stages/stages, details in documentation. To parse a string, we use a regular expression with named groups. The group name will be used in the following steps. There are standard settings for json or logfmt; there is no need to invent regular expressions to parse json.

  pipeline_stages:
    - regex:
        expression:
          \[(?P<time>.*?)\] \[(?P<tenantid>.*?)\] \[(?P<severity>.*?)\] \[(?P<module>.*?)\] \[(?P<msg>.*?)\]

Parsing result

[01.11.2023 17:07:59] [node001] [INFO] [Export.Csv] [rsvp_flow_stateMachine: entering state SESSIONED]
[inspect: regex stage]:
{stages.Entry}.Extracted["module"]:
        +: Export.Csv
{stages.Entry}.Extracted["msg"]:
        +: rsvp_flow_stateMachine: entering state SESSIONED
{stages.Entry}.Extracted["severity"]:
        +: INFO
{stages.Entry}.Extracted["tenantid"]:
        +: node001
{stages.Entry}.Extracted["time"]:
        +: 01.11.2023 17:07:59

Next is the tags stage, here we indicate what tags the entry will have.

    - labels:
        tenantid:
        severity:
        module:

The result is below, pay attention to what marks were before and what they were after.

[inspect: labels stage]:
{stages.Entry}.Entry.Labels:
        -: {filename="c:\\LogManagement\\logs\\test.log", job="logs"}
        +: {filename="c:\\LogManagement\\logs\\test.log", job="logs", module="Export.Csv", severity="INFO", tenantid="node001"}

Now the very important part is setting the recording time. In my example, the time zone is not explicitly specified in the logs, so the time zone is set through the config. If that’s not the case for you, location throw it away format change it to suit yours. Promtail uses the Go format for times and dates. For those unfamiliar with this format, here it is: crib. Timezone codes for the config can be viewed Here.

    - timestamp:
        format: 02.01.2006 15:04:05
        source: time
        location: Asia/Krasnoyarsk

The result of the stage is below, note that the date has changed to 2 days. The date in the log file is November 1st, processing occurs on November 3rd. By default, the log processing time will be indicated in the entry. Test this step thoroughly, especially if the logs are generated in the cloud.

[inspect: timestamp stage]:
{stages.Entry}.Entry.Entry.Timestamp:
        -: 2023-11-03 16:32:11.2903454 +0700 +07
        +: 2023-11-01 17:07:59 +0700 +07

Next, we throw out the extra labels, we don’t need filename, we don’t need time either, because we figured out the time of the log entries at the previous stage.

    - labeldrop:
        - filename
        - time

Result of the stage

[inspect: labeldrop stage]:
{stages.Entry}.Entry.Labels:
        -: {filename="c:\\LogManagement\\logs\\test.log", job="logs", module="Export.Csv", severity="INFO", tenantid="node001"}
        +: {job="logs", module="Export.Csv", severity="INFO", tenantid="node001"}

Now let’s create a new field msgData using a template, the data for the new field will be from the old field msg. It looks a little crooked, but that’s done on purpose. In order to show how you can change the string content of logs while Promtail is running, before they get to Loki. Here tenant, severity, node and time are removed from the line, because this data will still be in the labels. If you need to leave the line as is, then this and the next stage can simply be removed from the config.

    - template:
        source: msgData
        template: '{{ .msg }}'

The result of the stage is a log line

[inspect: template stage]:
{stages.Entry}.Extracted["msgData"]:
        +: rsvp_flow_stateMachine: entering state SESSIONED

The final stage – we indicate the data source for the log line, this is the field from the previous stage.

    - output:
        source: msgData

Final record – line and labels

[inspect: output stage]:
{stages.Entry}.Entry.Entry.Line:
        -: [01.11.2023 17:07:59] [node001] [INFO] [Export.Csv] [rsvp_flow_stateMachine: entering state SESSIONED]
        +: rsvp_flow_stateMachine: entering state SESSIONED
2023-11-01T17:07:59+0700{job="logs", module="Export.Csv", severity="INFO", tenantid="node001"}  rsvp_flow_stateMachine: entering state SESSIONED

Entire config

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: logs
      __path__: c:\LogManagement\logs\test.log
  pipeline_stages:
    - regex:
        expression:
          \[(?P<time>.*?)\] \[(?P<tenantid>.*?)\] \[(?P<severity>.*?)\] \[(?P<module>.*?)\] \[(?P<msg>.*?)\]
    - labels:
        tenantid:
        severity:
        module:
    - timestamp:
        format: 02.01.2006 15:04:05
        source: time
        location: Asia/Krasnoyarsk
    - labeldrop:
        - filename
        - time
    - template:
        source: msgData
        template: '{{ .msg }}'
    - output:
        source: msgData

Promtail has its own simple web client, the default port is 9080, it can be used for performance testing and administration, the client looks like this

Setting up Loki

Everything is simple here, use the default configuration file and everything works. By default, Loki sends some statistics to its servers. To disable this, uncomment these lines

#analytics:
#  reporting_enabled: false

To check the functionality of Loki, go to the link in your browser (or any web client) http://localhost:3100/ready. The answer should be readythere may still be such an answer Ingester not ready: waiting for 15s after being ready, wait the specified 15 seconds and try again. Loki has its own API, help here. Something useful at the initial stage might be an endpoint for metrics http://localhost:3100/metrics.

Setting up Grafana

Run the file grafana-10.2.0\bin\grafana-server.exe and go to http://localhost:3000. Login password when you first start admin/admin, Grafana will immediately prompt you to change it. Loki must be running and there must be data in it.

Add Loki Datasource

In the Home Connections Data sources menu, click on the Add new data source button

There we select Loki

In the Connection field enter http://127.0.0.1:3100

At the bottom of the form, click on Save & Test and wait for confirmation of a successfully established connection

In the upper right corner click on the explore data button

A form for creating and editing requests will open.

By clicking on the Label browser button, you can see what labels and what values ​​they have in the database, create and execute a request

If you have data in your database, it will look something like the picture below. This is the main place to write and debug queries

Creating the Raw Logs Panel

First you need to create a dashboard, in the Home Dashboards menu click on the New button, in the drop-down list select New dashboard
Next, click on the Add visualization button
Select the Loki datasource created in the previous step
In the upper right corner, change the visualization type to Logs

Change the Panel title to Raw logs and enable the Time and Enable log details checkboxes

Click on the Apply button

Go to dashboard properties

Here we go to the Variables tab and click on the Add variable button, the variable name is tenantid, the rest is as in the picture below

We create another variable, everything is the same, only for nodeid

Create a third variable called filter and type Text box

The panel now looks like this. Click edit to change the request

The request will be like this. pay attention to line_formatthis way you can change the display of log entries.

{job="logs", tenantid=~"$tenantid", severity=~"$severity"} |= "$filter" | line_format "{{.tenantid}} {{.module}} {{.severity}} {{__line__}}"

As a result, everything will look like the picture below, you can select tenant, severity and filter by log content.

Creating a Pie Chart Panel

Click on the Add button and select Visualization

Set the properties as shown in the picture below

The request will be like this

sum by(severity) (count_over_time({job="logs"} [$__range]))

Please note that the query does not contain the variables that are in the query for the Raw Logs panel. Therefore, the chart will use all data for the selected period of time, there will be no filtering. If desired, you can add variables to the request and then the data in the chart and in the Raw Logs panel will be the same.
Colors for specific values ​​can be selected by clicking on the thin colored bar
on the legend

Or by adding the required values ​​on the Overrides tab in the chart properties

As a result, our chart will look like this

Conclusion

Grafana is a great tool for data visualization, Loki and Promtail make it quite easy to set up the collection and storage of logs. The combination of these tools gives very good results with little effort. I hope the article will be useful.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *