Setting up centralized logging using Dokku, Loki and Grafana

Centralized logging plays an important role in modern server monitoring and management. This allows you to quickly analyze and monitor the operation of applications and servers in real time. In this article, we will look at the process of setting up a centralized logging system using powerful tools such as Dokku, Loki and Grafana. Using these tools together provides a flexible and scalable system for monitoring and analyzing your server data.

The content of the article:

• Step 1: Set up Dokku

• Step 2: Setting up Loki

• Step 3. Setting up Promtai

• Step 4: Setting up Grafana

• Step 5. Set up Promtail to monitor JSON logs on a remote server running Loki

• Step 6: Data Visualization with Grafana

Step 1: Set up Dokku

Dokku is a lightweight Docker container management tool that makes it easy to deploy and manage applications. To get started with Dokku, follow these steps:

Installation instructions for Dokku:

1. Loading the installation script:

wget -NP . https://dokku.com/bootstrap.sh

2. Running the installer:

sudo DOKKU_TAG=v0.32.3 bash bootstrap.sh

3. Domain setup:

dokku domains:set-global dokku.me

4. Adding an SSH Key: Replace "your-public-key-contents-here" to your public SSH key.

echo "your-public-key-contents-here" | dokku ssh-keys:add admin

These steps will create the foundation for running Dokku on your server.

Requirements

To complete this setup, you will need a server with Dokku installed, running Ubuntu 22.04. These instructions have been tested on Dokku v0.32.3, but they will likely work on other versions.

Public access points are secured by http-auth and tls, so make sure you have the appropriate plugins installed:

dokku plugin:install https://github.com/dokku/dokku-http-auth.git
dokku plugin:install https://github.com/dokku/dokku-letsencrypt.git

Adding a full monitoring solution will increase the resource demands on your server, but once configured, you’ll have metrics to help you determine if additional resources are needed.

5. Network setup

We will use the Docker private network to allow applications to communicate with each other on a private network.

Create a bridge network named “grafana-bridge”:

dokku network:create grafana-bridge

Step 2: Setting up Loki

To configure Loki, follow these steps:

1. Create a Loki application:

dokku apps:create loki

2. Configure ports for Loki to listen for HTTP requests on port 80 and forward them to port 3100 where Loki will run:

dokku ports:add loki http:80:3100

3. Set the environment variable for Loki to the path to the configuration file. This will allow Loki to use your configuration setting:

dokku config:set --no-restart loki DOKKU_DOCKERFILE_START_CMD="-config.file=/etc/loki/loki-config.yaml"

4. Create a directory to mount the Loki configuration store and create a configuration file:

mkdir -p /var/lib/dokku/data/storage/loki/config
touch /var/lib/dokku/data/storage/loki/config/loki-config.yaml

5. Set the correct access rights to the created directories and files:

chown -R nobody:nogroup /var/lib/dokku/data/storage/loki

6. Mount the Loki configuration directory into the Loki container using the following command:

dokku storage:mount loki /var/lib/dokku/data/storage/loki/config:/etc/loki

Loki configuration

7. Save the Loki configuration file in the following path: /var/lib/dokku/data/storage/loki/config/loki‑config.yaml

and add the following content to it:

auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 1h
  max_chunk_age: 1h
  chunk_target_size: 1048576
  chunk_retain_period: 30s
  max_transfer_retries: 0

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /tmp/loki/boltdb-shipper-active
    cache_location: /tmp/loki/boltdb-shipper-cache
    cache_ttl: 24h
    shared_store: filesystem
  filesystem:
    directory: /tmp/loki/chunks

compactor:
  working_directory: /tmp/loki/boltdb-shipper-compactor
  shared_store: filesystem

limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 168h

chunk_store_config:
  max_look_back_period: 0s

table_manager:
  retention_deletes_enabled: false
  retention_period: 0s

ruler:
  storage:
    type: local
    local:
      directory: /tmp/loki/rules
  rule_path: /tmp/loki/rules-temp
  alertmanager_url: http://localhost:9093
  ring:
    kvstore:
      store: inmemory
  enable_api: true

8. Save the Loki configuration file.

9. Setting up a Custom Domain:

• Assign a custom domain to Grafana: dokku domains:set loki loki.dokku.me

• Make sure the DNS records for loki.dokku.me correctly point to your Dokku server.

10. Enable HTTPS:

After installing the domain, enable HTTPS:

dokku letsencrypt:enable loki

10. Enable HTTP authentication for the Loki application. Replace <username> And <password> to your chosen username and password combination:

dokku http-auth:on loki <username> <password>

You now have Loki configured to centrally collect and aggregate logs on your Dokku server.

Step 3: Setting up Promtail

To set up Promtail, follow these steps:

1. Create a Promtail application:

dokku apps:create promtail

2. Set the environment variable for Promtail to the path to the configuration file. This will allow Promtail to use your configuration setting:

dokku config:set --no-restart promtail DOKKU_DOCKERFILE_START_CMD="-config.file=/etc/promtail/promtail-config.yaml"

3. Create a directory to mount the Promtail configuration store and create a configuration file:

mkdir -p /var/lib/dokku/data/storage/promtail/config
touch /var/lib/dokku/data/storage/promtail/config/promtail-config.yaml

4. Set the correct access rights to the created directories and files:

chown -R nobody:nogroup /var/lib/dokku/data/storage/promtail

5. Mount the Promtail configuration directory into the Promtail container:

dokku storage:mount promtail /var/lib/dokku/data/storage/promtail/config:/etc/promtail

6. Mount the log directory /var/log into the Promtail container:

dokku storage:mount promtail /var/log:/var/log

7. Connect the Promtail application to the network grafana-bridge after deployment:

dokku network:set promtail attach-post-deploy grafana-bridge

Promtail configuration

8. Create a Promtail configuration file in the following path: /var/lib/dokku/data/storage/promtail/config/promtail-config.yaml

and add the following content to it:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki.web:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          __path__: /var/log/*log
      - targets:
          - localhost
        labels:
          job: nginx
          __path__: /var/log/nginx/*log

9. Save the Promtail configuration file.

Promtail deployment

11. Deploy the Promtail application with one command:

dokku git:from-image promtail grafana/promtail

12. Disable the domain name for the Promtail application as it is not required for log collection:

dokku domains:disable promtail

You now have Promtail configured to collect and send logs to the Loki system on your Dokku server.

Step 4: Setting up Grafana

To configure Grafana, follow these steps:

1. Create a Grafana application:

dokku apps:create grafana

2. Configure ports for Grafana so that it listens for HTTP requests on port 80 and forwards them to port 3000, where Grafana will run:

dokku ports:add grafana http:80:3000

3. Create directories for mounting the data store and Grafana configuration, as well as a directory for plugins:

mkdir -p /var/lib/dokku/data/storage/grafana/{config,data,plugins}

4. Create a directory to store data source configuration files:

mkdir -p /var/lib/dokku/data/storage/grafana/config/provisioning/datasources

5. Set the correct access rights to the created directories and files:

chown -R 472:472 /var/lib/dokku/data/storage/grafana

6.Mount the Grafana data and configuration directories, as well as the plugins directory:

dokku storage:mount grafana /var/lib/dokku/data/storage/grafana/config/provisioning/datasources:/etc/grafana/provisioning/datasources
dokku storage:mount grafana /var/lib/dokku/data/storage/grafana/data:/var/lib/grafana
dokku storage:mount grafana /var/lib/dokku/data/storage/grafana/plugins:/var/lib/grafana/plugins

7.Create a configuration file for the Loki data source:

nano /var/lib/dokku/data/storage/grafana/config/provisioning/datasources/loki.yml

And add the following content to it:

datasources:
  - name: Loki
    type: loki
    access: proxy
    orgId: 1
    url: http://loki.web:3100
    basicAuth: false
    isDefault: false
    version: 1
    editable: true

8. Save the configuration files for the data sources.

9. Join the Grafana application to the network grafana-bridge after deployment:

dokku network:set grafana attach-post-deploy grafana-bridge

10. Setting up a Custom Domain:

Assign a custom domain to Grafana:

dokku domains:set grafana grafana.dokku.me

Make sure the DNS records for grafana.dokku.me correctly point to your Dokku server.

11. Enable HTTPS:

After installing the domain, enable HTTPS:

dokku letsencrypt:enable grafana

You now have Grafana configured with a domain name grafana.kaido.team and HTTPS enabled to ensure secure data access.

Step 5. Set up Promtail to monitor JSON logs on a remote server running Loki

To configure Promtail to monitor JSON logs on a remote server and send them to Loki, consider the following steps:

Setting up a remote server

1. Install and configure Promtail on a remote server. An example Promtail configuration file on a remote server might look like this:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki.dokku.me:3100/loki/api/v1/push
    basic_auth_user: ваш_пользователь
    basic_auth_password: ваш_пароль

scrape_configs:
  - job_name: json_logs
    static_configs:
      - targets:
          - /путь/к/файлу.json  # Укажите путь к JSON-лог файлу на удаленном сервере
        labels:
          job: json_logs

2. In this configuration file:

• http_listen_port And grpc_listen_port – these are the ports on which Promtail will listen for HTTP and gRPC requests.

• clients – points to the Loki address where Promtail will send the collected logs. Specify basic_auth_user And basic_auth_password for HTTP authorization on the Loki server.

• scrape_configs – this is a list of configurations for monitoring logs. In this case, we created a configuration named json_logs to monitor a JSON log file on a remote server.

• targets – replace /путь/к/файлу.json to the path to the JSON log file on the remote server.

3. Save the configuration file on a remote server, e.g. /etc/promtail/promtail-config.yaml

Promtail will now monitor the JSON log file on the remote server and send it to Loki using HTTP authorization on the Loki server available on your local server for further visualization in Grafana.

Step 6: Data Visualization with Grafana

Visualization of logs

1. Login to the Grafana web interface https://grafana.dokku.me.

2. Login to Grafana with the credentials login: admin password: admin (the default values ​​that you will change during your first login).

3. Go to the “Explore” section in the top navigation bar of Grafana.

4. In the Explore section, you can select a Loki data source and begin searching and visualizing your JSON logs.

5. In the “Log labels” field you can select job in the drop-down list to filter logs by job_name.

6. You can then enter queries and use Loki’s features to search, filter, and visualize logs.

7. Create and save panels to visualize the data you are interested in from the logs.

You now have the tools to visualize and analyze JSON logs using Grafana and Loki. You can create graphs, panels and dashboards to monitor your infrastructure and applications based on logs.

In conclusion, setting up centralized logging using Dokku, Loki and Grafana provides a powerful solution for modern server monitoring and management. This system not only provides flexibility and scalability, but also allows you to quickly analyze and monitor the performance of applications and servers in real time. Through the integration of Dokku for container management, Loki for log aggregation and centralization, and Grafana for data visualization and analysis, users receive a comprehensive set of tools to effectively monitor and respond to events in their infrastructure.

The guide provided in this article details each configuration step, from installing Dokku to configuring Grafana and Promtail. This provides in-depth insight into the process and helps users customize the system to their unique requirements. Taking into account all aspects of security and performance, such a logging system becomes a reliable solution for collecting, analyzing and visualizing log data.

Centralized logging not only simplifies the monitoring process, but also plays an important role in ensuring the stability and security of the server infrastructure. In today’s environment, where speed and accuracy of incident response is critical, such a system is an essential tool in the arsenal of any server administrator or developer.