Django application in docker. Logging and monitoring (also in docker)

The promised second part.

The first part is here https://habr.com/ru/articles/844280/

The main idea was to set everything up with a minimum amount of manual actions to make logging and monitoring work.

One manual action will still be required – specify the host (or IP) from which Prometheus will collect metrics in the file https://github.com/famer/loki/blob/main/prometheus.yml. Otherwise, everything will work just by docker-compose up.

Full code Here

Let's look at the main points.

The scheme is as follows:

Promtail collects logs from docker containers via docker socket. Application logs are written to stdout stderr accordingly. That is, to the log of the container itself. How logging is configured for a Django application can be seen here https://github.com/famer/django/blob/main/itjobs/settings/prod.py (from the first part):

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        'console': {
            'class': 'logging.StreamHandler',
        },
    },
    'root': {
        'handlers': ['console'],
        'level': 'INFO',
    },
    'loggers': {
        'django': {
            'handlers': ['console'],
            'level': 'INFO',
            'propagate': True,
        },
        'django.request': {
            'handlers': ['console'],
            'level': 'INFO',
            'propagate': False,
        },
    },
}

Promtail passes logs to Loki, which stores them.

Prometheus collects metrics based on the file specified https://github.com/famer/loki/blob/main/prometheus.yml host (by the way, pay attention to https://github.com/famer/django/blob/main/nginx.conf there access to metrics is prohibited from all IP addresses, except those specified for security purposes, in general /metrics is not recommended to be exposed to the outside)

Well, that's all, add Loki and Prometheus as data sources to Grafana and then visualize them as you want.

Example in graph for requesting and formatting logs from nginx:

{container="/itjobs-nginx-1"} | json request_method, request_uri, status | __error__=`` | line_format `{{.request_method}} {{.request_uri}} with HTTP status: {{.status}} ` 

That's all.