Collecting web application logs in Kibana
Why do we log?
We log to solve two very important tasks.
First — we want to understand how the software behaves on client devices. Of course, we test all the functionality that we bring to production on a large number of devices. But such testing is still a drop in the ocean compared to the variety of devices and client browsers.
In client logs, we encountered errors of blocked localStorage and added handling of inaccessible storage to the code. We met the error of the missing browser API – added a polyfill or raised the version of the minimum supported browser to display the stub of an outdated browser. We met errors in scripts that were not provided for by logic and testing – we corrected them.
We came across errors in browser extensions and even viruses on client devices, errors of blocked analytics, errors of loss of Internet connections. We worked out all these incidents. The strangest error we had was an error in an external script. It is reproduced only in mi‑browsers for some clients. If you know what this external script is – let me know in the comments.
Logging these errors can significantly improve the quality of the software being developed and fix problems before users notice them.
Second task – Incident handling. Every time a client has an error in the software, he calls the MTS hotline and reports the problem. The operator starts an incident in the internal system, then the information gets to our team and we start analyzing the problem from the logs. Using them, we can reproduce the chain of user actions and find the error. There was even a case when a person stopped using the service, but forgot to turn off the subscription. When contacting the call center, he reported this and said that he did not use the services. We checked the logs and really didn’t find any user activity after the debit date. The money was returned to the client.
So logging helps to significantly improve the quality of the software and better understand the errors that users have.
What are we logging?
On the side of the web application, exceptions within the application, global exceptions, important events in the logic (for example, payment) are usually logged. The set of these logs includes: error text, stacktrace, error location, navigator, user ID.
No personal data not logged for security reasons. Suffice it to recall the hacking of the Kibana (ELK) logging system, which allowed uploading someone else’s code to the server and, among other things, stealing someone else’s logs. In the absence of logging of personal data, it becomes much more difficult to steal them. And to identify the logs for the purpose of processing incidents, a user ID is enough.
How to log correctly?
Before we start collecting logs in Kibana or Loki, let’s first understand how to code the logging system.
It seems that the easiest way to log looks like this:
console.log("Это самое правильное логирование!");But there is a problem, such code can only be read on the client device. You can, of course, replace the browser console API to send it to the server, but this is an anti-pattern called decoy patching and we will not resort to it.
Kibana and Loki do not have their own logging modules in web applications. But you can find a well-known analogue called Sentry. This is a ready-made, convenient, and visual logging system for client errors. Then can log through Sentry?
import * as Sentry from "@sentry/browser";
Sentry.captureMessage("Вот теперь точно лучшая система логирования!!!");But no, with this method we will not see the logs locally, we will make a vendor lock on the Sentry system, we will not be able to correctly configure the logging levels.
We log correctly!
For logging, we use the method adopted in corporate development. We use an object that implements the helper pattern, it looks like this:
import * as Sentry from "@sentry/browser";
export class Logger {
public logLevel: LogLevels = LogLevels.Info;
public constructor () {
Sentry.init({
dsn: "https://examplePublicKey@o0.ingest.sentry.io/0",
maxBreadcrumbs: 50
});
}
public log(message: string, data: unknown) {
if (this.logLevel <= LogLevels.Info) {
console.log(message, data);
Sentry.captureMessage(message, "debug");
}
}
public warning(message: string, data: unknown) {
if (this.logLevel <= LogLevels.Warn) {
console.warn(message, data);
Sentry.captureMessage(message, "warning");
}
}
public error (message: string, data: unknown, error: Error) {
if (this.logLevel <= LogLevels.Error) {
console.error(message, data, error);
Sentry.captureMessage(message, "error");
Sentry.captureException(error);
}
}
public info/debug/trace (message: string, data: unknown) {...}
}
// singleton
export const logger = new Logger();
Such a helper object is called via DI, if it exists. If there is no DI – as an already created singleton logger object.
Having written such an object, we have created an abstraction that allows us to get rid of the binding to a specific logging system. Now, when logging in the code, we use this object everywhere and are not afraid that sooner or later the logging system will need to be changed. In addition, we got the opportunity to adjust the logging level depending on the needs and, if necessary, we can enable more detailed logs for an individual client.
Suddenly we lost Sentry, but gained Kibana
With the digital transformation of MTS, we needed to change the contours in which our applications are hosted. The old circuit had a well-functioning logging system built on Sentry, but for reasons beyond our control we had to abandon it (I hope it was temporary). But in the new circuit, the ELK logging system was already working with might and main, with a visualization panel on Kibana.
The only question left was: how to send web application logs to Kibana? After all, it can only work with server logs and does not have a module for web applications. And some applications (including microfrontends) are also published as static files and do not have their own server application.
The answer to the question turned out to be extremely simple – we made a microservice that receives logs from a web application and outputs them to the container console. Further, containers already have the functionality of reading logs and sending them to the logging system: Logstash (part of ELK), Loki and others.
First of all, we transformed our logger code by commenting out Sentry there in case it comes back. We also added a code that sends logs to the logging microservice.
The code took the following form:
export class Logger {
...
public log(message: string, data: unknown) {
if (this.logLevel <= LogLevels.Info) {
console.log(message, data);
// Sentry.captureMessage(message, "debug");
fetch("/logs/log/info", {
method: "POST",
body: JSON.stringify({
message: message,
data: data,
userAgent: navigator.userAgent,
location: location.href,
...other data
})
});
}
}
...
}
By adding only 10 lines to our logger, we taught it to work with the new logging microservice. But in general, such an abstraction as a logger helped us get rid of editing thousands of lines of code in which logging takes place.
Logging microservice
We created the microservice according to the already debugged process, which I wrote about earlier. They took the NestJS framework, created just one controller with logic, rolled it into a container and published it. We brought the microservice itself to Open source, code posted on githuband ready the container can be launched right now from dockerhub.
Instead of NestJS, there could be something lightweight. For example, C# or Go, but then there would be no one to support them except me.
The whole controller logic looks like this:
@Controller("logs")
export class LogController {
@Post("log")
public writeLog(@Body() body: object, @Headers() headers: object): void {
console.log(
JSON.stringify({
...body,
logLevel: LogLevels.Info,
time: Date.now(),
userIp: headers["x-real-ip"],
traceId: headers["x-trace-id"]
})
);
}
}This microservice runs in a container, the logs from which are read by the LogStash logging driver, as a result of which we observe the logs in Kibana.
Thus, the entire transformation of the logging system took only a couple of hours. They went to write and deploy the container, as well as edit the logger code.
What gives Kibana?
Kibana is an interface for working with logs, it allows you to filter them, build graphs, create your own dashboards and (most usefully) view end-to-end logs. You can build end-to-end logs throughout the system by user ID or other parameters, understand which button the client pressed on the front-end and what consequences this led to on the back-end.
It must be admitted that in Sentry it is much more convenient to analyze problems in order to understand in which browsers and on which devices it is played, to go through breadcrumbs, so to speak.
Thanks for taking the time to post! I hope my experience is useful to you. If you have any questions or comments, I will be happy to talk in the comments to the article!
