Parsing, storing and displaying logs from 3 different email delivery agents. MTA Log Parser
This is my first short article on Habré in an attempt to make a project that I did for myself for the purpose of self-education and application at work useful to someone else. It would be possible to write more letters, but due to the peculiarities of professional activity, there is not much time for this, to put it mildly. Those who are interested can always learn the details on their own on the github.
background
I remember that about 10 years ago I wanted to find a convenient GUI mail log viewer with filters by time and various key fields for Exim, and preferably with the ability to automatically clean up outdated entries. Reading the log files is, of course, good, but not very fast and convenient, especially when a person is standing over you (or hanging on the telephone line), who “right now” needs to find out why the most important letter from some LLC has not arrived. Horns and hooves”. Yes, and the issues of storing logs for a period longer than a week and accessing them were also acute.
Such a GUI viewer was subsequently found in the face of an old, but very effective exilog. However, over time, Postfix-based mailers began to appear on the farm (usually as part of mailcow-type combines), and for them nothing modern, youthful, and most importantly, simple and quick in scanning could not be found.
Then ELK appeared and there were my quite successful attempts to fasten the parsing, storage and display of all logs (not just mail logs) to this stack. But there was a strong desire to have something separate, more lightweight, less resource-intensive and universal specifically for mail.
Around this time on github, I came across a wonderful project Privex Postfix Log parser.
In it, I immediately liked the approach and there was a desire to develop the project, making it more convenient and expanding the scope, because. I had a lot of wishlists, but the project seemed somehow unfinished and abandoned.
So, meet MTA Log Parser.
Actually subject
The programmer from me is so-so, since this is not my main activity, but it turned out that it turned out. The request for the source code is not strongly scolded and not beaten with slippers.
Initial project on Quart (backend) and VueJS (frontend) was finalized by me, adapted to the more current vue 3. As a DBMS in the initial project, rethinkdb is used and migration to something else, or support for something else, I have not yet done for reasons – works – do not touch. If there is a reasoned opinion why it is worth redoing the DBMS backend to something else – write in the comments, it will be interesting to read.
The list of changes compared to the original project is quite large and can be found in CHANGES.md. Here I will write the most basic at the moment (v.1.1.2):
Support for the 3 most popular (according to some resources) MTA – Exim, Postfix and Sendmail
Dockerization – greatly simplifies and speeds up deployment to any production
More advanced GUI with more filters
Localization of the GUI (Russian and English locales are currently supported, but it is possible to add support for others)
Rotation of logs (removal of obsolete ones) by the number of days
The plans include all sorts of fake whistles, support for several simultaneous filters (now only one text filter for the selected field and a filter for time are applied at the same time) and code refactoring, because. what I’ve done is quite terrible in terms of code.
Criticism, PR’s and just advice from programming gurus are accepted =)
Well, a couple of screenshots for the seed:
