lnav is a super underrated log management tool

Good day. This is my first post on Habr, so don't be too hard on it.

In the world of development, system administration and DevOps, despite the fact that tools related to centralized collection, visualization and analysis of logs (graylog, ELK/EFK, loki, loggly and others) have long existed and occupied their niche, there is still a need to periodically pick up a saber and work with old/good (and maybe not so good) text logs. Over 21 years of my activity, I have managed to be a system administrator, DevOps engineer, developer, CTO and system analyst, but the need to periodically work with logs has always been present in one form or another. This could be parsing the output of a new service or container on a developer's machine, something that has not yet been added (or deliberately for some reason has not been added) to a centralized log collection system or, for example, a service temporarily enabled in debug mode to find the causes of a problem. There are many situations and situations are different, but text logs were, are and will be with us for a long time.

Everyone who is somehow connected with DevOps knows about such utilities as more, less, tail, head, grep, sed, awk (and some even a dozen more specific ones) and uses them when necessary, but of those I talked to, no one confirmed to me that they know about lnav. I myself did not know and was looking for something like this for more than ten years. lnav is not just a Swiss army knife in the world of working with logs, but an entire spaceship on which you can fly to a neighboring galaxy. My world was divided into “before” and “after” acquaintance with this utility. Where previously hours, or even tens of hours were required to analyze logs, now a few minutes are enough.

Briefly about the main features of lnav.

Positive and negative filters: lnav provides the ability to turn filters on and off (including temporarily), allowing you to focus only on the information you need. Positive filters find relevant records, and negative filters exclude unnecessary ones, and you can immediately see how many lines have fallen under the filter and are currently displayed. Naturally, regular expressions are supported. That is, you can, for example, quickly find all logged queries in the database, the execution time of which exceeded a certain number of seconds.
Convenient navigation: The utility offers an intuitive interface with VIM-style navigation. HJKL, jump to the beginning and end with gg and G and some other features work. You can easily move to the next (e) or previous (E) error, or to what lnav considers it an error, etc. The ability to quickly navigate to the necessary blocks of information significantly speeds up the work process.
Syntax highlighting and theming: Syntax highlighting makes logs more readable, and the ability to customize the theme allows you to adapt the interface to your preferences. This is especially important when working with large files, where details can easily slip away due to improper formatting. Anyone who has tried to unpack a log into several million SQL queries will understand and appreciate it. Also, lnav can prettify (yes, I still don't know how to translate this term correctly into Russian) for xml and json.
Statistics graph: lnav allows you to display a graph of the statistics of log entries per unit of time. The length of the filled line of the graph shows the number of lines per unit of time relative to neighboring lines and can have up to three sections of different colors: gray for normal entries, yellow for warning entries, and red for error entries. This allows you to quickly visualize the state of the system and understand when something went wrong in your system. That is, you will immediately see at what point the number of entries with errors sharply increased, even if their absolute number is less than a hundred for several tens/hundreds of thousands of normal entries.
Definition of loglevel: The utility allows you to explicitly specify the loglevel for display, which helps you focus on the most critical records and not get distracted by less important ones.
Multiple log formats: lnav supports the simultaneous display of several log files of different formats (yes, with individual syntax highlighting for each format), which simplifies the process of analyzing information from various sources and leads to a more complete understanding of what is happening.
Internal SQL and field partitioning: An interesting and perhaps controversial solution, but lnav translates the entire log into a SQLite database, which it stores in RAM. This allows you to use different parts of the log entry as separate fields and make SQL queries to your log data.
SELECT c_ip, count(*), sum(sc_bytes) AS total FROM access_log GROUP BY c_ip ORDER BY total DESC;
Working with logged SQL queries: When filtering lnav analyzes SQL queries and outputs all the lines that match the filter, even if the query consists of several lines. This is an invaluable feature. That is, by entering something like “uid=123” into the filter, you will get full records with both single-line queries and those where it occurred on the 105th line of the query. Naturally, the location of the entry will be highlighted.
Merging records by time: Even if the time formats in the logs are different, lnav will try to interpret them and display the entries on a single timeline. This allows you to see the alternation of lines from different files as they arrive, which improves the overall picture of what is happening and allows you to discover the relationships between events.
Export data: After applying all the necessary filters, you have the ability to select a block of lines, including data from different files, and export this data to a new file in text, JSON or CSV format. This significantly simplifies the preparation of reports and further analysis. In particular, you can select the data needed for export not in one block, but in several.
Creating your own log formats: You can use a special syntax to describe your log format to separate it into fields. This allows for more productive analysis in the future, as you can customize lnav to the specific requirements of your specific project. Moreover, you can either create a format from scratch or inherit the fields of an already described format.
Bookmarks and additional features: lnav allows you to create bookmarks, which helps you quickly return to the desired areas of data, similar to how it works in vim.
Possibility of non-interactive work with lnav and creation of scripts for data processing: One of the powerful features of lnav is its non-interactive operation, which allows you to write your own scripts to automate log analysis and data processing. This is especially useful when you need to process large amounts of information or perform recurring tasks.
$ lnav -n \ -c ';SELECT c_ip, count(*) AS total FROM access_log GROUP BY c_ip ORDER BY total DESC LIMIT 10' \ -c ':write-csv-to -' \ access.log c_ip,total 10.208.110.176,2989570 10.178.4.102,11183 10.32.110.197,2020 10.29.165.250,443
Saving and loading sessions: lnav allows you to save sessions. This allows you to save the current state of the log view, including applied filters, annotations, and all actions performed.
Saving a session: :save-session имя_сессии.lnav
Loading a previously saved session: :load-session имя_сессии.lnav
Working with pipes: If necessary, you can use pipes, for example: tail -n 1000 -f somefile.log | lnav.

More information can be found at official documentation.

Fly in the ointment

There are no tools that consist only of advantages, so below are the disadvantages I have identified:

Since the log is completely loaded into RAM, we are limited by the amount of this memory. If you currently have a shortage of free RAM, it is probably a bad idea to open several dozen logs of a gigabyte each on a production server in lnav. When working with a large number of logs, monitor the RAM or unload the logs to a local machine.
Because the log contents are completely translated into SQLite, opening large volumes of logs can take a significant amount of time on systems with weak CPUs.
Higher barrier to entry. “It's not grep”, knowledge of regex alone will not be enough if you are going to use lnav to its full potential.
Sometimes, for unexplained reasons, especially on large amounts of data, lnav can crash. Unfortunately, this happens from time to time.

Conclusion

In my opinion, the advantages of lnav clearly outweigh the disadvantages. lnav is a powerful and versatile tool that greatly simplifies the process of log analysis. The combination of its capabilities makes it an indispensable assistant for anyone who has to deal with logs. Over the years that I have been using lnavit has become a must-have tool on every host I have, both home and work. The effort it took to learn lnav paid for itself a hundredfold. If you haven't heard of this tool, I highly recommend you check it out.

You can find out more about lnav on official website.