Kibana. Using the KQL query language when searching for logs

For beginner testers

Author: Nadezhda Dudnik

Content

Introduction

Kibana (wiki) used to monitor and analyze IT infrastructure as part of the Elastic Stack, which, in addition to it, includes Elasticsearch and Logstash. Logstash is responsible for logging and delivers the incoming data stream to Elasticsearch for storage, classification, and retrieval. Kibanain turn, accesses the data elasticsearch for their visualization in various visual formats.

Kibana has its own KQL query language (Kibana Query Language)- official source.

Using this language, you can write queries that help you filter and find the information you need.

Connecting to Kibana to view logs

Go to URL Kibana- http://37.46.129.245:5601/app/home#/ (This information was shared Evgeny Sychev – PM and QA Lead for fintech projects, for the practice of searching for logs) .

Enter data for authorization:

Select Kibana → Discover:

The mode for KQL is enabled by default.

Select value in Change index pattern, where you need to check the logs. First, for parsing, I will choose ‘logstash‘ and then for parsing consider ‘forum‘, since the logs come from the clan’s forum for the game Lord of the Rings online – http://forum.free-peoples.ru/for which Evgeny Sychev is also responsible (warning: certificate requirement -> trust).

The main blocks for working with logs.

1. Block for entering the search for logs using a KQL query and without;

2. Block with a choice of time interval (Today, This week, Last 15 minutes (default), Last 30 minutes, Last 1 hour, Last 24 hours, Last 7 days, Last 30 days, Last 90 days, Last 1 year);

3. Block with selected filters for outputting logs (“Selected fields”);

4. Block with available filters (“available fields“);

5. Log search results block with or without selected filters;

6. Block of filters by data type “Filter by type“;

7. Buttons “Refresh / Update“.

Using KQL.

Point the cursor to the line for composing KQL queries.

Possible key attributes to search for logs are displayed.

Specify, for example, the attribute “message” (Filter results that contain message).

Screenshots from documentation about the description of the main parameters.

Let’s look at examples, the field “host” has data type “string“, at which the following parameters are displayedдля поля строки:

Field “geoip.ip” has data type “number“, which displays the following options для числового поля:

Compiling a simple KQL query.

host: jekil1.fvds.ru and severity : WARN

_{( : ) is an operator that is responsible for finding matches.}

Match results:

Следующий запрос: host: jekil1.fvds.ru and severity : WARN and message: *auth.log*

*auth.log* where

denotes a wildcard. *auth.log* – finds any values ​​that have “oauth.log”

Shows “oauth.log” in any position exc*- finds any values ​​that start with

with “exc”.

“exc”.KQL queries do not support suffix matching

values ​​that end with “exc” -> no such values

Double quotes in KQL query. To specify a clause in a KQL query, you must use the operator – double quotes” “

.

message: exception=>Errno::EACCES

For example, when specifying the following KQL query:

Search Error
Expected AND, OR, end of input, whitespace but ">" found. message: exception=>Errno::EACCES -------------------^

sentence without quotes And when specifying a KQL querywith quotes

message: "exception=>Errno::EACCES"

:

sentence with quotation marks

I want to note that double quotes are very necessary when searching for logs.

Filter by typeAdd field “_type

Press “+”

1. Ввести "host: jekil1.fvds.ru and severity : WARN and message: exc*" в блоке для запросов;
2. Выбрать "Today" в блоке временного интервала; 
3. Выбрать только "_type" в блоке используемых фильтров;
4. Нажать на кнопку "Refresh/Update"

Adding a filter by the _type field

Adding a filter by the _type field Search results by filter selected fields, choosing“_type” and “path”

Adding a filter by _type and path fields

KQL with “Filter for value” filter.Additionally, the field “host

The value “WARN” is highlighted in yellow, as it is specified in the filter by values ​​in the upper right corner.

1. Ввести, например, значение поля "source" = "filewatch.tailmode.handlers.create" в блок для запросов;
2. Выбрать "Today" в блоке с временным интервалом ;
3. Нажать на кнопку "Refresh";
4. Удалить выбранные фильтры из "Filter for value" и "Filter by type".

Finding logs without using KQL.

Finding logs without using KQL

1. Practice. Follow the link of the clan forum for the game The Lord of the Rings online –https://forum.free-peoples.ru/

2. . (enter at your own risk -> ‘Advanced’);

3. Enter invalid “Username:” = “test” and “Password:” = “test” on the forum page;

4. Click on the “Login” button;

5. Fix the time; Follow the link http://37.46.129.245:5601/app/home#/

6. -> go through authorization -> select “Discover” -> select “forum” in Change index pattern;

7. Enter the value “login” in the query string;

8. Click on “Refresh”;

   

9. Search for logs for authorization in the forum on the site Click on the “Register” link on the website https://forum.free-peoples.ru/ucp.php?mode=register

10. . An error is displayed on a new page;

11. Fix the time;

12. Find log with text “mode=register” in Kibana;

13. Choose to show the log in JSON format;

Make sure that logs with an error are found

Additional Information:

View log in JSON format view single document

View detailed log

Also available:

From the documentation: “It is possible to combine different parts of keyword queries using an opening bracket ” ( ” and a closing bracket ” ) “. Each opening bracket ” ( ” must have a corresponding closing bracket ” ) “. A space before or after the bracket has no effect on the query. “

You can also add filters from the block with all filters to the area with used (selected) filters. Availablequick reference to KQL

.

Conclusion

I usually look for logs in Kibana by the ERROR logging level, as well as specifying the necessary data on the system. Additionally, you can search for logs by sessionId, Id, UUID, messages from the server response, request data. By this instruction,relying on useful sources

I wanted to convey an acquaintance how to write KQL queries, teach how to read logs and apply this knowledge in my work.

Thanks for reading.Sincerely, Nadezhda Dudnik (protestinginfo)

chief test engineer in fintech and software testing mentor.