we compress data, encrypt traffic, help investigate incidents
Tarantool 2.10 has added new tools for both the Enterprise and Community Editions. We also launched Tarantool on the DBaaS model in the VK Cloud Solutions cloud. Next, we will talk in detail about the main updates: encryption of connections between replicas and the client, data compression in memory, advanced auditing capabilities and share a link to full list of changes.
Compression
Tarantool has always been able to compress data on disk. Now Tarantool Enterprise allows you to compress data that is stored in RAM. This saves memory space, which reduces the need to purchase additional hardware. This is especially true if you store long strings and/or JSON objects in memory. On some datasets, you can reduce the amount of data stored in RAM by up to 60%.
Compression can be run directly on a running cluster. At the same time, continue to serve requests as usual. This will reduce the size of the stored data and allow you to reduce the number of servers used or increase the possible waiting time for purchases.
Algorithms ZSTD, LZ4 are supported. The first compresses better, but spends a little more time on the compression process. The second compresses less efficiently, but also works faster.
To use compression, you do not need to recreate the data space. Just specify which fields need to be compressed in the function space:format(). Then run the background migration via space:upgrade({ background = true }). Then everything works transparently: when writing, the data itself is compressed, when reading it is decompressed.
You can also compress data in existing storages.
Encryption
As a rule, Tarantool is in an internal loop, and it does not have access to the Internet. This ensures the security of data storage.
Previously, Tarantool did not have built-in tools for secure communication with services within the company. There were custom solutions that were made for the client. Now such a tool has appeared in the product – in the new version of Tarantool Enterprise, all traffic is encrypted.
Encryption will allow:
protect the interaction of services with Tarantool;
protect the interaction of Tarantool instances with each other;
expose access to Tarantool directly to the Internet.
An attacker will not be able to read the data, even if he was able to intercept your traffic.
Setting up encryption is simple: you need to specify the port, key, certificate, and the required algorithm. Further – everything works.
The new version supports the TLS protocol. This works using the GOST 34.10-2018GOST algorithm, as well as popular foreign algorithms. This allows you to use Tarantool in a stack and meet the information security requirements of any company.
Audit
Employee misuse of data, hacking attempts, and other incidents are things that no company is immune to. In the new release, we have expanded the capabilities of the audit log in Tarantool Enterprise. Now it will be easier to investigate incidents.
The audit system will allow you to quickly understand:
what happened at one time or another in the system;
what requests and changes were made by specific users.
Logs can now be written in JSON, CSV. The records themselves are configurable. Works on the same principle as in other common databases. The types of events that need to be logged can be filtered. You can also create your own event types via the Tarantool API.
Audit log is still enabled via the option audit_log in box.cfg{}. Nothing has changed.
And what else?
Tarantool in the cloud
Community version of Tarantool is now available as managed database on the VK Cloud Solutions cloud platform. Tarantool in the cloud is already configured and ready to upload data. The cloud version lowers the entry threshold: to work with Tarantool in the cloud, you do not need to program in Lua, understand sharding, and administer the solution yourself.
For Community and Enterprise versions
These mechanisms are called Constraints and Foreign keys, respectively.
API example for constraints:
box.schema.func.create('check_day_of_year',
{ language="LUA",
is_deterministic = true,
body = 'function(x, c) return x >= 1 and x <= 366 end'})space:format({.., { name="day_of_year", constraint="check_day_of_year"}, ..})API example for Foreign keys:
space:format({.., { name="country_id",
foreign_key = {
country = { space="countries", field = 'id'}
}
}, ..
})Sharding speeded up: remote stored procedure calls with vshard accelerated up to 70%. If the module is used
crudthen you will also notice a performance boost.We saved space: small lines are now stored more optimally – one line in memory needs 4 bytes less. Due to this, you can save up to several hundred MB on some datasets. Available after update. You don’t need to configure anything additionally.
Extended support for transactions in the protocol for connectors: now you can keep transactions open and still perform other operations.
Only in Tarantool Enterprise:
⚡️ Learned how to change the data schema in the background. Now you do not need to manually migrate data to a new space and spend time writing, running and recording a migration script.
How it works? All this will work through the function
space:upgrade().command execution
space:upgrade()leads to an instant change in the space format (so that the insertion of new data and the selection must match the new format);the space is not blocked for reading / writing, instead, the selection returns tuples in a new format (regardless of whether they have already been converted or not);
command starts a background task that iterates over all space tuples and converts them to a new format according to
upgrage_function.
⚡️ Implemented LDAP support. Everything is familiar and is what is expected. You can read the documentation here.
⚡️ Started writing Tarantool internal information and statistics to disk. They called it Flightrec. This will allow our technical support to quickly understand what happened and how to solve the problem.
