static typing in Tarantool Lua scripts

Introduction

If you've ever written scripts for Tarantula, you can certainly understand my pain. Tarantula is an amazing tool that not only allows you to store relatively large amounts of data and perform amazingly fast CRUD operations on this data, but also provides very extensive capabilities for processing this data directly in the Tarantula environment. And by data processing, I mean not just validating them and performing some mathematical operations on them, but almost the entire range of capabilities provided by the Lua language and a whole bunch of useful modules included in the Tarantula distribution package or installed from third-party sources.

In order to write, for example, a full-fledged HTTP server on Tarantula (please don’t kick me for this wording), we need to know very little – the basic syntax of the Lua language and the API of the main modules of Tarantula itself. And if with Lua everything is quite simple – learning this language in one evening, I’m sure few people will find it an impossible task – then with Tarantula modules everything is a little more complicated. You can study all the official documentation up and down and just while writing the script you will encounter one unpleasant problem – writing relatively large things for Tarantula is terribly inconvenient.

Problem

And that's where my pain lies. I write everything in VSCode and quite often I have to write scripts for Tarantula. This means that during the process I don’t have the usual editor hints or at least some static type analysis. No, of course, the Lua language server provides tools for working directly with the Lua source code, but once it comes to using Tarantula's built-in modules, almost all the associated code has to be written at random. I can't be sure which modules contain which functions, what their signatures are, what they return, or how to work with them. Cramming documentation is, of course, not an option. (I’ve already overcome myself and read it, what else do you need?)and switching to the browser window every second to find the necessary information is a bit of a chore, to be honest.

Finding a solution

One day, sitting at my laptop and pressing keys while writing another Lua script, I came to an interesting thought. What if there was a tool that would allow you to describe the types provided by Tarantula, and then use the described representation in the process of developing new scripts? Experience with JavaScript and TypeScript suggests that solving this problem is not a bad idea. The idea is actually simple and very strange that it only occurred to me a year after I met Tarantula. A quick Google allowed me to find three promising areas for working on the problem.

1. LuaLS – extension for VSCode with language server for Lua. Provides the ability to write your own addon that describes the required type scheme. Actually, I already use this extension, and the idea of ​​adding the elements I need there looks quite interesting.

2. Teal – a dialect of Lua with static typing. Something like TypeScript, but for Lua. It also allows you to describe the types used, and then compile the program into a native Lua script. Looks interesting too.

3. TypeScriptToLua (TSTL) – but this is a real TypeScript that compiles to Lua.

Perhaps there are other tools that solve my problem, but I decided not to go further than these three. The first two look pretty easy to implement, so I thought it would be a good idea to do something more useful more complicated. My choice fell on the third instrument. Well, at the same time I’ll improve TypeScript.

Solution

I won't be here pull the cat It’s long and tedious to describe the process of writing my own package for TSTL, so as not to cause the reader to yawn, so I’ll immediately present the result of my suffering works TarantoolScript – a package that provides type declarations that can be useful when writing scripts for Tarantula. Mind your hands: I really like Lua, so I did my best to avoid writing in Lua 🙂

In order for this thing to work, you, of course, need to have the TypeScript and TSTL packages installed. Well, and be able to use TypeScript, of course. Well, take a little look at the TSTL documentation. That is, in addition to one Tarantula documentation, you now need to additionally study two other documentation. An excellent result of the work done, I think.

But seriously, yes – instead of writing Lua almost at random, the package makes it easy and almost painless to use almost all the features of TypeScript. TSTL will take care of compiling all this into Lua.

Examples

A database migration, if you can call it that, could of course look something like this:

{
        name: '_create_partners',
        up: function (this: Migration, userName: string): void {
            box.once(this.name, () => {
                const space = box.schema.space.create('partners', {
                    if_not_exists: true,
                    engine: 'memtx',
                    user: userName,
                });

                space.format([
                    { name: 'id', type: 'uuid' },
                    { name: 'name', type: 'string' },
                ]);

                space.create_index('primary', {
                    unique: true,
                    if_not_exists: true,
                    parts: ['id'],
                });

                box.space._schema.delete(`once${this.name}_down`);
            });
        },
        down: function (this: Migration): void {
            box.once(`${this.name}_down`, () => {
                box.space.get('partners').drop();
                box.space._schema.delete(`once${this.name}`);
            });
        },
    },

This is how it will be compiled:

{
        name = "_create_partners",
        up = function(self, userName)
            box.once(
                self.name,
                function()
                    local space = box.schema.space.create("partners", {if_not_exists = true, engine = "memtx", user = userName})
                    space:format({{name = "id", type = "uuid"}, {name = "name", type = "string"}})
                    space:create_index("primary", {unique = true, if_not_exists = true, parts = {"id"}})
                    box.space._schema:delete(("once" .. self.name) .. "_down")
                end
            )
        end,
        down = function(self)
            box.once(
                self.name .. "_down",
                function()
                    box.space.partners:drop()
                    box.space._schema:delete("once" .. self.name)
                end
            )
        end
    }

And this is how you can initialize the HTTP server:

import * as http_server from 'http.server';

const httpd = http_server.new_(host, port, {
    log_requests: true,
});

httpd.route({ path: '/sync/:partner_id/:partner_user_id', method: 'GET' }, handlerSync);
httpd.route({ path: '/match', method: 'GET' }, handlerMatch);
httpd.route({ path: '/partners', method: 'GET' }, handlerPartners);
httpd.route({ path: '/partners', method: 'POST' }, handlerPartners);
httpd.route({ path: '/partners/:partner_id', method: 'GET' }, handlerPartners);
httpd.route({ path: '/partners/:partner_id', method: 'DELETE' }, handlerPartners);
httpd.start();

TSTL will turn this into the following piece of code:

local http_server = require("http.server")
local httpd = http_server.new(host, port, {log_requests = true})
httpd:route({path = "/sync/:partner_id/:partner_user_id", method = "GET"}, handlerSync)
httpd:route({path = "/match", method = "GET"}, handlerMatch)
httpd:route({path = "/partners", method = "GET"}, handlerPartners)
httpd:route({path = "/partners", method = "POST"}, handlerPartners)
httpd:route({path = "/partners/:partner_id", method = "GET"}, handlerPartners)
httpd:route({path = "/partners/:partner_id", method = "DELETE"}, handlerPartners)
httpd:start()

And all this with editor hints and static type checking. It's a fairy tale!

Not a fairy tale

Of course, there are problems. TSTL itself, for example, although it has existed for quite a long time, has a number of problems. Just look at the number of open issues in the project repository (119 at the time of writing). Source maps do not work very correctly, there are very poor options for managing the import of modules, there is constant confusion about whether functions need to be explicitly specified as this or not, and more. The project, of course, is far from abandoned, but it’s not like it’s being updated every day.

What to pay attention to

If the reader has reached this point, then the article probably interested him. I assume that he might also be interested in trying to feel my library with his own hands. Therefore, here is a rough list of what you need to consider when working with TarantoolScript in particular and TSTL in general:

• Before accessing the module box, you must declare it explicitly. That is, somewhere at the top of the source code file there should be a piece like this:
  import { Box } from 'tarantoolscript';
declare const box: Box;

• If you want to use any module besides box , you need to make some small changes to tsconfig.json. For example, if there is a need to connect a module log then the config must have the following settings:
  {
"compilerOptions": {
"paths": {
"log": ["tarantoolscript/src/log.d.ts"]
}
},
"tstl": {
"noResolvePaths": [
"log"
]
}
}
  For luatest.helpersfor example, you need to add:
  {
"compilerOptions": {
"paths": {
"luatest.helpers": ["tarantoolscript/src/luatest.helpers.d.ts"] }
},
"tstl": {
"noResolvePaths": [
"luatest.helpers"
]
}
}
  After which the modules can be imported:
  import * as log from 'log';
  or
  import * as luatest_helpers from 'luatest.helpers';
  For other modules, if they already have type descriptions, everything is the same – just change the names.

• In order to be able to correctly obtain the values ​​of functions that return several values ​​at a time, you should use destructuring:
  const [ok, body] = pcall(() => req.json());

In general, a lot of useful information in terms of syntax can be gleaned from this And this projects. The first is just a set of samples that clearly demonstrate how TS code compiles in Lua (the main thing is not to forget about npm run build). The second is an example of a project that could be written using the tools I describe.

Conclusion

Towards the end of the article, I suddenly remembered that at the very beginning it would be nice to describe the goals of the work I did. But since I'm not completely normal I’m too lazy to rewrite, I’ll risk formulating my goals here.

The first goal is, of course, to have a convenient tool that will make my life easier. The second and, probably, main goal is to gain experience in Lua, TypeScript and Tarantool. The second goal has definitely been achieved, but the degree to which the first goal has been achieved will only be assessed after some time has passed.

I will assume that the most banal question that a skeptical reader can ask looks something like this: “Why is all this necessary?” My most banal answer is – it’s interesting! Otherwise, why would we all gather here then?

Links

Thank you for your attention!