Adding Starlark to a Go Application
What kind of bird?
Starlark (previously known as Skylark) is a python-like language originally developed for the Bazel build system, which eventually got out of it through interpreters for go And Rust.
The language is conducive to use as a configuration tool, however, thanks to a well-written Go interpreter and detailed specification descriptionit can be used as a programming language built into the application – for example, when you want to let the user interact with the application logic object, but do not want to constantly produce entities for almost the same cases through a hundred settings (my case).
I could not find enough complete tutorials in the process of work, so I got the idea to write a small material on this topic. In this article, we’ll walk through working with Starlark in Go, from simply running a script to adding a new built-in type.
Disclaimer
I’m not a real developer, only pretending I am writing a pet project in my free time, so the text may contain not quite correct definitions. About all jambs, ochepyatka and other possible errors, please report, I will be corrected.
Starting small
To get started, it is enough to do two things – write the script source and execute it:
# hello.star
print("Hello from Starlark script")// main.go
package main
import "go.starlark.net/starlark"
func main() {
_, err := starlark.ExecFile(&starlark.Thread{}, "hello.star", nil, nil)
if err != nil {
panic(err)
}
}ExecFile loads the source file (or source code if the third function argument is not nil), parses and executes the code in the specified thread, after which it returns a dictionary with objects of the script’s global scope – variables, functions. The global scope is frozen, which means that any attempt to change the global definitions will cause a runtime error.
We run, we get:
$ go run main.go
Hello from Starlark scriptA good start, however, there is little benefit from this – the whole point is that the script should not be considered as a separate program, but as a plug-in module, the functions of which we are going to call as needed.
Let’s try it differently:
# hello.star
def hello():
print("Hello from Starlark script")We re-run the Go code and… nothing happens (expectedly) because we just declared the function, but didn’t call it. So, you need to call it from the main program:
// main.go
package main
import "go.starlark.net/starlark"
func main() {
// выделим поток, в котором будем загружать и выполнять скрипт
thread := &starlark.Thread{}
globals, err := starlark.ExecFile(thread, "hello.star", nil, nil)
if err != nil {
panic(err)
}
_, err = starlark.Call(thread, globals["hello"], nil, nil)
if err != nil {
panic(err)
}
}Result:
$ go run main.go
Hello from Starlark scriptglobals this is the same dictionary mentioned earlier with global variables and functions. Through call we call our function hello() by name, getting it from a dictionary. Positional and named arguments can be passed to the function as the third and fourth arguments.
Passing and receiving values
“Out of the box” Starlark has eleven built-in types (and a few more available modulesbut now not about them):
Noneanalogue
nilused when you want to express the absence of a valueBooleanlogical
TrueorFalseIntegerinteger, type concatenates signed and unsigned integers
floatfloating point number
Stringstring in UTF-8
Listsheet, mutable sequence of values
Tuplea tuple like a leaf, only immutable (but the values contained in the tuple can be changed)
Dictionarya key-value dictionary, only hashable types are supported as keys
set, a set, uses a hash table under the hood, so the requirement for values is the same as for keys in a dictionary; Go-specific type that requires setting to use special flag
functionfunctions defined in the Starlark code
built-in functiona separate type for functions (or methods, more on that later) implemented in Go
From the Go side, all types are required to implement the interface valueapart from type-specific interfaces such as callable for functions – this information is useful when writing your own types.
So, let’s try to pass something to our function:
# hello.star
def hello(message):
print(message)// main.go
package main
import "go.starlark.net/starlark"
func main() {
thread := &starlark.Thread{}
globals, err := starlark.ExecFile(thread, "hello.star", nil, nil)
if err != nil {
panic(err)
}
// здесь готовим позиционные аргументы для вызываемой функции
args := starlark.Tuple{
starlark.String("Hello from Golang"),
}
_, err = starlark.Call(thread, globals["hello"], args, nil)
if err != nil {
panic(err)
}
}Result:
$ go run main.go
Hello from GolangThis way you can pass any reasonable number of arguments to the called function. It is worth noting that an attempt to pass more or fewer arguments is a script runtime error – if three arguments are specified in the signature, then three are passed.
Let’s try to get something from our function. Let’s write the addition of numbers as the simplest example:
# hello.star
def sum(x, y):
return x + y// main.go
package main
import "go.starlark.net/starlark"
func main() {
thread := &starlark.Thread{}
globals, err := starlark.ExecFile(thread, "hello.star", nil, nil)
if err != nil {
panic(err)
}
// здесь готовим позиционные аргументы для вызываемой функции
args := starlark.Tuple{
starlark.MakeInt(42),
starlark.MakeInt(451),
}
result, err := starlark.Call(thread, globals["sum"], args, nil)
if err != nil {
panic(err)
}
print(result.String()) // распечатаем результат
}We launch:
$ go run main.go
493In a variable reslut stored some the result of the called function. Now get the value came out by translation Value to a string, but in real use you will need to cast the interface to the desired type. As an example:
long function
func toGoValue(starValue starlark.Value) (any, error) {
switch v := starValue.(type) {
case starlark.String:
return string(v), nil
case starlark.Bool:
return bool(v), nil
case starlark.Int: // int, uint both here
if value, ok := v.Int64(); ok {
return value, nil
}
if value, ok := v.Uint64(); ok {
return value, nil
}
return nil, errors.New("unknown starlark Int representation")
case starlark.Float:
return float64(v), nil
case *starlark.List:
slice := []any{}
iter := v.Iterate()
defer iter.Done()
var starValue starlark.Value
for iter.Next(&starValue) {
goValue, err := toGoValue(starValue)
if err != nil {
return nil, err
}
slice = append(slice, goValue)
}
return slice, nil
case *starlark.Dict:
datamap := make(map[string]any, v.Len())
for _, starKey := range v.Keys() {
goKey, ok := starKey.(starlark.String)
if !ok { // datamap key must be a string
return nil, fmt.Errorf("datamap key must be a string, got %v", starKey.String())
}
// since the search is based on a known key,
// it is expected that the value will always be found
starValue, _, _ := v.Get(starKey)
goValue, err := toGoValue(starValue)
if err != nil {
return nil, err
}
datamap[goKey.String()] = goValue
}
return datamap, nil
default:
return nil, fmt.Errorf("%v is not representable as datamap value", starValue.Type())
}
}A small note: in the above code, you should pay attention to working with iterator – when it is no longer needed, it is required to explicitly call Done().
Adding a new type
The Starlark interpreter allows you to extend the language by adding new types – just implement the interface value.
Imagine that we have a type, let it be a user, a synthetic example:
type User struct {
name string
mail *mail.Address
}
// конструктор пригодится позже
func NewUser(name, address string) (*User, error) {
mail, err := mail.ParseAddress(address)
if err != nil {
return nil, err
}
if len(name) == 0 {
return nil, errors.New("name required")
}
return &User{name: name, mail: mail}, nil
}
func (u *User) Rename(newName string) {
u.name = newName
}
func (u *User) ChangeMail(newMail string) error {
mail, err := mail.ParseAddress(newMail)
if err != nil {
return err
}
u.mail = mail
return nil
}
func (u *User) Name() string {
return u.name
}
func (u *User) Mail() string {
return u.mail.String()
}and we want to make it available on Starlark. To do this, you need a wrapper type with the appropriate methods:
var _ starlark.Value = &StarlarkUser{}
type StarlarkUser struct {
user *User
}
func (e *StarlarkUser) String() string {
return fmt.Sprintf("name: %v, mail: %v", e.user.Name(), e.user.Mail())
}
func (e *StarlarkUser) Type() string {
return "user"
}
// для упрощения, не будем заморачиваться с реализацией методов ниже
func (e *StarlarkUser) Freeze() {}
func (e *StarlarkUser) Truth() starlark.Bool {
return len(e.user.Name()) > 0 && len(e.user.Mail()) > 0
}
func (e *StarlarkUser) Hash() (uint32, error) {
return 0, errors.New("not hashable")
}MethodsString() , Type() And Truth() are needed to use the type in functions built into the language str(), type() And bool() (in addition, the second carries information about the type), Hash() is used to hash a value for use in a hash map in dictionaries and sets, and Freeze() is needed to freeze an object (as you can see, the guarantee of the immutability of an object after freezing the global scope lies entirely with the type implementation).
This type can already be used somehow, let’s try:
# user.star
def user_info(user):
print(type(user)) # напечатает тип
print(user) # напечатает строковое представление объектаargs := starlark.Tuple{
&StarlarkUser{
&User{
name: "John",
mail: &mail.Address{
Name: "John",
Address: "John@gmail.com",
},
},
},
}
_, err = starlark.Call(thread, globals["user_info"], args, nil)
if err != nil {
panic(err)
}Result:
user
name: John, mail: "John" <John@gmail.com>We have a full-fledged type, however, no operations can be performed on it – to implement them, you need to support the appropriate interfaces, for example, Has Unary, HasBinary etc., which of course we will not do within the framework of this article, otherwise there is already a lot of text, and there is still the creation of built-in functions and methods ahead.
Adding a new feature
This is where the constructor comes in handy. NewUser(). Built-in functions are implemented via Built in:
// тело функции
func newUser(_ *starlark.Thread, b *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
var name, mail string
if err := starlark.UnpackPositionalArgs(b.Name(), args, kwargs, 2, &name, &mail); err != nil {
return starlark.None, err
}
user, err := NewUser(name, mail)
if err != nil {
return starlark.None, err
}
return &StarlarkUser{user: user}, nil
}
func main() {
thread := &starlark.Thread{}
// собираем наши встраиваемые функции, которые затем передаются в ExecFile()
builtins := starlark.StringDict{
"newUser": starlark.NewBuiltin("newUser", newUser),
}
globals, err := starlark.ExecFile(thread, "user.star", nil, builtins)
if err != nil {
panic(err)
}
_, err = starlark.Call(thread, globals["user_info"], nil, nil)
if err != nil {
panic(err)
}
}Several new things have appeared. First, inline functions must match the type func(thread *starlark.Thread, fn *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) .
Secondly, inline functions, or rather, all predefined objects, are collected in a dictionary and passed to ExecFile()to make them available to the Starlark code.
Third, to unpack arguments, you can use UnpackPositionalArgs – it will check the number and types of the passed arguments.
Let’s try to call:
# user.star
def user_info():
user = newUser("John", "john@gmail.com")
print(type(user))
print(user)
$ go run main.go
user
name: John, mail: <john@gmail.com>Works! It is worth noting that in this way you can pass not only functions, but also any other objects whose type implements Value – for example, you can pass a pre-assembled set of constants that can be useful in embedded code.
Adding Methods
Adding “methods” to custom objects is implemented in a similar way, through Buitlinwhile the type that has methods must implement the interface HasAttrs.
But first, let’s prepare our methods:
func userName(_ *starlark.Thread, b *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
if err := starlark.UnpackPositionalArgs(b.Name(), args, kwargs, 0); err != nil {
return starlark.None, err
}
// получаем ресивер, приводим к нужному типу и работаем уже с ним
name := b.Receiver().(*StarlarkUser).user.Name()
return starlark.String(name), nil
}
func userRename(_ *starlark.Thread, b *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
var name string
if err := starlark.UnpackPositionalArgs(b.Name(), args, kwargs, 1, &name); err != nil {
return starlark.None, err
}
b.Receiver().(*StarlarkUser).user.Rename(name)
return starlark.None, nil
}The mechanics of how methods work differs from functions in that from the transmitted Builtin the receiver is extracted, converted to the desired type, after which the necessary manipulations are performed on it.
Looking at the implementation of built-in types in the interpreter library, we collect our methods into a dictionary:
var userMethods = map[string]*starlark.Builtin{
"name": starlark.NewBuiltin("name", userName),
"rename": starlark.NewBuiltin("rename", userRename),
}And we implement HasAttrs:
func (e *StarlarkUser) Attr(name string) (starlark.Value, error) {
b, ok := userMethods[name]
if !ok {
return nil, nil // нет такого метода
}
return b.BindReceiver(e), nil
}
func (e *StarlarkUser) AttrNames() []string {
names := make([]string, 0, len(userMethods))
for name := range userMethods {
names = append(names, name)
}
sort.Strings(names)
return names
}BindReceiver() creates a new Builtinwhich carries the passed value, which we access in the method.
We try:
# user.star
def user_info():
user = newUser("John", "john@gmail.com")
user.rename("Jim")
print(user.name())$ go run main.go
JimIn such a not very tricky way, we managed to add methods to our custom type.
Bonus: modules, built-in and custom
Starlark has several built-in modules that can be useful, here are some of them:
math – a module with mathematical functions and a couple of constants
time – functions and types for working with time
There is a special function for loading modules loadhowever, it will not work just like that – the modules are loaded the Load() function in the thread, and by default it is not, which means that you need to implement it. Let’s expand our original flow:
import (
"go.starlark.net/lib/math"
"go.starlark.net/lib/time"
)
thread := &starlark.Thread{
Load: func(thread *starlark.Thread, module string) (starlark.StringDict, error) {
switch module {
case "math.star":
return starlark.StringDict{
"math": math.Module,
}, nil
case "time.star":
return starlark.StringDict{
"time": time.Module,
}, nil
default:
return nil, fmt.Errorf("no such module: %v", module)
}
},
}The function will be called for every load() in code. Let’s try to display the current time:
# modules.star
load("time.star", "time")
print(time.now()) # выведет текущее времяSecond (and subsequent) arguments load() define imported literals, while starting with _ are not imported. In the case of modules implemented in Go, it is imported structurethrough the fields of which we refer to functions and constants.
You can write modules on Starlark, and for convenience, use the extension starlarkstructso that working with our custom module does not differ from working with built-in ones:
builtins := starlark.StringDict{
"newUser": starlark.NewBuiltin("newUser", newUser),
"struct": starlark.NewBuiltin("struct", starlarkstruct.Make),
}Let’s support loading files in the module loading function:
thread := &starlark.Thread{
Load: func(thread *starlark.Thread, module string) (starlark.StringDict, error) {
switch module {
case "math.star":
return starlark.StringDict{
"math": math.Module,
}, nil
case "time.star":
return starlark.StringDict{
"time": time.Module,
}, nil
default: // внешний модуль, загружаем из файла
script, err := os.ReadFile(module)
if err != nil {
return nil, err
}
entries, err := starlark.ExecFile(thread, module, script, builtins)
if err != nil {
return nil, err
}
return entries, nil
}
},
}Let’s define a module:
# myModule.star
def hello():
print("hello from module")
# создаем структуру с экспортируемыми литералами
myModule = struct(
hello = hello
)And use it in the main code:
load("myModule.star", "myModule")
myModule.hello() # выведет hello from moduleDue to the fact that we have defined a structure with the same name with literals in the module, the module is very easy to import and use.
Conclusion
In my case, embedding Starlark into the application gave users a tool for flexible configuration and adding custom logic to some controlled stages with events. I will be glad if this material is useful to you, and Starlark, perhaps, will take its rightful place in your code.
