Setting up dynamic Gitlab Runners in Yandex Cloud

Hi all! I am Kirill, DevOps company sports.ru. Not so long ago, we started the process of moving to Yandex Cloud, I want to tell you how it was.

In parallel, we began to look for where else we can apply the strengths of the public cloud. I immediately remembered a long-standing problem with periodic bursts of developer activity, which led to the exhaustion of gitlab runners and, consequently, to a long wait in the queue. Previously, to solve this problem, it was necessary to add a new server for runners. But since it was episodic, horizontal scaling was impractical. But Yandex Cloud gives us the opportunity to quickly get resources for a short period. In Gitlab, similar functionality is implemented through the Docker Machine Executor.

It is worth noting that Docker Machine is now in deprecated status, but the gitlab team continues to support their fork, so this solution can be considered quite reliable. However, in the future, autoscale will be implemented on gitlab’s own technology, but there is no timeline for the transition at the moment.

Preparatory stage

Before installing runner, there will be a few preparatory steps.

First of all, we need the Docker Machine utility. Here it is enough to download the latest version of the binary file from the repository gitlab and mark it along the way /usr/bin .

To work with Yandex Cloud, we also need an official driver which also needs to be labeled /usr/bin. It is worth noting here that the name of the driver file is extremely important, as well as the rights to it, it must be executable.

The last preparatory step will be the generation of an “Authorized Key” in Yandex Cloud. To begin with, we create a service account with “compute.admin” rights, if your runner suddenly needs an external IP address, add “vpc.admin” rights to the account. Next, we generate a key, this can be done through the UI or through the yc utility. The last option is more convenient, the command itself looks like this yc iam key create --service-account-name <service-account-name> --output key.json --folder-id <folder-id>. As a result, we get json, which should be kept in encrypted form, for example, in ansible vault.

Installing gitlab runner

So how do we get an auto-scaling runner? First you need to raise the virtual machine on which GitlabRunner will be installed. For these purposes, we use a ready-made ansible-role debops.gitlab_runnerwhich served faithfully for many years, but in this case failed.

The maintainers abandoned the development of the role (the last commit was in 2018) and, as expected, unsupported parameters appeared, in particular, the “Off Peak time mode” was deprecated and a separate section appeared in the runner settings: “runners.machine.autoscaling”. That is why we had to fork the role and mark it on our side along the way. github.

There have been no major changes to this role, so you can use the official documentation to set it up. We just added the autoscaling section setting. It looks like this:

gitlab_runner__machine_autoscaling:
  - period: "* * 7-19 * * mon-fri *"
    idel_count: 0
    idel_time: 600
    timezone: "UTC"

Particular attention should be paid to setting up the driver for Yandex Cloud.

gitlab_runner__machine_idle_count: 0
gitlab_runner__machine_idle_time: 900
gitlab_runner__machine_name: "auto-scale-%s"
gitlab_runner__machine_driver: "yandex"
gitlab_runner__machine_options: ["yandex-sa-key-file=/etc/gitlab-runner/key.json", "yandex-folder-id={{ yc_qa__folder_id }}", "yandex-cloud-id={{ yc_cloud_id }}", "yandex-subnet-id={{ yc_qa__subnet_id }}", "yandex-use-internal-ip=true", "yandex-image-family=ubuntu-2004-lts", "yandex-cores=4", "yandex-disk-type=network-ssd", "yandex-memory=8", "yandex-preemptible=true"]

gitlab_runner__machine_idle_count – The number of runners that should always be available; 0 means the runner will only run on demand.

gitlab_runner__machine_idle_time – Time (in seconds) before the runner is removed; counted from the last completed job.

gitlab_runner__machine_name – the name of the automatically defined VM.

gitlab_runner__machine_driver – driver name for Docker Machine.

gitlab_runner__machine_options – list of passed parameters.

The driver has a lot of parameters, their list and values can be found in the official repository. Let’s dwell on just a few of them. The key that we generated at the preliminary stage must be specified in the parameter yandex-sa-key-file. If the gitlab instance is on the same network as the runner, then you can only use the internal IP address, for this we specify yandex-use-internal-ip=true. Otherwise, you must specify yandex-nat=true, then the VM will get a white IP address. And one more parameter worth specifying, yandex-preemptible=truethis will allow the creation of “interruptible machines”, which are much cheaper.

After setting up and rolling the role, we will get a runner. And its configuration file will look something like this:

concurrent = 50

[[runners]]
  name = "gitlab-runner-test-autoscale"
  url = "https://gitlab-test.test.ru/"
  token = "TOKEN"
  environment = [ "DOCKER_BUILDKIT=1", "DOCKER_DRIVER=overlay2" ]
  limit = 10
  executor = "docker+machine"
  [runners.docker]
    image = "docker:dind"
    privileged = true
    disable_cache = false
    cache_dir = "/home/gitlab-runner/cache"
    cap_drop = [ "NET_ADMIN", "SYS_ADMIN", "DAC_OVERRIDE" ]
    volumes = [ "/var/run/docker.sock:/var/run/docker.sock", "/home/gitlab-runner/cache" ]
  [runners.machine]
    IdleCount = 0
    IdleTime = 600
    MaxBuilds = 100
    MachineName = "auto-scale-%s"
    MachineDriver = "yandex"
    MachineOptions = [
      "yandex-sa-key-file=/etc/gitlab-runner/key.json",
      "yandex-folder-id=<ID>",
      "yandex-cloud-id=<ID>",
      "yandex-subnet-id=<ID>",
      "yandex-use-internal-ip=true",
      "yandex-image-family=ubuntu-2004-lts",
      "yandex-cores=4",
      "yandex-disk-type=network-ssd",
      "yandex-memory=8",
      "yandex-preemptible=true"
    ]
            [[runners.machine.autoscaling]]
          Periods = ['* * 7-19 * * mon-fri *']
          IdleCount = 2
          IdleTime  = 1800
          Timezone  = "UTC"
            [[runners.machine.autoscaling]]
          Periods = ['* * * * * sat,sun *']
          IdleCount = 0
          IdleTime  = 600
          Timezone  = "UTC"

Based on the settings, on weekdays from 7 to 19 Gitlab will keep two VMs on a “hot start”. But at night and on weekends, for the sake of economy, all virtual machines will be deleted. In addition, their waiting time will be reduced to 600 seconds. In case of a large influx of tasks, the runner will be able to create up to 10 virtual machines (defined by the limit = 10 parameter), which will be deleted if these runners no longer receive tasks within 30 minutes. It should be noted that it takes 2 to 3 minutes to create one machine, they will be added to the total execution time of the pipeline.

Conclusion

As you can see, the solution does not require any serious effort. In fact, all the necessary components are already there, they only need to be assembled together.

At the moment, we are just starting to use this approach and do not have empirical data in order to assess how much dynamic runners will improve the speed of processing the queue during peak periods, in contrast to the current implementation. In the future, we will definitely share our results and try to evaluate the effectiveness of this approach.

If you have had experience using dynamic runners in public clouds, then we are happy to discuss it in the comments.