Self-hosting macOS CI on Apple Silicon with the Cilicon app

In short: we have released a new app for macOS called Cilicon, which provides and runs ephemeral virtual machines for CI (Cilicon Installer is a standalone application). Using it, we were able to switch to our own Actions Runners and speed up our CI by a factor of 3, as well as give a second life to some of our damaged M1 MacBook Pro devices.

Our problems with CI on macOS

When we launched Trade Republicour mobile CI consisted of one 2013 Mac Pro running the agent Buildkite Agent. Its performance was satisfactory, but it quickly became clear that manual maintenance of more than one machine was not practical in the long run. In 2020, when the team grew and quarantines began due to COVID-19, we began to look for alternatives.

Although this meant performance degradation, we decided to switch to runners (a runner is a dedicated application designed to run tests, validate output, and provide tools for debugging and diagnosing tests and applications) hosted on GitHub. This allowed us to run more jobs in parallel and not worry about maintenance at all.

At the time, our iOS workflow only ran about 10 minutes (when using cached dependencies), but this quickly increased to 30 minutes as the team and codebase grew.

With a 10x price multiplier for macOS Launch Minutes, as well as the adoption of GitHub Actions in other repositories, we quickly began to exceed the 50,000 free minutes per month included in our enterprise plan. In addition, our teams were becoming increasingly frustrated with the performance of GitHub hosted runners, so we were looking for alternatives for a while.

Given our initial experience with manually maintaining the Buildkite Agent and the effort involved, self-hosting didn’t seem like a viable option.

However, one day I accidentally stumbled upon the Apple Virtualization Framework. Out of curiosity, I downloaded code example and launched it. Much to my surprise, the code turned out to be very simple and clear. The potential for creating and running virtual machines with code soon became apparent. Around the same time, I was informed that we had a number of MacBook Pro M1s in stock that were either broken or in too bad a condition to hand out to employees. Self-hosting our CI suddenly became more realistic and the idea of ​​Cilicon (CI + Silicon) was born.

Introducing Cilicon

The Cilicon concept boils down to a simple loop:

Duplicate Image (duplicate image)

Cilicon creates a clone of the package (folder) of your VM for every run. Thanks to the excellent clone functions in APFS this happens very quickly, even with large packets.

Provision Shared Folder (provide a shared folder)

Depending on the provider you choose, Cilicon places the files your guest OS needs in your package’s Resources folder.

GitHub Actions Provisioner Prepares an image with the artist’s download URL, registration token, name, and labels.

Process Provisioner runs the executable of your choice when preparing and removing a package.

You can also choose not to use a provider by setting the provider type to none. This might work fine with services like Buildkite that use enrollment tokens that don’t expire.

Start Virtual Machine (start a virtual machine)

Cilicon starts the virtual machine and automatically mounts the Resources package folder in the guest OS.

Listen for Shutdown (listen to shutdown)

Cilicon listens for a shutdown of the guest OS and removes the used image before starting again.

To create virtual machine packages, Cilicon comes with its own standalone application called “Cilicon Installer”. Once created, all that’s left to do is run the virtual machine in editor mode, install all the necessary dependencies, and add start.command from the resource share as the Login Item.

Our previous experience

After a two-week trial period in November, during which we ran both self-hosted and GitHub-hosted at the same time, we felt confident that we could switch. Since we’ve been using our fleet of eight M1 MacBook Pros exclusively and without a single glitch, we’ve been enjoying work 3x faster. Most of the MacBooks we used were considered unusable by employees because they had display, keyboard or cosmetic defects. While we originally planned to upgrade to the M1 Mac Mini after the test period, the MacBooks have done such a great job that we’ll continue to use them for now.

our initial setup consisting of 8 M1 Macbook Pro
our initial setup consisting of 8 M1 Macbook Pro
overview Github Actions Runner
overview Github Actions Runner

Return to the runners hosted on Github

Self-hosting always comes with additional risks. Even though our office server rooms have redundant power and network redundancy, we wanted to add the ability to easily fall back to GitHub hosted runners in case we couldn’t connect to ours.

To do this, we’ve added a new job to our workflow that checks if the PR contains a label named “Run on GitHub Hosted Runner” and selects the runs-on label for the next job accordingly. The reluctance to add a labeled trigger to our workflow meant that we needed to get the labels through the GitHub API rather than retrieving them from the provided context variables, as reruns provide an accurate snapshot of the data from the initial run. Since the runners hosted on Github run on x86, you can also include the runner’s arch in your cache keys using ${{ runner.arch }}.

jobs:
  runner_type_job:
    name: Runner Type Selection
    runs-on: ubuntu-latest
    env:
      GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    outputs:
      runner_type: ${{ steps.set_runner_type.outputs.runner_type }}
    steps:
      - id: set_runner_type
        run: |
          # Fetching fresh PR Labels using GH CLI, needed on Re-run Workflows.
          GH_PR_URL="https://github.com/${{ github.repository }}/pull/${{ github.event.number }}"
          GH_PR_LABELS=$((gh pr view $GH_PR_URL --json=labels --jq='.labels | map(.name) | @sh') | tr -d \')
          echo "Labels applied: $GH_PR_LABELS"
          RUN_ON_GH_HOSTED_RUNNER="Run on Github Hosted Runner"
          if [[ ! " ${GH_PR_LABELS[*]} " =~ " ${RUN_ON_GH_HOSTED_RUNNER} " ]]; then
            echo 'runner_type=["self-hosted", "macos-13", "ARM64", "xcode-14.1"]' >> $GITHUB_OUTPUT
          else
            echo 'runner_type="macos-12"' >> $GITHUB_OUTPUT
          fi
  test:
    name: Unit Tests
    needs: runner_type_job
    runs-on: ${{ fromJSON(needs.runner_type_job.outputs.runner_type) }}

Service

To keep maintenance efforts to a minimum, Cilicon offers a few tricks up its sleeve.

Since Cilicon does not support preparing images over the network, it is currently recommended to transfer the image via an SSD. So, to eliminate any interaction with the OS (especially since some of the devices we use have damaged displays), Cilicon can be configured to scan for a mounted volume with a specific name and automatically copy the VM.bundle. The start and end of the copying process are accompanied by system sounds, eliminating the need to open the lid or interact with the keyboard.

It also doesn’t hurt to restart devices from time to time, so Cilicon can be configured to restart the host machine after a given number of starts.

Conclusion

We hope that more companies and individuals will use Cilicon to reduce costs and speed up their CI. While self hosting hardware may not be suitable for many, Hetzner offers very affordable hosting. M1 Mac Mini for about 70 euros per month, and confirmedthat it works with Cilicon.

Contribution to the project is also welcome!

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *