Minecraft server hosting - and why such a complex architecture?

Hello! This is the second part of the articles about hosting Minecraft servers, which we are building. In the first part I talked about the physical part of the infrastructure – from the laptop to the server rack. In this one we will dive into its logical part without a long historical background. Through thorns to k8s, enjoy reading.

In our historical story we will not follow the logical parts, and will simply construct something like this agend:

I wonder why such architecture is needed for a regular minecraft hosting? (To be honest, I'm also interested) Then let's dive in together!

Part 0: a bit of history and disclaimer.

Everything described below is nothing more than our amateur vision and attempts to be architects, team leaders, CEOs of the company, and so on. Let me remind you from In the first part of the articles, we are actually a team of two people: I am responsible for the infrastructure, and the second “creator” is responsible for everything related to development.

We have built and continue to build our entire business on one important feature, which probably distinguishes us from others: we learn something new or try to understand something more than just on the surface (let's consider a simple example with infrastructure: at work “N” I learned what k8s is. It was already installed and worked well. At the same time, I did not understand how to cook it, install it and work with it a little more than write manifestos. And on a live pet project I can roll it out and test it, even despite the fact that it is not such a big highload).

In general, don't judge too harshly 🙂

Part 1: Zoo of servers and clouds.

Before diving further into the logical part, I will first dwell on a small physical part of the story – about the geography of infrastructure.

The important distinction should be as follows: infrastructural servers and custom. Those that directly host Minecraft docker containers we call “nodes” – just for convenience. The main part of server infrastructures moved from the “balcony” (if you don't know this concept, read the first part of the article about physical servers!) in collocation in one Moscow data center. The second part – one physical server – is located in Hetzner.

The situation with nodes is more interesting: buying servers all over the world and putting them in collocation is at least expensive, at most it does not make sense at this stage, so we, like most people, rent them from other hostings. Trying to build a distributed infrastructure, at this stage our geographic presence is in four countries:

The first two countries – Germany and Finland, I think you've already guessed, were chosen for a completely clear reason: that's where Hetzner provides dedicated servers for everyone. France is used exclusively for part of the antiddos infrastructure and there are no user nodes as such.

With the Russian part, everything is much more interesting. Our points of presence are in Moscow, St. Petersburg and a recent new location – Yekaterinburg.

The first two cities were chosen logically again – the European part of Russia has the most players, and the smallest ping from the client will be either to Moscow or to St. Petersburg. But Yekaterinburg appeared due to numerous requests from our clients who are closer to the Asian part of Russia or in the CIS countries (eg Kazakhstan). It does not sound so scary, but for Moscow alone we use several hosters, and here is why:

In total, we have about 30 physical servers and about the same number of virtual machines – they are located on infrastructure hyperhosts. We use a standard approach for this – Proxmox, and in Moscow we attached ceph to it to enable live migration (and, of course, to understand how it works).

In general, we've sorted out the infrastructure, and now let's dive further.

Part 2: automation.

With such a zoo, there is no desire to roll everything manually or even with bash scripts, and, of course, you need to improve your knowledge in the direction of ansible – that's why we use it. In total, we have two global repositories: under infrastructure and under nodes. Collocation is also divided into parts: fronts, monitoring and everything else. It turns out to be a small, but still branched structure. Let's take a closer look at what the process of deploying infrastructure on nodes looks like.

So, we rented a server, what next?

We commit IP addresses to the repository with terraform, which are then rolled out to Cloudflare DNS via a simple gitlab-ci, so there is no need to go to the admin panel yourself, edit something, overwrite or break something along the way
We write in hosts.yml in ansible a new server, add your ssh key to it and run the playbook. In half an hour we will roll everything out and start accepting users to the new node

roles:
  - {role: 'base',  tags: ['base', 'base']}
  - {role: 'docker',  tags: ['docker']}
  - {role: 'node-exporter',  tags: ['node-exporter']}
  - {role: 'promtail',  tags: ['promtail']}
  - {role: 'letsencrypt',  tags: ['le']}
  - {role: 'certbot', tags: ['le-ssl']}
  - {role: 'wings', tags: ['wings']}
  - {role: 'mysql-master-slave',
    mysql_db: [
      { name:  },
    ],
    mysql_users: [
      { name: , pass: , priv: «» },
    ], tags: [ 'mysql-master-slave' ]
  }
  - {role: 'cron-minecraft', tags: ['cron-minecraft']}
  - {role: 'nginx-placeholder', tags: ['nginx-placeholder']}
  - {role: 'superhub-node-monitoring', tags: ['superhub-node-monitoring']}
  - {role: 'superhub-25565',  tags: ['25565']}
  - {role: 'logrotate', tags: ['logrotate']}

The structure of roles also looks quite simple:

Base rolls the basic settings of the node itself: ssh-key authorization, sysctl, users, etc.
Dockeraccordingly, installs the docker itself – it is needed for the operation of Minecraft servers, since in the architecture of custom ones, all servers are launched in separate containers
Node-exporter basic monitoring rolls (unlike infra – in the Moscow data center we also use consul for SD, because virtual machines appear much more often than physical servers)
Promtail we need it to collect logs from wings demons (more on that later) and backup logs – we want to be sure that everything worked correctly
Certbot rolls certificates that are also needed for the correct operation of Wings
Wings – is an open-source daemon pterodactyl – a panel that we use to manage user servers. It is installed on each node and is centrally managed from the panel (I will tell you about the panel in the section about the cube)
Cron-minecraft, nginx-placeholder, superhub-node-monitoring and the rest of the roles below are all our custom roles that run our internal demons and various scripts. Here we include custom monitoring for our backend, automatic backups of data and databases

After rolling this playbook, we consider the node ready to receive traffic and mark it in our database as “open” – From now on, users can rent resources on it.

Infrastructure playbooks differ in their role, but have a similar structure. We do not use any rocket science, do not make nested roles and do not write our own modules (but we should), so there is no point in copying the example with infrastructure. I think it is obvious to everyone that we do not roll prom/nginx and its various configurations with both nodes and infrastructure – Ansible is responsible for all this, and in some cases we also use gitlab-ci – to further minimize manual actions for launch.

So what we include in the word “automation” on hosting:

Terraform in a minimal form for Cloudflare
Ansible for everything
Kickstart+PXE, Cloudinit for deploying virtual machines in the data center

Part 3: Favorite – monitoring.

Instead of a thousand words, I suggest you look at the picture and take it apart piece by piece:

We have 3 main monitorings:

Let's look at the diagram and figure out what is needed for what.

Elasticsearch

This is where we send all netflow data. We decided to use a ready-made open source solution from elastiflow with a basic license that allows you to get up to 4000 flow/sec, this is quite enough for us. The need for this system is basically due to the peculiarity of the Minecraft community (and, probably, game hosting in general) – they periodically experience small or even large DDOS attacks, which can easily disable a hyper/node for some time. Unfortunately, we are not able to detect an attack proactively, but at the same time we react to it retrospectively and quickly during any problems. Netflow collects data both from src/dst by IP addresses, and (which is especially important for us) to which port the traffic is being uploaded. This detector allows us to contact the client directly and inform him about what happened and respond to the incident to him – connect external antiddos protection.

On user nodes there is a softflowd daemon, which sends the entire netflow to flowcoll via UDP, which in turn sends data to ELK. At the “balcony” level, mikrotik is responsible for sending such data – since this is the entry point for all traffic. This is how it looks at the Kibana level:

Prometheus and so on

Historically, we wanted a more fault-tolerant infrastructure: in case of a balcony failure, we would still have access to metrics. That's why prometheus was initially installed on an infrastructure machine in hetzner. Victoria metrics for long-term storage of metrics is also located there, remote_write. This prom collects data from all nodes and other servers not within the balcony perimeter. We monitor both basic metrics like node_exporter, and some others: cadvisor and blackbox (as an alternative uptime-kuma for internal monitoring). This scheme, I am sure, is quite basic for most of all current companies or any other installations (let's not forget about alertmanager and all that).

On the other side there is prometheus on the balcony, which sends long-term metrics to the remote_write cluster as well. The only difference of the internal prom is consul SD – virtual machines on hypervisors appear and disappear more often than dedicated servers are purchased, and in order not to accidentally forget anything, we tried to reduce the manual factor in the form of an unspilled config. The internal prom also collects metrics for nginx vts for alerting on 5xx errors and so on. We have the same scheme in k8s: prometheus in cube sends metrics to Victoria, and then we draw them in grafana on the necessary dashboards.

By the way, about Grafana. Here we also don't have any rocket science and we use the most standard dashboards for everything possible. The only difference is the cool dashboard, which I like :). I tried to make something like “business metrics” of the application, in which we build data on:

Number of payments
The number of balance replenishments and the balance replenishment amount
Number of registered users
Metrics for active running servers
Backup fails, 504 panels (this is important for us) and several other indicators.

Uptime kuma

This is an alternative to paid status page services that we wanted to use to provide clients with high-level information about the “liveness” of a particular system. If you are interested, you can check it out at the link – status.superhub.host.

Unfortunately, kuma v1 has several global architectural problems, which the developers promise to fix in v2. The main problem we encountered is storing all data in sqlite, which is insanely slow for us (despite a small number of probes).

How can I fix it?

By the way, if you use kuma in your project and want speed up her work – reduce data storage to 1 day and clean up the sqlite database – this will help significantly and the increase in page loading will be significant.

We are looking at alternatives to Kuma and possibly looking at our own status page based on Blackbox, but we don't have the resources to develop our own right now.

Part 4: almost finished reading – backups.

We are almost coming to the end of our story, there are only a few parts left, and now I wanted to stop at one important thing that some people forget.

Why do they forget?

During the 4 years of our hosting operation, we have encountered several security incidents. Some were hacked due to vulnerability, some – because they did not have 2FA. However, no matter who we talked to after the hacks – no one had even the slightest hint of backups. (It is true that it is worth considering that if they hack you not just in the admin panel and delete the database, but gain access to the server or, even worse, to the server with backups – it still will not help you, but let's not talk about sad things)

Count how many times this article mentions the division into user nodes and infrastructure servers? And add +1 to that. First, we'll talk about infrastructure backups, and then we'll mention the latter.

We have 2 s3 providers that we use: minio on a server in hezner, which has hot-storage with NVME disks and cold-storage with HDD. For a number of buckets, hot-backup rotation to cold is configured, since cold storage Tb >> hot storage Tb.

Database backups

In our architecture, we use Xtradb Cluster with PMM on an external server for monitoring. Backups are configured via xtrabackup, which creates them via cron every 3 hours. This way, we can always roll back to the most current version of the database, losing only 3 hours of user data. We also use Postgres for Jira and a number of other service infrastructures, the backups of which fly to s3.

Backups on nodes

One part of the backups flies to minio s3 – these are user backups, and the second – to the external s3 of the hoster. Let's understand why this is so.

The minecraft server control panel has the ability to automatically or manually create a backup. The same daemon is responsible for this wingswhich simply archives the server and sends it to our hot-s3 storage. When a user deletes a server, backups are also deleted, and we reduce the space used. The most important point in choosing your own system versus ready-made and fault-tolerant ones is the price. We experience relatively large network loads and the number of GET/POST/PUT requests when loading user backups. (Unfortunately, some clients do not know what cron is and set up backups * 0 * * * Yes, this happens too.). If we host with s3 providers, then according to preliminary estimates we will pay for storing user backups approximately 1/3 of the cost of all server infrastructures per month. It seems that this is not practical.

On the other hand, we still use external s3 storage, but for our personal automated backups, namely: in case of possible problems on the hosting side – server hacking, data deletion via the panel or rm -rf, data deletion by the user (or user hacking) we add a small layer in the form of security and backup of all data once a day at the least loaded time – from 00:00 to 06:00 (the time is randomly selected by ansible in the cron). We send these backups to external storage for two reasons: the volume and amount of data. In this case, it is cheaper and safer for us to store this data ourselves and pay for it and also control the possible consumption, which can be reduced by optimizing various backup options.

Part 5: k8s.

This is probably the longest part of the story, but I'll answer the question right away: Yes, we wanted a cube. Yes, we're just learning something new. Now let's dive into the diagram.

I think the scheme before k8s is clear to everyone, logical and does not need discussion, so we will immediately go to the level after ingress. The entry point of any user can be 2 domains: superhub.host And panel.superhub.host. The second one is solely responsible for managing his server, while all the magic of buying, paying, changing server characteristics is on our main site

You've come to main front – this is nextjs, which cooked our page for you. On it you also saw the counter of running servers – for this you contacted /v2and the API in turn received these metrics from monitoring. We also take the rest of the data on server costs, reviews, and so on from /v2. If the client wants to go to /wikithen we will render this page using a proxy in GitHub – each user can send us a PR for editing or creating a new article, and we beautifully wrap it in the current design of the site.

When authorizing, we use our auth-server and a backend that stores authorization data in redis for each user, and checks logins in a separate database.

When a client purchases a server, we record this data in a separate “collector base» (no, we do not pass this data to collectors (!), but it sounds rather neo – we store a lot of information there: the day and time of the write-off, whether it was successful or not, the amount, how the user's balance changed and much more. Collector-workers are engaged in processing and writing off, and the master gives the workers tasks for processing payments.

If we talk about the panel, we use an open source panel. pterodactyl with minor changes in security – we wrote our own layer for seamless user authorization, so all requests first go to ours authand then – to the panel. We also removed the panel 2FA and screwed our own (previously, the client could have 2FA in the panel, and in the personal account authorized only by password). Pterodactyl also has schedulerwhich is responsible for creating backups, sending data to wings and communicating with nodes.

One feature of our panel is the sawing of the monolith and entry into the cube. If you look at the documentation of pterodactyl, the panel is installed as a normal one git clone; apt-get install php-fpm and that's it. At our facilities we decided to use the cube for it too – together with the fashionable HPA, so that it would be beautiful (Recently our vault got 'seal'ed, so the panel lived on one pod. We didn't notice it right away, but it was painful.)

That's probably all. We didn't include all the microservices that we wrote, because the diagram would have turned into a big mess, but we try to fit everything under the cube that we can (including all the infrastructure services – Zhira, Grafana and everything-everything-everything).