Setting up an Anycast address within a budget test bench

Preface

Within IPv4 and IPv6 there is a concept Anycast addresses. To simplify, these are IP addresses that look like ordinary “gray” or “white” addresses, but which can work simultaneously on one server or on many. There is an opinion that this is difficult to configure, requires many additional layers of routing equipment, etc. But in this article I will try to describe setting up an Anycast address anywhere and at minimal cost. As a test bench, we will have a Mikrotik hAP lite router and a Proxmox cluster on RockPi mini-computers (analogous to Raspberry Pi) with virtual machines based on the Debian distribution.

Why exactly a Proxmox cluster? Yes, just like that 🙂

It would be possible to get by with virtual machines on a computer/laptop created using VirtualBox or one mini-computer with Proxmox, but I have this thing for home experiments, so I used it.

Hi all! My name is Artyom Druz, I work as a senior engineer at the Kontur company, where I set up and maintain digital telephony based on Asterisk and OpenSIPS. For digital telephony in the “classic form”, UDP transport is used, and accordingly Anycast addressing. To create fault-tolerant telephony, this is just what the doctor ordered. Some might argue that using UDP is a thing of the past and is now in vogue. WebSocket/WebRTC or even QUIC. But if you turn on WireShark and try to find the exchange of voice packets in your favorite instant messenger in the huge flow of your traffic, then with a high degree of probability you will find UDP traffic. It is for this reason that configuring fault tolerance of services using UDP transport using “old methods and technologies” does not lose relevance.

Designing a network

Before we start setting up, let's sketch out a diagram of what we want to see as a result. Within the existing stand there is a limitation in the form of MikroTik hAP lite: due to the chip on the architecture smips we are not allowed to use the protocol BGPwhich is recommended for things like this. Therefore, let's take the protocol OSPF for IP address announcements from virtual machines. For severe production with hundreds of thousands of requests per second, where every packet of data is important, this, of course, is not suitable. But such loads are not planned within the test bench 😉

For a network project, these inputs are sufficient, since the rest of the scheme will be implemented at the software level.

Setting up OSPF on MikroTik

To configure MikroTik to receive announcements, we need a combination of three elements:

1. OSPF Instance – here we set the name for our OSPF installation

2. OSPF Area – description of the OSPF zone that is associated with OSPF Instance

3. OSPF Interface Template – a template that will be used to authenticate announcements from virtual machines and create interfaces in connection with OSPF Area. In it we define the interfaces within which we need to work with announcements (in my case this is a common bridge bridge-lan). Timers are also set here, according to which announcements will be executed and processed, and the absence of announcements for Dead Interval will be interpreted as a trigger to remove the advertising server from the routing table. These intervals have a relationship with each other, so it is impossible to assign 1 second everywhere to speed up convergence when the announcer fails.

Below is a screenshot of my stand settings.

Setting up address advertising on virtual machines

First let's create dummy interface on each of the virtual machines and run it.

To file /etc/network/interfaces add this construction:

auto dummy0
iface dummy0 inet static
    address 172.18.0.64/32
    pre-up ip link add dev dummy0 type dymmy
    pre-down ip link delete dev dummy0 type dummy

Important. The IP address in the interface configuration must be outside existing subnetsotherwise the circuit will not work correctly.

Next, we launch the interface with the command ifup dummy0

For those who have a Linux distribution using NetworkManager There is a command that essentially does the same thing, but in one action:

nmcli connection add type dummy ifname dummy0 ipv4.method manual ipv4.addresses 172.18.0.64/32

Next we install bird

apt install bird2

And we make changes to /etc/bird/bird.conf

log syslog all;
protocol device {
}
protocol kernel {
        ipv4 {
              import none;
              export all;
        };
}
protocol static {
        ipv4;
}
protocol ospf myAnycast { # myAnycast - любое произвольное имя для инстанса OSPF на уровне виртуальной машины
        ipv4 {
                import all;
                export all;
        };
        area 0 {
                stubnet 172.18.0.64/32 { # 172.18.0.64 - Anycast-адрес не входящий в существующие подсети в локальной сети
                };
                interface "enp0s11" { # enp0s11 - имя сетевого интерфейса с которого должны производиться анонсы Anycast-адреса
                        cost 1;
                        type broadcast;
                        hello 3; # Hello Interval из шаблона в MikroTik
                        retransmit 2; # Retransmit Interval из шаблона в MikroTik
                        wait 2;
                        dead 4; # Dead Interval из шаблона в MikroTik
                        authentication simple; # Authentication из шаблона в MikroTik
                        password "myOSPFpassword"; # Auth. Key из шаблона в MikroTik
                };
                interface "dummy0" { # Имя dummy интерфейса, на котором настроен Anycast-адрес
                        stub;
                };
        };
}

Now we restart bird with the command systemctl restart birdand after a few seconds in the MikroTik interface (provided that everything is configured correctly) we should see something like this:

Checking the operation of the Anycast address…

…from the router side

Here we'll get by with a simple ping from the console:

There is contact! We could stop there, but I’d like a more lively example 🙂

…from other virtual machines

In this test, I decided to test the operation of the telephony service in a scheme where 4 Asterisks from different addresses connect to the Anycast address used by OpenSIPS (one on each of the virtual machines where the Anycast address is configured).

To check, the reader can use, instead of SIP servers and software PBXs, some voice service operating via UDP transport. In my screenshot of tmux running sngrep on two virtual machines (I turned off the third during the test to get a readable screenshot). It is clear that:

• Asterisk with address 172.18.0.24 was distributed by MiroTik to the first OpenSIPS

• Asterisks with addresses 172.18.0.22, 172.18.0.23 and 172.18.0.25 were distributed to the second OpenSIPS

How to create a different traffic distribution strategy (for example, so that each packet is distributed to Anycast address advertisers using the round-robin strategy) on MikroTik is a separate topic, and we will not touch on it in this article.

Thus, the goal can be considered achieved and you can start experimenting with your own Anycast address 😉

Conclusion

In the article I did not mention anything that would tie the reader to Proxmox in general or to its specific implementation on mini-computers with an ARM processor in particular. The article is instructions for setting up an Anycast address on your computer or on a local network in order to try to implement a service that uses features of this type of address. In my example, Proxmox was needed only as a general environment for creating virtual machines.

As a result, we now have our own Anycast address advertised via OSPF, with which we can experiment and test hypotheses. Of course, this scheme has shortcomings (for example, routing outgoing packets through the interfaces of virtual machines performing the announcement). But in order to begin to understand this topic, this springboard, in my opinion, is enough.

Important reservations, which have already been mentioned above in one form or another:

• The solution described in the article is not for heavy-duty production. It can be used for small businesses or self-development tasks, but certainly not for systems with thousands of R.P.S.. At a minimum, for fast routing convergence when one of the nodes fails, you need to use BGP instead of OSPF.

• Anycast addressing is about UDP traffic, in which there is no concept of guaranteed connection and data delivery (joke). If you have TCP traffic (websites, REST API, connections to databases, etc.), then for fault tolerance and/or high availability of services you need balancers that process such traffic or specialize for a specific service (HAProxy, PgBouncer, Galera Cluster , Redis Sentinel and others).

That's all. Thank you for reading the article to the end.