Node-Local Virtual IP in OpenStack

Our Cloud Tech Research Team at Huawei’s Advanced Software Technology Lab often needs to optimize the performance of certain cloud infrastructure components. When solving one of these tasks of updating 5G Core applications on very large clusters, we proposed and developed the Local IP technology in OpenStack Neutron.

Local IP is a virtual IP that runs within a physical server and is used to forward requests to a local port/virtual machine.

An example of using Local IP for local caching.

Local IP technology will be useful:

(generally) when implementing side-car proxies, load balancers or service meshes;
(in large-scale and/or high-performance) when building distributed data caching and CDN services. Also, the technology will simplify the use of distributed caches in clouds built on the basis of OpenStack. For example, by running virtual machines with Docker Registry (in passthrough mode) on each physical server, you can achieve a significant reduction in the download time of images and the start time of containers run in virtual machines;
(in large-scale) when implementing the load-balance-at-source pattern.

There are two modes of operation of Local IP:

translate (default) – Local IP behavior will be similar to DNAT;
passthrough – packets will be forwarded without changing the IP headers. In this case, the guest OS of the virtual machines is expected to be responsible for configuring the Local IP address on the interface. This mode will be useful for implementing Fault-tolerance or similar scenarios.

How it works?

Neutron is a network management component in OpenStack (we will consider a standard installation with an ML2 plugin and OpenvSwitch). For those readers who are not familiar with OpenStack Networking, I recommend paying attention to this article, which describes all aspects in sufficient detail.

Neutron, using agents on each physical server, configures OpenvSwitch (OVS), which in turn creates several breeches:

the “br-int” bridge (the so-called “integration bridge”) is used to work with local ports/virtual machines;
the “br-ex” (or “provider bridge”) bridge is used to interact with external networks;
the “br-tun” bridge (or “tunnel bridge”) is designed for communication between hosts running OpenStack.

Each bridge in OVS is described using several tables that contain sets of rules for matching packets to certain criteria and a list of actions to be taken on packets that meet these criteria.

An example of circulating packages inside OVS with Local IP enabled (green curve) and without (gray dotted curve).

All the logic of Local IP is implemented inside “br-int”, so address translation occurs locally, without going beyond the boundaries of the physical server. This allows you to use the same address for Local IP on different physical servers.

Local IP implementation inside “br-int”.

The figure, in a simplified form, shows the OpenFlow pipeline for “br-int” with the rules and a set of actions for the operation of Local IP. Rules are arranged in order of execution priority from highest to lowest. You can find out more about which tables “br-int” contains and what rules these tables contain using the following command “ovs-ofctl dump-flows br-int” by running it on a physical server running OpenStack with OVS. For those readers who want to understand how OpenvSwitch is configured, I recommend this article for review.

The logic of Local IP, at first glance, is very simple: using DNAT, you need to ensure translation (substitution) of dst MAC and IP addresses from Local IP to Fixed IP. This is described by the Static Local IP Translation rules in Table 31 “Local IP”. NAT is implemented in two ways and can be configured using the static_nat option:

(static_nat = False, default value) via contrack using ct nat action;
(static_nat = True) using static NAT rules added with LEARN action in reverse flow using the following information from the packet header: origin/destination MAC address, origin/destination IP address, protocol and origin/destination ports. Supports VLAN isolation, the ability to use Local IP along with OpenvSwitch Firewall and allows SmartNIC for OpenvSwitch offloading and DPDK functionality. I note right away that in this case, Local IP uses more flows in the tables, which can affect performance.

In addition, the following boundary scenarios must be taken into account:

A local host with a Local IP must be able to access an external host with the same IP address;
The local host can issue an ARP request to find out the MAC address of the host with Local IP;
On one physical server for different networks, the same Local IP can be set to different local hosts;
The external host may send a “Gratuitous ARP” or “ARP Announcement” packet with the same IP address as is already used for the Local IP.

In order for a local host with Local IP to access an external host with the same IP address, it is necessary for such packets to ignore all the “Static Local IP Translation” translation rules and not create an entry in DNAT. This is implemented using the “Self NAT Avoiding” rule in Table 31 “Local IP”.

Due to the fact that the local ports “br-int” are not tagged and do not contain information about the network, then using the rule described in Table 30 “Local Egress”, the VLAN identifier is set in reg6 of each packet and is then used to identify the network .

In order for the local host to be able to determine the MAC address of the host with Local IP, it is necessary to implement a local ARP Responder that will intercept all ARP requests from local ports and generate responses with the MAC address of the local host to such requests. This is implemented by the “ARP Responder” rule in Table 31 “Local IP”.

In order to block gARP announcements sent by external hosts, you need to implement the so-called gARP Blocker, which will block only ARP packets containing Local IP in order to prevent updates to ARP addresses in local host caches.

And of course, you need to take into account that Neutron in “br-int” creates rules to prevent ARP / MAC Spoofing (tables 24 and 25). Moreover, this functionality is optional and can be disabled during the installation of OpenStack. In order for Local IP to be compatible with it, a redirect to Local Egress has been added to table 25 as well.

How to use?

Let’s look at an example where we will redirect traffic using Local IP to a local proxy. Imagine that there are two virtual machines on a physical server: host “A” (MAC: aa:aa:aa:aa:aa:aa, IP: 10.0.0.101) hosting the client and host “B” (MAC: bb :bb:bb:bb:bb:bb, IP: 10.0.0.102) that hosts the proxy, and there is also an external host “E” (MAC: ee:ee:ee:ee:ee:ee, IP: 10.0 .0.10) which hosts the service that the client will access. Without a Local IP, the client on host ‘A’ is accessing the service on host ‘E’.

An example of traffic redirection to a local proxy with Local IP enabled (solid line) and without (dashed line).

By assigning a Local IP with address 10.0.0.10 to 10.0.0.102 (host “B”) we will direct our client’s traffic through the proxy server without changing our client’s configuration. This can be done by creating a Local IP object and association by issuing the following commands:

$ openstack local ip create --network public --name local_proxy --local-ip-address 10.0.0.10
+------------------+--------------------------------------+
| Field            | Value                                |
+------------------+--------------------------------------+
| created_at       | 2022-02-07T11:39:55Z                 |
| description      |                                      |
| id               | e1918998-84c1-40d3-abd3-950e5bcb4cad |
| ip_mode          | translate                            |
| local_ip_address | 10.0.0.10                            |
| local_port_id    | 49207b7d-52d5-4b18-b0bc-3c38bb2d08bb |
| name             | local_proxy                          |
| network_id       | 8bf58077-2a80-4f23-84b4-80ea1f40835e |
| project_id       | 183b5dc87b774aac937f431f4e161e4e     |
| revision_number  | 0                                    |
| updated_at       | 2022-02-07T11:39:55Z                 |
+------------------+--------------------------------------+
  
$ openstack local ip association create e1918998-84c1-40d3-abd3-950e5bcb4cad f8c935ce-24fc-4d55-b537-7f7bec6c2c79
+------------------+--------------------------------------+
| Field            | Value                                |
+------------------+--------------------------------------+
| fixed_ip         | 10.0.0.102                           |
| fixed_port_id    | f8c935ce-24fc-4d55-b537-7f7bec6c2c79 |
| host             | uscpepper00644                       |
| local_ip_address | 10.0.0.10                            |
+------------------+--------------------------------------+

Thanks to NAT, for every outgoing packet from host “A” to MAC bb:bb:bb:bb:bb:bb and IP address: 10.0.0.10, the destination address will be translated to 10.0.0.102 and the packets will be forwarded to the host’s local port” B”. In packets sent in response from host “B”, also thanks to NAT, the departure address from 10.0.0.102 to 10.0.0.10 will be translated and thus the packets will be delivered to host “A”.

If there is no MAC address for 10.0.0.10 in the local ARP cache of host “A”, then host “A” will send a broadcast ARP request. In response to this request, host “A” will receive the MAC address: “bb:bb:bb:bb:bb:bb”, which corresponds to host “B”. In the event that everything worked for us as usual, the response would contain the MAC ee:ee:ee:ee:ee:ee of the host “E”.

This example turned out to be completely synthetic, but if we imagine that we have several virtual machines and several hypervisors, and the proxy can cache requests, then we will get a decrease in the load on the network and host “E” at times.

Degree of readiness of Local IP technology

This technology will be available starting with the release of Yoga, which is scheduled for March 30, 2022. Currently implemented:

Neutron API – done;
OpenStack CLI – done;
VLAN support (ML2 driver) – done;
Support for VxLAN and GRE (Native driver) – done;
Support in DevStack – done;
Support for “passthrough” mode – not planned in the release of Yoga (can be implemented through an allowed address pair);
IPv6 support – not planned in the near future;
Support in HeatStack – not planned in the near future.