On the knee: VPN aggregation, or Reliable communication on unreliable channels

Imagine a problem: you need to provide stable Internet and cover a building with an area of ​​300 m with seamless Wi-Fi2 with a possible design load of up to 100 people. At first glance, it looks like a fault. But just add a couple of details, and the task becomes more complicated:

  • the building is located in a forested area where there are no optics, so our option is mobile communications;

  • you need to ensure regular video broadcasts, that is, to achieve a stable Internet with a single GSM provider;

  • the budget is limited.

Total: losses and dumps from the base station sneak up at the most inopportune time.

I have encountered such problems at call centers without dedicated communication channels, mobile reporter complexes, critical remote systems. Difficulties can arise not only in the case of VoIP and video streaming, but also with any request for a guaranteed delivery channel for loss-sensitive traffic. Unfortunately, it is not always possible to supply optics and purchase expensive equipment.

In the article I will show how in one project I solved these problems “cheap and cheerful” – this option is suitable for a small business. In the end, I will offer solutions for a less modest budget – for large customers.

Solution outline at a glance

So, when I first ran into the dumps problem, I started with frequency aggregation and found that it didn’t help. Changing the category of the LTE modem from Cat4 to Cat6 or – even better – Cat12 gave an advantage in speed, but not in losses and dumps. I came to the conclusion that we need a second LTE provider. At the same time, when switching, not a single frame should be lost and the broadcast should not fall off.

This bundle came to the rescue: aggregation, aka bonding, and a TCP-OpenVPN tunnel on top of it.

  • created an “aggregation server” in the cloud – a virtual machine with CLOUD HOSTED ROUTER (CHR) based on Router OS;

  • on it I raised an L2TP server with IPsec encryption enabled;

  • created two EoIP tunnels over L2TP over IPsec;

  • EoIP tunnels are aggregated by a bonding interface;

  • the cherry on top is a TCP-shny OpenVPN tunnel.

Final scheme:

Instead of a virtual machine in a data center, any piece of hardware with sufficient performance can act as R1. For example, the same MikroTik CCR series, a computer located anywhere. The main thing is to take care of performance and stable communication channels, use active redundancy schemes (VRRP to help).

OpenVPN UDP support is implemented only in the 7th version of RouterOS, therefore, in this configuration, the TCP protocol is used without any alternative.

Now the circuit works stably, but there is no limit to perfection. For reliability, you can add more LTE providers or wired communication channels when such an opportunity arises.

Now I will tell you more about the construction of the circuit. Let’s start with R1 (cloud router) and then R2 (branch).

Router R1

  1. First, we take the second white IP in the data center. My CHR was behind Edge in the VMware cloud, so be sure to forward ports to Edge UDP 1701, 500 and 4500 NAT-T – IPSec Network Address Translator Traversal. We also make an allow rule in the Edge firewall.

  2. Add to the firewall filter table a rule allowing access to the router from the outside for UDP ports 1701, 500 and 4500. If you have white IP directly on the router without forwarding through the Edge, the NAT Traversal checkbox SHOULD BE UNCONTINUED!

    Checking the default IPsec profile:

    /ip ipsec profile 
    set [ find default=yes ] dh-group=modp1024 enc-algorithm=3de
  3. Create a profile for L2TP tunnels:

    /ppp profile 
    add change-tcp-mss=no name=profile01 use-compression=no use-encryption=no use-mpls=no use

    and set up accounts:

    /ppp secret
    add local-address=172.16.0.1 name=l2tp_R1-R2_ISP1 password=ros7.elements.forever profile=profile01 remote-address=172.16.0.2 service=l2tp
    add local-address=172.16.0.5 name=l2tp_R1-R2_ISP2 password=ros7.elements.forever profile=profile01 remote-address=172.16.0.6 service=l2tp
  4. We activate L2TP server and enable IPsec encryption:

    /interface l2tp-server server 
    set authentication=mschap2 caller-id-type=number default-profile=profile01 enabled=yes ipsec-secret=ВАШ КРУТОЙ ПАРОЛЬ use-ipsec=yes
  5. We set up two EoIP tunnels over L2TP / IPsec tunnels:

    /interface eoip
    add keepalive=1s,5 local-address=172.16.0.1 mac-address=00:00:00:00:00:A1 name=eoip-tun1_over_l2tp_R1-R2_ISP1 remote-address=172.16.0.2 tunnel-id=1
    add keepalive=1s,5 local-address=172.16.0.5 mac-address=00:00:00:00:00:B1 name=eoip-tun2_over_l2tp_R1-R2_ISP2 remote-address=172.16.0.6 tunnel-id=2

    Be sure to specify the minimum keepalive timeout equal to 1 second and specify a unique ID for each EoIP tunnel.

  6. We set up bonding and assign an IP address to it:

    /interface bonding
    add lacp-rate=1sec mii-interval=1ms mode=broadcast name=bonding1 slaves=eoip-tun1_over_l2tp_R1-R2_ISP1,eoip-tun2_over_l2tp_R1-R2_ISP2
    /ip address 
    add address=172.16.1.1/30 interface=bonding1

    It is important to note here that in the mode field (the mode of operation of the bonding interface), I specified the broadcast so that the packets are sent through two tunnels at once. Thus, packet loss on either of the two interfaces will not result in packet loss on the bonding interface. Set the rest of the values ​​as in the picture.

We activate the OpenVPN server

Since I also used OpenVPN for external connections, I pre-generated certificates and imported them into CHR. I will not dwell on this in detail. Create / ppp profile and / ppp secret for OpenVPN:

/ppp profile 
add change-tcp-mss=no name=profile02 use-compression=no use-encryption=no use-mpls=no use
/ppp secret 
add local-address=172.16.2.1 name=ovpn_over_bonding1 password=ros7.elements.forever profile=profile02 remote-address=172.16.2.2 service=ovpn
/interface ovpn-server server
set auth=sha1 certificate=server.crt_0 cipher=aes256 default-profile=profile02 enabled=yes keepalive-timeout=30 port=1194 require-client-certificate=yes

Be sure to write a rule for our gray branch network behind router R2 in the nat table of the firewall so that traffic goes outside through R1:

/ip firewall nat
add action=masquerade chain=srcnat out-interface-list=WAN src-address=192.168.1.0/24

We indicate the return route to the gray network behind router R2 through the OpenVPN tunnel:

/ip route
add check-gateway=ping distance=1 dst-address=192.168.1.0/24 gateway=172.16.2.2

Router R2

  1. First of all, we register routes from one LTE modem interface to one white IP address of the data center. We prohibit the passing of packets from another interface in the firewall settings in the output chain:

    /ip route
    add distance=1 dst-address= 198.51.100.10/32 gateway=lte1
    add distance=1 dst-address= 198.51.100.20/32 gateway=lte2
    /ip firewall filter
    add action=drop chain=output dst-address= 198.51.100.10 out-interface=lte2
    add action=drop chain=output dst-address= 198.51.100.20 out-interface=lte1
  2. We bring the default config / ip ipsec profile in line with R1:

    /ip ipsec profile 
    set [ find default=yes ] dh-group=modp1024 enc-algorithm=3de
  3. Create / ppp profile:

    and two L2TP / IPsec connections to the data center for each of the providers:

    /ppp profile 
    add change-tcp-mss=no name=profile01 use-compression=no use-encryption=no use-mpls=no use
    /interface l2tp-client
    add allow=mschap2 connect-to= 198.51.100.10 disabled=no ipsec-secret= ros7.elements.forever keepalive-timeout=30 name=l2tp_to_R1_over_ISP1 password=ros7.elements.forever
        profile=profile01 use-ipsec=yes user=l2tp_R1-R2_ISP1
    add allow=mschap2 connect-to= 198.51.100.20 disabled=no ipsec-secret= ros7.elements.forever keepalive-timeout=30 name=l2tp_to_R1_over_ISP2 password=ros7.elements.forever
        profile=profile01 use-ipsec=yes user=l2tp_R1-R2_ISP2
  4. We create EoIP tunnels by analogy with R1, only we swap the local and remote IP L2TP / IPsec links of R2. The bonding interface is the same as on R1:

    /interface eoip
    add keepalive=1s,5 local-address=172.16.0.2 mac-address=00:00:00:00:00:A2 name=eoip-tun1_over_l2tp_R1-R2_ISP1 remote-address=172.16.0.1 tunnel-id=1
    add keepalive=1s,5 local-address=172.16.0.6 mac-address=00:00:00:00:00:B2 name=eoip-tun2_over_l2tp_R1-R2_ISP2 remote-address=172.16.0.5 tunnel-id=2
    /interface bonding
    add lacp-rate=1sec mii-interval=1ms mode=broadcast name=bonding1 slaves=eoip-tun1_over_l2tp_R1-R2_ISP1,eoip-tun2_over_l2tp_R1-R2_ISP2
    /ip address 
    add address=172.16.1.2/30 interface=bonding1
  5. We also import certificates, create a profile:

    Configuring OpenVPN client on R2:

    /ppp profile 
    add change-tcp-mss=no name=profile02 use-compression=no use-encryption=no use-ipv6=no use-mpls=no use-upnp=no
    /interface ovpn-client
    add certificate=client.crt_0 cipher=aes256 connect-to=172.16.1.1 mac-address=00:00:00:00:00:C2 name=ovpn_over_bonding1 password=ВАШ КРУТОЙ ПАРОЛЬ profile=profile02 use-peer-dns=no user="ovpn_over_bonding1 " verify-server-certificate=yes
  6. The tunnels lit up with the magic letter R, and EoIP – also RS. OpenVPN started up too. Now you can direct traffic from the broadcast computer to our sandwich sandwich – to the OpenVPN tunnel. To do this, create a rule / ip firewall mangle and immediately register a new routing table:

    /ip firewall mangle
    add action=mark-routing chain=prerouting dst-address-list=google_sites dst-port=1935 new-routing-mark=pc_to_stream-youtube_over_R1 passthrough=yes protocol=tcp src-address=192.168.1.1
  7. Create a route through our OpenVPN tunnel with this routing table:

    /ip route
    add check-gateway=ping distance=1 gateway=172.16.2.1 routing-mark=pc_to_stream-youtube_over_R1

And you’re done!

Trouble shooting

  • When deploying the configuration on the operating hardware, it is imperative to switch the forward and reverse routes from L2TP tunnels to the OpenVPN tunnel. If, for example, you switch only the direct route, and leave the reverse route to L2TP instead of OpenVPN, aggregation will not work completely and packets will still be lost.

  • The RouterOS utilities in the / tools section are very useful for troubleshooting. The bundle / tools Packet Sniffer + Wireshark also works well.

  • Don’t forget to play around with mtu to get the best tunnel performance.

  • Nobody canceled the signal quality. RSRP, RSRQ and SINR will show how good everything is. If the distance from the base station is long and the signal is poor, external directional antennas will help.

  • Important! If the provider filters traffic and L2TP is blocking, then you can set up other tunnels as the basis for EoIP, for example: OpenVPN or SSTP.

  • To check everything in action, you can simulate a failure. Disable any of the LTE interfaces or artificially create losses: add a rule for partial blocking of packets to the firewall and specify the value in the random field when creating a new rule.

What else can be improved and optimized

  • I do not recommend wrapping all Internet traffic, as this will cause increased overhead costs (utilization of processors, channels, etc.). It is better to use labeling for guaranteed delivery of really necessary traffic, and send everything else to LTE providers. For example, I did this with uploading video files to a cloud drive.

  • QOS is a good thing, especially on LTE channels, and especially with VoIP. Do not forget about this so that the rest of the traffic does not clog the channel that is not too wide.

  • You can enhance security by restricting external connectivity to the L2TP and IPsec ports on R1. We indicate the white IP of the LTE provider using a firewall and address lists. Although the address is from NAT and more than one client hangs on it, it will still be better. Since the IP is dynamic, you need to enable the ip – cloud function on MikroTik so that the DNS servers always know the current IP, DDNS technology.

Of course, the circuit has commercial counterparts with out-of-the-box capabilities, for example: peplink MAX HD4 LTE and similar equipment – aggregation connections. Here the business evaluates their value for themselves.

I leave links to similar topics:

Also, for those who are interested in this topic, I recommend digging towards MPTCP (Multipath TCP).

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *