How IPv6 was implemented in RNKB

The transition to IPv6 and the problems associated with it are now being heard. We at RNKB have recently experienced this for ourselves – we have implemented IPv6 for mobile applications. The task turned out to be non-trivial, so we decided to tell other habravchans why not everything went smoothly for us and what we encountered. In other banks of the Russian Federation, including those from the top 10, according to our information, IPv6 is not yet supported.

Why do we need IPv6?

Fintech customers are increasingly using online services, so banks are closely monitoring the availability and convenience of their mobile applications, funding their own developments and trying to improve customer experience in every possible way. At RNKB, we have always taken our Internet economy very seriously, including Internet connectivity: we have our own AS (autonomous system) and blocks of IP addresses that allow you to flexibly manage the distribution of routes to the Bank’s resources on the Internet. Therefore, when we began to receive feedback from customers like “your application does not work, although the Internet is available – Google is opening”, we began to understand the reasons.

We know from experience that it is often the networks of Internet operators themselves that are responsible for problems. But in our case, not all failures could be attributed to them, and we suspected that the problem might be in the incorrect operation of the mobile Internet. For example, we have repeatedly noticed that cellular providers sometimes “issued” IPv6 addresses to mobile devices without assigning IPv4 addresses.

The task was obvious: it was time to introduce IPv6 connectivity for our mobile applications so that the bank’s services remained available to customers.

Not all providers know how to prepare IPv6

In fact, our autonomous system has been assigned a block of IPv6 addresses for a long time, it’s just that our hands didn’t reach to use it. So, when we decided that we still need to implement IPv6, we first had to deal with IPv6 connectivity on the uplinks of our AS. It turned out to be not so easy: we were not immediately able to get IPv6 performance from upstream providers, and some of them were not able to do it at all. It even began to seem to us that in large providers, all smart engineers and competent managers are employed in inter-operator divisions, while the clients – legal entities are left with specialists who at least somehow understand only DHCP and static IPv4 assignment. At least when we tried to figure out if we could get IPv6 connectivity over BGP, we were simply not understood. It got to the point that we corresponded with one provider on this issue for more than a year!

But although the process took a long time, and not all providers were able to participate, we eventually managed to get IPv6 connectivity on many of our uplinks. Then it was up to us.

Choosing the optimal solution

So, we had a mobile application that worked over IPv4, and the FQDNs of resources were described by A-records on DNS servers. “White” IPv4 addresses were translated (NAT / PAT) on the network equipment of the perimeter of our network into “gray” (RFC1918) addresses of Nginx proxy servers. These proxy servers have multiple network interfaces, and they did the initial processing of https requests and balancing across application servers. It was necessary to somehow introduce IPv6 into this scheme.

The first thing that came to our mind was to add an AAAA record to DNS and 6-to-4 NAT address translation on network equipment. But, having considered this option, we still recognized it as unsuccessful. There were many reasons for this, the most important of which was the inability to keep track of client IPv6 addresses on Nginx. Such accounting is implemented in RNKB in existing solutions and is needed in some cases. In particular, when countering DDoS attacks.

Then we thought about the option to make a special Nginx with an IPv6 interface towards the Internet and traditional IPv4 interfaces towards application servers. But this version also had a significant drawback: “exposing” the server with a “white” IPv6 address on the Internet is quite dangerous. So we “hid” it behind NAT, in our case – destination nat 6-to-6. Yes, we know that this decision will seem controversial to many: after all, best security practices, and not only they do not recommend using NAT in this way. But this practice is widespread and has proven itself well.

The final solution looked like this: AAAA records, 6-to-6 NAT, and dual-connected proxies, IPv6 and IPv4.

Testing and “pitfalls” of smartphones

To test our solution, we had to deploy a Wi-Fi network for mobile devices, an access network with IPv6 connectivity but no IPv4. To achieve the goal, we used a MikroTik router, initially with an uplink on a cellular modem. We configured the necessary AAAA records on the caching DNS in the router and started the tests.

The first results were depressing: the application did not work more stable, but there were all sorts of glitches: when connecting smartphones, pinging, with DNS queries, etc. Then we tried to improve the situation and organized a ground uplink of the stand with IPv6 connectivity. But this did not help either: various glitches also occurred on the terrestrial channel. What’s the matter? They began to find out – and found the cause. The culprit was … the smartphones themselves.

Initially, we performed the settings and the first checks from a laptop, and only later did different smartphones come into play. In detail, the process was as follows: we set up the router, then connected to its SSID with the same laptop and checked the availability of the frontend with curl or a browser. Then, smartphones with the installed banking application were connected to this SSID and the operability was already fully checked. If the application did not work, then we checked the pings and the correctness of DNS name resolution. If necessary, dumps were taken using MikroTik or on the uplink of the stand.

IPv6 offers 2 different client autoconfiguration mechanisms: DHCPv6 and ND (Neighbor Discovery). In addition, there are a number of non-obvious settings (more details can be found here: https://wiki.mikrotik.com/wiki/Manual:IPv6). So: different models of smartphones define the performance of the Internet via Wi-Fi in different ways: some of them need only IPv6 Neighbor Discovery, someone prefers DHCPv6, and some don’t want to consider a Wi-Fi network working without standard DHCP at all ( v4).

We had to modify our solution once again to eliminate this problem as well. The final configuration included ND and the distribution of IPv4 addresses in Wi-Fi via DHCP, but without an IPv4 address on the uplink interface of the router. Approximately the following MikroTik configuration in terms of IPv6 turned out to be workable:

…

/ipv6 pool

add name=poolipv6 prefix=xxxx:xxxx:xxxx:xxxx::/xx prefix-length=xx

…

/ip dns static

add address=xxxx:xxxx:0:xxxx::xx name=service.rncb.ru type=AAAA
…

/ipv6 route

add disabled=no distance=1 dst-address=::/0 gateway=xxxx:xxxx:xxxx:xxxx::x \

routing-table=main scope=30 target-scope=10

…

/ipv6 address

add address=xxxx:xxxx:xxxx:xxxx::x/xx advertise=no interface=XXXX

add address=::1 from-pool=poolipv6 interface=XXXX

…

/ipv6nd

set [ find default=yes ] disabled=yes

add dns=xxxx:xxxx:xxxx:xxxx::1, xxxx:xxxx:xxxx:xxxx::1 interface=XXXX\

ra-interval=20s-1m ra-lifetime=13m20s

…

Testing with this version of the configuration on the bench showed that the mobile application connects and works stably from all smartphones and tablets – at least those that were at our disposal.

Further testing was done without any details worth mentioning. We identified a few more errors in the settings and application logic in relation to IPv6 clients and fixed them.

Well, the result of our experiments is already in practice: after the work done, customers stopped complaining about problems with the application when the mobile Internet is working and communication is only via IPv6. Success.

Results

Was the task worth the effort? Depending on which side to look at. For example, customers are satisfied, and this is important! But, when, after launching into production, we studied the share of IPv6 traffic in the total volume of requests to the service, it turned out that it was small – about 10%. Probably because the IPv6 protocol is still not widely used on the networks of mobile operators and Wi-Fi.

Nevertheless, working on the implementation of IPv6 was a positive experience for us. It allowed us to better understand the new technology and get comfortable with the appropriate hardware settings. It is possible that this experience will help you better prepare for the IPv6 encounter in the future.

And if you, like us, have already encountered IPv6 in your work and also found pitfalls or useful solutions, write in the comments.