DN42 – large BGP sandbox

DN42 is a network of VPN tunnels with BGP routing. Participants establish connections between them using various tunnel technologies (Wireguard, OpenVPN, GRE, Tinc, ZeroTier and others).

DN42 uses addresses in the range 172.20.0.0/14 (for IPv4), fd00::/8 (for IPv6), as well as autonomous system (AS) numbers from the private range 4242420000-4242429999.

At the moment (September 2023) there are 314 IPv4 blocks available for allocation (15% of the range), ~72058 billion /64 networks (0.00006% of the range), 1404 ASN (65% of the range). ASN can be obtained in the range 4242420000-AS4242423999 (the remaining blocks are currently closed for registration).

Unlike the Internet, in DN42 the minimum routable block is /29 (IPv4) and /64 (IPv6). What dual-stack systems allow you to do.

For comparison, on the Internet the minimum routable IPv4 block is /24, which costs ~11000 EUR (purchase), ~200 EUR/month (rent).

Why is it needed?

• You can experiment and explore routing technologies.

  The main benefit of DN42 is the ability to gain experience setting up tunnels and BGP connections. This is how internet routing works.

• You can learn how the Internet works and feel like a provider.

  DN42 has a registry, which is very similar to the same RIPE Database, a set internal services and your root domain .dn42 and root DNS servers.

What is needed for this?

1. Have some device with Internet access. Any Linux/BSD VM can be a DN42 router.

   Ideally, the VM/Computer/Router should be connected to the network 24/7 and have static addresses.

2. Place this device in a segment where network tunnels are allowed (Wireguard, GRE, and so on).

   I use a cheap VPS (2.50 euros per month) in Suomi.

3. Have at least basic knowledge of networks and routing.

Keys and initial setup

These steps can be done either on your personal computer (which is safer) or on a VPS (if you don’t have a Linux/BSD machine at hand). By the way, it will not work to use Windows-PC, since when creating objects inet6num And
route6 is used in the file name :but in Windows this is not allowed.

SSH

Most likely you already have SSH keys, but if not, then we generate them with the command ssh-keygen:

ssh-keygen -t ecdsa -b 521

It will create a key pair ~/.ssh/id_ecdsa (private key) and /home/asm/.ssh/id_ecdsa.pub (public key).

PGP

PGP keys for configuring DN42 are not required, but are recommended. Here good instructions from GitHub.

Git

We will also need to install git. DN42 Registry is a git repository.

Creating objects in DN42 Registry

Setting up an account and adding keys in Gitea

In KeyName indicate the name of the key, in Content – the contents of the file ~/.ssh/id_ecdsa.pub

• If the PGP key email address does not match the one used when registering in the Registry, then it is advisable to confirm it too. This can be done in account settings

Creating Objects

Finally we start creating objects.

• To do this we go to Registry and click the Fork button.
  This will create a fork of the project that will be available on address

• Cloning our fork

git clone git@git.dn42.dev:username/registry.git

cd registry

1. First we create the object person.

   Here and further USERNAME, Username, username are used as an example. Replace them with your values.

   vim data/person/USERNAME-DN42

   person:             username
   e-mail:             me@domain.tld
   nic-hdl:            USERNAME-DN42
   mnt-by:             USERNAME-MNT
   source:             DN42

   person – your username (aka nickname),
   email — email address (preferably the one specified in the PGP key).
   nic-hdl – must match the file name
   mnt-by is a reference to the mntner object (we will make it next).
   source – DN42

2. Create an mntner object.

   vim data/mntner/USERNAME-MNT

   mntner:             USERNAME-MNT
   admin-c:            USERNAME-DN42
   tech-c:             USERNAME-DN42
   mnt-by:             USERNAME-MNT
   source:             DN42
   auth:               ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAAEVWYzitbGM5wyH07zY3iieXH28drL24rYpglGMlnH8QK4Tz+sCCg5LI/XpGd7At+liXWa90zQjAoYjk3q4Vi1+QFseJUC...
   auth:               pgp-fingerprint 88361AC....

   mntner – object name (must match the file name)
   admin-c – administrative contact. Object reference person
   tech-c – technical contact. Also object reference person
   mnt-by – same as mntner field
   source – DN42
   auth – file contents ~/.ssh/id_ecdsa.pub
   auth – fingerprint pgp key (if created).

3. For the following objects we need to select the autonomous system number (ASN), IPv4 and IPv6 prefixes (networks).
   To find free ASNs and prefixes go to FreeExplorer and choose.

   Typically people choose one ASN, /27 IPv4 network (32 addresses) and /48 IPv6 network. If you have a small network, then I advise you to take the /28th network (16 addresses). If on the contrary, then /26 – it will be given without additional questions. Anything more than /26 (for IPv4) and /48 (for IPv6) will require an explanation of why you need such a large network.

   So, let’s choose. Again, I took random values ​​​​from FreeExplorer and they must be replaced with those chosen by you.

   AS4242422076
   172.23.109.240/28
   fd46:b10:16::/48

4. We return to the cloned repository and create an object for our ASN

   vim data/aut-num/AS4242422076

   aut-num:            AS4242422076
   as-name:            EXAMPLE-AS
   descr:              My Username Personal AS or another useful desciption
   admin-c:            USERNAME-DN42
   tech-c:             USERNAME-DN42
   remarks:            Peer with me at me@domain.tld or another human-oriented message
   mnt-by:             USERNAME-MNT
   source:             DN42
   

   as-name – the name of our autonomous system
   descr – description (for people)
   admin-c, tech-c and mnt-by – links to the mntner object
   remarks – optional field for notes (Usually information about peering methods is indicated there)
   source – like everywhere else DN42

5. Next we describe the blocks

   vim data/inetnum/172.23.109.240_28

   inetnum:            172.23.109.240 - 172.23.109.255
   cidr:               172.23.109.240/28
   netname:            EXAMPLE-NETWORK
   descr:              My Personal IPv4 Network
   remarks:            Peer with me at me@domain.tld
   country:            FI
   admin-c:            USERNAME-DN42
   tech-c:             USERNAME-DN42
   mnt-by:             USERNAME-MNT
   status:             ASSIGNED
   source:             DN42
   

   The file name must contain our subnet and mask (/ is replaced with _).
   inetnum – indicates the range of possible addresses (can be calculated using IP calculator)
   cird – subnet in CIRD notation
   netname – network name
   descr – description for people
   remarks – the same as for an autonomous system (you can write it, or you can not).
   country is also an optional field
   admin-c, tech-c and mnt-by – links to the mntner object
   status – ASSIGNED (there are others, but it seems you need to specify ASSIGNED)
   source – as usual, DN42

6. IPv6 block is almost the same

   vim data/inet6num/fd46:b10:16::_48

   inet6num:           fd46:0b10:0016:0000:0000:0000:0000:0000 - fd46:0b10:0016:ffff:ffff:ffff:ffff:ffff
   cidr:               fd46:b10:16::/48
   netname:            EXAMPLE-NETWORK
   descr:              My Personal IPv6 Network
   remarks:            peer with me at me@domain.tld
   country:            FI
   admin-c:            USERNAME-DN42
   tech-c:             USERNAME-DN42
   mnt-by:             USERNAME-MNT
   status:             ASSIGNED
   source:             DN42

7. We have created the objects for the blocks, now it’s time to connect the blocks with our autonomous system. There are objects for this route And route6.

   vim data/route/172.23.109.240_28

   route:              172.23.109.240/28
   origin:             AS4242422076
   mnt-by:             USERNAME-MNT
   source:             DN42

   Everything is clear here too. The only thing in official instructions it says that another attribute is needed max-length, which must be equal to the network mask. This attribute tells other routers that the maximum possible block will be /28, and if we try to advertise /29, then this announcement will not pass validation.
   I do not recommend specifying it, because if you want to announce your network in parts, then this attribute will have to be removed. As shown practice – this is not at all easy, since the main users are against the growth of the routing table.

   vim data/route6/fd46:b10:16::_48

   route6:             fd46:b10:16::/48
   origin:             AS4242422076
   mnt-by:             USERNAME-MNT
   source:             DN42

8. There are still objects dns (for your domain in the .dn42 zone) and organisation (for the organization), but they are optional and I will write about them in another post.

Review, Commit, Sign and Pull Request

At this point, all the necessary objects have been created and now you need to check, commit and create a Pull Request.

./fmt-my-stuff USERNAME-MNT

./check-my-stuff USERNAME-MNT

git add -A && git commit -am "Join DN42"

./sign-my-commit --ssh --key ~/.ssh/id_ecdsa USERNAME-MNT

git push --force

• Let’s go along link in messagecheck the changes, and also that the commit is signed and make a New Pull Request

• Next, your commit will be checked using automatic verification.

  If all is well, then all you have to do is wait for your PR to be merged with the main branch. This process may take 1-2 days. In practice, it has been verified that mergers occur in the morning (at 9-10 EEST) and in the evening (17-20 EEST)

  If something went wrong

  vim data/path/to/file

  fixing the error

  git add -A && git commit -am "Join DN42"

./squash-my-commits

./sign-my-commit --ssh --key ~/.ssh/id_ecdsa USERNAME-MNT

git push --force

As you can see from the message, there is no need to create a new PR and the changes have already been sent to the existing request.

This time both checks were successful and all we can do is wait

Preparing the server

First rule fight club dn42: always disable rp_filter.

Actually, rp_filter This is a useful thing and blocks the packet if the packet’s return route uses a different interface than the one from which it arrived.
But just like on the Internet, routing in DN42 is asymmetric, meaning packets can go along different routes. This is why rp_filter needs to be disabled.

vim /etc/sysctl.conf

We also enable forwarding between interfaces:

vim /etc/sysctl.conf

net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
net.ipv6.conf.default.forwarding = 1

Apply on the fly

sysctl -p

Checking

sysctl -a | grep rp_filter

sysctl -a | grep forwarding

Finding a peering partner

After our changes are in the main branch (master), we can look for a peering partner or several peering partners at once.

But I still advise you to wait a couple of hours. Like the Internet, DN42 has a mechanism for protecting and verifying ROA (Route Origin Authorization) routes.

ROA is a configuration file of the form

route 172.23.67.0/25 max 29 as 4242420022;
route 172.20.175.96/27 max 29 as 4242421508;
...

It tells the BGP daemon that ASN 4242420022 can advertise the prefix 172.23.67.0/25 or its subnets up to /29. This file is generated by dn42regsrv ROA Generator and is available for download from here.
Most (IMHO) peers download this file via cronjob (or systemd timer) once every 15-20 minutes and you need to wait precisely so that information about your objects is distributed across the network.

To find a peering partner, you can use the service DN42 PeerFinder.

It will show the routers closest to your location, but since the majority (again IMHO) are peered manually and you have to write to people by email/telegram/IRC, I would advise starting with automatic peering.

Here list networks that provide automatic peering.

Personally, I liked the network service the most Lutoma. It provides a user-friendly interface and looking grasswhich allows you to view the session status from Lutoma.

Automatic peering

The peering process consists of 3 steps:

• Verifying that you own the AS you are using for peering

• Creating a tunnel

• Creating a BGP session

Let’s show how this looks in practice (using the example of Lutoma and our ArchLinux server).

AS Registration and Verification

• Opening Peering Portal and click the yellow “Sign up” button

• Enter your maintainer object USERNAME-MNTand receive a verification link via email.

Setting up a Wireguard tunnel

For automatic peering, Lutoma uses Wireguard tunnels (this is the most popular tunnel type in DN42), so you need to install wireguard-tools

pacman -Sy wireguard-tools

Next we need to generate keys for encryption in the tunnel.

cd /etc/wireguard

wg genkey | (umask 0077 && tee 64719.pri) | wg pubkey > 64719.pub

Hereinafter, I use the autonomous network number of the peering partner in the name of the configuration files.
After successful verification and account creation, you can finally create a tunnel and set up a BGP session.

For this:

• Click Create Peering

• select the appropriate Router and enter the name of the connection. Something like myuserame (later it will be useful when debugging)

• VPN Type – already selected.

• In Wireguard endpoint – you need to insert the IP of your router (this is the external IP through which the router is accessible on the Internet) and the port on which our Wireguard will work. It is also possible to enter a domain name, which in theory allows you to establish a connection with a dynamic IP (DynDNS), but I have not tried this.

• Next, Lutome needs your public key. Copy it and paste it into the Wireguard public key field.

cat 64719.pub

Filling out the BGP section:

• enter the autonomous system number 4242422076

• In the Internal IPv4 address field you need to enter one of the IPs from our DN42 subnet, for example,172.23.109.249. I did it this way: I divided my /28th network into two /29ths. The first part (172.23.109.240/29) is used for LAN, the second (172.23.109.248/29) is used for peering.

• In Link-local IPv6 address, enter any address from the range fe80::/64 (For example, fe80::4719).

• Leave the checkbox for Multi-protocol BGP over IPv6 (Bird2 can do this).

• In Link bandwidth, select the bandwidth of your Internet connection.

Creating peering

We return to the server and set up a tunnel.

vim /etc/wireguard/wg64719.conf

[Interface]
PrivateKey = <Private_key_from_64719.pri>
ListenPort = 64719
Table = off

PostUp = /sbin/ip addr add dev %i 172.23.109.241/32 peer 172.22.119.1/32
PostUp = /sbin/ip addr add dev %i fe80::4719/128 peer fe80::acab/128
PostUp = sysctl -w net.ipv4.conf.%i.rp_filter=0
PostUp = sysctl -w net.ipv4.conf.%i.forwarding=1
PostUp = sysctl -w net.ipv6.conf.%i.autoconf=0
PostUp = sysctl -w net.ipv6.conf.%i.accept_ra=0
PostUp = sysctl -w net.ipv6.conf.%i.forwarding=1

[Peer]
Endpoint = de-fra.dn42.lutoma.org:43229
PublicKey = <Lutoma_Public_Key>
AllowedIPs = 172.20.0.0/14, fd00::/8, fe80::/64
PersistentKeepalive = 20

In PrivateKey we insert our private key from the file 64719.pri
In Endpoint-e you need to insert the host and port from Your configuration.
Lutoma_Public_Key is also taken from configurations.

Launching the tunnel

systemctl enable --now wg-quick@wg64719

After this you can check the connection.

ping 172.22.119.1

ping fe80::acab%wg64719

If both addresses are pinged, then you can proceed to setting up BGP; if not, then check the status using the command

wg

Setting up a BGP session

You can use it as a BGP daemon on Linux/BSD Bird/Quagga/FRRouting and others. I use Bird2 if you plan to use another daemon, but DN42 will most likely have instructions by setting it up.

Installing Bird2

pacman -S bird

Creating a directory structure

mkdir -p /etc/bird/{roa,peers}

ROA

As I wrote earlier, ROA allows you to check ASN-prefix compliance, this allows you to protect against BGP hi-jacking.
You don’t have to do this, but I recommend using an ROA check, especially since it’s not difficult.

To synchronize the ROA configuration, I use curl, which is launched every 15 minutes by the Systemd Timer.

vim /etc/systemd/system/roa.service

[Unit]
Description=Update DN42 ROA

[Service]
Type=oneshot
ExecStart=curl -sfSLR -o /etc/bird/roa/roa_v4.conf -z /etc/bird/roa/roa_v4.conf https://dn42.burble.com/roa/dn42_roa_bird2_4.conf
ExecStart=curl -sfSLR -o /etc/bird/roa/roa_v6.conf -z /etc/bird/roa/roa_v6.conf https://dn42.burble.com/roa/dn42_roa_bird2_6.conf
ExecStart=birdc configure

vim /etc/systemd/system/roa.timer

systemctl enable --now roa.timer

As a result, we should have two new files /etc/bird/roa/roa_v4.conf And /etc/bird/roa/roa_v6.conf

Bird2

Let’s move on to setting up Bird. To begin with I took the configuration from documentation.
In this configuration, we get all the routes, filter out those that did not pass the ROA check, and install the rest in the kernel route table.
We export (inform the peering partner) all the routes that we have, but first we filter out non-DN42 networks.

################################################
#               Variable header                #
################################################

define OWNAS =  4242422076;
define OWNIP =  172.23.109.241;
define OWNIPv6 = fd46:b10:16::1;
define OWNNET = 172.23.109.240/28;
define OWNNETv6 = fd46:b10:16::/48;
define OWNNETSET = [172.23.109.240/28+];
define OWNNETSETv6 = [fd46:b10:16::/48+];

################################################
#                 Header end                   #
################################################

router id OWNIP;

protocol device {
    scan time 10;
}

/*
 *  Utility functions
 */

function is_self_net() {
  return net ~ OWNNETSET;
}

function is_self_net_v6() {
  return net ~ OWNNETSETv6;
}

function is_valid_network() {
  return net ~ [
    172.20.0.0/14{21,29}, # dn42
    172.20.0.0/24{28,32}, # dn42 Anycast
    172.21.0.0/24{28,32}, # dn42 Anycast
    172.22.0.0/24{28,32}, # dn42 Anycast
    172.23.0.0/24{28,32}, # dn42 Anycast
    172.31.0.0/16+,       # ChaosVPN
    10.100.0.0/14+,       # ChaosVPN
    10.127.0.0/16{16,32}, # neonetwork
    10.0.0.0/8{15,24}     # Freifunk.net
  ];
}

roa4 table dn42_roa;
roa6 table dn42_roa_v6;

protocol static {
    roa4 { table dn42_roa; };
    include "/etc/bird/roa_dn42.conf";
};

protocol static {
    roa6 { table dn42_roa_v6; };
    include "/etc/bird/roa_dn42_v6.conf";
};

function is_valid_network_v6() {
  return net ~ [
    fd00::/8{44,64} # ULA address space as per RFC 4193
  ];
}

protocol kernel {
    scan time 20;

    ipv6 {
        import none;
        export filter {
            if source = RTS_STATIC then reject;
            krt_prefsrc = OWNIPv6;
            accept;
        };
    };
};

protocol kernel {
    scan time 20;

    ipv4 {
        import none;
        export filter {
            if source = RTS_STATIC then reject;
            krt_prefsrc = OWNIP;
            accept;
        };
    };
}

protocol static {
    route OWNNET reject;

    ipv4 {
        import all;
        export none;
    };
}

protocol static {
    route OWNNETv6 reject;

    ipv6 {
        import all;
        export none;
    };
}

template bgp dnpeers {
    local as OWNAS;
    path metric 1;

    ipv4 {
        import filter {
          if is_valid_network() && !is_self_net() then {
            if (roa_check(dn42_roa, net, bgp_path.last) != ROA_VALID) then {
              print "[dn42] ROA check failed for ", net, " ASN ", bgp_path.last;
              reject;
            } else accept;
          } else reject;
        };

        export filter { if is_valid_network() && source ~ [RTS_STATIC, RTS_BGP] then accept; else reject; };
        import limit 1000 action block;
    };

    ipv6 {
        import filter {
          if is_valid_network_v6() && !is_self_net_v6() then {
            if (roa_check(dn42_roa_v6, net, bgp_path.last) != ROA_VALID) then {
              print "[dn42] ROA check failed for ", net, " ASN ", bgp_path.last;
              reject;
            } else accept;
          } else reject;
        };
        export filter { if is_valid_network_v6() && source ~ [RTS_STATIC, RTS_BGP] then accept; else reject; };
        import limit 1000 action block;
    };
}

include "/etc/bird/peers/*.conf";

In fact, you just need to substitute your values ​​in Header:

• With OWNAS everything is clear – this is the autonomous system number

• In OWNIP we indicate the address of the router (I use the address of the LAN adapter).

• With OWNIPv6 – the same as with OWNIP

• In OWNNET and OWNNETv6 – we indicate our IPv4 subnet (and in the IPv6 subnet, respectively).

• In OWNNETSET/OWNNETSETv6 – the same thing.

Let’s add the configuration for our peering partner.

vim /etc/bird/peers/lutoma.conf

Let’s take off Let’s launch!

systemctl enable --now bird

Checking status

We wait 2-3 minutes and check that the routes have arrived:

ip route && ip -6 route

If the session is established, then you will see ~565 IPv4 and ~740 IPv6 routes.

Congratulations! You are connected to DN42. You can even ping some IP (root DNS server DN42, for example):

ping -c 2 172.20.0.53

Debugging

If not everything is so rosy and the route table is almost empty, then we go to LutomaWhere myuserame is the name of our connection.

We also look at this from our side:

birdc show protocols lutoma

birdc show route

DNS Settings

Pinging and tracing IP addresses is good, but it gets boring quickly. DN42 has the same a lot of serviceswhich are only accessible from the inside.

Since the zone .dn42 is internal and the root DNS servers of the Internet do not know about it, then using the same Cloudflare DNS or Google DNS will not work. You can use a local DNS daemon instead dnsmasqwhich will address all requests to dn42 domains to the root DN42 server, and the rest to Cloudflare/Google/Quad9.

Installation and configuration of dnsmasq

pacman -S dnsmasq

vim /etc/dnsmasq.conf

# Listen on this standard DNS port
port=53

# Never forward plain names (without a dot or domain part)
domain-needed

# Don't read /etc/hosts file
no-hosts

# Don't read /etc/resolv.conf or any other file.
# Use only the configuration provided by this file.
no-resolv

# Don't poll changes from external files (like /etc/resolv.conf)
no-poll

# Upstream DNS servers
server=2606:4700:4700::1111
server=2001:4860:4860::8844
server=1.1.1.1
server=8.8.8.8

# Force the upstream servers to be used in order
strict-order

# Increase the cachesize
cache-size=1500

# Don't store in cache the invalid resolutions
no-negcache

# Forward DN42
rebind-domain-ok=dn42
server=/dn42/172.20.0.53

# Reverse Forwards for DN42
server=/10.in-addr.arpa/172.20.0.53
server=/20.172.in-addr.arpa/172.20.0.53
server=/21.172.in-addr.arpa/172.20.0.53
server=/23.172.in-addr.arpa/172.20.0.53

Dnsmasq can also block ads, but that’s a completely different story.

All that remains is to launch dnsmasq and register it as the default DNS server.

systemctl enable --now dnsmasq

vim /etc/resolv.conf

Let’s check.

ping -c 2 asm.dn42

I suggest we stop here. The following topics were not covered:

• Multi-homing (peering with several peering partners).

• BGP communities

• Creating your own domain in the zone .dn42

• Creation of an organization

• Different types of peering: Peer, Upsteam, Client

• Anycast

• Other VPN types (OpenVPN/GRE/ZeroTier)

• 2+ servers in DN42 and internal routing

• IXP

• Setting up all kinds of Mikrotik.

If someone finds this interesting and useful, I can write a continuation article.

And of course, Peering with me.

useful links