Emulating arm64 OpenWRT router in Proxmox VE

In my home lab, the main virtualization platform is Proxmox VE. Since this is a house, it is connected to the Internet along with all other devices via a regular router with OpenWRT firmware.

In most of my experiments I practice the approach “if something goes wrong… we'll figure it out and then just restart terraform”. During one of these experiments I needed to reconfigure a couple of things deep inside the OpenWRT router, and suddenly I realized that a home router is not an ephemeral resource at all. To the question: “If I brick the router, will I be able to revive it without a screwdriver, a soldering iron and, most importantly, without access to the Internet?” the answer was “¯\_(ツ)_/¯“.

Since this is the case, first we need to practice on cats, which means we will need an OpenWRT router emulator. Moreover, the arm64 version, since the packet base between arm and x86 can differ significantly. On the Internet, I have not found an instruction that would work immediately and would be easily automated. I hope this guide will work for you “out of the box”, and the time saved will go to experiments.

We will need:

Experimental plan:

  • Let's try Openwrt x86_64 emulation first as a fallback if the next step fails.

  • Next we will configure Proxmox to emulate the arm64 version of Openwrt.

  • And finally, let's add some virtual hardware to make the emulated router look more like a real device.

Step 1: OpenWRT x86_64

In the catalog with releasesselect the version, then targets -> x86 -> 64. In the Proxmox console (I used Shell in the GUI, but there is no reason why it wouldn’t work via ssh) download squashfs-combined-efi version and unpack the archive:

cd /var/lib/vz/template/iso

wget -c https://downloads.openwrt.org/releases/23.05.3/targets/x86/64/openwrt-23.05.3-x86-64-generic-squashfs-combined-efi.img.gz

gunzip openwrt-23.05.3-x86-64-generic-squashfs-combined-efi.img.gz

Disclaimer: The chosen path to store the image may not be best practice, but it is present on the Proxmox node by default. Also, I have storage called local-zfsif you have a different path, then just replace local-zfs to your own in the parameters --efidisk0 And --scsi0 in the commands below.

Actually, it is already possible to create a machine

qm create $(pvesh get /cluster/nextid) \
--name "openwrt-amd64" \
--description "openwrt-amd64" \
--arch x86_64 \
--tags "openwrt" \
--bios ovmf \
--efidisk0 file=local-zfs:4,efitype=4m,pre-enrolled-keys=0 \
--sockets 1 \
--cores 2 \
--memory 256 \
--vga type=serial0 \
--serial0 socket \
--boot order=scsi0 \
--scsihw virtio-scsi-pci \
--scsi0 file=local-zfs:0,import-from="/var/lib/vz/template/iso/openwrt-23.05.3-x86-64-generic-squashfs-combined-efi.img" \
--net0 model=virtio,bridge=vmbr0,firewall=1,link_down=0,mtu=1

and launch

qm start <VM ID> ; qm terminal <VM ID>

If everything went well, then we will see something like this in the terminal:

BusyBox v1.36.1 (2024-03-22 22:09:42 UTC) built-in shell (ash)

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt 23.05.3, r23809-234f1a2efa
 -----------------------------------------------------
=== WARNING! =====================================
There is no root password defined on this device!
Use the "passwd" command to set up a new password
in order to prevent unauthorized SSH logins.
--------------------------------------------------
root@OpenWrt:/$ cat /etc/os-release 
NAME="OpenWrt"
VERSION="23.05.3"
ID="openwrt"
ID_LIKE="lede openwrt"
PRETTY_NAME="OpenWrt 23.05.3"
VERSION_ID="23.05.3"
HOME_URL="https://openwrt.org/"
BUG_URL="https://bugs.openwrt.org/"
SUPPORT_URL="https://forum.openwrt.org/"
BUILD_ID="r23809-234f1a2efa"
OPENWRT_BOARD="x86/64"
OPENWRT_ARCH="x86_64"
OPENWRT_TAINTS=""
OPENWRT_DEVICE_MANUFACTURER="OpenWrt"
OPENWRT_DEVICE_MANUFACTURER_URL="https://openwrt.org/"
OPENWRT_DEVICE_PRODUCT="Generic"
OPENWRT_DEVICE_REVISION="v0"
OPENWRT_RELEASE="OpenWrt 23.05.3 r23809-234f1a2efa"

exit the terminal via ctrl+oclean up after ourselves and move on to the next step

qm stop <VM ID> ; qm destroy <VM ID>

Step 2: OpenWrt ARM64

IN release notes for PVE 8.1 It is mentioned that to emulate (U)EFI BIOS in virtual machines on ARM64, you need to install the package pve-edk2-firmware-aarch64:

apt install pve-edk2-firmware-aarch64

In the catalog with releasesselect the version, then targets -> armsr -> armv8.

In the Proxmox console, download squashfs-combined version and unpack the archive:

cd /var/lib/vz/template/iso

wget -c https://downloads.openwrt.org/releases/23.05.3/targets/armsr/armv8/openwrt-23.05.3-armsr-armv8-generic-squashfs-combined.img.gz

gunzip openwrt-23.05.3-armsr-armv8-generic-squashfs-combined.img.gz

We create a machine with the following command:

qm create $(pvesh get /cluster/nextid) \
--name "openwrt-aarch64" \
--description "openwrt-aarch64" \
--tags "openwrt" \
--arch aarch64 \
--bios ovmf \
--efidisk0 file=local-zfs:4,efitype=4m,pre-enrolled-keys=0 \
--sockets 1 \
--cores 2 \
--memory 256 \
--vga type=serial0 \
--serial0 socket \
--boot order=scsi0 \
--scsihw  virtio-scsi-pci \
--scsi0 file=local-zfs:0,import-from="/var/lib/vz/template/iso/openwrt-23.05.3-armsr-armv8-generic-squashfs-combined.img" \
--net0 model=virtio,bridge=vmbr0,firewall=1,link_down=0,mtu=1

loading

qm start <VM ID> ; qm terminal <VM ID>

And if this time we were successful, then in the terminal we will see something like:

BusyBox v1.36.1 (2024-03-22 22:09:42 UTC) built-in shell (ash)

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt 23.05.3, r23809-234f1a2efa
 -----------------------------------------------------
=== WARNING! =====================================
There is no root password defined on this device!
Use the "passwd" command to set up a new password
in order to prevent unauthorized SSH logins.
--------------------------------------------------
root@OpenWrt:/$ cat /etc/os-release 
NAME="OpenWrt"
VERSION="23.05.3"
ID="openwrt"
ID_LIKE="lede openwrt"
PRETTY_NAME="OpenWrt 23.05.3"
VERSION_ID="23.05.3"
HOME_URL="https://openwrt.org/"
BUG_URL="https://bugs.openwrt.org/"
SUPPORT_URL="https://forum.openwrt.org/"
BUILD_ID="r23809-234f1a2efa"
OPENWRT_BOARD="armsr/armv8"
OPENWRT_ARCH="aarch64_generic"
OPENWRT_TAINTS=""
OPENWRT_DEVICE_MANUFACTURER="OpenWrt"
OPENWRT_DEVICE_MANUFACTURER_URL="https://openwrt.org/"
OPENWRT_DEVICE_PRODUCT="Generic"
OPENWRT_DEVICE_REVISION="v0"
OPENWRT_RELEASE="OpenWrt 23.05.3 r23809-234f1a2efa"

exit the terminal via ctrl+oclean up after ourselves and move on to adding network interfaces.

qm stop <VM ID> ; qm destroy <VM ID>

Step 3: Turn the virtual machine into a router

At the moment, there is only one network interface on the virtual OpenWRT, which is automatically connected in br-lan and is initialized via a DHCP server on a real router. To make the router a router, we need to add at least one more network interface. To do this, in ProxmoxVE we will create Linux Bridge interface vmbr1.

Via shell:

cp /etc/network/interfaces /etc/network/interfaces.new

cat <<EOF>>/etc/network/interfaces.new
auto vmbr1
iface vmbr1 inet static
        address 192.168.1.0/24
        bridge-ports none
        bridge-stp off
        bridge-fd 0
EOF

systemctl start pvenetcommit

systemctl restart networking

or the path via Proxmox GUI: Datacenter->pve->System->Network->Create, Apply Configuration

If two network interfaces are available in OpenWRT, the first (eth0) will be assigned as LAN, and the second (eth1) as a WAN. To simulate a typical home network and avoid interference between DHCP servers, we need the Proxmox interface vmbr1 became eth0 in OpenWRT, and Proxmox interface vmbr0 became eth1.

Given the above, the command to create a machine would be:

qm create $(pvesh get /cluster/nextid) \
--name "openwrt-aarch64" \
--description "openwrt-aarch64" \
--tags "openwrt" \
--arch aarch64 \
--bios ovmf \
--efidisk0 file=local-zfs:4,efitype=4m,pre-enrolled-keys=0 \
--sockets 1 \
--cores 2 \
--memory 256 \
--vga type=serial0 \
--serial0 socket \
--boot order=scsi0 \
--scsihw  virtio-scsi-pci \
--scsi0 file=local-zfs:0,import-from="/var/lib/vz/template/iso/openwrt-23.05.3-armsr-armv8-generic-squashfs-combined.img" \
--net1 model=virtio,bridge=vmbr0,firewall=1,link_down=0,mtu=1 \
--net0 model=virtio,bridge=vmbr1,firewall=1,link_down=0,mtu=1

Let's launch:

qm start <vmid>; qm terminal</vmid>

we check:

root@OpenWrt:/$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br-lan state UP qlen 1000
    link/ether bc:24:11:25:4f:f2 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP qlen 1000
    link/ether bc:24:11:dc:6b:e7 brd ff:ff:ff:ff:ff:ff
    inet 10.1.2.105/24 brd 10.1.2.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::be24:11ff:fedc:6be7/64 scope link 
       valid_lft forever preferred_lft forever
4: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether bc:24:11:25:4f:f2 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.1/24 brd 192.168.1.255 scope global br-lan
       valid_lft forever preferred_lft forever
    inet6 fd4f:be36:41c0::1/60 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::be24:11ff:fe25:4ff2/64 scope link 
       valid_lft forever preferred_lft forever
root@OpenWrt:/$ cat /etc/config/network 

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fd4f:be36:41c0::/48'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'eth0'

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option ipaddr '192.168.1.1'
        option netmask '255.255.255.0'
        option ip6assign '60'

config interface "https://habr.com/ru/articles/826526/wan"
        option device 'eth1'
        option proto 'dhcp'

config interface 'wan6'
        option device 'eth1'
        option proto 'dhcpv6'

As you can see, the wan interface is bound to eth1which received an address from the real router 10.1.2.105.

OpenWRT by default opens ports for management only on the side lanand rightly so! But we started all this for risky experiments, therefore, to simplify further setup, we will open ports 22, 80And 443 on the side wanwhich should never be repeated on real hardware:

uci add firewall rule
uci set firewall.@rule[-1].name="Allow-Admin"
uci set firewall.@rule[-1].enabled='true'
uci set firewall.@rule[-1].src="https://habr.com/ru/articles/826526/wan"
uci set firewall.@rule[-1].proto='tcp'
uci set firewall.@rule[-1].dest_port="22 80 443"
uci set firewall.@rule[-1].target="ACCEPT"
uci add firewall rule
uci commit firewall  
service firewall restart

We check the connection via ssh from a host on the physical network:

$ ssh root@10.1.2.105
The authenticity of host '10.1.2.105 (10.1.2.105)' can't be established.
ED25519 key fingerprint is SHA256:VkLVqOud2GArohqiV+cHh1uIWXKVbpBJFHQJjUhrKzg.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '10.1.2.105' (ED25519) to the list of known hosts.


BusyBox v1.36.1 (2024-03-22 22:09:42 UTC) built-in shell (ash)

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt 23.05.3, r23809-234f1a2efa
 -----------------------------------------------------
=== WARNING! =====================================
There is no root password defined on this device!
Use the "passwd" command to set up a new password
in order to prevent unauthorized SSH logins.
--------------------------------------------------
root@OpenWrt:~#

It works! You can confidently implement your wildest fantasies on your router.

Instead of a conclusion

In my case, the virtual router helped me figure out how to raise a DNS zone for a local network with dynamic record updating on OpenWRT without painful rollbacks of a real device (RFC 2136).

I'll tell you about this “20-minute adventure” in the next article.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *