Emulating arm64 OpenWRT router in Proxmox VE
![](https://habrastorage.org/getpro/habr/upload_files/edd/aab/a77/eddaaba778dc7e71975a500dd02257d1.png)
In my home lab, the main virtualization platform is Proxmox VE. Since this is a house, it is connected to the Internet along with all other devices via a regular router with OpenWRT firmware.
In most of my experiments I practice the approach “if something goes wrong… we'll figure it out and then just restart terraform”. During one of these experiments I needed to reconfigure a couple of things deep inside the OpenWRT router, and suddenly I realized that a home router is not an ephemeral resource at all. To the question: “If I brick the router, will I be able to revive it without a screwdriver, a soldering iron and, most importantly, without access to the Internet?” the answer was “¯\_(ツ)_/¯
“.
Since this is the case, first we need to practice on cats, which means we will need an OpenWRT router emulator. Moreover, the arm64 version, since the packet base between arm and x86 can differ significantly. On the Internet, I have not found an instruction that would work immediately and would be easily automated. I hope this guide will work for you “out of the box”, and the time saved will go to experiments.
We will need:
Experimental plan:
Let's try Openwrt x86_64 emulation first as a fallback if the next step fails.
Next we will configure Proxmox to emulate the arm64 version of Openwrt.
And finally, let's add some virtual hardware to make the emulated router look more like a real device.
Step 1: OpenWRT x86_64
In the catalog with releasesselect the version, then targets -> x86 -> 64
. In the Proxmox console (I used Shell in the GUI, but there is no reason why it wouldn’t work via ssh) download squashfs-combined-efi
version and unpack the archive:
cd /var/lib/vz/template/iso
wget -c https://downloads.openwrt.org/releases/23.05.3/targets/x86/64/openwrt-23.05.3-x86-64-generic-squashfs-combined-efi.img.gz
gunzip openwrt-23.05.3-x86-64-generic-squashfs-combined-efi.img.gz
Disclaimer: The chosen path to store the image may not be best practice, but it is present on the Proxmox node by default. Also, I have storage called local-zfs
if you have a different path, then just replace local-zfs
to your own in the parameters --efidisk0
And --scsi0
in the commands below.
Actually, it is already possible to create a machine
qm create $(pvesh get /cluster/nextid) \
--name "openwrt-amd64" \
--description "openwrt-amd64" \
--arch x86_64 \
--tags "openwrt" \
--bios ovmf \
--efidisk0 file=local-zfs:4,efitype=4m,pre-enrolled-keys=0 \
--sockets 1 \
--cores 2 \
--memory 256 \
--vga type=serial0 \
--serial0 socket \
--boot order=scsi0 \
--scsihw virtio-scsi-pci \
--scsi0 file=local-zfs:0,import-from="/var/lib/vz/template/iso/openwrt-23.05.3-x86-64-generic-squashfs-combined-efi.img" \
--net0 model=virtio,bridge=vmbr0,firewall=1,link_down=0,mtu=1
and launch
qm start <VM ID> ; qm terminal <VM ID>
If everything went well, then we will see something like this in the terminal:
BusyBox v1.36.1 (2024-03-22 22:09:42 UTC) built-in shell (ash)
_______ ________ __
| |.-----.-----.-----.| | | |.----.| |_
| - || _ | -__| || | | || _|| _|
|_______|| __|_____|__|__||________||__| |____|
|__| W I R E L E S S F R E E D O M
-----------------------------------------------------
OpenWrt 23.05.3, r23809-234f1a2efa
-----------------------------------------------------
=== WARNING! =====================================
There is no root password defined on this device!
Use the "passwd" command to set up a new password
in order to prevent unauthorized SSH logins.
--------------------------------------------------
root@OpenWrt:/$ cat /etc/os-release
NAME="OpenWrt"
VERSION="23.05.3"
ID="openwrt"
ID_LIKE="lede openwrt"
PRETTY_NAME="OpenWrt 23.05.3"
VERSION_ID="23.05.3"
HOME_URL="https://openwrt.org/"
BUG_URL="https://bugs.openwrt.org/"
SUPPORT_URL="https://forum.openwrt.org/"
BUILD_ID="r23809-234f1a2efa"
OPENWRT_BOARD="x86/64"
OPENWRT_ARCH="x86_64"
OPENWRT_TAINTS=""
OPENWRT_DEVICE_MANUFACTURER="OpenWrt"
OPENWRT_DEVICE_MANUFACTURER_URL="https://openwrt.org/"
OPENWRT_DEVICE_PRODUCT="Generic"
OPENWRT_DEVICE_REVISION="v0"
OPENWRT_RELEASE="OpenWrt 23.05.3 r23809-234f1a2efa"
exit the terminal via ctrl+o
clean up after ourselves and move on to the next step
qm stop <VM ID> ; qm destroy <VM ID>
Step 2: OpenWrt ARM64
IN release notes for PVE 8.1 It is mentioned that to emulate (U)EFI BIOS in virtual machines on ARM64, you need to install the package pve-edk2-firmware-aarch64
:
apt install pve-edk2-firmware-aarch64
In the catalog with releasesselect the version, then targets -> armsr -> armv8
.
In the Proxmox console, download squashfs-combined
version and unpack the archive:
cd /var/lib/vz/template/iso
wget -c https://downloads.openwrt.org/releases/23.05.3/targets/armsr/armv8/openwrt-23.05.3-armsr-armv8-generic-squashfs-combined.img.gz
gunzip openwrt-23.05.3-armsr-armv8-generic-squashfs-combined.img.gz
We create a machine with the following command:
qm create $(pvesh get /cluster/nextid) \
--name "openwrt-aarch64" \
--description "openwrt-aarch64" \
--tags "openwrt" \
--arch aarch64 \
--bios ovmf \
--efidisk0 file=local-zfs:4,efitype=4m,pre-enrolled-keys=0 \
--sockets 1 \
--cores 2 \
--memory 256 \
--vga type=serial0 \
--serial0 socket \
--boot order=scsi0 \
--scsihw virtio-scsi-pci \
--scsi0 file=local-zfs:0,import-from="/var/lib/vz/template/iso/openwrt-23.05.3-armsr-armv8-generic-squashfs-combined.img" \
--net0 model=virtio,bridge=vmbr0,firewall=1,link_down=0,mtu=1
loading
qm start <VM ID> ; qm terminal <VM ID>
And if this time we were successful, then in the terminal we will see something like:
BusyBox v1.36.1 (2024-03-22 22:09:42 UTC) built-in shell (ash)
_______ ________ __
| |.-----.-----.-----.| | | |.----.| |_
| - || _ | -__| || | | || _|| _|
|_______|| __|_____|__|__||________||__| |____|
|__| W I R E L E S S F R E E D O M
-----------------------------------------------------
OpenWrt 23.05.3, r23809-234f1a2efa
-----------------------------------------------------
=== WARNING! =====================================
There is no root password defined on this device!
Use the "passwd" command to set up a new password
in order to prevent unauthorized SSH logins.
--------------------------------------------------
root@OpenWrt:/$ cat /etc/os-release
NAME="OpenWrt"
VERSION="23.05.3"
ID="openwrt"
ID_LIKE="lede openwrt"
PRETTY_NAME="OpenWrt 23.05.3"
VERSION_ID="23.05.3"
HOME_URL="https://openwrt.org/"
BUG_URL="https://bugs.openwrt.org/"
SUPPORT_URL="https://forum.openwrt.org/"
BUILD_ID="r23809-234f1a2efa"
OPENWRT_BOARD="armsr/armv8"
OPENWRT_ARCH="aarch64_generic"
OPENWRT_TAINTS=""
OPENWRT_DEVICE_MANUFACTURER="OpenWrt"
OPENWRT_DEVICE_MANUFACTURER_URL="https://openwrt.org/"
OPENWRT_DEVICE_PRODUCT="Generic"
OPENWRT_DEVICE_REVISION="v0"
OPENWRT_RELEASE="OpenWrt 23.05.3 r23809-234f1a2efa"
exit the terminal via ctrl+o
clean up after ourselves and move on to adding network interfaces.
qm stop <VM ID> ; qm destroy <VM ID>
Step 3: Turn the virtual machine into a router
At the moment, there is only one network interface on the virtual OpenWRT, which is automatically connected in br-lan
and is initialized via a DHCP server on a real router. To make the router a router, we need to add at least one more network interface. To do this, in ProxmoxVE we will create Linux Bridge
interface vmbr1
.
Via shell:
cp /etc/network/interfaces /etc/network/interfaces.new
cat <<EOF>>/etc/network/interfaces.new
auto vmbr1
iface vmbr1 inet static
address 192.168.1.0/24
bridge-ports none
bridge-stp off
bridge-fd 0
EOF
systemctl start pvenetcommit
systemctl restart networking
or the path via Proxmox GUI: Datacenter->pve->System->Network->Create, Apply Configuration
If two network interfaces are available in OpenWRT, the first (eth0
) will be assigned as LAN, and the second (eth1
) as a WAN. To simulate a typical home network and avoid interference between DHCP servers, we need the Proxmox interface vmbr1
became eth0
in OpenWRT, and Proxmox interface vmbr0
became eth1
.
Given the above, the command to create a machine would be:
qm create $(pvesh get /cluster/nextid) \
--name "openwrt-aarch64" \
--description "openwrt-aarch64" \
--tags "openwrt" \
--arch aarch64 \
--bios ovmf \
--efidisk0 file=local-zfs:4,efitype=4m,pre-enrolled-keys=0 \
--sockets 1 \
--cores 2 \
--memory 256 \
--vga type=serial0 \
--serial0 socket \
--boot order=scsi0 \
--scsihw virtio-scsi-pci \
--scsi0 file=local-zfs:0,import-from="/var/lib/vz/template/iso/openwrt-23.05.3-armsr-armv8-generic-squashfs-combined.img" \
--net1 model=virtio,bridge=vmbr0,firewall=1,link_down=0,mtu=1 \
--net0 model=virtio,bridge=vmbr1,firewall=1,link_down=0,mtu=1
Let's launch:
qm start <vmid>; qm terminal</vmid>
we check:
root@OpenWrt:/$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br-lan state UP qlen 1000
link/ether bc:24:11:25:4f:f2 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP qlen 1000
link/ether bc:24:11:dc:6b:e7 brd ff:ff:ff:ff:ff:ff
inet 10.1.2.105/24 brd 10.1.2.255 scope global eth1
valid_lft forever preferred_lft forever
inet6 fe80::be24:11ff:fedc:6be7/64 scope link
valid_lft forever preferred_lft forever
4: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
link/ether bc:24:11:25:4f:f2 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.1/24 brd 192.168.1.255 scope global br-lan
valid_lft forever preferred_lft forever
inet6 fd4f:be36:41c0::1/60 scope global noprefixroute
valid_lft forever preferred_lft forever
inet6 fe80::be24:11ff:fe25:4ff2/64 scope link
valid_lft forever preferred_lft forever
root@OpenWrt:/$ cat /etc/config/network
config interface 'loopback'
option device 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'
config globals 'globals'
option ula_prefix 'fd4f:be36:41c0::/48'
config device
option name 'br-lan'
option type 'bridge'
list ports 'eth0'
config interface 'lan'
option device 'br-lan'
option proto 'static'
option ipaddr '192.168.1.1'
option netmask '255.255.255.0'
option ip6assign '60'
config interface "https://habr.com/ru/articles/826526/wan"
option device 'eth1'
option proto 'dhcp'
config interface 'wan6'
option device 'eth1'
option proto 'dhcpv6'
As you can see, the wan interface is bound to eth1
which received an address from the real router 10.1.2.105
.
OpenWRT by default opens ports for management only on the side lan
and rightly so! But we started all this for risky experiments, therefore, to simplify further setup, we will open ports 22
, 80
And 443
on the side wan
which should never be repeated on real hardware:
uci add firewall rule
uci set firewall.@rule[-1].name="Allow-Admin"
uci set firewall.@rule[-1].enabled='true'
uci set firewall.@rule[-1].src="https://habr.com/ru/articles/826526/wan"
uci set firewall.@rule[-1].proto='tcp'
uci set firewall.@rule[-1].dest_port="22 80 443"
uci set firewall.@rule[-1].target="ACCEPT"
uci add firewall rule
uci commit firewall
service firewall restart
We check the connection via ssh from a host on the physical network:
$ ssh root@10.1.2.105
The authenticity of host '10.1.2.105 (10.1.2.105)' can't be established.
ED25519 key fingerprint is SHA256:VkLVqOud2GArohqiV+cHh1uIWXKVbpBJFHQJjUhrKzg.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '10.1.2.105' (ED25519) to the list of known hosts.
BusyBox v1.36.1 (2024-03-22 22:09:42 UTC) built-in shell (ash)
_______ ________ __
| |.-----.-----.-----.| | | |.----.| |_
| - || _ | -__| || | | || _|| _|
|_______|| __|_____|__|__||________||__| |____|
|__| W I R E L E S S F R E E D O M
-----------------------------------------------------
OpenWrt 23.05.3, r23809-234f1a2efa
-----------------------------------------------------
=== WARNING! =====================================
There is no root password defined on this device!
Use the "passwd" command to set up a new password
in order to prevent unauthorized SSH logins.
--------------------------------------------------
root@OpenWrt:~#
It works! You can confidently implement your wildest fantasies on your router.
Instead of a conclusion
In my case, the virtual router helped me figure out how to raise a DNS zone for a local network with dynamic record updating on OpenWRT without painful rollbacks of a real device (RFC 2136).
I'll tell you about this “20-minute adventure” in the next article.