Debian+Proxmox 7 on Orange Pi 5

I bought myself a pair of Orange PI 5B. It seemed that Wi-Fi was very good. There is a lot of memory, I want to put different things on them. But I don’t want to constantly rearrange everything and generally crave convenience. In the end, I searched and found that Proxmox allows you to make a farm with virtual machines and then demolish them. I liked the idea right away, there is a lot of memory on the device, you can cut it into virtual machines and put k3s somewhere, where else. I will actually describe the installation procedure.

To begin with, this version of the hardware does not have an m2 slot for connecting an ssd, so it is connected externally to a usb3 port. I just bought a portable SSD drive, and at the same time regretted that I didn’t wait for the release of a version with both a wafer and an m2, but it will do for my modest tasks. That is, I need the boot to be from emmc (soldered into the board), the bootloader should be there, and the OS should be on the SSD, which is plugged into the USB3 port. Because I wouldn’t want to wear out the soldered emmc.

Putting the OS on an SD card and loading it, and even putting it on an SD card and loading from a USB-SSD is not difficult, it seems, the default installers can do this (it didn’t work out, but I didn’t try), we are not looking for easy ways. This means you need to install a bootloader that can immediately hook up a USB-SSD with emmc. This bootloader is called EDK II (I liked it among the alternatives to those that allow you to boot freely from anywhere). Below is the procedure for installing the bootloader that worked. Based on this document https://wiki.radxa.com/Rock5/install/rockchip-flash-tools I assembled rkdeveloptool, this is the emmc firmware, which is soldered on board orangepi 5b.

sudo apt-get install libudev-dev libusb-1.0-0-dev dh-autoreconf
может еще (см. доку) sudo apt-get install pkg-config libusb-1.0 

git clone https://github.com/radxa/rkdeveloptool.git
 cd rkdeveloptool
 autoreconf -i
 ./configure
 make
sudo cp rkdeveloptool /usr/local/bin/
sudo ldconfig
rkdeveloptool -vй

Next we have Tula, which can sew emmc directly from Linux without Windows, which is nice. Further based on this document https://wiki.radxa.com/Rock5/install/spi I downloaded the EDK2 bootloader
here is the download starter
https://dl.radxa.com/rock5/sw/images/loader/rock-5b/rk3588_spl_loader_v1.08.111.bin
Next is the UEFI bootloader itself
https://github.com/edk2-porting/edk2-rk3588/releases
There, for Orange 5b you just need to select the version for Orange 5.
https://github.com/edk2-porting/edk2-rk3588/releases/download/v0.9.1/orangepi-5_UEFI_Release_v0.9.1.img

These files need to be sewn into the device itself, after which EDK2 can load both the OS and other bootloaders from anywhere. I need GRUB to load, it’s more familiar. Before sewing, you need to switch the device to special mode. This is done according to the manufacturer’s instructions (Orange Pi 5 User Manual) from here http://www.orangepi.org/html/hardWare/computerAndMicrocontrollers/service-and-support/Orange-Pi-5B-32GB.html

The procedure is as follows:

You need to disconnect everything from the device. And food.
Connect only one USB port. Moreover, strictly defined, to the computer (see user manual p. 41). The one that does NOT supply power.
Press the special button on it near the port (see user manual page 41)
Next, connect the power, for me this is also a USB port.
Release the special button.

I took a napkin, took the board through it and did everything quite easily, managing not to burn it. Damn tightrope walker, I thought at that moment.

When the device is connected to the computer in this way, it is in maskrom mode. You can flash previously downloaded files using these commands. Doc is here https://wiki.radxa.com/Rock5/install/spi

посмотреть на список девайсов подключенных и их maskmode режим проверить
sudo rkdeveloptool ld
sudo rkdeveloptool db /путь к файлу/orange/rk3588_spl_loader_v1.08.111.bin
sudo rkdeveloptool wl 0 /путь к файлу/orange/orangepi-5_UEFI_Release_v0.9.1.img 
rkdeveloptool rd

As a result, you will have an EDK II UEFI bootloader on your emmc, which, when started and the keyboard is connected, allows you to twist anything in the menu and you will need to twist it. You need to go to the Device manager->Rockchip platform configuration->ACPI/Device tree menu and select Device tree mode. Without this, the distros will not load further.

The next step is to get the distro that we would like to put on the SSD. And there will be two stages. The first is a basic distro, and the second is the installation of Proxmox on it, which is built for arm64. I found the simplest one. Official distro from the authors of Orange PI5. Moreover, for Proxmox 7 you need Debian Bullseye because I chose the easiest way to install the Proxmox 7 script from here https://github.com/pimox/pimox7

So we take the distro from here http://www.orangepi.org/html/hardWare/computerAndMicrocontrollers/service-and-support/Orange-Pi-5B-32GB.html
After which you need to upload it to the SSD disk. The problem is that this distro does not have the files that are needed to load it from the UEFI bootloader; you need GRUB (for me, really). Therefore, you can take official Debian, for example debian-bookworm-live-arm64.hybrid.iso, and take two folders from it. The first is the EFI folder, which is actually the bootloader. We copy this folder to the first partition of the SSD disk, which will be there after uploading the official image to it. I did it on my computer

sudo mount /dev/sdX1 /mnt
и скопировал папку в /mnt с помощью 
sudo mc

The second folder that we need contains the GRUB files. This is the /boot/grub folder, copy it to the second partition, there is already a boot folder there, so go straight into it. This partition is mounted directly in nautilus, so that’s where I did everything. After which you need to edit the grub menu. I didn’t try to be clever and simply edited /boot/grub/grub.cfg manually in the folder on the SSD where we copied the files. You need to write the following there

set root=(hd0,gpt1)
linux /vmlinuz-5.10.160-rockchip-rk3588 root=/dev/sda2
initrd /initrd.img-5.10.160-rockchip-rk3588
boot

This will tell GRUB where to get the kernel and how to boot it. Next, you can connect our SSD to Orange PI5 via USB3 and turn it all on. We will see the EDK2 bootloader (let me remind you that we have already configured Device tree mode or you can do this now in EDK2). In it, you need to put our SSD disk in the Boot Maintenace manager menu instead of NONE so that it boots from it by default. In general, you can do without GRUB, but I’m more used to it this way. As a result, EDK2 will search for GRUB files on the first partition in the /EFI folder, transfer control to them, and GRUB will find its files and configs on the second partition in the /boot/grub folder, and the Linux kernel will start. You can go to the root menu in the bootloader by saying yes to all his questions whether to save the changes and press reset. The official Debian from the authors of the hardware will start.

At the first start, you will have to wait, since the official distro will understand that it is on a large disk and will stretch its file system over the entire disk, so you won’t have to do it manually. That’s it, we’re in the console, the default password is orangepi. We need to run the Proxmox7 installation script

sudo -s
curl https://raw.githubusercontent.com/pimox/pimox7/master/RPiOS64-IA-Install.sh > RPiOS64-IA-Install.sh
chmod +x RPiOS64-IA-Install.sh
./RPiOS64-IA-Install.sh

Next, follow the instructions of the script. But there is one thing. After the restart, when the script is finalized, you still need to execute

systemctl restart networking

Without this, my network would not start. Please note that your hardware is now given a static IP address. That is, on the router you probably need to attach it to it. Since the restart, I took a laptop that was connected to the same network and went to the Proxmox7 web interface
https://192.168.1.130:8006

I haven’t played the virtual machines themselves yet, until I forgot and decided to write down how to get to the console of this Proxmox7. But I suspect everything will work as is customary on arm for now, so-so and there will still be crap. That’s why I’ll leave it here https://github.com/jiangcuo/Proxmox-Port/wiki/Install-Proxmox-VE-on-Debian-bullseye
There are notes there to add firmware specifically for Rockchip. I’ll do this later too, since qemu didn’t notice anything right away, but so far I’ve pressed two buttons and haven’t tried anything.
Yes, a newer version of Proxmox can be installed from here https://github.com/jiangcuo/Proxmox-Port/wiki/Install-Proxmox-VE-on-Debian-bookworm but I didn’t bother since the automatic script was made to match the old one. Even without that, it took a lot of time to find all this. Maybe later. The article may also be useful for those interested https://medium.com/@aj.abdelwahed/openwrt-on-proxmox-ve-turn-your-raspberry-pi-into-a-networking-powerhouse-ee61bd2f342c
There the guy seems to be assembling isos ready for Proxmox with everything installed, I haven’t tried it, but in general, the above is enough to start them, like any distro.