Installing ubuntu 20.04 rooted on ZFS encrypted mirror and UEFI boot

A pair of disks combined in soft raid1 has been running on my home machine for 7 years now. And just the other day, one disk in the mirror finally began to crumble. There was a reason to reinstall the system from scratch and start using encryption, which was not used 7 years ago. In the process of googling about the state of affairs in the LUKS configuration on top of mdadm, I came up with article comparing performance of zfs vs mdadm/ext4. And then I found another article with performance testing of encrypted disks using LUKS and zfs. According to both articles, zfs demonstrates very good performance and I decided to try it in action.

On Habré some time ago there were already articles on the same topic:
2019 Ubuntu 18.04 Root on ZFS
2018 Installing Debian rooted on a ZFS encrypted mirror
2018 /boot on a ZFS mirror

I decided to write my article, because I use UEFI to boot (in past articles, legacy boot was used), and plus, 3 years have passed since the last article, and it seemed to me that a modern instruction proven in practice could be useful for the community.

When installing, I mainly focused on these articles:

Ubuntu 20.04 Root on ZFS

Installing UEFI ZFS Root on Ubuntu 20.04 with Native Encryption

I will describe the installation on a virtualbox virtual machine. Installing on real hardware is no different.

So, I created a virtual machine with a couple of 20GB disks and 8GB of memory. Booted from ubuntu-20.04.1-desktop-amd64.iso, clicked on try ubuntu and launched a terminal. In the terminal, I immediately switched to root, since all the commands used require root privileges. First of all, I defined a few variables:

export DISK1=/dev/disk/by-id/ata-VBOX_HARDDISK_VBad5107ca-df268eef
export DISK2=/dev/disk/by-id/ata-VBOX_HARDDISK_VBaf134a71-943e2d11
export HOSTNAME=ubuntu-zfs-vm
export USERNAME=toor

Now you can partition the disks:

sgdisk --zap-all $DISK1
sgdisk --zap-all $DISK2
sgdisk -n1:1M:+256M -t1:EF00 -c1:EFI $DISK1
sgdisk -n1:1M:+256M -t1:EF00 -c1:EFI $DISK2
sgdisk -n2:0:+1024M -t2:be00 -c2:Boot $DISK1
sgdisk -n2:0:+1024M -t2:be00 -c2:Boot $DISK2
sgdisk -n3:0:0 -t3:bf00 -c3:Ubuntu $DISK1
sgdisk -n3:0:0 -t3:bf00 -c3:Ubuntu $DISK2

I will use UEFI to boot, so I need to create a vfat-formatted EF00 drive:

mkfs.msdos -F 32 -n EFI ${DISK1}-part1
mkfs.msdos -F 32 -n EFI ${DISK2}-part1

It’s time to create zfs. I will be using grub, which supports booting from zfs, although not with all the options. So creating a boot partition only requires explicitly specifying what grub understands:

zpool create -f -o cachefile=/etc/zfs/zpool.cache -o ashift=12 
    -o autotrim=on -d -o feature@async_destroy=enabled 
    -o feature@bookmarks=enabled -o feature@embedded_data=enabled 
    -o feature@empty_bpobj=enabled -o feature@enabled_txg=enabled 
    -o feature@extensible_dataset=enabled 
    -o feature@filesystem_limits=enabled -o feature@hole_birth=enabled 
    -o feature@large_blocks=enabled -o feature@lz4_compress=enabled 
    -o feature@spacemap_histogram=enabled -O acltype=posixacl -O canmount=off 
    -O compression=lz4 -O devices=off -O normalization=formD -O relatime=on 
    -O xattr=sa -O mountpoint=/boot -R /mnt 
    bpool mirror ${DISK1}-part2 ${DISK2}-part2

The boot partition is created, you can create a root partition (since we are using encryption, we will need to enter a password):

zpool create -f -o ashift=12 -o autotrim=on -O encryption=aes-256-gcm 
    -O keylocation=prompt -O keyformat=passphrase -O acltype=posixacl 
    -O canmount=off -O compression=lz4 -O dnodesize=auto 
    -O normalization=formD -O relatime=on -O xattr=sa -O mountpoint=/ 
    -R /mnt rpool mirror ${DISK1}-part3 ${DISK2}-part3

You can create datasets. I decided to limit myself to a minimum:

zfs create -o canmount=off -o mountpoint=none rpool/ROOT
zfs create -o canmount=off -o mountpoint=none bpool/BOOT
UUID=$(dd if=/dev/urandom bs=1 count=100 2>/dev/null 
    |tr -dc 'a-z0-9' | cut -c-6)
zfs create -o mountpoint=/ -o com.ubuntu.zsys:bootfs=yes 
    -o com.ubuntu.zsys:last-used=$(date +%s) 
    rpool/ROOT/ubuntu_$UUID
zfs create -o mountpoint=/boot bpool/BOOT/ubuntu_$UUID
zfs create -o canmount=off -o mountpoint=/ rpool/USERDATA
zfs create -o com.ubuntu.zsys:bootfs-datasets=rpool/ROOT/ubuntu_$UUID 
    -o canmount=on -o mountpoint=/home/$USERNAME 
    rpool/USERDATA/${USERNAME}_$UUID

To install the system, let’s use debootstrap:

apt-get install -y debootstrap
debootstrap focal /mnt

Copy the missing components to the new file system:

echo $HOSTNAME >/mnt/etc/hostname
sed '/cdrom/d' /etc/apt/sources.list > /mnt/etc/apt/sources.list
sed "s/ubuntu/$HOSTNAME/" /etc/hosts > /mnt/etc/hosts
cp /etc/netplan/*.yaml /mnt/etc/netplan/

And we mount the pseudofs needed to continue the installation:

mount --make-private --rbind /dev  /mnt/dev
mount --make-private --rbind /proc /mnt/proc
mount --make-private --rbind /sys  /mnt/sys

We go into the chroot environment:

chroot /mnt /usr/bin/env DISK1=$DISK1 DISK2=$DISK2 USERNAME=$USERNAME 
    /bin/bash –login

Update binary package indexes, set locale:

apt-get update
locale-gen --purge "en_US.UTF-8"
update-locale LANG=en_US.UTF-8 LANGUAGE=en_US
dpkg-reconfigure --frontend noninteractive locales

Set the time zone we need:

dpkg-reconfigure tzdata

Mount the EFI partition. Usually it is mounted under /boot/efi, but in our case there are 2 partitions, plus there is a problem with the order in which disks are mounted. I decided to mount the drive in a different hierarchy and use the symlink:

mkdir /run/efi1
mount $DISK1-part1 /run/efi1
ln -s /run/efi1 /boot/efi
echo /dev/disk/by-uuid/$(blkid -s UUID -o value 
    ${DISK1}-part1) /run/efi1 vfat defaults 0 0 >> /etc/fstab
echo /dev/disk/by-uuid/$(blkid -s UUID -o value 
    ${DISK2}-part1) /run/efi2 vfat defaults 0 0 >> /etc/fstab

Install other required packages:

apt-get install -y grub-efi-amd64 grub-efi-amd64-signed linux-image-generic 
    shim-signed zfs-initramfs zsys ubuntu-minimal network-manager

Because of regression you need to add the kernel parameter init_on_alloc=0:

sed -ie 's/(GRUB_CMDLINE_LINUX_DEFAULT="[^"]*)/1 init_on_alloc=0/' 
    /etc/default/grub

I prefer to have a small swap:

zfs create -V 4G -b $(getconf PAGESIZE) -o compression=off 
    -o logbias=throughput -o sync=always -o primarycache=metadata 
    -o secondarycache=none rpool/swap
mkswap -f /dev/zvol/rpool/swap
echo "/dev/zvol/rpool/swap none swap defaults 0 0" >> /etc/fstab
echo RESUME=none > /etc/initramfs-tools/conf.d/resume

Adding a user:

adduser $USERNAME
find /etc/skel/ -type f|xargs cp -t /home/$USERNAME
chown -R $USERNAME:$USERNAME /home/$USERNAME
usermod -a -G adm,cdrom,dip,plugdev,sudo $USERNAME

Update inird and grub and exit the chroot environment:

update-initramfs -c -k all
update-grub
grub-install --target=x86_64-efi --efi-directory=/boot/efi 
    --bootloader-id=ubuntu --recheck --no-floppy
exit

Unmount what was mounted in the chroot environment and reboot:

mount | grep -v zfs | tac | awk '//mnt/ {print $3}' | xargs -i{} umount -lf {}
zpool export -a
reboot

Since everything happens in virtualbox, I note that with UEFI enabled, the virtual machine refused to boot from the optical drive. So at this point I remove the disk from the virtual drive and enable UEFI boot.

If nothing unexpected happened, you will see the grub menu. But do not rush to press enter! Boot into recovery mode instead, because importing the zfs root pool will cause an error because the pool was last used on a machine with a different name. The fix is ​​simple:

zpool import -f root
exit

After that, you will be asked for a password to access the disk and the download will continue until the moment when you are prompted to use the emergency console (because we booted into recovery mode) or press ctrl-d to boot in normal mode. Press ctrl-d. After a few seconds, you will be able to log in using the user you created earlier. However, our troubles do not end there. Look at the /boot directory and you will see that it is empty. The boot pool was not imported either. Let’s fix this:

zpool import bpool

And the final touch – mark both EFI partitions as requiring updating when grub changes:

dpkg-reconfigure grub-efi-amd64

Now the installation of the system is completely completed and you can reboot and use the default grub menu item. Due to the fact that quiet is present in the kernel parameters by default, you will see a black screen. The password for accessing the disk will have to be entered blindly a few seconds after the start of the download. You can remove quiet from the options or just set the desktop environment.

All commands above can be downloaded as a single scriptwhich needs to define the variables DISK1, DISK2, HOSTNAME, and USERNAME in order to work.