Block storage performance based on soft raids (mdadm, LVM and ZFS) using iSER and NVMe-oF

The first article was devoted to performance from the local system (https://habr.com/ru/articles/753322/).

The point of this article remains the same, to show maximum performance when the issue of data safety is resolved by replicas or backups.

The purpose of this article is to test the performance of three systems for combining physical devices into one logical system using iSER and NVMe-oF.

Within the framework of this article, three systems will be compared that demonstrate the highest performance according to the test results from the first part.

1. mdadm raid0

2. LVM stripe

3. ZFS stripe (default lz4)

And also the most functional solution of the available free backends:

4. ZFS stripe with compression and deduplication

When connected via iSER and NVMe-oF.

Test stand:

Virtualization server 1:

• Motherboard: Supermicro H11SSL-i

• CPU: EPYC 7302

• RAM: 4x64GB Micron 2933MHz

• Network: 40GbE ConnectX-3 Pro, 10GbE/25GbE ConnectX-4 LN EN

• OS: ESXi 7U3 build 20036586

Virtualization server 2:

• Motherboard: Tyan S8030 (ver 1GbE)

• CPU: EPYC 7302

• RAM: 4x64GB Micron 2933MHz

• Network: 40GbE ConnectX-3 Pro x2 (One adapter inserted into the VM)

• OS: ESXi 7U3 build 21930508

Storage VM on virtualization server #2which will provide block access for host 1 and host 2:

• OS: Ubuntu 22.04, kernel 6.2.0-26 (mitigatios=off)

• vCPU: 8

• RAM: 64GB (expanded to 128GB for ZFS)

• Drives: 6xPM9A3 1.92TB (upgraded to GDC5902Q version)

Connection diagram:

Note: There is no documentation for HCIbench on how to change the testing code to use /dev/nvme0nX instead of /dev/sdX, so tests are run using pvscsi.
According to tests, the difference between the pvscsi and nvme controller lies within the measurement error of 5% (more details in the test table) for version 7U3, since full NVMe back-to-back was implemented only in 8U2, which is no longer supported by ConnectX-3 🙁

The Prepare Virtual Disk Before Testing parameter is set to Random. Tests are carried out using – 4 VMs with two 100GB disks, 8 vCPUs, 16GB RAM.

To select tests, the reference values ​​for the parameters were taken from next source.

• Test 1:

  VDbench – 4k – 80% rng – 50/50 r/w – 8 thread per disk

• Test 2:

  VDbench – 8k – 80% rng – 75/25 r/w – 8 thread per disk

• Test 3:

  VDbench – 64k – 80% seq – 75/25 r/w – 8 thread per disk

• Test 4:

  VDbench – max_iops_read – 4k – 100% rng – 100/0 r/w – 8 thread per disk

• Test 5:

  VDbench – max_iops_write – 4k – 100% rng – 0/100 r/w – 8 thread per disk

To test the drop in speed as a result of adding a layer in the form of a VM, tests from the first part were used, the results are below in the table and graph (error + -5%):

The results of iperf3 tests between ConnectX-3 Pro were performed by forwarding ConnectX-3 Pro to a VM on each of the hosts, i.e. the diagram looks like this:

The result was:

Rounding down, we get:

34 Gbits/sec (or 4.25 Gbytes/s)

Regarding OFED. Ubuntu 20.04 with kernel 5.5 and OFED installed for it was also used for tests, but Ubuntu 22.04.03 with kernel 6.2 showed higher performance and repeatability in iperf3.

s1:  [ ID] Interval           Transfer     Bitrate         Retr
s1:  [  5]   0.00-10.00  sec  15.2 GBytes  13.0 Gbits/sec  1062    sender
s1:
s1:  iperf Done.
s2:  - - - - - - - - - - - - - - - - - - - - - - - - -
s2:  [ ID] Interval           Transfer     Bitrate         Retr
s2:  [  5]   0.00-10.00  sec  14.8 GBytes  12.7 Gbits/sec  1856    sender
s2:
s2:  iperf Done.
s3:  - - - - - - - - - - - - - - - - - - - - - - - - -
s3:  [ ID] Interval           Transfer     Bitrate         Retr
s3:  [  5]   0.00-10.00  sec  10.0 GBytes  8.59 Gbits/sec  1027    sender

s1:  [ ID] Interval           Transfer     Bitrate         Retr
s1:  [  5]   0.00-10.00  sec  13.4 GBytes  11.5 Gbits/sec  1705    sender
s1:  [  5]   0.00-10.04  sec  13.2 GBytes  11.3 Gbits/sec          receiver
s1:
s1:  iperf Done.
s2:  - - - - - - - - - - - - - - - - - - - - - - - - -
s2:  [ ID] Interval           Transfer     Bitrate         Retr
s2:  [  5]   0.00-10.00  sec  10.4 GBytes  8.93 Gbits/sec  2171    sender
s2:  [  5]   0.00-10.05  sec  10.1 GBytes  8.61 Gbits/sec          receiver
s3:  - - - - - - - - - - - - - - - - - - - - - - - - -
s3:  [ ID] Interval           Transfer     Bitrate         Retr
s3:  [  5]   0.00-10.00  sec  16.8 GBytes  14.5 Gbits/sec  1513    sender
s3:  [  5]   0.00-10.04  sec  16.7 GBytes  14.3 Gbits/sec          receiver
s3:
s3:  iperf Done.

Comparing the performance of NVMe drives when they are connected directly to the OS or through VMware with PCIe passthrough, we can say that the performance does not change, since the results are within 2% of error.

Excel table for tests of disks connected directly to a physical host and transferred to a virtualization host.

2 scenarios will be considered:

1. Issuing a block device via iSER using LIO (configuration is done using targetcli-fb from the master branch of the repository)

2. Block device issuance over NVMe-oF using SPDK.

   A small digression – at first it was planned to use the native kernel implementation – nvmet (https://enterprise-support.nvidia.com/s/article/howto-configure-nvme-over-fabrics–nvme-of–target-offload), but it is not supported by VMware (https://koutoupis.com/2022/04/22/vmware-lightbits-labs-and-nvme-over-tcp/ https://communities.vmware.com/t5/ESXi-Discussions/NVMEof-Datastore-Issues/td-p/2301440), so SPDK was used (https://spdk.io/doc/nvmf.html, https://spdk.io/doc/bdev.html)

iSER

From the Linux side:

In the LIO settings enable_iser boolean=true is specified at the portal level:

set attribute authentication=0 demo_mode_write_protect=0 generate_node_acls=1 cache_dynamic_acls=1

Additionally, the following options are specified for the device (https://documentation.suse.com/ses/7/html/ses-all/deploy-additional.html):

set attribute emulate_3pc=1 emulate_tpu=1 emulate_caw=1 max_write_same_len=65535 emulate_tpws=1 is_nonrot=1

From the VMware side:

esxcli rdma iser add

Then the settings are similar to the settings for the iSCSI array.

The performance of mdadm with the FIO tests from the first part on 1 VM is as follows:

The final graph looks like this:

MDADM

mdadm --create --verbose /dev/md0 --level=0 --raid-devices=6 /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 /dev/nvme5n1

LVM

lvcreate -i6 -I64 --type striped -l 100%VG -n nvme_stripe nvme

ZFS

v1 (without dedup)

zpool create -o ashift=12 -O compression=lz4 -O atime=off -O recordsize=128k nvme /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 /dev/nvme5n1 zfs create -s -V 10T -o volblocksize=16k -o compression=lz4 nvme/iser

v2 (with dedup)

zpool create -o ashift=12 -O compression=lz4 -O atime=off -O dedup=on -O recordsize=128k nvme /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 /dev/nvme5n1 zfs create -s -V 10T -o volblocksize=16k -o compression=lz4 nvme/iser

NVMe-oF:

From the Linux side:

In addition to the banal installation of SPDK (https://spdk.io/doc/getting_started.html), there is 1 caveat. At the time of writing, SPDK from the wizard does not work with VMware due to the implementation of a check for the responder_resources == 0 parameter. On the VMware side, this parameter is equal to 1 (https://github.com/spdk/spdk/issues/3115). Therefore it is necessary to build version 23.05.x, so the spdk installation process will not start from – fixed in c8b9bba

git clone https://github.com/spdk/spdk --recursive
modprobe nvme-rdma
modprobe rdma_ucm
modprobe rdma_cm
scripts/setup.sh
screen - либо другая сессия, либо можно написать демона, который будет запускать его сам
build/bin/nvmf_tgt
ctrl+a d - отключение от сессии screen и возврат в консоль
scripts/rpc.py nvmf_create_transport -t RDMA -u 8192 -i 131072 -c 8192
scripts/rpc.py nvmf_create_subsystem nqn.2016-06.io.spdk:cnode1 -a -s SPDK00000000000001 -d SPDK_Controller
scripts/rpc.py bdev_aio_create /dev/md0 md0
scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode1 md0
scripts/rpc.py nvmf_subsystem_add_listener nqn.2016-06.io.spdk:cnode1 -t rdma -a 10.20.0.1 -s 4420
scripts/rpc.py nvmf_subsystem_add_listener nqn.2016-06.io.spdk:cnode1 -t rdma -a 10.20.0.5 -s 4420

From the VMware side:

esxcli system module parameters set -m nmlx4_core -p “enable_rocev2=1” (otherwise there will be an error “Underlying device does not support requested gid/RoCE type.”)

Then the VMware NVME over RDMA Storage Adapter is created on the web, in the controller the IP address of the VM specified earlier is set in the port field – port 4420

The performance of mdadm with the FIO tests from the first part on 1 VM is as follows:

The final graph looks like this:

MDADM

mdadm --create --verbose /dev/md0 --level=0 --raid-devices=6 /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 /dev/nvme5n1

LVM

lvcreate -i6 -I64 --type stripe  -l 100%VG -n lvm_stripe stripe

ZFS

v1 (without dedup)

zpool create -o ashift=12 -O compression=lz4 -O atime=off -O recordsize=128k nvme /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 /dev/nvme5n1 zfs create -s -V 10T -o volblocksize=16k -o compression=lz4 nvme/iser

v2 (with dedup)

zpool create -o ashift=12 -O compression=lz4 -O atime=off -O dedup=on -O recordsize=128k nvme /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 /dev/nvme5n1 zfs create -s -V 10T -o volblocksize=16k -o compression=lz4 nvme/iser

Conclusion

LVM shows the highest results, so if we take it as 100%, the test results will look like this:

ZFS, as expected, does not show the best results, but this is also due to the CPU load, but in general this is influenced by the fact that ZFS is not suitable for fast work with NVMe.

All test results are available here And excel spreadsheet here (yes, all the tables are in the archive, since Google sheets break the formulas).

The graph of all tests grouped by the connection protocol used is as follows:

Graph of all tests grouped by software raid used

PS If you think that these tests are incomplete, something is missing in them or they are incorrect, I am open to any suggestions to supplement them.