E1.S: micro … Supermicro

In the comments to one of the previous posts, we were asked about the tests of the Supermicro platform based on E1.S.

Fortunately, it turned out that at that moment we had a group of servers available in a laboratory in the Netherlands. So we ran the tests and are now ready to tell you how it went.

But first, let’s briefly explain what kind of disks they are and what they are for. Let’s start from the king of peas from the basics.

How the EDSFF form factor came about

Having got rid of outdated data access methods, NVMe drive manufacturers have taken up the form factor. This is how the new EDSFF standard appeared, which, according to Intel developers, was better adapted to operate in a data center environment. EDSFF, which stands for Enterprise and Data Center SSD Form Factor, and for many is known as Ruler, was created to solve just one problem: to provide the minimum total cost of storage on Flash drives on a data center scale on a simple principle “The drive takes up less space => you can shove more 1PB drives into 1 unit => we get less cost at the exit”.

But, of course, there are indicators that had to be sacrificed with this approach – for example, performance per drive.

Supermicro has servers that support two types of drives: long and short. These form factors are described in the SNIA specifications:

In addition to the direct density of TB, IOps and GBps, drives from the EDSFF family can significantly reduce power consumption. Intel and Supermicro in their marketing materials say tens of percent compared to U.2.

Unfortunately, in our laboratory we cannot verify this statement 🙂

How did our tests go?

So, our colleagues from Supermicro have installed an SSG-1029P-NES32R server in a Dutch laboratory. It is positioned as a server for working with databases, IOps-intensive applications and hyper-converged infrastructures. It is based on the X11DSF-E motherboard with 2 sockets for installing the second generation Intel Xeon Scalable processor – in our case there were a pair of Intel® Xeon® Gold 6252 processors, 8 32GB DDR4-2933 MHz memory sticks and 32 Intel® SSDs DC P4511 Series. The platform, by the way, supports Intel® Optane ™ DCPMM.

There were the following interfaces for communicating with the “outside world”:

• 2 PCI-E 3.0 x16 (FHHL) slots,

• 1 PCI-E 3.0 x4 (LP) slot.

It is worth noting that Supermicro supplies such platforms only assembled. As a vendor we understand them, but as a potential buyer we do not approve 🙂

We will not list the rest of the technical characteristics – if anything, all the information is available at website vendor. Better to pay a little more attention to the features of the test configuration and test results.

FIO configurations

[global]

filename=/dev/era_dimec

ioengine=libaio

direct=1

group_reporting=1

runtime=900

norandommap

random_generator=tausworthe64

[seq_read]

stonewall

rw=read

bs=1024k

offset_increment=10%

numjobs=8

[seq_write]

stonewall

rw=write

bs=1024k

offset_increment=10%

numjobs=8

[rand_read]

stonewall

rw=randread

bs=4k

numjobs=48

[rand_write]

stonewall

rw=randwrite

bs=4k

numjobs=48

iodepth was varied with a script.

system configuration

OS: Ubuntu Server 20.04.3 LTS

Kernel: 5.11.0-34-generic

raidix-era-3.3.1-321-dkms-ubuntu-20.04-kver-5.11

BOOT_IMAGE=/vmlinuz-5.11.0-34-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro noibrs noibpb nopti nospectre_v2 nospectre_v1 l1tf=off nospec_store_bypass_disable no_stf_barrier mds=off tsx=on tsx_async_abort=off mitigations=off

So, we had 32 Intel® SSD DC P4511 installed in our system.

Like us and advised earlier, you need to start with the calculation – and how much can you squeeze out of them in theory?

According to the specification, the capabilities of each drive are as follows:

• maximum sequential read speed – 2800 MB / s;

• maximum sequential write speed – 2400 MB / s;

• random read speed – 610 200 IOps (4K Blocks).

• random write speed – 75,000 IOps (4K Blocks).

But when we ran a simultaneous test of the performance of all disks, we reached 9999 thousand IOps.

Almost 10 million! Although the expected performance should be close to 20 million IOps … We immediately thought about the need to straighten our arms, but a detailed study showed that the problem lies in oversubscription over PCIe lines. In such a system, the maximum per drive can be obtained only at half load by drives:

By reducing the block size to 512b, we were able to achieve a total disk performance of 12 million IOps for reading and 5 million IOps for writing.

It’s a shame, of course, to lose half the performance, but 12 million IOPs per 1U is more than enough. It is unlikely that you will find an application that with 2 sockets will provide such a load on the storage system.

But we wouldn’t be us …

if we hadn’t run two tests with RAIDIX on board 🙂

As usual, we tested RAIDIX ERA in comparison with mdraid. Here is a summary of the results at 1U: