InfiniBand on Windows made easy

From the meager information that I was able to find, I isolated and wrote simple instructions for beginners on InfiniBand at home: what equipment is needed, how to install drivers, how to set up an IB network between several PCs and an unmanaged switch. So let's get started!

Below are the devices with which I configured InfiniBand:

1. Unmanaged switch Mellanox IS5022, 40Gb/s, InfiniBand.

2. Two PCs with operating systems Windows 10 Pro And Windows 11 Pro.

3. Two network cards Mellanox CX354A HP 544QSFP, 56Gb IB, 40GbE, 2x QSFP. One for each PC.

4. Two cable cables Mellanox HP 670759-B25, QSFP+ → QSFP+, 56Gb, 3m. One for each PC.

First we need to install drivers and configure our hardware:

1. Switch configuration: Not required as it is uncontrollable. Just plug it into the network and connect it with cables to the network cards.

2. Installing drivers for Mellanox network cards: For Mellanox network cards, you need to install the latest driver, depending on the adapter and operating system used: WinOF For ConnectX-3 And ConnectX-3 Proor WinOF-2 For ConnectX-4 and newer models. We have an operating system Windows 10 (also supported on Windows 11), and the network card is ConnectX-3. Therefore, we download the driver WinOF versions 5.50.53000 from the official website Nvidia and install it on both computers.

   

3. Checking the operating mode of the network card: Now we need to make sure that the card is in working mode InfiniBand. For this we need official utilities MFT. The latest current version for our map is MFT 4.22.1-406-LTS. We also download from the official website Nvidia and install it on both computers.

   

   Launch a terminal or command line in the folder with the installed program and enter the command:

   mst status

   

   We are looking for the name of our map – mt4099_pciconf0. We look at the operating mode of the network card with the command: mlxconfig -d mt4099_pciconf0 q

   

   In line LINK_TYPE_P1 (1 port) and LINK_TYPE_P2 (2 port) value ETH(2). Our card is currently operating in Ethernet mode. Need to switch to operating mode InfiniBand (IB). To do this we will use the command:
   mlxconfig -d /dev/mst/mt4099_pciconf0 set LINK_TYPE_P1=1 LINK_TYPE_P2=1
   Where:
   1 = InfiniBand – IB(1)

   2 = Ethernet – ETH(2)

   3 = VPI – VPI(3)

   

   Confirm the action and restart the computer. Don't forget to do the same with the second computer.
   Note 1: You can additionally check the card's operating mode in Device Manager:

   

   

   Note 2: We can also customize the switching mode from InfiniBand V Ethernet and back on the fly without rebooting. To do this, you need to switch the card to mode VPI:

   mlxconfig -d /dev/mst/mt4099_pciconf0 set LINK_TYPE_P1=3 LINK_TYPE_P2=3

   Reboot and try to switch the operating mode of the network card in the driver Mellanox. Not in the driver that is in the section “Network adapters”and the one that is in the section “System devices”. We launch the device manager, find the driver and switch to the operating mode we need.

   

4. Checking the connection: Now let's check the connection between the network card and the switch. If we have configured everything correctly, the indicators of the involved ports should light orange.

   

5. We raise the network: InfiniBand LAN requires a subnet manager OpenSM. OpenSM is very useful if you have an unmanaged switch or connect two or more PCs directly. It takes over the functions of a switch and manages your network. And most importantly, if the “server” computer fails, its functions will be taken over by a neighbor. And the network will continue to function without interruption. We have already installed OpenSM Network Manager along with WinOF/WinOF2 drivers. All that remains is to launch OpenSM on the first PC. Enter the command in the terminal:

   opensm.exe

   OpenSM started and our PC switched to master (server) mode. The network manager scans our network and automatically picks up connected IB devices. Check the indicators on the switch. If the network has risen, they should light up green.

   

   Let's try to ping our “client” PC. Both computers received IP addresses on the same subnet, so I didn’t change anything and simply entered the IP that I took in the adapter properties.

   

   Enter the command in the terminal:

   ping 169.254.228.137

   

   Great! There is ping! Now let's configure the automatic launch of the OpenSM service. Enter the command in the terminal:

   New-Service --Name "OpenSM" --BinaryPathName "`"C:\Program Files\Mellanox\MLNX_VPI\IB\Tools\opensm.exe`" --service -L 128" -DisplayName "OpenSM" --Description "OpenSM for IB subnet" -StartupType Automatic

   The InfiniBand network is configured and starts automatically.

As you can see, there is nothing complicated about a home network on InfiniBand. Of course, this is just the first step. You can then configure InfiniBand for various tasks, for example, configure Samba Direct to transfer data between PCs at 40 Gbps. But now we will not delve into this topic. Today I just wanted to show that anyone can easily and inexpensively set up a network for a home laboratory or for the production needs of a small company with tens of gigabits per second using InfiniBand technology.

The author of the article is Svyatoslav Sokolov, system engineer at ServerBay