Using Infiniband on Ubuntu 16.04

The following was done with Mellanox Technologies MT26428 adapters. It looks like the MT26428 requires MSI-X. I had a 2nd gen i5 system that would not accept the MT26428 no matter if I was on Linux, FreeBSD or Windows

Verify the adapter's core driver is loaded. The output of dmesg | grep mlx4 should look like this

    2.486355] mlx4_core: Mellanox ConnectX core driver v2.2-1 (Feb, 2014)
[    2.539227] mlx4_core: Initializing 0000:01:00.0
[    4.816873] mlx4_core 0000:01:00.0: PCIe BW is different than device's capability
[    4.816914] mlx4_core 0000:01:00.0: PCIe link speed is 2.5GT/s, device supports 5.0GT/s
[    4.816951] mlx4_core 0000:01:00.0: PCIe link width is x8, device supports x8
[    4.829392] mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.2-1 (Feb 2014)

Next add these modules to /etc/modules or load them with modprobe
```
ib_umad
ib_ipoib
mlx4_ib
```
The first modules allows for management of infiniband adapters, ip_ipoib enables the use of TCP/IP over infiniband and mlx4_ib turns on infiniband on your adapter.
Next install opensm and infiniband-diags
```
apt install opensm infiniband-diags
```
OpenSM is a subnet manager for infiniband you need one of these for every infiniband network, but it doesn't hurt to have multiple. infiniband-diags is the package that installs all the infiniband status and management programs

For the infiniband link to go up you need to have opensm running the infiniband network and issue a ifconfig ib0 up. Check the infiniband link with ibstat

CA 'mlx4_0'
        CA type: MT26428
        Number of ports: 1
        Firmware version: 2.9.1000
        Hardware version: b0
        Node GUID: 0x0002c903000c01ae
        System image GUID: 0x0002c903000c01b1
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 10
                Base lid: 1
                LMC: 0
                SM lid: 1
                Capability mask: 0x0251086a
                Port GUID: 0x0002c903000c01af
                Link layer: InfiniBand

Once the Link is up just bind a ip to ib0 with ifconfig like a ethernet device

here is a ifup network entry

auto ib0
iface ib0 inet static
        address 192.168.221.2
        netmask 255.255.255.0

ISD 820 Knowledge Base

User Tools

Site Tools

Using Infiniband on Ubuntu 16.04

Page Tools