User Tools

Site Tools


Using Infiniband on Ubuntu 16.04

  • The following was done with Mellanox Technologies MT26428 adapters. It looks like the MT26428 requires MSI-X. I had a 2nd gen i5 system that would not accept the MT26428 no matter if I was on Linux, FreeBSD or Windows
  1. Verify the adapter's core driver is loaded. The output of dmesg | grep mlx4 should look like this
        2.486355] mlx4_core: Mellanox ConnectX core driver v2.2-1 (Feb, 2014)
    [    2.539227] mlx4_core: Initializing 0000:01:00.0
    [    4.816873] mlx4_core 0000:01:00.0: PCIe BW is different than device's capability
    [    4.816914] mlx4_core 0000:01:00.0: PCIe link speed is 2.5GT/s, device supports 5.0GT/s
    [    4.816951] mlx4_core 0000:01:00.0: PCIe link width is x8, device supports x8
    [    4.829392] mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.2-1 (Feb 2014)
  2. Next add these modules to /etc/modules or load them with modprobe

    The first modules allows for management of infiniband adapters, ip_ipoib enables the use of TCP/IP over infiniband and mlx4_ib turns on infiniband on your adapter.

  3. Next install opensm and infiniband-diags
    apt install opensm infiniband-diags

    OpenSM is a subnet manager for infiniband you need one of these for every infiniband network, but it doesn't hurt to have multiple. infiniband-diags is the package that installs all the infiniband status and management programs

  4. For the infiniband link to go up you need to have opensm running the infiniband network and issue a ifconfig ib0 up. Check the infiniband link with ibstat
    CA 'mlx4_0'
            CA type: MT26428
            Number of ports: 1
            Firmware version: 2.9.1000
            Hardware version: b0
            Node GUID: 0x0002c903000c01ae
            System image GUID: 0x0002c903000c01b1
            Port 1:
                    State: Active
                    Physical state: LinkUp
                    Rate: 10
                    Base lid: 1
                    LMC: 0
                    SM lid: 1
                    Capability mask: 0x0251086a
                    Port GUID: 0x0002c903000c01af
                    Link layer: InfiniBand
  5. Once the Link is up just bind a ip to ib0 with ifconfig like a ethernet device
  6. here is a ifup network entry
    auto ib0
    iface ib0 inet static
ubuntu/infiniband.txt · Last modified: 2017/02/24 14:23 by tschulz