ISD 820 Knowledge Base

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Using Infiniband on Ubuntu 16.04  ======

  * The following was done with Mellanox Technologies MT26428 adapters.  **It looks like the MT26428 requires MSI-X**.  I had a 2nd gen i5 system that would not accept the MT26428 no matter if I was on Linux, FreeBSD or Windows

  - Verify the adapter's core driver is loaded.  The output of dmesg | grep mlx4 should look like this<file>    2.486355] mlx4_core: Mellanox ConnectX core driver v2.2-1 (Feb, 2014)
[    2.539227] mlx4_core: Initializing 0000:01:00.0
[    4.816873] mlx4_core 0000:01:00.0: PCIe BW is different than device's capability
[    4.816914] mlx4_core 0000:01:00.0: PCIe link speed is 2.5GT/s, device supports 5.0GT/s
[    4.816951] mlx4_core 0000:01:00.0: PCIe link width is x8, device supports x8
[    4.829392] mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.2-1 (Feb 2014)</file>
  - Next add these modules to /etc/modules or load them with modprobe<file>ib_umad
ib_ipoib
mlx4_ib</file>The first modules allows for management of infiniband adapters, ip_ipoib enables the use of TCP/IP over infiniband and mlx4_ib turns on infiniband on your adapter.
  - Next install opensm and infiniband-diags<file>apt install opensm infiniband-diags
</file>OpenSM is a subnet manager for infiniband you need one of these for every infiniband network, but it doesn't hurt to have multiple.  infiniband-diags is the package that installs all the infiniband status and management programs
  - For the infiniband link to go up you need to have opensm running the infiniband network and issue a ifconfig ib0 up.  Check the infiniband link with ibstat<file>CA 'mlx4_0'
        CA type: MT26428
        Number of ports: 1
        Firmware version: 2.9.1000
        Hardware version: b0
        Node GUID: 0x0002c903000c01ae
        System image GUID: 0x0002c903000c01b1
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 10
                Base lid: 1
                LMC: 0
                SM lid: 1
                Capability mask: 0x0251086a
                Port GUID: 0x0002c903000c01af
                Link layer: InfiniBand
</file>
  - Once the Link is up just bind a ip to ib0 with ifconfig like a ethernet device
  - here is a ifup network entry<file>auto ib0
iface ib0 inet static
        address 192.168.221.2
        netmask 255.255.255.0</file>
ISD 820 Knowledge Base

User Tools

Site Tools

Page Tools
