====== Using Infiniband on Ubuntu 16.04 ======
* The following was done with Mellanox Technologies MT26428 adapters. **It looks like the MT26428 requires MSI-X**. I had a 2nd gen i5 system that would not accept the MT26428 no matter if I was on Linux, FreeBSD or Windows
- Verify the adapter's core driver is loaded. The output of dmesg | grep mlx4 should look like this 2.486355] mlx4_core: Mellanox ConnectX core driver v2.2-1 (Feb, 2014)
[ 2.539227] mlx4_core: Initializing 0000:01:00.0
[ 4.816873] mlx4_core 0000:01:00.0: PCIe BW is different than device's capability
[ 4.816914] mlx4_core 0000:01:00.0: PCIe link speed is 2.5GT/s, device supports 5.0GT/s
[ 4.816951] mlx4_core 0000:01:00.0: PCIe link width is x8, device supports x8
[ 4.829392] mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.2-1 (Feb 2014)
- Next add these modules to /etc/modules or load them with modprobeib_umad
ib_ipoib
mlx4_ibThe first modules allows for management of infiniband adapters, ip_ipoib enables the use of TCP/IP over infiniband and mlx4_ib turns on infiniband on your adapter.
- Next install opensm and infiniband-diagsapt install opensm infiniband-diags
OpenSM is a subnet manager for infiniband you need one of these for every infiniband network, but it doesn't hurt to have multiple. infiniband-diags is the package that installs all the infiniband status and management programs
- For the infiniband link to go up you need to have opensm running the infiniband network and issue a ifconfig ib0 up. Check the infiniband link with ibstatCA 'mlx4_0'
CA type: MT26428
Number of ports: 1
Firmware version: 2.9.1000
Hardware version: b0
Node GUID: 0x0002c903000c01ae
System image GUID: 0x0002c903000c01b1
Port 1:
State: Active
Physical state: LinkUp
Rate: 10
Base lid: 1
LMC: 0
SM lid: 1
Capability mask: 0x0251086a
Port GUID: 0x0002c903000c01af
Link layer: InfiniBand
- Once the Link is up just bind a ip to ib0 with ifconfig like a ethernet device
- here is a ifup network entryauto ib0
iface ib0 inet static
address 192.168.221.2
netmask 255.255.255.0