User Tools

Site Tools


freebsd:iband_mlx4

Infiniband MLX4 on FreeBSD

Add the Following to your Kernel and Compile

options        OFED            # Infiniband protocol stack and support
options        SDP             # Sockets Direct Protocol for infiniband
options        IPOIB_CM        # Use connected mode IPoIB

options COMPAT_LINUXKPI
device         ipoib           # IP over IB devices
device         mlx4ib          # ConnectX Infiniband support
device         mlxen           # ConnectX Ethernet support
device         mthca           # Infinihost cards
device mlx5
device mlx5en

Add to or create /etc/src.conf

WITH_OFED='yes'

Using Ethernet mode

  • it seems like the mlx4 adapter start by default in ethernet mode which seems the most stable
  • below is a rc.conf for using ethernet mode
  • on 11.1 opensm could not find the adapter in infiniband mode
  • It seems like using 9k jumbo frames yield a significant performance gain
ifconfig_mlxen0="inet x.x.x.x netmask x.x.x.x mtu 9000"

Add to /boot/loader.conf (optional)

mlx4_load="YES"
ibcore_load="YES"
mlx4ib_load="YES"
ipoib_load="YES"
freebsd/iband_mlx4.txt · Last modified: 2017/09/29 13:29 by tschulz