Infiniband MLX4 on FreeBSD

options        OFED            # Infiniband protocol stack and support
options        SDP             # Sockets Direct Protocol for infiniband
options        IPOIB_CM        # Use connected mode IPoIB

options COMPAT_LINUXKPI
device         ipoib           # IP over IB devices
device         mlx4ib          # ConnectX Infiniband support
device         mlxen           # ConnectX Ethernet support
device         mthca           # Infinihost cards
device mlx5
device mlx5en

WITH_OFED='yes'

• it seems like the mlx4 adapter start by default in ethernet mode which seems the most stable
• below is a rc.conf for using ethernet mode
• on 11.1 opensm could not find the adapter in infiniband mode
• It seems like using 9k jumbo frames yield a significant performance gain

ifconfig_mlxen0="inet x.x.x.x netmask x.x.x.x mtu 9000"

mlx4_load="YES"
ibcore_load="YES"
mlx4ib_load="YES"
ipoib_load="YES"