Infiniband MLX4 on FreeBSD
Add the Following to your Kernel and Compile
options OFED # Infiniband protocol stack and support
options SDP # Sockets Direct Protocol for infiniband
options IPOIB_CM # Use connected mode IPoIB
options COMPAT_LINUXKPI
device ipoib # IP over IB devices
device mlx4ib # ConnectX Infiniband support
device mlxen # ConnectX Ethernet support
device mthca # Infinihost cards
device mlx5
device mlx5en
Add to or create /etc/src.conf
Using Ethernet mode
it seems like the mlx4 adapter start by default in ethernet mode which seems the most stable
below is a rc.conf for using ethernet mode
on 11.1 opensm could not find the adapter in infiniband mode
It seems like using 9k jumbo frames yield a significant performance gain
ifconfig_mlxen0="inet x.x.x.x netmask x.x.x.x mtu 9000"
Add to /boot/loader.conf (optional)
mlx4_load="YES"
ibcore_load="YES"
mlx4ib_load="YES"
ipoib_load="YES"