ISER/RMDA with ZFS on Ubuntu Server 17.04/17.10
RDMA Infiniband Modules
- /etc/modules
rdma_cm
ib_uverbs
rdma_ucm
ib_ucm
ib_umad
ib_ipoib
mlx4_ib
mlx4_en
mlx5_ib
iw_cxgb3
iw_cxgb4
iw_nes
iw_c2
rpcrdma
- /etc/security/limits.conf
...
* soft memlock unlimited
* hard memlock unlimited
root soft memlock unlimited
root hard memlock unlimited
...
- /etc/network/interfaces
auto ib0
iface ib0 inet static
address 192.168.113.1
netmask 255.255.255.0
- /usr/share/perl5/PVE/Storage/ISCSIPlugin.pm
#my $cmd = [$ISCSIADM, '--mode', 'discovery', '--type', 'sendtargets', '--portal', $portal];
my $cmd = [$ISCSIADM, '--mode', 'discovery', '--type', 'sendtargets', '--portal', $portal, '-I', 'iser'];
URLS
Disable ISCSI Authentication
cd iscsi/iqn.*/tpgt1/
set attribute authentication=0 demo_mode_write_protect=0 generate_node_acls=1 cache_dynamic_acls=1
Enable ISCSI iSER
cd iscsi/iqn.*/tpgt1/portals/192.168.221.1:3260
enable_iser true
Create File based iSCSI backstore
cd /backstores/fileio/
create file_or_dev=/tank/vol1/iservol2 name=iservol2 size=20G write_back=true
Assign a new lun to a target
cd /iscsi/iqn.*/tpg1/luns/
create /backstores/fileio/iservol2
ISCSI Admin Commands
Discover Targets from ISCSI Portal (TCP)
iscsiadm -m discovery -p <IP>:<PORT>
Discover Targets from ISCSI Portal using iSER
iscsiadm -m discovery -p <IP>:<PORT> -I iser
Refresh LUNS of ISCSI Target
Logout of all Targets
Logout to all Targets
Logout of a Target
iscsiadm -m node -T <TARGETNAME> -u
Logout to a Target
iscsiadm -m node -T <TARGETNAME> -l
Other Commands
Check IO Delay/Sats
apt install sysstat
iostat -x 1
Check ZFS IO Stats
fio --rw=read --bs=128k --numjobs=1 --iodepth=3 --size="200m" --loops=1 --ioengine=libaio --direct=1 --invalidate=1 --fsync_on_close=1 --randrepeat=1 --norandommap --exitall --name "test1" --filename="/dev/sdf" --output=/tmp/fio_verify.sh-22334.log --time_based --runtime=305