User Tools

Site Tools


ubuntu:iser_rdma

ISER/RMDA with ZFS on Ubuntu Server 17.04/17.10

RDMA Infiniband Modules

/etc/modules
rdma_cm
ib_uverbs
rdma_ucm
ib_ucm
ib_umad
ib_ipoib
 
mlx4_ib
mlx4_en
mlx5_ib
iw_cxgb3
iw_cxgb4
iw_nes
iw_c2
rpcrdma
/etc/security/limits.conf
...
* soft memlock unlimited
* hard memlock unlimited
root soft memlock unlimited
root hard memlock unlimited
...
/etc/network/interfaces
auto ib0
iface ib0 inet static
        address 192.168.113.1
        netmask 255.255.255.0

Comment out discovery command and add a new ISER Command

/usr/share/perl5/PVE/Storage/ISCSIPlugin.pm
#my $cmd = [$ISCSIADM, '--mode', 'discovery', '--type', 'sendtargets', '--portal', $portal];
my $cmd = [$ISCSIADM, '--mode', 'discovery', '--type', 'sendtargets', '--portal', $portal, '-I', 'iser'];

URLS

Configure TargetCLI

Disable ISCSI Authentication

  • Remember on new version of Debian and Ubuntu targetcli has been replaced with targetcli-fb
cd iscsi/iqn.*/tpgt1/ 
set attribute authentication=0 demo_mode_write_protect=0 generate_node_acls=1 cache_dynamic_acls=1

Enable ISCSI iSER

cd iscsi/iqn.*/tpgt1/portals/192.168.221.1:3260
enable_iser true

Create File based iSCSI backstore

cd /backstores/fileio/
create file_or_dev=/tank/vol1/iservol2 name=iservol2 size=20G write_back=true

Assign a new lun to a target

cd /iscsi/iqn.*/tpg1/luns/
create /backstores/fileio/iservol2

ISCSI Admin Commands

Discover Targets from ISCSI Portal (TCP)

iscsiadm -m discovery -p <IP>:<PORT>

Discover Targets from ISCSI Portal using iSER

iscsiadm -m discovery -p <IP>:<PORT> -I iser

Refresh LUNS of ISCSI Target

iscsiadm -m node -R

Logout of all Targets

iscsiadm -m node -u

Logout to all Targets

iscsiadm -m node -l

Logout of a Target

iscsiadm -m node -T <TARGETNAME> -u

Logout to a Target

iscsiadm -m node -T <TARGETNAME> -l

Other Commands

Check IO Delay/Sats

apt install sysstat
iostat -x 1

Check ZFS IO Stats

zpool iostat -v 1

FIO Performance Test

fio --rw=read --bs=128k --numjobs=1 --iodepth=3 --size="200m" --loops=1 --ioengine=libaio --direct=1 --invalidate=1 --fsync_on_close=1 --randrepeat=1 --norandommap --exitall --name "test1" --filename="/dev/sdf" --output=/tmp/fio_verify.sh-22334.log --time_based --runtime=305
ubuntu/iser_rdma.txt · Last modified: 2018/02/23 15:12 by tschulz