User Tools

Site Tools


ubuntu:iser_rdma

This is an old revision of the document!


ISER/RMDA with ZFS on Ubuntu Server 17.04/17.10

gg

RDMA Infiniband Modules

/etc/modules
rdma_cm
ib_uverbs
rdma_ucm
ib_ucm
ib_umad
ib_ipoib
 
mlx4_ib
mlx4_en
mlx5_ib
iw_cxgb3
iw_cxgb4
iw_nes
iw_c2
rpcrdma
/etc/security/limits.conf
...
* soft memlock unlimited
* hard memlock unlimited
root soft memlock unlimited
root hard memlock unlimited
...
/etc/network/interfaces
auto ib0
iface ib0 inet static
        address 192.168.113.1
        netmask 255.255.255.0

Comment out discovery command and add a new ISER Command

/usr/share/perl5/PVE/Storage/ISCSIPlugin.pm
#my $cmd = [$ISCSIADM, '--mode', 'discovery', '--type', 'sendtargets', '--portal', $portal];
my $cmd = [$ISCSIADM, '--mode', 'discovery', '--type', 'sendtargets', '--portal', $portal, '-I', 'iser'];

URLS

Configure TargetCLI

Disable ISCSI Authentication

  • Remember on new version of Debian and Ubuntu targetcli has been replaced with targetcli-fb
cd iscsi/iqn.*/tpgt1/ 
set attribute authentication=0 demo_mode_write_protect=0 generate_node_acls=1 cache_dynamic_acls=1

Enable ISCSI iSER

cd iscsi/iqn.*/tpgt1/portals/192.168.221.1:3260
enable_iser true

Create File based iSCSI backstore

cd /backstores/fileio/
create file_or_dev=/tank/vol1/iservol2 name=iservol2 size=20G write_back=true

Assign a new lun to a target

cd /iscsi/iqn.*/tpg1/luns/
create /backstores/fileio/iservol2

ISCSI Admin Commands

Discover Targets from ISCSI Portal (TCP)

iscsiadm -m discovery -p <IP>:<PORT>

Discover Targets from ISCSI Portal using iSER

iscsiadm -m discovery -p <IP>:<PORT> -I iser

Refresh LUNS of ISCSI Target

iscsiadm -m node -R

Logout of all Targets

iscsiadm -m node -u

Logout to all Targets

iscsiadm -m node -l

Logout of a Target

iscsiadm -m node -T <TARGETNAME> -u

Logout to a Target

iscsiadm -m node -T <TARGETNAME> -l

Other Commands

Check IO Delay/Sats

apt install sysstat
iostat -x 1

Check ZFS IO Stats

zpool iostat -v 1
ubuntu/iser_rdma.1519238698.txt.gz · Last modified: 2018/02/21 12:44 by tschulz