This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
ubuntu:iser_rdma [2017/07/20 09:58] tschulz [RDMA Infiniband Modules] |
ubuntu:iser_rdma [2018/02/23 15:12] tschulz [Check ZFS IO Stats] |
||
---|---|---|---|
Line 1: | Line 1: | ||
- | ====== ISER/RMDA with ZFS on Ubuntu Server 17.04 ====== | + | ====== ISER/RMDA with ZFS on Ubuntu Server 17.04/17.10 ====== |
- | gg | ||
===== RDMA Infiniband Modules ===== | ===== RDMA Infiniband Modules ===== | ||
<file bash /etc/modules> | <file bash /etc/modules> | ||
Line 18: | Line 18: | ||
iw_nes | iw_nes | ||
iw_c2 | iw_c2 | ||
+ | rpcrdma | ||
</file> | </file> | ||
Line 37: | Line 38: | ||
</file> | </file> | ||
==== Comment out discovery command and add a new ISER Command ==== | ==== Comment out discovery command and add a new ISER Command ==== | ||
- | <file bash /usr/share/perl5/PVE/Storage/ISCSIPlugin.pm> | + | <file perl /usr/share/perl5/PVE/Storage/ISCSIPlugin.pm> |
#my $cmd = [$ISCSIADM, '--mode', 'discovery', '--type', 'sendtargets', '--portal', $portal]; | #my $cmd = [$ISCSIADM, '--mode', 'discovery', '--type', 'sendtargets', '--portal', $portal]; | ||
my $cmd = [$ISCSIADM, '--mode', 'discovery', '--type', 'sendtargets', '--portal', $portal, '-I', 'iser']; | my $cmd = [$ISCSIADM, '--mode', 'discovery', '--type', 'sendtargets', '--portal', $portal, '-I', 'iser']; | ||
Line 44: | Line 45: | ||
+ | ==== URLS ==== | ||
+ | * [[https://community.mellanox.com/docs/DOC-1468|HowTo Configure LIO enabled with ISER for Ubuntu 14.04 Inbox driver]] | ||
+ | * [[https://community.mellanox.com/docs/DOC-1465|HowTo Setup RDMA Connection using Inbox Driver (RHEL, Ubuntu)]] | ||
- | ===== Other Commands ===== | + | |
- | ==== Check IO Delay/Sats ==== | + | |
+ | ===== Configure TargetCLI ===== | ||
+ | ==== Disable ISCSI Authentication ==== | ||
+ | * Remember on new version of Debian and Ubuntu **targetcli** has been replaced with **targetcli-fb** | ||
<file> | <file> | ||
- | apt install sysstat | + | cd iscsi/iqn.*/tpgt1/ |
+ | set attribute authentication=0 demo_mode_write_protect=0 generate_node_acls=1 cache_dynamic_acls=1 | ||
</file> | </file> | ||
+ | ==== Enable ISCSI iSER ==== | ||
<file> | <file> | ||
- | iostat -x 1 | + | cd iscsi/iqn.*/tpgt1/portals/192.168.221.1:3260 |
+ | enable_iser true | ||
</file> | </file> | ||
- | ==== Check ZFS IO Stats ==== | + | ==== Create File based iSCSI backstore ==== |
<file> | <file> | ||
- | zpool iostat -v 1 | + | cd /backstores/fileio/ |
+ | create file_or_dev=/tank/vol1/iservol2 name=iservol2 size=20G write_back=true | ||
</file> | </file> | ||
- | ==== Discover Targets from ISCSI Portal ==== | + | |
+ | ==== Assign a new lun to a target ==== | ||
+ | <file> | ||
+ | cd /iscsi/iqn.*/tpg1/luns/ | ||
+ | create /backstores/fileio/iservol2 | ||
+ | </file> | ||
+ | |||
+ | ===== ISCSI Admin Commands ===== | ||
+ | ==== Discover Targets from ISCSI Portal (TCP) ==== | ||
<file> | <file> | ||
iscsiadm -m discovery -p <IP>:<PORT> | iscsiadm -m discovery -p <IP>:<PORT> | ||
+ | </file> | ||
+ | |||
+ | ==== Discover Targets from ISCSI Portal using iSER ==== | ||
+ | <file> | ||
+ | iscsiadm -m discovery -p <IP>:<PORT> -I iser | ||
</file> | </file> | ||
==== Refresh LUNS of ISCSI Target ==== | ==== Refresh LUNS of ISCSI Target ==== | ||
Line 69: | Line 93: | ||
iscsiadm -m node -R | iscsiadm -m node -R | ||
</file> | </file> | ||
- | ==== Logout of Target ==== | + | ==== Logout of all Targets ==== |
<file> | <file> | ||
iscsiadm -m node -u | iscsiadm -m node -u | ||
</file> | </file> | ||
- | ==== Logout to Target ==== | + | ==== Logout to all Targets ==== |
<file> | <file> | ||
iscsiadm -m node -l | iscsiadm -m node -l | ||
+ | </file> | ||
+ | |||
+ | ==== Logout of a Target ==== | ||
+ | <file> | ||
+ | iscsiadm -m node -T <TARGETNAME> -u | ||
+ | </file> | ||
+ | ==== Logout to a Target ==== | ||
+ | <file> | ||
+ | iscsiadm -m node -T <TARGETNAME> -l | ||
+ | </file> | ||
+ | |||
+ | ===== Other Commands ===== | ||
+ | ==== Check IO Delay/Sats ==== | ||
+ | <file> | ||
+ | apt install sysstat | ||
+ | </file> | ||
+ | <file> | ||
+ | iostat -x 1 | ||
+ | </file> | ||
+ | ==== Check ZFS IO Stats ==== | ||
+ | <file> | ||
+ | zpool iostat -v 1 | ||
+ | </file> | ||
+ | |||
+ | ==== FIO Performance Test ==== | ||
+ | <file> | ||
+ | fio --rw=read --bs=128k --numjobs=1 --iodepth=3 --size="200m" --loops=1 --ioengine=libaio --direct=1 --invalidate=1 --fsync_on_close=1 --randrepeat=1 --norandommap --exitall --name "test1" --filename="/dev/sdf" --output=/tmp/fio_verify.sh-22334.log --time_based --runtime=305 | ||
</file> | </file> | ||