ZFS and VFIO PCI Passthrough on Debian 8

/etc/defaults/grub

..
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
..

/etc/modules

...
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
...

• If a driver is using a pci device that you want to passthough you might want to blacklist the driver, although you can try unbinding the driver before you tell vfio to claim the device
• driver name is found be running:

  lspci -vn | less -p VGA

  Look for “Kernel driver in use:”

  /etc/modprobe.d/blacklist.conf

  ...
  blacklist mgag200
  blacklist nouveau
  ...

/etc/network/interfaces

...
auto eth0
iface eth0 inet manual
 
auto br0
iface br0 inet dhcp
        bridge_ports eth0
...

• add the following line to /etc/apt/sources.list

  deb http://ftp.debian.org/debian jessie-backports main

• Update package list and update packages

  apt-get update
  apt-get -t jessie-backports upgrade

• Download current release from https://github.com/zfsonlinux/zfs/releases. You will need spl-x.x.x.x.tar.gz and zfs-x.x.x.x.tar.gz
• Install dependency packages

  apt-get install build-essential gawk alien fakeroot linux-headers-$(uname -r) zlib1g-dev uuid-dev libblkid-dev libselinux-dev parted lsscsi

• Build and install spl

  tar xvzf spl-x.x.x.x.tar.gz
  cd spl-x.x.x.x
  ./configure
  make deb-utils deb-kmod
  dpkg -i *.deb

• Build and install zfs

  tar xvzf zfs-x.x.x.x.tar.gz
  cd zfs-x.x.x.x
  ./configure
  make deb-utils deb-kmod
  dpkg -i *.deb

1. To grab the hexadecimal PCI id. run

   lspci -nv

2. e.g. for

   07:00.0 0300: 10de:0322 (rev a1) (prog-if 00 [VGA controller])

   the PCI ID would be 10de:0322. The slot ID/address is 07:00.0

3. Then take the slot address in the previous example *07:00.0 and prepend “0000:” so that you have something like 0000:07:00.0. Then execute the following command to unbind the current driver

   echo 0000:07:00.0 > /sys/bus/pci/devices/0000:07:00.0/driver/unbind

4. Then take the PCI ID in the previous example 10de:0322 and replace the “:” with a space then run the following command

   echo 102b 0522 > /sys/bus/pci/drivers/vfio-pci/new_id

• If you are passing a physical video card to a VM, you will need to also pass usb input devices to the VM since the VM will not have a SDL or Spice virtual display.
• This is done through USB passthrough.
• To find the USB id's to pass through run

  lsusb | grep -v "root hub"

• This will give you something like this

  Bus 005 Device 002: ID 046b:ff10 American Megatrends, Inc. Virtual Keyboard and Mouse
  Bus 003 Device 003: ID 0557:2205 ATEN International Co., Ltd 
  Bus 003 Device 002: ID 0451:2046 Texas Instruments, Inc. TUSB2046 Hub

• What we want is the hexadecimal ID ffff:ffff
• then this to pass the usb device to the vm

  -usbdevice host:ffff:ffff

#!/bin/sh
 
echo 0000:07:00.0 > /sys/bus/pci/devices/0000:07:00.0/driver/unbind
echo 102b 0522 > /sys/bus/pci/drivers/vfio-pci/new_id
 
qemu-system-x86_64 -enable-kvm -M q35 -m 2048 \
-cpu host -smp 4,sockets=1,cores=4,threads=1 \
-net nic,model=virtio,vlan=0 -net tap,vlan=0,ifname=tap0,script=/etc/qemu-ifup \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=07:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-drive file=/vol1/iso/bie2k8r2411.iso,index=2,media=cdrom \
-drive file=/vol1/iso/virtio-win.iso,index=3,media=cdrom \
-drive file=/vol1/hd-img/win2kr3.qed,format=qed,if=virtio,cache=writeback,index=1 \
-usb -usbdevice host:0557:2205 \
-boot menu=on \
-vga none

• The following is used to tell kvm to use the pci video card for display not a SDL or Spice canvas

  -vga none

• The following is used to tell kvm to setup bridged networking

  -net nic,model=virtio,vlan=0 -net tap,vlan=0,ifname=tap0,script=/etc/qemu-ifup

• The following is used to add a pcie bridge to the vm (needed for pci passthrough)

  -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1

• The following adds ICH9 emulation that enables smoother pci passthrough

  -M q35

• The following unbinds the current driver and tells vfio to claim the pci device

  echo 0000:07:00.0 > /sys/bus/pci/devices/0000:07:00.0/driver/unbind
  echo 102b 0522 > /sys/bus/pci/drivers/vfio-pci/new_id

• The following passes a video card to the vm and binds it to the pcie bridge

  -device vfio-pci,host=07:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \

• The following passes a usb hub to the vm so that keyboard and mouse input can be used with the physical display

  -usbdevice host:0557:2205

• Add Spice Display

  -vga qxl -spice port=5900,addr=127.0.0.1,disable-ticketing