i followed these directions and my system works perfectly. I tried to make another system for my other daughter (i had 2 cores avail, 6G ram and another radeon video card (the host is headless)) both video cards are claimed by vfio-pci and my script is the same except for the changes to use the proper card (and changing tun0 to tun1 for the nic)
also thank you for the excellent write up, i was beating my head on vfio for 4 days before i found your thread and got it working very quickly. big change from how i used to run my kvm guests and MUCH faster!
I figured i would share my configs and build to give back
Code: Select all
Motherboard: ASRock Micro ATX H170M PRO4
CPU: Intel 3.7GHz Core i3-6100
RAM: Patriot Extreme Performance 8G PC4-25600 (3200MHz) Viper Elite Series (i just ordered a 2nd kit to give 16G total)
Video: 2x MSI AMD Radeon R7 240 2GB DDR3 Low Profile PCIE
Audio: 2x Sabrent USB External Stero Sound Adapter (C-Media)
Storage: 2x Adata SU800 128G 3D-NAND 2.5" SATA3 SSD
Host Storage: 1x NVMe 250G Samsung 960 EVO
Sadly, there appears to be only 1 USB controller in the system so i had to passthrough devices themselves.
Code: Select all
-[0000:00]-+-00.0 Intel Corporation Skylake Host Bridge/DRAM Registers [8086:190f]
+-01.0-[01]--+-00.0 Advanced Micro Devices, Inc. [AMD/ATI] Oland PRO [Radeon R7 240/340] [1002:6613]
| \-00.1 Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0]
+-02.0 Intel Corporation HD Graphics 530 [8086:1912]
+-14.0 Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller [8086:a12f]
+-14.2 Intel Corporation Sunrise Point-H Thermal subsystem [8086:a131]
+-16.0 Intel Corporation Sunrise Point-H CSME HECI #1 [8086:a13a]
+-17.0 Intel Corporation Sunrise Point-H SATA controller [AHCI mode] [8086:a102]
+-1b.0-[02]--
+-1c.0-[03]--
+-1c.4-[04]--+-00.0 Advanced Micro Devices, Inc. [AMD/ATI] Oland PRO [Radeon R7 240/340] [1002:6613]
| \-00.1 Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0]
+-1d.0-[05]--
+-1f.0 Intel Corporation Sunrise Point-H LPC Controller [8086:a144]
+-1f.2 Intel Corporation Sunrise Point-H PMC [8086:a121]
+-1f.3 Intel Corporation Sunrise Point-H HD Audio [8086:a170]
+-1f.4 Intel Corporation Sunrise Point-H SMBus [8086:a123]
\-1f.6 Intel Corporation Ethernet Connection (2) I219-V [8086:15b8]
Code: Select all
01:00.0 0300: 1002:6613 (rev 87)
Subsystem: 1462:8094
Kernel driver in use: vfio-pci
Kernel modules: radeon
01:00.1 0403: 1002:aab0
Subsystem: 1462:aab0
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
04:00.0 0300: 1002:6613 (rev 87)
Subsystem: 1462:8094
Kernel driver in use: vfio-pci
Kernel modules: radeon
04:00.1 0403: 1002:aab0
Subsystem: 1462:aab0
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
I include the radeon driver in /etc/initramfs-tools/modules
I blacklist the radeon driver in /etc/modprobe.d/blacklist.conf
i downloaded the latest ovmf from git and modified /etc/libvirt/qemu.conf to have
Code: Select all
nvram = [
"/srv/ovmf-x64/OVMF_CODE-pure-efi.fd:/srv/ovmf-x64/OVMF_VARS-pure-efi.fd"
]
my grub commandline is as follows:
Code: Select all
GRUB_CMDLINE_LINUX_DEFAULT="rd.driver.pre=vfio-pci quiet splash intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 iommu=pt"
There are a few items to add in here since i also have a 915 for the onboard skylake video, but not including those commands had no negatives that I've seen as of yet.
i load into /etc/modules the following
Code: Select all
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
vhost-net
kvm
kvm_intel
/etc/vfio-pci1.cfg
/etc/vfio-pci2.cfg
My /etc/network/interfaces is as such:
Code: Select all
auto br0
iface br0 inet dhcp
bridge_ports enp0s31f6
bridge_stp off
bridge_maxwait 0
bridge_fd 0
post-up ip link set dev enp0s31f6 mtu 9000
in /etc/udev/rules.d/70-persistent-net.rules i have
Code: Select all
SUBSYSTEM=="net", ACTION=="add", KERNEL=="tap*", ATTR{mtu}="9000"
my launch script for system 1 is
Code: Select all
#!/bin/bash
configfile=/etc/vfio-pci1.cfg
vmname="system1"
vfiobind() {
dev="$1"
vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
device=$(cat /sys/bus/pci/devices/$dev/device)
if [ -e /sys/bus/pci/devices/$dev/driver ]; then
echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
fi
echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
}
if ps -A | grep -q $vmname; then
echo "$vmname is already running."
exit 1
else
cat $configfile | while read line;do
echo $line | grep ^# >/dev/null 2>&1 && continue
vfiobind $line
done
if [ ! -f /tmp/system1.fd ]; then
cp /srv/ovmf-x64/OVMF_VARS-pure-efi.fd /tmp/system1.fd
fi
taskset -c 0,3 qemu-system-x86_64 \
-name $vmname,process=$vmname \
-serial none \
-parallel none \
-machine type=q35,accel=kvm \
-cpu host,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time \
-smp 2,sockets=1,cores=1,threads=2 \
-enable-kvm \
-m 3G \
-mem-prealloc \
-balloon none \
-rtc clock=host,base=localtime \
-vga none \
-nographic \
-usb -usbdevice host:040b:2000 -usbdevice host:046d:c01e -usbdevice host:0d8c:0014 -usbdevice host:046d:0819 \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,romfile=/srv/rom1.rom \
-device vfio-pci,host=01:00.1,bus=root.1 \
-drive if=pflash,format=raw,readonly,file=/srv/ovmf-x64/OVMF_CODE-pure-efi.fd \
-drive if=pflash,format=raw,file=/tmp/system1.fd \
-boot order=c \
-device virtio-scsi-pci,id=scsi \
-drive file=/dev/sdc1,id=disk0,if=none,format=raw,cache=none,aio=native \
-device scsi-hd,drive=disk0 \
-netdev type=tap,id=net0,ifname=tap0,vhost=on \
-device virtio-net-pci,netdev=net0,mac=52:54:00:ea:c8:8d
# -mem-path /dev/hugepages1 \
exit 0
fi
system2 is
Code: Select all
#!/bin/bash
configfile=/etc/vfio-pci2.cfg
vmname="system2"
vfiobind() {
dev="$1"
vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
device=$(cat /sys/bus/pci/devices/$dev/device)
if [ -e /sys/bus/pci/devices/$dev/driver ]; then
echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
fi
echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
}
if ps -A | grep -q $vmname; then
echo "$vmname is already running."
exit 1
else
cat $configfile | while read line;do
echo $line | grep ^# >/dev/null 2>&1 && continue
vfiobind $line
done
if [ ! -f /tmp/system2.fd ]; then
cp /srv/ovmf-x64/OVMF_VARS-pure-efi.fd /tmp/system2.fd
fi
taskset -c 1,2 qemu-system-x86_64 \
-name $vmname,process=$vmname \
-serial none \
-parallel none \
-machine type=q35,accel=kvm \
-cpu host,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time \
-smp 2,sockets=1,cores=1,threads=2 \
-enable-kvm \
-m 3G \
-mem-prealloc \
-balloon none \
-rtc clock=host,base=localtime \
-vga none \
-nographic \
-usb -usbdevice host:04f2:0939 -usbdevice host:04f2:0833 \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,romfile=/srv/rom2.rom \
-device vfio-pci,host=04:00.1,bus=root.1 \
-drive if=pflash,format=raw,readonly,file=/srv/ovmf-x64/OVMF_CODE-pure-efi.fd \
-drive if=pflash,format=raw,file=/tmp/system2.fd \
-boot order=c \
-device virtio-scsi-pci,id=scsi \
-drive file=/dev/sdd1,id=disk0,if=none,format=raw,cache=none,aio=native \
-device scsi-hd,drive=disk0 \
-netdev type=tap,id=net0,ifname=tap1,vhost=on \
-device virtio-net-pci,netdev=net0,mac=52:54:00:ea:c8:8e
# -mem-path /dev/hugepages2 \
exit 0
fi
i create a screen session for each system and launch from within screen. This gives me the qemu console so i can wake the system up from sleep and such if needed.
I plan to reintroduce hugepages once i get the extra memory installed, although the guide here doesnt go into detail on using hugepages for 2 guests, you can simply add it to fstab from what i can tell and point each system to its relevant hugepage mount point.
---------------------------------------------------------------------
Issue1:Resolved!
that all said, the second system won't use the video card. i deleted the vga none line and built the whole system in qemu and in windows, i dont even see the video card listed in any state. It seems like the pci passthrough is being completely ignored. the strange thing is, it worked once and never again. Any thoughts come to mind?
got my second system issues resolved (rebooted host...)
---------------------------------------------------------------------
Issue2:
anyway, both systems are running great, my windows score on each is a 5.1. each system has 3G ram, 1core and 2threads (pinned) and a radeon r7 240 3G ddr3. I pass through a usb audio device, mouse, and keyboard to each system. for disk, each system has an SSD passed through except windows detects it as a thin provisioned disk (how do i fix this???)
I passed the SSD through as such:
Code: Select all
-device virtio-scsi-pci,id=scsi \
-drive file=/dev/sdc1,id=disk0,if=none,format=raw,cache=none,aio=native \
-device scsi-hd,drive=disk0 \
and
Code: Select all
-device virtio-scsi-pci,id=scsi \
-drive file=/dev/sdd1,id=disk0,if=none,format=raw,cache=none,aio=native \
-device scsi-hd,drive=disk0 \
for each machine respectively. I'm assuming i did something wrong here, but setting up lvm didn't interest me as I dont need snapshots or growth potential.
---------------------------------------------------------------------
Issue3: Resolved!
My next issue (which really doesnt matter much) is MTU
the host nic is 9000MTU, the br0 is 9000MTU, until a tap device joins, then it drops to 1500. I'm not sure how to get the tap device MTU set, but i'm actively researching it.
using this guide:
https://linuxaleph.blogspot.com/2013/01 ... guest.html
i manually resolved the issue, kid's are on their VMs so i cant test persistence yet.
---------------------------------------------------------------------
Issue4:
another issue is a slight crackle in the audio, no delays.. just crackly. im passing through the usb device via
I'm still researching this particular issue.
---------------------------------------------------------------------
Issue5:
aside from all those, i've trained my kids to put their computers to sleep when they finish, which they do... to my dismay since they can't wake the system up! i have to go into qemu and
a couple times and it wakes up. I checked the ovmf config to see if there was a mouse wakeup event, but i found none.
