HOW-TO make dual-boot obsolete using kvm VGA passthrough

[craigbett]

Turns out that the pci-stub wasn't grabbing the "0000:01:00.0" ( the actual graphics card ), nouveau was grabbing it before pci-stub could. Putting nouveau.blacklist=1 in the grub file wasn't enough to blacklist the kernel drivers

instead I added a new file /etc/modprobe.d/nvidia-graphics-drivers.conf and put

Code: Select all

blacklist nouveau
blacklist lbm-nouveau
blacklist nvidia-173
blacklist nvidia-96
blacklist nvidia-current
blacklist nvidia-173-updates
blacklist nvidia-96-updates
alias nvidia nvidia_current_updates
alias nouveau off
alias lbm-nouveau off

( recommended from this http://askubuntu.com/questions/112302/h ... nel-driver. )
and the pci-stub was able to get the card. The kvm works now

[powerhouse]

Hi powerhouse,

Thank you for your tutorial, I was able to follow everything and almost got it working. The VM starts fine, but nothing appears in the monitor with the guest gpu, like no signal is being received.

I have two gpus:
01:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] (rev a1)
02:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 760] (rev a1)

The gtx 970 will be the guest, and the gtx 760 will be the host, each on a different monitor (usind HDMI cables). The gtx 970 is used to boot the system, and it shows everything until the login screen, when the image is switched to the host gpu/monitor, and the guest monitor turns off. But the guest fans never stop also. I can't switch de PCIE slots because the guest is on x16 and host in a x4.

Looks like my gtx 970 doesn't have EFI support (the EfiRom returns code 0x00). But when I try to run it using seabios, nothing happens, just like OVMF. The VM starts and there is no error message (besides an error with the netdev).

And all gpus are isolated in their IOMMU groups.

Here is my VM start script:

Code: Select all

#!/bin/bash

configfile=/etc/vfio-pci.cfg
vmname="windows7vm"

vfiobind() {
   dev="$1"
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
   
}


#if ps -A | grep -q $vmname; then
#   echo "$vmname is already running." &
#   exit 1

#else


   cat $configfile | while read line;do
   echo $line | grep ^# >/dev/null 2>&1 && continue
      vfiobind $line
   done

# use pulseaudio
export QEMU_AUDIO_DRV=pa

cp /usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd /tmp/my_vars.fd

qemu-system-x86_64 \
  -name $vmname,process=$vmname \
  -machine type=q35,accel=kvm \
  -cpu host,kvm=off \
  -smp 4,sockets=1,cores=2,threads=2 \
  -enable-kvm \
  -m 4G \
  -mem-prealloc \
  -balloon none \
  -rtc clock=host,base=localtime \
  -bios /usr/share/seabios/bios.bin -vga none \
  -nographic \
  -serial none \
  -parallel none \
  -soundhw hda \
  -device vfio-pci,host=01:00.0,x-vga=on,multifunction=on \
  -device vfio-pci,host=01:00.1 \
  -boot order=dc \
  -device virtio-scsi-pci,id=scsi \
  -drive id=disk0,if=virtio,cache=none,format=raw,file=/media/neuromancer/vm/win.img \
  -drive file=/home/neuromancer/Downloads/os/win7ultimate/win7ultimate.iso,id=isocd,format=raw,if=none -device scsi-cd,drive=isocd \
  -drive file=/home/neuromancer/Downloads/os/virtio-win-0.1.118.iso,id=virtiocd,format=raw,if=none -device ide-cd,bus=ide.1,drive=virtiocd \
  -netdev type=tap,id=net0,ifname=tap0,vhost=on \
  -device virtio-net-pci,netdev=net0,mac=00:16:3e:00:01:01

   exit 0
#fi

Here is the vm output:

Code: Select all

W: /etc/qemu-ifup: no bridge for guest interface found
QEMU 2.1.2 monitor - type 'help' for more information

EfiRom output (I had to boot in safe mode to be able to echo 1 > rom, don't know if this can be the cause of the error:

Code: Select all

Image 1 -- Offset 0x0
  ROM header contents
    Signature              0xAA55
    PCIR offset            0x01A0
    Signature               PCIR
    Vendor ID               0x10DE
    Device ID               0x13C2
    Length                  0x0018
    Revision                0x0000
    DeviceListOffset        0x00
    Class Code              0x030000
    Image size              0xE800
    Code revision:          0x0001
    MaxRuntimeImageLength   0x00
    ConfigUtilityCodeHeaderOffset 0x8B2E
    DMTFCLPEntryPointOffset 0x2EC0
    Indicator               0x00
    Code type               0x00
EfiRom: ERROR 3001: Not supported
  Failed to read PCI ROM header from file!

And the guest Gpu is bind to vfio-pci:

Code: Select all

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:3160]
	Kernel driver in use: vfio-pci
01:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:3160]
	Kernel driver in use: vfio-pci

Do you have any idea why this could be happening? Am I doing the seabios part right? Thanks for the help!

If your graphics card does NOT support UEFI, then this tutorial won't work. You cannot use OVMF then. I have posted links to other tutorials that might work - check out the link to the Arch Linux forum.

[powerhouse]

Turns out that the pci-stub wasn't grabbing the "0000:01:00.0" ( the actual graphics card ), nouveau was grabbing it before pci-stub could. Putting nouveau.blacklist=1 in the grub file wasn't enough to blacklist the kernel drivers

instead I added a new file /etc/modprobe.d/nvidia-graphics-drivers.conf and put

Code: Select all

blacklist nouveau
blacklist lbm-nouveau
blacklist nvidia-173
blacklist nvidia-96
blacklist nvidia-current
blacklist nvidia-173-updates
blacklist nvidia-96-updates
alias nvidia nvidia_current_updates
alias nouveau off
alias lbm-nouveau off

( recommended from this http://askubuntu.com/questions/112302/h ... nel-driver. )
and the pci-stub was able to get the card. The kvm works now

Great news! I'm glad you got it working.

If possible, please post your Passmark or other benchmark results here: viewtopic.php?f=225&t=153482

[head_v_wall]

Hi Powerhouse, had been thinking my old XEN setup (following your previous guide) needed upgrading, then I found you wrote this...

my HW:
CPU: i7-3770
M/B: Asrock Z77 Extreme4-M
RAM: 16GB
Host GPU: integrated
VM GPU: AMD HD7950
Storage: 120G OCZ Agility 3 (for host) & Samsung EVO 850 (for guest)

*I'm also using the new Linux Mint 18 release

Couple things I had to do differently:


1 - change kernel parameters
radeon.blacklist=1 to radeon.modeset=0

With radeon.blacklist=1:
radeon does not get disabled, grabs half the GPU before pci-stub gets a chance, the pci-stub gets the audio-half

Code: Select all

$ dmesg
<...>
[    0.746882] radeon: unknown parameter 'blacklist' ignored
<...>
[    0.750454] radeon 0000:01:00.0: enabling device (0000 -> 0003)
<...>
[    4.295363] pci-stub: add 1002:679A sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    4.295367] pci-stub: add 1002:AAA0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    4.295420] pci-stub 0000:01:00.1: claimed by stub

confirmed with lspci:

Code: Select all

$ lspci -nnk
<...>
01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti PRO [Radeon HD 7950/8950 OEM / R9 280] [1002:679a]
	Subsystem: Gigabyte Technology Co., Ltd Tahiti PRO [Radeon HD 7950/8950 OEM / R9 280] [1458:254c]
	Kernel driver in use: radeon
	Kernel modules: radeon
01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series] [1002:aaa0]
	Subsystem: Gigabyte Technology Co., Ltd Tahiti XT HDMI Audio [Radeon HD 7970 Series] [1458:aaa0]
	Kernel driver in use: pci-stub
	Kernel modules: snd_hda_intel
<...>

Then with radeon.modeset=0:

Code: Select all

$ dmesg
<...>
[    0.745078] [drm:radeon_init [radeon]] *ERROR* No UMS support in radeon module!
<...>
[    1.207983] pci-stub: add 1002:679A sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    1.208036] pci-stub 0000:01:00.0: claimed by stub
[    1.208040] pci-stub: add 1002:AAA0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    1.208060] pci-stub 0000:01:00.1: claimed by stub
<...>

once more confirmed with lspci:

Code: Select all

$ lspci -nnk
<...>
01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti PRO [Radeon HD 7950/8950 OEM / R9 280] [1002:679a]
	Subsystem: Gigabyte Technology Co., Ltd Tahiti PRO [Radeon HD 7950/8950 OEM / R9 280] [1458:254c]
	Kernel driver in use: pci-stub
	Kernel modules: radeon
01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series] [1002:aaa0]
	Subsystem: Gigabyte Technology Co., Ltd Tahiti XT HDMI Audio [Radeon HD 7970 Series] [1458:aaa0]
	Kernel driver in use: pci-stub
	Kernel modules: snd_hda_intel
<...>

For reference my full kernel command line:

Code: Select all

GRUB_CMDLINE_LINUX_DEFAULT="radeon.modeset=0 quiet splash intel_iommu=on pci-stub.ids=1002:679a,1002:aaa0"

2 - Mess with the GPU passthrough (a lot!)
The pass-through initially worked fine for me, I just got stuck when installing the GPU drivers - the VM would immediately crash no matter what, then guest reboots after that would be highly unreliable and crash at random.
First I found: https://www.redhat.com/archives/vfio-us ... 00120.html
After putting my GPU behind a 'ioh3420' device, the VM did not crash, but the display never came back - so progress i guess?
Then: https://lists.fedoraproject.org/piperma ... 03911.html
Which lead on to: https://bbs.archlinux.org/viewtopic.php?pid=1361071
which has lead me to construct my current abomination of a qemu command-line (still in the win10vm.sh script):

Code: Select all

qemu-system-x86_64 \
  -name $vmname,process=$vmname \
  -cpu host \
  -machine type=q35,accel=kvm \
  -smp 4,sockets=1,cores=2,threads=2 \
  -enable-kvm \
  -m 4G \
  -mem-prealloc \
  -balloon none \
  -rtc clock=host,base=localtime \
  -vga none \
  -nographic \
  -serial none \
  -parallel none \
  -soundhw hda \
  -usb -usbdevice host:046d:c31d -usbdevice host:046d:c05a \
  -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
  -device vfio-pci,host=01:00.0,multifunction=on,bus=root.1,addr=00.0 \
  -device vfio-pci,host=01:00.1,bus=pcie.0,multifunction=on \
  -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \
  -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \
  -drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd \
  -drive if=pflash,format=raw,file=/tmp/my_vars.fd \
  -boot order=dc \
  -device virtio-scsi-pci,id=scsi \
  -drive id=disk0,if=virtio,cache=none,format=raw,file=/media/user/win.img \
  -drive file=/media/isos/Win10_1511_2_EnglishInternational_x64.iso,id=isocd,format=raw,if=none -device scsi-cd,drive=isocd \
  -drive file=/media/isos/virtio-win-0.1.118.iso,id=virtiocd,format=raw,if=none -device ide-cd,bus=ide.1,drive=virtiocd \
  -netdev type=tap,id=net0,ifname=tap0,vhost=on \
  -device virtio-net-pci,netdev=net0,mac=00:16:3e:00:01:01

Note: i have flashed the GPU rom to get UEFI support.

With that I am able to boot the VM, install windows AND install the GPU drivers!

From here I plan to experiment some more trying to trim down the command line to find what it was that eventually made the pass-through succeed, then i'll blow it all away and do a 'proper' LVM-based install, but thought I should share first

[powerhouse]

Hello head_v_wall, thanks for sharing your experiences and the workarounds.

It gave me a laugh when I read that radeon would grab "half the GPU" - thanks for pointing out the new radeon.modeset=0 grub option.

I myself am not that sure that switching from Xen to kvm improved performance. I actually really liked Xen and how well it performed. But some practical reasons brought me to use kvm.

Good luck with optimising your startup script, and please share when you are done. I did a lot of experimenting to get decent disk performance. In Xen it would just work using the PV drivers. With kvm I got a great variance of results, depending on the parameters in the qemu command. See viewtopic.php?f=225&t=153482&start=20#p1109525.

[powerhouse]

Those of you who followed the how-to and have a running Windows VM, please post some benchmarks here: viewtopic.php?f=231&t=197754.

[Ludo_Kressh]

Firstly, thank you for posting such a great guide, both powerhouse and O_Shovah for the same graphics card part.

I followed the guide perfect, very easy to do as its so well written, but I have a problem I cant seem to find a way around. I have tried many things and searched for different methods but cant get this bit solved.

So am simply trying to pass through one of my cards (both the same nvidia titans) to the VM and use the vfio-pci driver instead of the Nvidia nouveau one. But no matter what I do the nouveau driver always seems to take priority. I even tried to completely disable that to see if the vfio would take over it then, but in thay case no driver appears in the kernal driver in use output from

Code: Select all

lspci -nnk

I can show you any of the config files I have used, but they are all just variation on what the two of you have tried, and other people in other forums. I dont really understand why the vfio driver never seems to take the graphics card. Any help you or anyone else can give me would be appreciated.

[powerhouse]

Firstly, thank you for posting such a great guide, both powerhouse and O_Shovah for the same graphics card part.

I followed the guide perfect, very easy to do as its so well written, but I have a problem I cant seem to find a way around. I have tried many things and searched for different methods but cant get this bit solved.

So am simply trying to pass through one of my cards (both the same nvidia titans) to the VM and use the vfio-pci driver instead of the Nvidia nouveau one. But no matter what I do the nouveau driver always seems to take priority. I even tried to completely disable that to see if the vfio would take over it then, but in thay case no driver appears in the kernal driver in use output from

Code: Select all

lspci -nnk

I can show you any of the config files I have used, but they are all just variation on what the two of you have tried, and other people in other forums. I dont really understand why the vfio driver never seems to take the graphics card. Any help you or anyone else can give me would be appreciated.

Look at the top of this page, the post by craigbett here.

The nouveau driver can be nasty and you need to assign the vfio-pci driver before nouveau kicks in. I have two Nvidia cards and use the Nvidia proprietary driver for Linux. Here is my configuration:

In /etc/modprobe.d create or edit the file local.conf:

Code: Select all

install vfio-pci /sbin/vfio-pci-override-vga.sh

In /sbin create the file vfio-pci-override-vga.sh:

Code: Select all

#!/bin/sh

DEVS="0000:02:00.0 0000:02:00.1"

for DEV in $DEVS; do
    echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override
done

modprobe -i vfio-pci

and replace the DEVS=... with the PCI IDs of the graphics card you want to pass through and make the file executable.

Edit the /etc/default/grub file. You should have a line like that:

Code: Select all

GRUB_CMDLINE_LINUX_DEFAULT="nouveau.blacklist=1 quiet nomodeset intel_iommu=on"

If not, edit and add nouveau.blacklist=1 or nouveau.modeset=0, whatever works for you (the latter option might be needed for newer kernels, I'm running a 3.19 kernel).

I also have the /etc/modprobe.d/nvidia-graphics-drivers.conf file, which in my case was created when I installed the proprietary Nvidia driver onto Linux. It reads:

Code: Select all

# This file was installed by nvidia-352
# Do not edit this file manually

blacklist nouveau
blacklist lbm-nouveau
blacklist nvidia-current
blacklist nvidia-173
blacklist nvidia-96
blacklist nvidia-current-updates
blacklist nvidia-173-updates
blacklist nvidia-96-updates
blacklist nvidia-352-updates
alias nvidia nvidia_352
alias nvidia-uvm nvidia_352-uvm
alias nouveau off
alias lbm-nouveau off

Let us know if this works.

[Ludo_Kressh]

Thank you for such a quick reply.

I just went through what you suggested and aside from having nouveau.blacklist=1, everything you showed I already had in from the previous part of the guide. I have after I posted changed to the nvidia driver away from the nouveau driver, so I just added nouveau.blacklist=1, but the same thing. They are now bound to the nvidia driver.

I just cant seem to get it to ever go to the vfio one. If there is anything that will help you understand please let me know. Im sure its something very simple but Im at that state where I dont think I would ever find it.

[Ludo_Kressh]

Just been experiementing and I stripped down another system of mine and stuck a 960 in, and since its a different card I used the first method you showed and that worked first time no problems, straight to the pci-stub driver

Code: Select all

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK110 [GeForce GTX TITAN] [10de:1005] (rev a1)
	Subsystem: eVga.com. Corp. GeForce GTX Titan SC [3842:2791]
	Kernel driver in use: nvidia
	Kernel modules: nvidiafb, nouveau, nvidia_361
01:00.1 Audio device [0403]: NVIDIA Corporation GK110 HDMI Audio [10de:0e1a] (rev a1)
	Subsystem: eVga.com. Corp. GK110 HDMI Audio [3842:2791]
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel
02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK110 [GeForce GTX TITAN] [10de:1005] (rev a1)
	Subsystem: eVga.com. Corp. GeForce GTX Titan SC [3842:2791]
	Kernel driver in use: nvidia
	Kernel modules: nvidiafb, nouveau, nvidia_361
02:00.1 Audio device [0403]: NVIDIA Corporation GK110 HDMI Audio [10de:0e1a] (rev a1)
	Subsystem: eVga.com. Corp. GK110 HDMI Audio [3842:2791]
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel
04:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM206 [GeForce GTX 960] [10de:1401] (rev a1)
	Subsystem: eVga.com. Corp. GM206 [GeForce GTX 960] [3842:2966]
	Kernel driver in use: pci-stub
	Kernel modules: nvidiafb, nouveau, nvidia_361
04:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fba] (rev a1)
	Subsystem: eVga.com. Corp. Device [3842:2966]
	Kernel driver in use: pci-stub
	Kernel modules: snd_hda_intel

So at least that works, Ill play some more and see if I can get this one now to go with the vfio driver using the PCI bus address instead of the IDs.


..... No it just went back to the nvidia driver..


Ive been spending a few hours on this today and I think I am close to identifying the problem. Whenm I run the windows10vm.sh file I get the following output

Code: Select all

$ sudo /etc/windows7vm.sh 
/sbin/vfio-pci-override-vga.sh: 10: /sbin/vfio-pci-override-vga.sh: Syntax error: Unterminated quoted string
modprobe: ERROR: ../libkmod/libkmod-module.c:977 command_do() Error running install command for vfio_pci
modprobe: ERROR: could not insert 'vfio_pci': Unknown symbol in module, or unknown parameter (see dmesg)
Plugging 0000:04:00.0 into vfio-pci
/etc/windows7vm.sh: line 20: /sys/bus/pci/drivers/vfio-pci/new_id: No such file or directory
Plugged 0000:04:00.0 into vfio-pci
Plugging 0000:04:00.1 into vfio-pci
/etc/windows7vm.sh: line 20: /sys/bus/pci/drivers/vfio-pci/new_id: No such file or directory
Plugged 0000:04:00.1 into vfio-pci

Showing, to me at least, that something is wrong in the layout of the vfio-pci-override-vga.sh file and the modprode, local.conf set up

Code: Select all

install vfio-pci /sbin/vfio-pci-override-vga.sh

Code: Select all

#!/bin/sh

DEVS="0000:04:00.0 0000:04:00.1""

for DEV in $DEVS; do
    echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override
done

modprobe -i vfio-pci

As far as I can tell I havent made any mistakes in there so is ther something wrong with the installation command for vfio-pci?

[O_Shovah]

Hello Everybody,
Im reporting back from my exams and on my endevour to a KVM machine.

This post describes the Variant B of how to get two identical GPUs sorted for KVM using vfio-pci.

Version B :

In contrast to my first guide Version A this version uses the default tools included in Ubuntu 16.04 in covergence with powerhouse's setup.
This example uses AMD GPUs. Modify the grub example accordingly if you use Nvidia or other Manufacturers.

So first modify grub with sudo nano /etc/default/grub

Code: Select all

GRUB_DEFAULT=0
GRUB_HIDDEN_TIMEOUT=0
GRUB_HIDDEN_TIMEOUT_QUIET=true
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet radeon.modeset=0 intel_iommu=on"
GRUB_CMDLINE_LINUX=""

The paramter radeon.modeset=0 prevents the loading of all radeon drivers
until the X-Server is started(Namely your Desktop comes up). Therefore the boot up time
can be used by the vfio-pci driver to grab the GPU we want to use in the KVM.

Rebuild grub:

Code: Select all

sudo update-grub

Next add the modules we want to get loaded at boot in their respective order to /etc/modules
sudo nano /etc/modules

Code: Select all

# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
vfio
vfio_iommu_type1
vfio_pci
vhost-net

Create the file /etc/modprobe.d/local.conf and add:

Code: Select all

install vfio-pci /sbin/vfio-pci-override-vga.sh

Now look up the devices you want to be locked by vfio-pci to be later passed through using:

Code: Select all

lspci -nn

Wich yields in my case:

Code: Select all

04:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii PRO [Radeon R9 290] [1002:67b1]
04:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii HDMI Audio [1002:aac8]
05:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller [1b21:1242]
06:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii PRO [Radeon R9 290] [1002:67b1]
06:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii HDMI Audio [1002:aac8]

Remember to pass through both the graphics and the audio in case of a graphics card.

Create the following script:
/sbin/vfio-pci-override-vga.sh

Code: Select all

#!/bin/sh

DEVS="0000:06:00.0 0000:06:00.1"

for DEV in $DEVS; do
    echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override
done

modprobe -i vfio-pci

And replace the DEV with the ones of the devices you want to pass through. This works for both graphics cards and other devices like sound cards.

And make it executable:

Code: Select all

sudo chmod 755 /sbin/vfio-pci-override-vga.sh

Update yout initramfs :

Code: Select all

sudo update-initramfs -u

Now reboot your machine

Now run lspci -nk it should yield an output alike:

Code: Select all

04:00.0 0300: 1002:67b1
	Subsystem: 1462:3081
	Kernel modules: radeon
04:00.1 0403: 1002:aac8
	Subsystem: 1462:aac8
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel
06:00.0 0300: 1002:67b0
	Subsystem: 174b:e285
	Kernel driver in use: vfio-pci
	Kernel modules: radeon
06:00.1 0403: 1002:aac8
	Subsystem: 174b:aac8
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel

With the vfio-pci driver having grabbed the card you specified in the sbin file
and the Video output on the specified card having ceased.

In the hope to have given a guide to someone with a similar problems i had

[O_Shovah]

And now to the more specific path of mine.
I finally found a problem that might have kept me from getting a KVM to run in the first place.

Both my graphics cards did not posses a UEFI BIOS Neither the R9 290 nor the R9 290X.
It seems i never checked properly. -> Skipping steps in Manuals -> not so smart....

But thanks to the people over at Techpowerup i sucessfully flashed my card with
a UEFI BIOS. Still the ability to read the bios from the card as in STEP 1 works at times and
at times it just yields

Code: Select all

cat rom > /tmp/image.rom
cat: rom: Input/output error

Use the great Techpowerup BIOS database
and the very good AMD/ATI Flashing Guide
in case you experience a similar problem.
Nontheless: i take no responsibility for any potential damage to your card

With the vfio-pci driver now properly grabbing my R9 290X card using standard tools and the card
fielding a proper UEFI BIOS i will go on in the hope to get thins working soon

[peavey2787]

I've followed this tutorial and several others now and I can get everything to work up until its time to run the actual VM. qemo starts up but I have no video signal. I've tried with a gtx 960 and an 8800gs. Any ideas as to what could be wrong? I got no one to help Thanks for your time.

[powerhouse]

I've followed this tutorial and several others now and I can get everything to work up until its time to run the actual VM. qemo starts up but I have no video signal. I've tried with a gtx 960 and an 8800gs. Any ideas as to what could be wrong? I got no one to help Thanks for your time.

Sometimes it's a simple thing:

Have you switched your screen to the output of the second graphics card? Make sure the cable is properly connected. If you have only one screen, you need to connect two cables coming from the two graphics cards.

If you have two screens, the second screen should show the output of the Windows VM. Make sure everything is connected properly and turned on.

And finally, make sure your graphics card actually supports UEFI. See the post right above.

[powerhouse]

And now to the more specific path of mine.
I finally found a problem that might have kept me from getting a KVM to run in the first place.

Both my graphics cards did not posses a UEFI BIOS Neither the R9 290 nor the R9 290X.
It seems i never checked properly. -> Skipping steps in Manuals -> not so smart....

But thanks to the people over at Techpowerup i sucessfully flashed my card with
a UEFI BIOS. Still the ability to read the bios from the card as in STEP 1 works at times and
at times it just yields

Code: Select all

cat rom > /tmp/image.rom
cat: rom: Input/output error

Use the great Techpowerup BIOS database
and the very good AMD/ATI Flashing Guide
in case you experience a similar problem.
Nontheless: i take no responsibility for any potential damage to your card

With the vfio-pci driver now properly grabbing my R9 290X card using standard tools and the card
fielding a proper UEFI BIOS i will go on in the hope to get thins working soon

Hi O_Shovah: Thanks for the update!

Yep, my tutorial won't work unless you got an UEFI BIOS in your graphics card. Thanks for the links! I'm sure some readers will find them very useful.

[Ludo_Kressh]

I finally got mine working with two simple changes to force the vfio to load first.

Firstly I changed the /etc/modules file to this

Code: Select all

lp
rtc
pci_stub
vfio
vfio_iommu_type1
vfio_pci
kvm
kvm_intel

Then the second is the main one, changing the /sbin/vfio-pci-override-vga.sh script to this

Code: Select all

#!/bin/sh

for i in $(find /sys/devices/pci* -name boot_vga); do
        if [ $(cat $i) -eq 0 ]; then
                GPU=$(dirname $i)
                AUDIO=$(echo $GPU | sed -e "s/0$/1/")
                echo "vfio-pci" > $GPU/driver_override
                if [ -d $AUDIO ]; then
                        echo "vfio-pci" > $AUDIO/driver_override
                fi
        fi
done

modprobe -i vfio-pci

This forces the binding of any non-boot gpu to the vfio driver. These two together work for me and I now have a fully functional VM. No idea why the normal method wouldnt work.

Thank you for the guide and your help

[powerhouse]

@Ludo_Kressh: Sorry for not noticing this earlier. I just saw that you had posted an error log and looking at the log and your old /sbin/vfio-pci-override-vga.sh script is saw the problem:

Code: Select all

DEVS="0000:04:00.0 0000:04:00.1""

There are two double quotes ("") at the end of the line - it should be only one quote ("). It was a simple syntax error.

[powerhouse]

I finally got mine working with two simple changes to force the vfio to load first.

Firstly I changed the /etc/modules file to this

Code: Select all

lp
rtc
pci_stub
vfio
vfio_iommu_type1
vfio_pci
kvm
kvm_intel

Then the second is the main one, changing the /sbin/vfio-pci-override-vga.sh script to this

Code: Select all

#!/bin/sh

for i in $(find /sys/devices/pci* -name boot_vga); do
        if [ $(cat $i) -eq 0 ]; then
                GPU=$(dirname $i)
                AUDIO=$(echo $GPU | sed -e "s/0$/1/")
                echo "vfio-pci" > $GPU/driver_override
                if [ -d $AUDIO ]; then
                        echo "vfio-pci" > $AUDIO/driver_override
                fi
        fi
done

modprobe -i vfio-pci

This forces the binding of any non-boot gpu to the vfio driver. These two together work for me and I now have a fully functional VM. No idea why the normal method wouldnt work.

Thank you for the guide and your help

Thanks for sharing your solution. I think it would have worked with the script in my tutorial, once the syntax error is fixed. Could you test that?

Also, I doubt you need the pci_stub module.

The script you listed has one advantage: you don't need to look up the graphics cards pci bus ids as it just searches for non-boot graphics cards and binds them all to vfio-pci. In most cases this is fine, unless you use multiple GPUs in your host. I have to think about that and I might want to replace the script I posted with yours.

[powerhouse]

Just for information: Alex Williamson, a kvm developer, has written a How-To series on VGA passthrough that can be found at http://vfio.blogspot.com/2015/05/vfio-g ... e.html?m=0.

The downside of his tutorial is that it's Fedora-based. The part of creating a VM using libvirt and virt-manager, though it looks easy using the GUI, can be a real challenge. If you try that, use the latest Linux Mint 18 and kernel 4.x as well as a recent release of virt-manager.

Alex' tutorial excels in in-depth explanations and is well worth reading, even if you follow my how-to.

[Ludo_Kressh]

The guide by Alex Williamson is the one that I combined with yours to get mine to work.

The syntax error, I knew it would be something so simple. If O_Shovah reads this I copied that from his guide so if you have chance edit that part so no more fools like me copy it.

I am a little busy but as soon as I get chance I was going to experiment more and I will try again with the method you showed without the mistakes. I had the pci-stub in there just because I was passing a keyboard/mouse receiver through using that.

Thank you for taking the time to reply, and Ill come back once I have played some more. Another thing I was going to try is to pass through multiple SSDs. Got one working earlier this week, so going to try a few more.