HOW-TO make dual-boot obsolete using kvm VGA passthrough

Questions about virtualization software
Forum rules
Before you post read how to get help. Topics in this forum are automatically closed 6 months after creation.
steakikan

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by steakikan »

powerhouse wrote:
steakikan wrote:
steakikan wrote:Ah I see, that's mean I have to use some other means to shutdown the guest os. One more question, I have difficulty in connecting the network connection, can you help? In qemu there is an error message : W: /etc/qemu-ifup: no bridge for guest interface found. Thanks for your help.
Just updating my own post, there is a way for safe shutdown guest using telnet protocol and qmp, i haven't found a way to lock the telnet to localhost but the idea is to use the telnet json protocol to send shutdown command and wait for qmp to reply then you execute the host shutdown command.
Thanks for this - very helpful.

Did you manage to solve the PCI passthrough issue?
Yep it works, I stupidly passing through pci bridges and that's the cause of the error :mrgreen:
For those interested in safe shutdown you can get some refference from here
http://serverfault.com/questions/563724 ... s-finished
I still not sure how to limit ip address so that qmp telnet can be accessed locally but this is my current script for safe shutdown (I separate it into 2 file so that I can choose which to shutdown)
Add the following configuration to qemu start script

Code: Select all

  -qmp tcp:127.0.0.1:4444,server,nowait \
Guest shutdown code (guestshutdown.sh)

Code: Select all

#/bin/bash

set timeout -1

spawn telnet 127.0.0.1 4444

expect "QMP"
send "{ 'execute': 'qmp_capabilities' }\n"

expect "return"
send "{ 'execute': 'system_powerdown' }\n"

expect "SHUTDOWN"
Guest+host shutdown code

Code: Select all

#/bin/bash
# int-or-string.sh

if [[ $UID != 0 ]]; then
    echo "Please run this script with sudo:"
    echo "sudo $0 $*"
    exit 1
fi


echo "Shutting down vm"
./guest.sh

echo "done vm shutdown"

echo "shutting down host"
shutdown -h now

If anyone can find out how to limit the telnet access to localhost that will be better security wise. Hopefully this helps and cmiiw.
powerhouse
Level 6
Level 6
Posts: 1138
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

Trainguyrom wrote:Hey, it looks like I'm having issues with both of my graphics cards being in the same iommu group, as well as the network bridge failing.

Code: Select all

W: /etc/qemu-ifup: no bridge for guest interface found
QEMU 2.1.2 monitor - type 'help' for more information
(qemu) qemu-system-x86_64: -device vfio-pci,host=02:00.0,multifunction=on: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,multifunction=on: vfio: failed to get group 1
qemu-system-x86_64: -device vfio-pci,host=02:00.0,multifunction=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,multifunction=on: Device 'vfio-pci' could not be initialized
Looking in /sys/bus/pci/devices/<device name>/iommu_group/devices for both cards. I see everything is in the same group. It looks like I need to move 0000:02:00.0 and 0000:02:00.1 to their own group.
Regarding the IOMMU issue, you can try to move one of the cards to a difference PCIe slot. See also http://vfio.blogspot.co.il/2014/08/iomm ... d-out.html for an indepth explanation. I'm not sure the current kernel already has the ACS override patch, so if repositioning of the graphics card doesn't help, you may want to check that out (that is, patch the kernel).
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
powerhouse
Level 6
Level 6
Posts: 1138
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

@steakikan: Thanks for the update and the script (which I hope to bring to use in my own box). I haven't had much time lately, and will be on tour for nearly a month, so I doubt I will be able to help on the Telnet issue. I'm still thinking about some guest additions, but am not sure.

EDIT: Perhaps this helps: https://technet.microsoft.com/en-us/lib ... 10%29.aspx Check the registry settings part.
EDIT 2: http://serverfault.com/questions/562992 ... s-in-cisco - this looks even more to the point.
(I believe you want to restrict access to the Windows Telnet server to localhost.)

Glad you solved the other issues! :D
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
O_Shovah

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by O_Shovah »

@ powerhouse

I'm reporting back from my issue with the Seabios for Windows 7.

I was abled to install Windows 7 using Seabios and Windows 8.1 using Uefi bios.

Both system could run with the connected Monitor an install regular windowss software.
But as soon as i installed the Radeon drivers i got a "nice" pink striped screen on the next boot.

In addition i got my hands on a second Radeon R9 290 so now both the Client an the Host will have access to some
video power.
But thereby i gained another luxury problem:
Both cards have the same pci_id and therefore i can either lock both with pci_stub or none of them.

I've read about using xen-pciback for this purpose, but i havent tried this so far.
Do you know of any other way to choose one of the cards for a specific purpose ?
powerhouse
Level 6
Level 6
Posts: 1138
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

O_Shovah wrote:@ powerhouse

I'm reporting back from my issue with the Seabios for Windows 7.

I was abled to install Windows 7 using Seabios and Windows 8.1 using Uefi bios.

Both system could run with the connected Monitor an install regular windowss software.
But as soon as i installed the Radeon drivers i got a "nice" pink striped screen on the next boot.

In addition i got my hands on a second Radeon R9 290 so now both the Client an the Host will have access to some
video power.
But thereby i gained another luxury problem:
Both cards have the same pci_id and therefore i can either lock both with pci_stub or none of them.

I've read about using xen-pciback for this purpose, but i havent tried this so far.
Do you know of any other way to choose one of the cards for a specific purpose ?
xen-pciback in fact uses the PCI bus number and that would work. You can find more about it in my Xen VGA passthrough tutorial (see sticky thread, steps 8-11 - viewtopic.php?f=231&t=112013#p628768). I haven't tried xen-pciback with kvm, though.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
steakikan

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by steakikan »

powerhouse wrote:@steakikan: Thanks for the update and the script (which I hope to bring to use in my own box). I haven't had much time lately, and will be on tour for nearly a month, so I doubt I will be able to help on the Telnet issue. I'm still thinking about some guest additions, but am not sure.

EDIT: Perhaps this helps: https://technet.microsoft.com/en-us/lib ... 10%29.aspx Check the registry settings part.
EDIT 2: http://serverfault.com/questions/562992 ... s-in-cisco - this looks even more to the point.
(I believe you want to restrict access to the Windows Telnet server to localhost.)

Glad you solved the other issues! :D
Yep, at least I can safely remote shutdown guest vm from ssh. Currently there is some quirk though, if the vm screen is dimmed/turned off, you need to send 2 acpi command to shutdown (which can easily solved by adding another acpi command before expect, but not the prettiest solution). I just realized the telnet is actually blocked by default in linux so nothing to worry.
mani

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by mani »

Just in case anyone else has the same two problems....

I could not for the life of me get any drive to boot using the ovmf files linked on the edk site ( they're rebuilt daily at least, and I see no archive ).

After pulling my hair out for a couple days I found http://www.ubuntuupdates.org/package/co ... /base/ovmf and found a build from Jan 2015...a little old, but it can boot drives at least. To be clear the only difference between working and not working is this file.

Second problem is that I have two gtx 970's with the same pciid's. I got passed this by not using the pci-stub method and following http://turlucode.com/qemu-kvm-on-arch-l ... revfio-pci

So my script differs in that I'm not using the config file or binding the vfio drivers ( the card comes up bound to vfio on boot ).
powerhouse
Level 6
Level 6
Posts: 1138
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

mani wrote:Just in case anyone else has the same two problems....

I could not for the life of me get any drive to boot using the ovmf files linked on the edk site ( they're rebuilt daily at least, and I see no archive ).

After pulling my hair out for a couple days I found http://www.ubuntuupdates.org/package/co ... /base/ovmf and found a build from Jan 2015...a little old, but it can boot drives at least. To be clear the only difference between working and not working is this file.

Second problem is that I have two gtx 970's with the same pciid's. I got passed this by not using the pci-stub method and following http://turlucode.com/qemu-kvm-on-arch-l ... revfio-pci

So my script differs in that I'm not using the config file or binding the vfio drivers ( the card comes up bound to vfio on boot ).
Hello mani, thanks for the information! The links you posted are very useful. I wasn't aware of the ovmf files causing problems, and I will incorporate your link into my tutorial.

There are different methods to assign drivers to the graphics cards or other PCI devices. pci-stub uses the PCI vendor ID, which is useless if you have two of the same devices (like two identical graphics cards). Another method I used with Xen is xen-pciback (see viewtopic.php?f=231&t=112013 step 10 and 11), which uses PCI bus IDs. If I find the time I will try your suggested method and add it to the tutorial.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
powerhouse
Level 6
Level 6
Posts: 1138
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

O_Shovah wrote:@ powerhouse

I'm reporting back from my issue with the Seabios for Windows 7.

I was abled to install Windows 7 using Seabios and Windows 8.1 using Uefi bios.

Both system could run with the connected Monitor an install regular windowss software.
But as soon as i installed the Radeon drivers i got a "nice" pink striped screen on the next boot.

In addition i got my hands on a second Radeon R9 290 so now both the Client an the Host will have access to some
video power.
But thereby i gained another luxury problem:
Both cards have the same pci_id and therefore i can either lock both with pci_stub or none of them.

I've read about using xen-pciback for this purpose, but i havent tried this so far.
Do you know of any other way to choose one of the cards for a specific purpose ?
Forum member mani posted a link to a tutorial using the driver override feature in kernel 3.16 and above together with vfio-pci that would solve the pci_stub limitations - see http://turlucode.com/qemu-kvm-on-arch-l ... revfio-pci.

About the pink screen: What happens when you shut down both Windows and Linux and reboot? There is a known issue with AMD graphics cards where you can boot the Windows guest once, but you cannot reboot the Windows VM a second time. Is that the issue you are facing? As far as I remember this issue is specific to SeaBios boot, and may not occur when using the UEFI method. It has to do with resetting the graphics card after the Windows VM shuts down. In other words, if your Windows VM boots fine the first time you turn on the PC, but not when you shut Windows down and boot a second time, it's most likely related to this bug. There are work-arounds such as a Windows script that "ejects" the graphics card on Windows shutdown, which will trigger a card reset and allow a Windows reboot. Hope this helps.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
odarnat

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by odarnat »

Hi

I'm a bit new at this but I need your lights with my setup.
Great post by the way... if only someone could make an app to streamline the process it would be amazing..
So i'm getting very close to installing (Part 8), and I get a message that the vfio-pci I try to bind to my VM is not in the right group (because it is the same group as my other graphic card, used by the host.
Both my GTX 780 and my GT 610 are on group 1, I did tried to unbound one to assign it to a different group but it just does not.
This is what i get when a start the VM :

Code: Select all

W: /etc/qemu-ifup: no bridge for guest interface found
QEMU 2.1.2 monitor - type 'help' for more information
(qemu) qemu-system-x86_64: -device vfio-pci,host=02:00.0: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
qemu-system-x86_64: -device vfio-pci,host=02:00.0: vfio: failed to get group 1
qemu-system-x86_64: -device vfio-pci,host=02:00.0: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=02:00.0: Device 'vfio-pci' could not be initialized
I know i'm really close to get it to work, I just need to separate the 2 graphic cards in the IOMMU GROUPS.

Please help.

Olivier
(Sorry if my english is bad, Froggy French here ;-)
powerhouse
Level 6
Level 6
Posts: 1138
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

@odarnat: You are right - you need to separate the two graphics cards into different IOMMU groups. You will need to install the ACS override patch. See the links I posted above and in the tutorial. I hope to have a look at it sometime during the week (am in a hurry now).

See http://vfio.blogspot.co.il/2014/08/vfiovga-faq.html !!!
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
TDash

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by TDash »

How exactly can I use my integrated GPU (Intel i5-4460) and discrete GPU (NVIDIA GTX 970) together? Do I have to get the iGPU working first? Do I need to mess around with drivers? My kernel version is 4.4.0-18. I tried applying the VGA arbiter patch with

Code: Select all

sudo modprobe i915 enable_hd_vgaarb=1
to no avail. Before switching my NVIDIA drivers to noveau, I had to change the grub config to have nomodeset after quiet splash to get past a black screen when I enabled iGPU Multi Monitor, which enables the iGPU. Thanks in advance!
O_Shovah

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by O_Shovah »

Reporting back on the Matter of running two identical Graphics card on my Ubuntu 16.04.

I have two Radeon R9 290 graphics cards on my machine an finally found
a way to lock them from the radeon driver individually using vfio-pci instead of pci-stub.

There are two ways to get this working.
Version A is described below and uses the external dracut package to specify the nessesary scripts.

And Version B using the Mint/Ubuntu default tools including the info of powerhouse about two
post below this one, for the same cause.


So for anybody struggling with passing through devices with identical pci-id's try the following
modification to powerhouse's manual replacing the pci-stub part:

Please Note !!! :

This actions require at least kernel 3.19+ and a exchange of the initramfs-tools stack against dracut


First install the dracut package for modifying the initramfs image:

Code: Select all

sudo apt-get install dracut dracut-core
Next modify /etc/default/grub and add rd.driver.pre=vfio-pci the line GRUB_CMDLINE_LINUX=
So you get a grub file similar to this:

Code: Select all

GRUB_DEFAULT=0
GRUB_HIDDEN_TIMEOUT=0
GRUB_HIDDEN_TIMEOUT_QUIET=true
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
GRUB_CMDLINE_LINUX="intel_iommu=on "
Rebuild grub:

Code: Select all

sudo update-grub
For more information on the dracut package and its Grub commandline options see:
dracut-core
dracut.cmdline

Create the file /etc/modprobe.d/local.conf and add:

Code: Select all

install vfio-pci /sbin/vfio-pci-override-vga.sh
Now look up the devices you want to be locked by vfio-pci to be later passed through using:

Code: Select all

lspci -nn


Wich yields in my case:

Code: Select all

04:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii PRO [Radeon R9 290] [1002:67b1]
04:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii HDMI Audio [1002:aac8]
05:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller [1b21:1242]
06:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii PRO [Radeon R9 290] [1002:67b1]
06:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii HDMI Audio [1002:aac8]
Remember to pass through both the graphics and the audio in case of a graphics card.

Create the following script:
/sbin/vfio-pci-override-vga.sh

Code: Select all

#!/bin/sh

DEVS="0000:06:00.0 0000:06:00.1""

for DEV in $DEVS; do
    echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override
done

modprobe -i vfio-pci
And replace the DEV with the ones of the devices you want to pass through. This works for both graphics cards and other devices like sound cards.

And make it executable:

Code: Select all

sudo chmod 755 /sbin/vfio-pci-override-vga.sh
Create the file /etc/dracut.conf.d/local.conf and add:

Code: Select all

add_drivers+="vfio vfio_iommu_type1 vfio_pci vfio_virqfd"
install_items+="/sbin/vfio-pci-override-vga.sh /usr/bin/find /usr/bin/dirname"
Regenerate initramfs via dracut using:

Code: Select all

dracut -f --kver `uname -r`
Check if the script was sucessfully added:

Code: Select all

sudo lsinitramfs -l /boot/initramfs-4.4.0-22-generic.img | grep vfio
Adapt your kernel version accordingly.

It should show all scripts and modules you just added:

Code: Select all

drwxr-xr-x   3 root     root            0 Jun  2 00:04 lib/modules/4.4.0-22-generic/kernel/drivers/vfio
drwxr-xr-x   2 root     root            0 Jun  2 00:04 lib/modules/4.4.0-22-generic/kernel/drivers/vfio/pci
-rw-r--r--   1 root     root        72390 May 13 01:57 lib/modules/4.4.0-22-generic/kernel/drivers/vfio/pci/vfio-pci.ko
-rw-r--r--   1 root     root        46078 May 13 01:57 lib/modules/4.4.0-22-generic/kernel/drivers/vfio/vfio.ko
-rw-r--r--   1 root     root        23126 May 13 01:57 lib/modules/4.4.0-22-generic/kernel/drivers/vfio/vfio_iommu_type1.ko
-rw-r--r--   1 root     root        11486 May 13 01:57 lib/modules/4.4.0-22-generic/kernel/drivers/vfio/vfio_virqfd.ko
-rwxr-xr-x   1 root     root          157 Jun  1 01:43 sbin/vfio-pci-override-vga.sh
Now Reboot

And check if the vfio-pci was loaded as intendend:

Code: Select all

lspci -nnk
Showing:

Code: Select all

04:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii PRO [Radeon R9 290] [1002:67b1]
	Subsystem: Hightech Information System Ltd. Hawaii PRO [Radeon R9 290] [1787:2343]
	Kernel driver in use: radeon
	Kernel modules: radeon
04:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii HDMI Audio [1002:aac8]
	Subsystem: Hightech Information System Ltd. Hawaii HDMI Audio [1787:aac8]
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel
05:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller [1b21:1242]
	Subsystem: ASUSTeK Computer Inc. ASM1142 USB 3.1 Host Controller [1043:8696]
	Kernel driver in use: xhci_hcd
06:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii PRO [Radeon R9 290] [1002:67b1]
	Subsystem: Micro-Star International Co., Ltd. [MSI] Hawaii PRO [Radeon R9 290] [1462:3081]
	Kernel driver in use: vfio-pci
	Kernel modules: radeon
06:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii HDMI Audio [1002:aac8]
	Subsystem: Micro-Star International Co., Ltd. [MSI] Hawaii HDMI Audio [1462:aac8]
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel
It should have attached to the devices you had set in the vfio-pci-override-vga.sh file.

I hope this may solve the problem for those of you with dual graphics cards.

References taken:
VFIO tips and tricks
vfio-pci users maillinglist
Last edited by O_Shovah on Mon Aug 01, 2016 6:37 pm, edited 4 times in total.
powerhouse
Level 6
Level 6
Posts: 1138
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

Thanks O_Shovah !!! This is very useful.

I'm actually using the vfio-pci driver on my system. When I read your earlier post about your problem running identical graphics cards, I checked my configuration and wanted to share it here. Unfortunately travel, moving to a new apartment and a ton of things that took my time prevented me from posting here. I'm so glad you found a solution on your own.

EDIT: I added a link to your post into my tutorial (see part 4).
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
powerhouse
Level 6
Level 6
Posts: 1138
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

O_Shoval has posted an important update on my tutorial, which allows the use of identical graphics cards - one for the Linux host, the other for the Windows guest. See 2 posts above.

Here are some comments to that update:

1. I'm using the vfio-pci driver on kernel 3.19, NOT 4.1+ and it works like a charm.
2. I do NOT require the dracut package in my configuration.
3. My /etc/default/grub file does NOT have this entry: rd.driver.pre=vfio-pci

Explanation:
The pci-stub method uses PCI vendor IDs to identify the devices to bind to. As long as you have different graphics cards vendors and/or models, that will work. But if you use identical graphics cards for both Linux and the Windows VM, you cannot tell them apart using the pci-stub method. The vfio-pci driver uses PCI bus identifications such as DEVS="0000:06:00.0 0000:06:00.1". These are unique for each device.

Here is my /etc/default/grub file:

Code: Select all

GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=countdown
GRUB_TIMEOUT=0
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet nomodeset intel_iommu=on"
GRUB_CMDLINE_LINUX=""
My /etc/modprobe.d/local.conf file:

Code: Select all

install vfio-pci /sbin/vfio-pci-override-vga.sh
And my /sbin/vfio-pci-override-vga.sh file:

Code: Select all

#!/bin/sh

DEVS="0000:02:00.0 0000:02:00.1"

for DEV in $DEVS; do
    echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override
done

modprobe -i vfio-pci
This last one needs to be executable.

As far as I know, these are all the modifications to my original tutorial to make it work with identical graphics cards.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
O_Shovah

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by O_Shovah »

@powerhouse

I will update my part tomorrow accordingly. I preffer the dracut module as i fell it to be more comfortable but thats subjective.
And will try to test your method as well.
As software in general seems not to like me ;) i am still en route to my VM . Having passed the vfio issue i still dont get a VGA output on the screen
although when instlaling windows 10 using the emulated graphics cards for debug i can confirm both the VGA PCI and the USB controller have been sucessfully passed through. Atempts to install radeon drivers on the VM fail for obvious reasons.
I will keep looking into that problem.
powerhouse
Level 6
Level 6
Posts: 1138
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

O_Shovah wrote:@powerhouse

I will update my part tomorrow accordingly. I preffer the dracut module as i fell it to be more comfortable but thats subjective.
And will try to test your method as well.
As software in general seems not to like me ;) i am still en route to my VM . Having passed the vfio issue i still dont get a VGA output on the screen
although when instlaling windows 10 using the emulated graphics cards for debug i can confirm both the VGA PCI and the USB controller have been sucessfully passed through. Atempts to install radeon drivers on the VM fail for obvious reasons.
I will keep looking into that problem.
Well, good luck! Does the vfio-pci driver attach to your graphics card? You also must have the drivers ready to install when you install Windows.

Since my method uses UEFI, your graphics card must support UEFI boot. I have explained how to check that. If it doesn't, this method will fail. I provided a link to another how-to that describes how to flash another BIOS to the graphics card, if needed. However, that should be the very last step and only if determined necessary. The risk to brick a graphics card are high! I won't take any responsibility.

However, if your card can UEFI boot then it may be an issue with either the OVMF file (check that you downloaded the right file, and download the latest that was published today). It also may have to do with IOMMU groups - check that each graphic card is in a different IOMMU group, and make sure to pass through all components in that IOMMU group (usually video and sound). If nothing helps, there are some kernel patches that may help (these are CPU specific issues, so you should search for your CPU and VGA passthrough). If your CPU has issues with passthrough, you should find explanations and patches via a Google search.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
craigbett

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by craigbett »

This is a wonderful tutorial, thanks for sharing. Very easy to understand for beginners/intermediate Linux users. but I've run into an issue. This is a fresh install of Linux Mint 18 Cinnamon and when I attempt to start up the vm the console says :

Code: Select all

./windows.sh: line 64:  3652 Done                    cat $configfile
      3653 Killed                  | while read line; do
    echo $line | grep ^# > /dev/null 2>&1 && continue; vfiobind $line;
done
W: /etc/qemu-ifup: no bridge for guest interface found
QEMU 2.5.0 monitor - type 'help' for more information
(qemu) qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: vfio: error opening /dev/vfio/1: No such file or directory
qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: vfio: failed to get group 1
qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: Device initialization failed
Anyone else run into this issue and/or know how to fix it?
powerhouse
Level 6
Level 6
Posts: 1138
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

craigbett wrote:This is a wonderful tutorial, thanks for sharing. Very easy to understand for beginners/intermediate Linux users. but I've run into an issue. This is a fresh install of Linux Mint 18 Cinnamon and when I attempt to start up the vm the console says :

Code: Select all

./windows.sh: line 64:  3652 Done                    cat $configfile
      3653 Killed                  | while read line; do
    echo $line | grep ^# > /dev/null 2>&1 && continue; vfiobind $line;
done
W: /etc/qemu-ifup: no bridge for guest interface found
QEMU 2.5.0 monitor - type 'help' for more information
(qemu) qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: vfio: error opening /dev/vfio/1: No such file or directory
qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: vfio: failed to get group 1
qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: Device initialization failed
Anyone else run into this issue and/or know how to fix it?

Code: Select all

W: /etc/qemu-ifup: no bridge for guest interface found
Any chance you didn't set up the bridge? Install bridge-utils and use brctl to check and/or set up the bridge.

Code: Select all

qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: vfio: failed to get group 1
You are attempting to pass through the GPU at 01:00.0 - have you bound the device to pci-stub or better vfio-pci (see one of my posts above)? Check with

Code: Select all

dmesg | grep pci-stub
or

Code: Select all

dmesg | grep vfio-pci
You should get something like that:

Code: Select all

dmesg | grep vfio-pci
[    4.926942] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)
Find out which devices are in your IOMMU group 1:

Code: Select all

find /sys/kernel/iommu_groups/ -type l
Among others you should something like:

Code: Select all

/sys/kernel/iommu_groups/17/devices/0000:00:1f.2
/sys/kernel/iommu_groups/17/devices/0000:00:1f.3
/sys/kernel/iommu_groups/18/devices/0000:01:00.0
/sys/kernel/iommu_groups/18/devices/0000:01:00.1
/sys/kernel/iommu_groups/19/devices/0000:02:00.0
/sys/kernel/iommu_groups/19/devices/0000:02:00.1
If you have more than the two devices 0000:01:00.0 and 0000:01:00.1 you need to separate the other devices from that group (there is a kernel patch for that).
Let us know what you found.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
tortov

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by tortov »

Hi powerhouse,

Thank you for your tutorial, I was able to follow everything and almost got it working. The VM starts fine, but nothing appears in the monitor with the guest gpu, like no signal is being received.

I have two gpus:
01:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] (rev a1)
02:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 760] (rev a1)

The gtx 970 will be the guest, and the gtx 760 will be the host, each on a different monitor (usind HDMI cables). The gtx 970 is used to boot the system, and it shows everything until the login screen, when the image is switched to the host gpu/monitor, and the guest monitor turns off. But the guest fans never stop also. I can't switch de PCIE slots because the guest is on x16 and host in a x4.

Looks like my gtx 970 doesn't have EFI support (the EfiRom returns code 0x00). But when I try to run it using seabios, nothing happens, just like OVMF. The VM starts and there is no error message (besides an error with the netdev).

And all gpus are isolated in their IOMMU groups.

Here is my VM start script:

Code: Select all

#!/bin/bash

configfile=/etc/vfio-pci.cfg
vmname="windows7vm"

vfiobind() {
   dev="$1"
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
   
}


#if ps -A | grep -q $vmname; then
#   echo "$vmname is already running." &
#   exit 1

#else


   cat $configfile | while read line;do
   echo $line | grep ^# >/dev/null 2>&1 && continue
      vfiobind $line
   done

# use pulseaudio
export QEMU_AUDIO_DRV=pa

cp /usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd /tmp/my_vars.fd

qemu-system-x86_64 \
  -name $vmname,process=$vmname \
  -machine type=q35,accel=kvm \
  -cpu host,kvm=off \
  -smp 4,sockets=1,cores=2,threads=2 \
  -enable-kvm \
  -m 4G \
  -mem-prealloc \
  -balloon none \
  -rtc clock=host,base=localtime \
  -bios /usr/share/seabios/bios.bin -vga none \
  -nographic \
  -serial none \
  -parallel none \
  -soundhw hda \
  -device vfio-pci,host=01:00.0,x-vga=on,multifunction=on \
  -device vfio-pci,host=01:00.1 \
  -boot order=dc \
  -device virtio-scsi-pci,id=scsi \
  -drive id=disk0,if=virtio,cache=none,format=raw,file=/media/neuromancer/vm/win.img \
  -drive file=/home/neuromancer/Downloads/os/win7ultimate/win7ultimate.iso,id=isocd,format=raw,if=none -device scsi-cd,drive=isocd \
  -drive file=/home/neuromancer/Downloads/os/virtio-win-0.1.118.iso,id=virtiocd,format=raw,if=none -device ide-cd,bus=ide.1,drive=virtiocd \
  -netdev type=tap,id=net0,ifname=tap0,vhost=on \
  -device virtio-net-pci,netdev=net0,mac=00:16:3e:00:01:01

   exit 0
#fi
Here is the vm output:

Code: Select all

W: /etc/qemu-ifup: no bridge for guest interface found
QEMU 2.1.2 monitor - type 'help' for more information
EfiRom output (I had to boot in safe mode to be able to echo 1 > rom, don't know if this can be the cause of the error:

Code: Select all

Image 1 -- Offset 0x0
  ROM header contents
    Signature              0xAA55
    PCIR offset            0x01A0
    Signature               PCIR
    Vendor ID               0x10DE
    Device ID               0x13C2
    Length                  0x0018
    Revision                0x0000
    DeviceListOffset        0x00
    Class Code              0x030000
    Image size              0xE800
    Code revision:          0x0001
    MaxRuntimeImageLength   0x00
    ConfigUtilityCodeHeaderOffset 0x8B2E
    DMTFCLPEntryPointOffset 0x2EC0
    Indicator               0x00
    Code type               0x00
EfiRom: ERROR 3001: Not supported
  Failed to read PCI ROM header from file!
And the guest Gpu is bind to vfio-pci:

Code: Select all

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:3160]
	Kernel driver in use: vfio-pci
01:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:3160]
	Kernel driver in use: vfio-pci
Do you have any idea why this could be happening? Am I doing the seabios part right? Thanks for the help!
Last edited by tortov on Fri Jul 08, 2016 9:21 pm, edited 2 times in total.
Locked

Return to “Virtual Machines”