HOW-TO make dual-boot obsolete using kvm VGA passthrough

[powerhouse]

Hello Tech-Junk1e:

Your 09:0.2 device is using the wrong driver, it should also be bound to vfio-pci.

Perhaps you didn't list it as an alias, see https://heiko-sieger.info/running-windo ... assthrough

Make sure you have 4 alias entries AND the PCI device identifiers listed in this file, for example:

options vfio-pci ids=10de:13c2,10de:0fbb

You should have 4 comma separated entries after the ids=...

Please also note that my tutorial is written for Windows 10, not Windows 7 !!!

Windows 7 doesn't support UEFI boot out of the box. You would have to create an updated version with UEFI boot.

The alternative is to use regular Seabios (legacy) boot, without the ovmf part. This legacy boot might work but can have disadvantages, which is why I don't describe it. See my links, in particular the link to the archlinux forum, for more on the Seabios legacy boot.

[Tech-Junk1e]

Unfortunately i have created all of the alias entries correctly, no matter what I do I cannot stop that particular driver from taking hold. at one point on a previous install I even tried blacklisting it (though not recently).

Code: Select all

root@TJs-WorkHorse:~# cat /sys/bus/pci/devices/0000:09:00.0/modalias
pci:v000010DEd00002182sv00003842sd00001263bc03sc00i00
root@TJs-WorkHorse:~# cat /sys/bus/pci/devices/0000:09:00.1/modalias
pci:v000010DEd00001AEBsv00003842sd00001263bc04sc03i00
root@TJs-WorkHorse:~# cat /sys/bus/pci/devices/0000:09:00.2/modalias
pci:v000010DEd00001AECsv00003842sd00001263bc0Csc03i30
root@TJs-WorkHorse:~# cat /sys/bus/pci/devices/0000:09:00.3/modalias
pci:v000010DEd00001AEDsv00003842sd00001263bc0Csc80i00

Code: Select all

alias pci:v000010DEd00002182sv00003842sd00001263bc03sc00i00 vfio-pci
alias pci:v000010DEd00001AEBsv00003842sd00001263bc04sc03i00 vfio-pci
alias pci:v000010DEd00001AECsv00003842sd00001263bc0Csc03i30 vfio-pci
alias pci:v000010DEd00001AEDsv00003842sd00001263bc0Csc80i00 vfio-pci
options vfio-pci ids=10de:2182,10de:1aeb,10de:1aec,10de:1aed
options vfio-pci disable_vga=1

I had forgotten about windows 7 not being UEFI compatible and will look into the alternative method you mentioned.

Also another question of inexperience, can I use my ISO in virtualbox and use qemu-utils to convert the .vdi file to a .img or would UEFI still get in the way?

EDIT: I have just tried to pci-stub the usb controller via both /etc/initramfs-tools/modules and etc/default/grub with no success. other users are encountering the same issue.

Code: Select all

root@TJs-WorkHorse:~# dmesg | grep pci-stub
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.15.0-47-generic root=/dev/mapper/mint--vg-root ro quiet splash pci-stub.ids=10de:1aec amd_iommu=on iommu=pt vt.handoff=1
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.15.0-47-generic root=/dev/mapper/mint--vg-root ro quiet splash pci-stub.ids=10de:1aec amd_iommu=on iommu=pt vt.handoff=1
[    3.436830] pci-stub: add 10DE:1AEC sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
root@TJs-WorkHorse:~# lspci -kn | grep -A 2 09:00
09:00.0 0300: 10de:2182 (rev a1)
	Subsystem: 3842:1263
	Kernel driver in use: vfio-pci
--
09:00.1 0403: 10de:1aeb (rev a1)
	Subsystem: 3842:1263
	Kernel driver in use: vfio-pci
--
09:00.2 0c03: 10de:1aec (rev a1)
	Subsystem: 3842:1263
	Kernel driver in use: xhci_hcd
09:00.3 0c80: 10de:1aed (rev a1)
	Subsystem: 3842:1263
	Kernel driver in use: vfio-pci

here https://askubuntu.com/questions/719390/ ... assthrough
and here https://www.reddit.com/r/unRAID/comment ... his_erroe/

and I am still too green to understand the solution.

I also now have a UEFI enabled windows 7 ISO and a Win 10 ISO

[powerhouse]

Hello Tech-Junk1e,

My instructions work with UEFI.

It seems that the USB driver is very persistent. Have a look at the other options to bind the vfio-pci driver. One way is via grub file. Make sure whenever you make any changes in the /etc/initramfs you also run update-initramfs, similar to when you change anything in /etc/default/grub to run update-grub.

I'm quite sure that you did all that and I'm afraid I don't have a better answer, sorry.

Search for your graphics card and vfio-pci and see if anyone else solved the issue. In the worst case, sign up on the vfio list and post your dilemma. I hope I didn't miss a link in my references.

If you find a solution, please post it here! I will also try to keep an eye open but am very busy now.

[powerhouse]

Hello Tech-Junk1e,

My instructions work with UEFI.

It seems that the USB driver is very persistent. Have a look at the other options to bind the vfio-pci driver. One way is via grub file. Make sure whenever you make any changes in the /etc/initramfs you also run update-initramfs, similar to when you change anything in /etc/default/grub to run update-grub.

I'm quite sure that you did all that and I'm afraid I don't have a better answer, sorry.

Search for your graphics card and vfio-pci and see if anyone else solved the issue. In the worst case, sign up on the vfio list and post your dilemma. I hope I didn't miss a link in my references.

If you find a solution, please post it here! I will also try to keep an eye open but am very busy now.

I have updated my tutorial on my website and added a section on using the "driver override" feature of Linux that should solve the issue - see Using the “driver_override” feature in conjunction with the initial RAM disk (initramfs).

[bash64]

Powerhouse,

I want to install the i915 vga arbiter patch.
I am running linux mint 19.1.
What is the quickest way to get a kernel installed that has this?
I am willing to compile, but so far internet instructions on doing this fail at the patch stage.
Thanks

[powerhouse]

Powerhouse,

I want to install the i915 vga arbiter patch.
I am running linux mint 19.1.
What is the quickest way to get a kernel installed that has this?
I am willing to compile, but so far internet instructions on doing this fail at the patch stage.
Thanks

The question should be: Do you have to install the VGA arbiter patch?

This is from my tutorial:

If you plan to use the Intel IGD (integrated graphics device) for your Linux host, UEFI boot is the way to go. UEFI overcomes the VGA arbitration problem associated with the IGD and the use of the legacy Seabios.
If, for some reason, you cannot boot the VM using UEFI, and you want to use the Intel IDG for the host, you need to compile the i915 VGA arbiter patch into the kernel. Before you do, check the note below. For more on VGA arbitration, see here. For the i915 VGA arbiter patch, look here or under Part 15 – References.

Note: If your GPU does NOT support UEFI, there is still hope. You might be able to find a UEFI BIOS for your card at TechPowerUp Video BIOS Collection. A Youtube blogger calling himself Spaceinvader has produced a very helpful video on using a VBIOS.

See also here: https://heiko-sieger.info/running-windo ... ration_bug

If your "pass-through" graphics card supports UEFI, you don't need the patch. Please let me know if you use UEFI boot, as described in my tutorial.

Patching the kernel should be your last resource to solve the problem. Trust me, it's a pain in the neck.

[bash64]

Hey Powerhouse.

I have built a new PC, full amd threadripper 2nd gen.
My video card is the Radeon Duo Pro R9 Fury X nano series.
It has two gpus.

Is it possible to hold a gpu for vfio in a card with two gpus?
So far no success.

Thanks

[bash64]

p.s.

I solved my need for vga arbitration.
It was as simple as switching from uefi mode to uefi with csm in my bios.
The hardware issue I had went away.
Basically, my vga 15 pin port was disabled when using nomodeset on boot.
No i915 driver...no vga port signal.
Switching to uefi with csm kept the vga port alive regardless and I was able to use my 23" led monitor with the vga port for linux.
I always thought it was a software issue.

[powerhouse]

p.s.

I solved my need for vga arbitration.
It was as simple as switching from uefi mode to uefi with csm in my bios.
The hardware issue I had went away.
Basically, my vga 15 pin port was disabled when using nomodeset on boot.
No i915 driver...no vga port signal.
Switching to uefi with csm kept the vga port alive regardless and I was able to use my 23" led monitor with the vga port for linux.
I always thought it was a software issue.

Sorry I didn't respond earlier - I was actually having a vacation

I'm glad you managed to fix the issue. However, for the benefit of me and others, could you describe the problem that you fixed with it?

I understand that it has to do with the Radeon Duo Pro R9 Fury X nano, but I'm not exactly sure of what issue you ran into.

Thanks in advance!

[bash64]

Sorry, I had two different posts for two different PCs.
The one was for the msi gt70 laptop I have used for 5 years.
Basically, the vga 15 pin out was disabled when doing qemu passthrough.
I was using the laptop screen for linux and the display port for qemu.
I wanted to have the laptop screen appear on my 23" monitor hooked up to the vga 15 pin port, but that port was strangely disabled.
It turned out that it was a bios issue.
In uefi mode the vga port is disabled.
When I switched to uefi+csm it worked. The laptop screen and the vga port mirrored each other.


As to the new PC with threadripper 2 cpu.
I managed to share an amd radeon with dual gpus out to qemu.
Both gpus have the same vid:pid numbers.
I simply listed them twice as vfio options as such:
options vfio-pci ids=1002:7300,1002:aae8,1002:7300

Both gpus and the 4k monitor audio both went over to windows just fine and gaming works.
Also discovered that windows 7 does NOT have a threadripper driver and will not.
Had to switch to using windows 10.

Thanks for responding.

I built this new desktop to get away from the issues of my aging laptop.
It was great figuring out how to make it work on the laptop despite it having nvidia optimus, but time to move on.
I now have a desktop with two video cards and qemu couldn't be any easier.
Running both OSs at the same time with out the limitations of the laptop.

I moved on to qemu 4.0 also and no issues. Seems backwards compatible with my exisiting script.

Question: Does qemu really require a bridge for tap0 to work?
When I have a bridge defined I cannot have wireless. If I turn it on it causes issues.

I'd really love if tap0 were independant and could get an ip on its own.

[powerhouse]

@bash64: Thanks for responding and sharing your experience - this is what makes the forum so valuable. (I knew I had seen you on the forum long ago and was a little confused.)

You do not need to define a bridge, and if you skip the network definition in the script, it should automatically configure a layer 3 (IP) network that works with WiFi.

The reason I advocate a bridge is performance and simplicity. But there are plenty of use cases where bridging doesn't work. If your PC is connected to the router via wireless, then bridging is a no go (well, there are ways to get bridging working over WiFi, but it's a pain to set up and I didn't succeed).

In truth, I haven't tested IP networking (layer 3) in a while and performance may well be good. On my bridged network, I get the following results when copying a 10GiB file:
HD (host, accessed via Samba) to SSD (on Windows 10 VM) - 46 sec, or 217 MByte/sec
SSD (Windows 10 VM on Samsung EVO 860) to SSD (host, via Samba, a Sandisk 120GB disk) - 20 sec, or 500 MByte/sec

By the way, creating the file on a HD using dd if=/dev/zero of=file.txt count=1024 bs=10485760 takes the same time (20 sec) as copying the file between SSD.

Copying the same 10GiB file from my Linux host to my server over Gigabit Ethernet takes 1:41 minutes, or 100 MByte/sec.

Perhaps you can use the above benchmarks for comparison.

[powerhouse]

I wrote a new passthrough tutorial, this time using the Virtual Machine Manager. It's based on the AMD Ryzen 9 3900X / X570 platform. I chose an Ubuntu 19.10 based distribution, because of the more up-to-date QEMU, libvirt and virt-manager packages.

The upcoming Linux Mint 20 will be based on Ubuntu 20.04 which is scheduled to be released by end of April. Linux Mint 20 is expected to be released about a month after the Ubuntu release. Why I'm telling that? Because if you wait only a couple of months you can enjoy the new QEMU 4 goodies on Linux Mint.

Linux Mint 19.3 uses the old QEMU 2.11 which is not suitable for the tutorial I wrote.

Here the link: Creating a Windows 10 VM on the AMD Ryzen 9 3900X using Qemu 4.0 and VGA Passthrough

[bash64]

I wish to add a physical blueray drive to my existing virtual machine script.
I need help forming the appropriate line.
It is mounted in linux to /dev/sr0.
I do use redhat virtual i/o.

I used the el cheapo method of -cdrom /dev/sr0, but everytime I eject and insert a new cd I have to reboot windows so it can see it.
taskset is only there to set the cpu affinity. I have 24 cpus and 12 cores.
$1 is the number of cpus passed to the script that qemu can have for personal use. It is used in the -smp line.
Just in case you were wondering what all that was for.

Here is my script:

#!/bin/bash
cd /media/dad/QEMU-SSD/qemu-wd
export QEMU_AUDIO_DRV=pa
export QEMU_AUDIO_TIMER_PERIOD=200
export QEMU_PA_SERVER=/run/user/1000/pulse/native

taskset -p -c 0-7 `pidof lightdm | cut -d' ' -f2`
taskset -c 8-23 qemu4.0-system-x86_64 -machine type=q35,accel=kvm -cpu EPYC \
-chardev stdio,id=seabios -device isa-debugcon,iobase=0x402,chardev=seabios \
-smp $1,sockets=1,cores=$1 \
-bios /usr/share/seabios/bios.bin \
-m 16G \
-vga none \
-soundhw hda \
-mem-prealloc \
-rtc clock=host,base=localtime \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=0d:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=0d:00.1,bus=root.1,addr=00.1,multifunction=on \
-device vfio-pci,host=0e:00.0,bus=root.1,addr=00.2,multifunction=on \
-device virtio-scsi-pci,id=scsi \
-drive id=disk0,if=virtio,cache=none,format=raw,file=/media/dad/QEMU-SSD/qemu-wd/windows10.img \
-drive file=/media/dad/QEMU-SSD/qemu-wd/'virtio-win-0.1.171.iso',id=isocd,format=raw,if=none -device scsi-cd,drive=isocd \
-netdev type=tap,id=net0,ifname=tap0 \
-device virtio-net-pci,netdev=net0,mac=00:16:3e:00:01:01 \
-usb -device usb-host,vendorid=0x046d,productid=0xc534 \
-usb -device usb-host,vendorid=0x05ac,productid=0x12aa \
-boot menu=on \
-boot order=c
exit 0

[powerhouse]

Hello bash64, nice to see that you are still using passthrough (well, why not?).

To be frank, I haven't used the command line a lot lately, and my current rig uses libvirt. By the way, do you run a Ryzen 9 3900X? If yes, please share your hardware specs (m/b, BIOS release, etc.).

My suggestions to your problem:

1. Can you pass through the controller where the blueray drive is connected to? That would be the easiest.

2. I would have to check the new qemu commands myself, but there have been some additions. Check out the documentation section in my new tutorial: https://heiko-sieger.info/creating-a-wi ... umentation.

Sorry I can't be of more help now. Will have a look into qemu tomorrow.

By the way, I really look forward to the Linux Mint 20 release. When I wrote the Pop!_OS tutorial I chose that distro because it is Ubuntu-based and more up-to-date with regards to qemu. So Linux Mint 20 should come with all of the new goodies that Ubuntu 20.04 delivers.

I'm currently on Manjaro because Pop!_OS is perhaps nice for fanboys, but it got too many small issues and practically no support.

[bash64]

I built this so that I could have two video cards and run two OSs at the same time efficiently.
Overall cost was $3000

My parts list:
Antec High Current Pro HCP-1300 PLATINUM 1300W ATX 12V Power Supply PSU
AMD 2nd Gen Ryzen Threadripper 2920X 12-Core, 24-Thread
StormShield cpu liquid cooler with 2 fans
MSI X399 GAMING PRO CARBON AC SocketTR4 motheroard
G.SKILL Sniper X Series 32GB (4 x 8GB) ram
Corsair air 540 case
XFX AMD Radeon Pro Duo 8GB R9 Liquid Cooled Dual GPU Video Card
Nvidia 980 TI OC 6gb 2nd video card
Bought a 3rd monitor to go with this build.

not part of cost – borrowed from previous pc:
(2) 4TB 2.5” hard drives
(1) Intel SSD 1TB

[bash64]

Issue with cd eject resolved.
I am on the qemu mailing list and the guys on there solved it for me.

First, list out the scsi dev node associated with your burner (you might have to install lsscsi):
>lsscsi -g

[0:0:0:0] disk ATA ST2000LM003 HN-M 0006 /dev/sda /dev/sg0
[4:0:0:0] disk ATA ST4000LM024-2AN1 0001 /dev/sdb /dev/sg1
[5:0:0:0] disk ATA INTEL SSDSC2KW01 042C /dev/sdc /dev/sg2
[6:0:0:0] cd/dvd MATSHITA BD-MLT UJ260AF 1.00 /dev/sr0 /dev/sg3
[7:0:0:0] disk ATA ST4000LM024-2AN1 0001 /dev/sdd /dev/sg4

my burner is /dev/sr0 but is also /dev/sg3

now add the following two lines to your script:
-device scsi-generic,drive=cd0 \
-drive file=/dev/sg3,format=raw,id=cd0,if=none,unit=1 \

replace sg3 with your node name

the key is that the scsi-generic device sends genuine scsi commands directly to the drive and bypasses any emulation like scsi-cd would.
this solves the issues with ejection and , apparently, all other issues.
the device known as scsi-cd also does not like music/audio cds. they have no filesystem.
this causes windows to lockup when accessing audio cds


NOTE: you will have to eject using the linux side, but if you reinsert a disc windows will see it fine.

[powerhouse]

This is great. Good that you found the solution. I was away for some days but in the meantime I found this:

https://www.linux-kvm.org/page/Change_cdrom

Edit:

You may be able to make use of the qemu-ga guest agent to pass commands to qmp, for example to eject the DVD. See:

https://wiki.qemu.org/Features/GuestAgent
https://wiki.qemu.org/Documentation/QMP

[powerhouse]

Linux Mint 20 is expected to be released in June. It's based on the Ubuntu 20.04 LTS release so it will come with QEMU 4.2.

This is good news as QEMU 4.0 and later have solved some nasty issues with audio and better supports new architectures such as the popular AMD Zen architecture.

I myself switched to a AMD Ryzen 9 3900X system with a X570 motherboard. For those who play with the thought of switching from Intel to AMD, beware! There are still a number of issues that haven't been resolved. The most prevalent flaw is the lack of proper L3 cache architecture support in QEMU. See https://bugs.launchpad.net/qemu/+bug/1856335. A more serious issue was last years BIOS update that crippled IOMMU. It's been fixed later but it seems kinda hit-and-miss with BIOS updates.

Looking at the poor track record with AMD's graphics cards and their notorious FLR reset bug as well as various comments on other forums and websites (check out Phoronix) I am not able to recommend AMD at this time. They seem to have very little interest in Linux and virtualization (their server market share is very small compared to Intel), and their main market is still consumers / enthusiasts running Windows. Perhaps one day they realize they need solid Linux and virtualization support to play in the big league (cloud / grid computing).

Back to the topic: Once Linux Mint 20 is out you should be able to use my Pop!_OS tutorial with only minor changes to get VGA passthrough up and running in Linux Mint. One more note: I cannot endorse Pop_OS and I have switched to another Linux distro. You better stay with Linux Mint.

[bash64]

Thanks for mentioning ryzen.
I switched to amd threadripper and have had various issues.
I had to switch from using the -cpu host to using -cpu EPYC to resolve some windows 10 updates issues (thread exception unhandled).
Turned out windows 10 was an upgrade from windows 7 and that was causing all the issues. A reformat fixed it all.
I also bought the latest amd video card, an asrock amd rx5500 xt 8gb.
vfio-pci still cannot reset the card so I have the nasty navi 14 reset bug.
linux mint 20 hasnt helped.

Share your amd issues here so we can collaborate.

[bash64]

I have been using a radeon pro duo with 2 gpus for a while.
However, I could only use 1 gpu in passthrough.

Well, here is the fix to get both working:

I had to change my lines from:
-device vfio-pci,host=0c:00.0,bus=root_port1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=0c:00.1,bus=root_port2,addr=00.1,multifunction=on \
-device vfio-pci,host=0d:00.0,bus=root_port3,addr=01.0,multifunction=on \

to:
-device vfio-pci,host=0c:00.0,bus=root_port1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=0c:00.1,bus=root_port2,addr=00.0,multifunction=on \
-device vfio-pci,host=0d:00.0,bus=root_port3,addr=00.0,multifunction=on \


Apparently, this is a known bug.
All cards must be on the same addr and it needs to be 00.0.

I now have both gpus and 8gb of vmem instead of just 1 gpu and 4gb.

Nice to know you CAN have more than one video card with gpu passthrough.