HOW-TO make dual-boot obsolete using kvm VGA passthrough

Questions about virtualization software
Forum rules
Before you post please read how to get help
User avatar
catweazel
Level 14
Level 14
Posts: 5464
Joined: Fri Oct 12, 2012 9:44 pm
Location: Australian Antarctic Territory

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by catweazel » Fri Jun 15, 2018 2:48 am

bash64 wrote:
Fri Jun 15, 2018 2:31 am
I installed linux mint 19 beta.
It's beta, and unsupported. You may get some responses but your best bet is to post your comments or questions to the Mint blog.

Linux Mint 19 "Tara" Cinnamon – BETA Release
Linux Mint 19 "Tara" MATE – BETA Release
Linux Mint 19 "Tara" Xfce – BETA Release
A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it. - Max Planck

powerhouse
Level 6
Level 6
Posts: 1017
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse » Fri Jun 15, 2018 2:10 pm

bash64 wrote:
Fri Jun 15, 2018 2:31 am
Hello,

I seem to have failed to click submit on my last post.

I'll shorten it here:
I installed linux mint 19 beta.

I am getting kvm internal errors.
Have you had time to look into this?
I did not change my qemu setup in any way and it always worked.
I keep my full setup on an external 1TB SSD including all files, scripts, virtual hds.
The only difference is the new version all of the qemu files.
I did fail, at first to remove the nvidia dkms modules from the kernel i used for qemu.
Also nouveau loaded anyway (pci_stub showed it had the card regardless).
So I did a rmmod nouveau and it left memory.

KVM internal error. Suberror: 1
emulation failure
EAX=0000b422 EBX=0000b4b4 ECX=00004e59 EDX=000003d4
ESI=00009e48 EDI=00001682 EBP=0000b348 ESP=0000167a
EIP=0000b4b4 EFL=00010086 [--S--P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 ffffffff 00809300
CS =b4b4 000b4b40 ffffffff 00809b00
SS =0000 00000000 ffffffff 00809300
DS =0000 00000000 ffffffff 00809300
FS =0000 00000000 ffffffff 00809300
GS =0000 00000000 ffffffff 00809300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT= 00000000 00000000
IDT= 00000000 000003ff
CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff <ff> ff ff ff ff ff ff ff ff ff ff ff 55 aa 74 eb 4b 37 34 30 30 e9 4c 19 77 cc 56 49 44 45

>uname -a
Linux 4.8.17-040817-generic #201701090438 SMP Mon Jan 9 09:40:28 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
>dkms status
nvidia, 390.48, 4.15.0-20-generic, x86_64: installed
nvidia, 390.48, 4.15.0-23-generic, x86_64: installed
nvidia, 390.48, 4.8.16-040816-generic, x86_64: installed
dad@msi-GT70-2PE:~$ lsmod | grep nouveau
dad@msi-GT70-2PE:~$ dmesg | grep stub
[ 5.369546] pci-stub: add 10DE:1198 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[ 5.370702] pci-stub 0000:01:00.0: claimed by stub
[ 5.371705] pci-stub: add 10DE:0E0A sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[ 5.372709] pci-stub 0000:01:00.1: claimed by stub
[ 47.615229] pci-stub 0000:01:00.0: optimus capabilities: enabled, status dynamic power, hda bios codec supported
dad@msi-GT70-2PE:~$ lsmod | grep vf
vfio_pci 45056 1
vfio_iommu_type1 20480 1
vfio_virqfd 16384 1 vfio_pci
vfio 28672 5 vfio_iommu_type1,vfio_pci
irqbypass 16384 3 kvm,vfio_pci
Hi bash64,

1. Your uname -a shows that you are running an old 4.8.17 kernel. Try the latest one you installed.

2. Can't you use vfio-pci to bind the graphics card? You are using pci-stub.

3. Make sure you load the kernel modules in the right order. They should be listed in the following order in the /etc/initramfs-tools/modules file:

Code: Select all

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
vhost-net
Something is wrong with your modules list:
dad@msi-GT70-2PE:~$ lsmod | grep vf
vfio_pci 45056 1
vfio_iommu_type1 20480 1
vfio_virqfd 16384 1 vfio_pci
vfio 28672 5 vfio_iommu_type1,vfio_pci
irqbypass 16384 3 kvm,vfio_pci
The first and second line with a "1" at the end should list a module, or have "0". Here is mine as an example:

Code: Select all

 $ lsmod | grep vf
vfio_pci               40960  0
vfio_virqfd            16384  1 vfio_pci
irqbypass              16384  2 kvm,vfio_pci
vfio_iommu_type1       24576  0
vfio                   28672  2 vfio_iommu_type1,vfio_pci
4. Make sure you installed all relevant packages.

5. Don't forget it's a beta.

6. Sorry, I haven't had time to try it. I'm waiting for the stable release before I upgrade, and not without a complete backup before the upgrade.
Asus Sabertooth X79, i7 3930K CPU, 8x4GB Kingston DDR3, Noctua NH-D14 CPU cooler, GTX 970 + Quadro 2000 GPU, Asus Xonar Essence STX, Sandisk 120GB + Samsung EVO 850 250GB SSD + 4 HDD, Corsair 500R, SeaSonic 660W Gold X PS https://heiko-sieger.info

bash64
Level 2
Level 2
Posts: 82
Joined: Mon Mar 23, 2015 12:19 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by bash64 » Sun Jun 17, 2018 6:03 am

powerhouse,

beware. you need a full one for one backup.
i normally just use aptik to create a list of packages and to backup deb files.
i even went to the trouble of reinstalling mint 18.3 to a different partition and recovering.
i still cannot boot qemu.
i regret not veryfying all of the files it uses and what versions.

some were from repo, some were compiled and the seabios could have come from anywhere.

to clarify:
my virtual machines work fine. they boot in seconds if i use a vga adapter.
its when i boot with the nvidia card that they cannot boot.
i even left them running for hours and no boot.

i have switched to vfio-pci now. no more pci_stub:

~$ lsmod | grep vfio
vfio_pci 45056 0
vfio_virqfd 16384 1 vfio_pci
vfio_iommu_type1 20480 0
vfio 28672 2 vfio_iommu_type1,vfio_pci
irqbypass 16384 2 kvm,vfio_pci

still no dice.
is there another virtual machine that i can try to do the same thing.
willing to give it a shot.
p.s. i will step back through your tutorial and try again with mint 18.3.

bash64
Level 2
Level 2
Posts: 82
Joined: Mon Mar 23, 2015 12:19 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by bash64 » Sun Jun 17, 2018 10:18 am

i am stepping through your tutorial fresh.

i would like to get your help looking over my results to spot problems.

right now i am at the point of installing windows.

i am using seabios instead of uefi due to my laptop using nvidia optimus technology.

i cannot boot any iso image at all.
not windows 7 and not windows 10.
code 0005 error

or it just hangs after it tries to start booting
i also tried just a plain jane "-cdrom /pathtoiso"
no dice


i will wait to post long reports until i can at least get a fresh copy of windows installed.

i am not concerned about nvidia right now.
just need to get a test copy of any windows installed.
i am using all of your instructions.
i dont know what causes this bug but it is massively annoying.

p.s. using a compiled copy of qemu 2.12.50
any files it does not install come from the repo.


update: wrote w7 iso to usb thumb drive.
manually select thumb drive from seabios boot menu (hit esc).
no dice


#!/bin/bash

vmname="testvm"

if ps -A | grep -q $vmname; then
echo "$vmname is already running." &
exit 1

else

# use pulseaudio
export QEMU_AUDIO_DRV=pa
export QEMU_PA_SAMPLES=8192
export QEMU_AUDIO_TIMER_PERIOD=99
export QEMU_PA_SERVER=/run/user/1000/pulse/native

qemu-system-x86_64 \
-name $vmname,process=$vmname \
-machine type=q35,accel=kvm \
-cpu host,kvm=off \
-smp 8,sockets=1,cores=4,threads=2 \
-m 20G \
-bios /home/dad/seabios/out/bios.bin \
-mem-prealloc \
-rtc clock=host,base=localtime \
-serial none \
-parallel none \
-display gtk \
-soundhw hda \
-usb -device usb-host,hostbus=3,hostaddr=14 \
-device vfio-pci,host=01:00.0,multifunction=on \
-device vfio-pci,host=01:00.1 \
-boot order=dc \
-drive id=disk0,if=virtio,cache=none,format=raw,file=/home/dad/qemu/test7.img \
-drive file=/media/dad/QEMU-SSD/qemu-wd/w7u-sp1.iso,index=3,media=cdrom \
-drive file=/media/dad/QEMU-SSD/qemu-wd/virtio-win-0.1.149.iso,index=4,media=cdrom \
-netdev type=tap,id=net0,ifname=tap0,vhost=on \
-device virtio-net-pci,netdev=net0,mac=00:16:3e:00:01:01 \
-boot menu=on \
-boot order=dc

exit 0
fi

powerhouse
Level 6
Level 6
Posts: 1017
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse » Mon Jun 18, 2018 7:22 am

Hi bash64,

This does sound annoying, especially if you don't have a (full) backup.

One thing that sticks out is the qemu script:

Code: Select all

-smp 8,sockets=1,cores=4,threads=2 \
-m 20G \
It means that you pass 4 cores/8 threads to the VM, as well as 20 GB of memory. Seems rather a lot for a laptop and might prevent the guest from booting if you don't have that amount of memory.

If your PC is a 4 core machine with say 16GB memory, start with 2cores/4 threads and 8GB of memory. You can always add more and fine tune.

EDIT: I haven't tested / tried Ubuntu 18.04 / Linux Mint 19 yet, and will definitely wait until it is officially released. qemu / kvm is undergoing massive development and unfortunately they are changing their syntax on the way. Check the dmesg output. Also, make sure to run the script in a terminal window that allows you to follow the messages as they appear. Often they are the key to figuring out what's happening.

Before specifying the amount of memory for the VM, check how much PC memory you actually got:

Code: Select all

free
Recently I had two memory modules fail and it took me some head scratching until I actually figured out what was wrong and why my PC sometimes got "stuck".

Please note that my most up-to-date version of the tutorial is here: https://heiko-sieger.info/running-windo ... ssthrough/

At the end of the tutorial you'll find plenty of links and references to other tutorials and sites. Also don't forget to read your own posts here at this forum. If I remember correctly you posted some details on how you got it working with your Optimus laptop.
Asus Sabertooth X79, i7 3930K CPU, 8x4GB Kingston DDR3, Noctua NH-D14 CPU cooler, GTX 970 + Quadro 2000 GPU, Asus Xonar Essence STX, Sandisk 120GB + Samsung EVO 850 250GB SSD + 4 HDD, Corsair 500R, SeaSonic 660W Gold X PS https://heiko-sieger.info

bash64
Level 2
Level 2
Posts: 82
Joined: Mon Mar 23, 2015 12:19 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by bash64 » Mon Jun 18, 2018 8:10 am

thanks,

i upgraded the laptop to 32gb of ram.
when running fallout 4 i gave it 20gb because the game was a huge resource hog.
giving the machine all 4 cores with 8 threads was normal for me.
i can reduce it for now.
been trying endless seabios files.
no luck yet.
i'll wait a few weeks till i have tried quite a few things.
i am trying mint 18.3 as well as mint 19.
this is going to take some time.
i want to play fallout 76 online in november.
is there another gp passthrough virtual machine that i can try?

powerhouse
Level 6
Level 6
Posts: 1017
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse » Mon Jun 18, 2018 9:35 am

bash64 wrote:
Mon Jun 18, 2018 8:10 am
thanks,

i upgraded the laptop to 32gb of ram.
when running fallout 4 i gave it 20gb because the game was a huge resource hog.
giving the machine all 4 cores with 8 threads was normal for me.
i can reduce it for now.
been trying endless seabios files.
no luck yet.
i'll wait a few weeks till i have tried quite a few things.
i am trying mint 18.3 as well as mint 19.
this is going to take some time.
i want to play fallout 76 online in november.
is there another gp passthrough virtual machine that i can try?
OK, thanks for the answers. I read some of your earlier posts which may provide hints on how to get it to work. See viewtopic.php?f=231&t=212692&start=260#p1300634

When things don't work, start checking the basics. See if you installed all necessary packages including the seabios package. Start with an unmodified installation ISO, add the packages, make sure your PC is setup for passthrough (IOMMU enabled, BIOS settings, etc.). Update the packages, and install the latest kernel from the repository.

In your case, when using an Optimus laptop, you should probably stick to seabios, not the OVMF / UEFI method. This is slightly different as described in my tutorial.

With regards to other passthrough technologies or hypervisors, I don't think that switching horses will do much good. However, please check my links on my website for other tutorials that may be more relevant to your configuration.
Asus Sabertooth X79, i7 3930K CPU, 8x4GB Kingston DDR3, Noctua NH-D14 CPU cooler, GTX 970 + Quadro 2000 GPU, Asus Xonar Essence STX, Sandisk 120GB + Samsung EVO 850 250GB SSD + 4 HDD, Corsair 500R, SeaSonic 660W Gold X PS https://heiko-sieger.info

bash64
Level 2
Level 2
Posts: 82
Joined: Mon Mar 23, 2015 12:19 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by bash64 » Wed Jun 20, 2018 2:26 pm

Hey,

I have decided to work things out myself.
Sorry if panic mode got turned on.
I have worked on qemu for years, even providing how to docs here.
No need to panic now.

I will stick to helpful posts when I find helpful information.

I have determined that the issue is with seabios.
I was able to turn on debugging with seabios with this line:
-chardev stdio,id=seabios -device isa-debugcon,iobase=0x402,chardev=seabios \

here is the ouput of the important lines:
....
PCI: Enabling 01:00.0 for primary VGA

NOTE: 01.00.0 is my nvidia gtx 880m 8gb video card.....this line reads 00.01.0 when using the vga adapter...

PCI: Setting VGA enable on bridge 00:1c.0
handle_smp: apic_id=1
handle_smp: apic_id=5
handle_smp: apic_id=6
handle_smp: apic_id=2
handle_smp: apic_id=3
handle_smp: apic_id=7
handle_smp: apic_id=4
Found 8 cpu(s) max supported 8 cpu(s)
Copying PIR from 0x7ffbfca4 to 0x000f1710
Copying MPTABLE from 0x00006e0c/7ffa7230 to 0x000f15e0
Copying SMBIOS entry point from 0x00006e0c to 0x000f13f0
Scan for VGA option rom
Running option rom at c000:0003
Start SeaVGABIOS (version rel-1.11.0-0-g63451fca13-prebuilt.qemu-project.org)

You see that last line that says "Start SeaVGABIOS"?
that line never appears when using -vga none and getting a failed boot.

It only appears when I allow a vga adapter to be used and , thus, windows always boots...7 or 10. Doesn't matter which.

Running option rom is the last thing that happens on a failed boot.

I gotta go look this up, but I hope the seabios debug line is helpful.

bash64
Level 2
Level 2
Posts: 82
Joined: Mon Mar 23, 2015 12:19 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by bash64 » Thu Jun 21, 2018 12:26 pm

update:

kvm is definitely an issue.

seabios 1.7.x cannot boot my machines.
seabios 1.8.x does, but only with the vga adapter.
seabios 1.9.x and higher (up to 1.11.x) results in a kvm error.

make you wonder if kvm is the whole issue.
anyone else having problems?
strange that i can put back my system the way it was (mint 18.3, etc files...) and it wont boot without the vga adapter. windows 7/10 sees my gaming card perfectly. windows is not corrupted. seabios sees my gaming card. everything is right, but nothing boots.

p.s. got no proof this worked on mint 18.3 right before going to min 19.
i didn't exactly run it before the change.
i have both mint 18.3 and mint 19 installed at this time.
i am debugging mint 18.3.

an update could have come down right before the upgrade that broke it.
if that happened then mint 18.3 reinstalled has that update also.
i will try to revert to the earliest qemu packages in mint 18.3.

bash64
Level 2
Level 2
Posts: 82
Joined: Mon Mar 23, 2015 12:19 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by bash64 » Thu Jun 21, 2018 5:48 pm

VICTORY

sort of.

In the end I had to add nomodeset to my boot line.

Something is grabbing the video card that did not before.

I have tried:
1) modprobe.blacklist=nouveau
2) uninstall all nvidia software (only had to remove the dkms module before)

only nomodeset works which is a bummer. that means i have to operate in VESA mode in linux just so windows will boot.
no video acceleration. the linux side will be terrible at streaming videos.
ideas accepted.

bash64
Level 2
Level 2
Posts: 82
Joined: Mon Mar 23, 2015 12:19 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by bash64 » Thu Jun 21, 2018 9:03 pm

update:

the only driver loaded is i915.
but somehow it is grabbing the nvidia card.
i am using vfio-pci to grab the card on boot per the tutorial.
i can only think its the kms (kernel mode setting) kernel doing this.
i'd still rather not use nomodeset.

bash64
Level 2
Level 2
Posts: 82
Joined: Mon Mar 23, 2015 12:19 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by bash64 » Thu Jun 21, 2018 11:13 pm

TRIPLE VICTORY!!!

It not only works again in mint 18.3...I got it working in mint 19 beta.

The short answer on what broke....drumroll......i915.
The i915 driver causes the issue. Changes to this driver by Intel is causing issues with qemu with gpu passthrough for Optimus owners.

The quick fix is to blacklist i915.
On mint 18.3 add "nomodeset" to your boot line in /etc/default/grub and "sudo update-grub".

On mint 19 the i915 beast is harder to tame.
I made a blacklist file:
/etc/modprobe.d/blacklist-i915.conf
blacklist i915
install i915 /bin/false

I do not know if you need both lines...I used both and it worked.

The latest i915 driver will grab hold of you nvidia card if you have Optimus.
NOTE: One bad side effect is that any external monitors that the intel chip has access to are no longer available. I lost my external 15pin vga 26" monitor for the linux side. :-(


Victory #1 is mint 18.3 works once again.
Victory #2 is mint 19 also works.
Victory #3? I have always complained of it taking an excessively long time to boot windows. 4-8 minutes.
It now boots in exactly 60 seconds!

Perseverance has paid off again.
I am ready for you Fallout 76. Take my money please.

Post Reply

Return to “Virtualization”