HOW-TO make dual-boot obsolete using kvm VGA passthrough

Questions about virtualization software
Forum rules
Before you post please read how to get help
User avatar
catweazel
Level 17
Level 17
Posts: 7750
Joined: Fri Oct 12, 2012 9:44 pm
Location: Australian Antarctic Territory

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by catweazel » Fri Jun 15, 2018 2:48 am

bash64 wrote:
Fri Jun 15, 2018 2:31 am
I installed linux mint 19 beta.
It's beta, and unsupported. You may get some responses but your best bet is to post your comments or questions to the Mint blog.

Linux Mint 19 "Tara" Cinnamon – BETA Release
Linux Mint 19 "Tara" MATE – BETA Release
Linux Mint 19 "Tara" Xfce – BETA Release
¡uʍop ǝpısdn sı buıɥʇʎɹǝʌǝ os ɐıןɐɹʇsnɐ ɯoɹɟ ɯ,ı

powerhouse
Level 6
Level 6
Posts: 1066
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse » Fri Jun 15, 2018 2:10 pm

bash64 wrote:
Fri Jun 15, 2018 2:31 am
Hello,

I seem to have failed to click submit on my last post.

I'll shorten it here:
I installed linux mint 19 beta.

I am getting kvm internal errors.
Have you had time to look into this?
I did not change my qemu setup in any way and it always worked.
I keep my full setup on an external 1TB SSD including all files, scripts, virtual hds.
The only difference is the new version all of the qemu files.
I did fail, at first to remove the nvidia dkms modules from the kernel i used for qemu.
Also nouveau loaded anyway (pci_stub showed it had the card regardless).
So I did a rmmod nouveau and it left memory.

KVM internal error. Suberror: 1
emulation failure
EAX=0000b422 EBX=0000b4b4 ECX=00004e59 EDX=000003d4
ESI=00009e48 EDI=00001682 EBP=0000b348 ESP=0000167a
EIP=0000b4b4 EFL=00010086 [--S--P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 ffffffff 00809300
CS =b4b4 000b4b40 ffffffff 00809b00
SS =0000 00000000 ffffffff 00809300
DS =0000 00000000 ffffffff 00809300
FS =0000 00000000 ffffffff 00809300
GS =0000 00000000 ffffffff 00809300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT= 00000000 00000000
IDT= 00000000 000003ff
CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff <ff> ff ff ff ff ff ff ff ff ff ff ff 55 aa 74 eb 4b 37 34 30 30 e9 4c 19 77 cc 56 49 44 45

>uname -a
Linux 4.8.17-040817-generic #201701090438 SMP Mon Jan 9 09:40:28 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
>dkms status
nvidia, 390.48, 4.15.0-20-generic, x86_64: installed
nvidia, 390.48, 4.15.0-23-generic, x86_64: installed
nvidia, 390.48, 4.8.16-040816-generic, x86_64: installed
dad@msi-GT70-2PE:~$ lsmod | grep nouveau
dad@msi-GT70-2PE:~$ dmesg | grep stub
[ 5.369546] pci-stub: add 10DE:1198 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[ 5.370702] pci-stub 0000:01:00.0: claimed by stub
[ 5.371705] pci-stub: add 10DE:0E0A sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[ 5.372709] pci-stub 0000:01:00.1: claimed by stub
[ 47.615229] pci-stub 0000:01:00.0: optimus capabilities: enabled, status dynamic power, hda bios codec supported
dad@msi-GT70-2PE:~$ lsmod | grep vf
vfio_pci 45056 1
vfio_iommu_type1 20480 1
vfio_virqfd 16384 1 vfio_pci
vfio 28672 5 vfio_iommu_type1,vfio_pci
irqbypass 16384 3 kvm,vfio_pci
Hi bash64,

1. Your uname -a shows that you are running an old 4.8.17 kernel. Try the latest one you installed.

2. Can't you use vfio-pci to bind the graphics card? You are using pci-stub.

3. Make sure you load the kernel modules in the right order. They should be listed in the following order in the /etc/initramfs-tools/modules file:

Code: Select all

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
vhost-net
Something is wrong with your modules list:
dad@msi-GT70-2PE:~$ lsmod | grep vf
vfio_pci 45056 1
vfio_iommu_type1 20480 1
vfio_virqfd 16384 1 vfio_pci
vfio 28672 5 vfio_iommu_type1,vfio_pci
irqbypass 16384 3 kvm,vfio_pci
The first and second line with a "1" at the end should list a module, or have "0". Here is mine as an example:

Code: Select all

 $ lsmod | grep vf
vfio_pci               40960  0
vfio_virqfd            16384  1 vfio_pci
irqbypass              16384  2 kvm,vfio_pci
vfio_iommu_type1       24576  0
vfio                   28672  2 vfio_iommu_type1,vfio_pci
4. Make sure you installed all relevant packages.

5. Don't forget it's a beta.

6. Sorry, I haven't had time to try it. I'm waiting for the stable release before I upgrade, and not without a complete backup before the upgrade.
Asus Sabertooth X79, i7 3930K CPU, 8x4GB Kingston DDR3, Noctua NH-D14 CPU cooler, GTX 970 + Quadro 2000 GPU, Asus Xonar Essence STX, Sandisk 120GB + Samsung EVO 860 1TB SSD + 4 HDD, Corsair 500R, SeaSonic 660W Gold X PS https://heiko-sieger.info

bash64
Level 2
Level 2
Posts: 96
Joined: Mon Mar 23, 2015 12:19 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by bash64 » Sun Jun 17, 2018 6:03 am

powerhouse,

beware. you need a full one for one backup.
i normally just use aptik to create a list of packages and to backup deb files.
i even went to the trouble of reinstalling mint 18.3 to a different partition and recovering.
i still cannot boot qemu.
i regret not veryfying all of the files it uses and what versions.

some were from repo, some were compiled and the seabios could have come from anywhere.

to clarify:
my virtual machines work fine. they boot in seconds if i use a vga adapter.
its when i boot with the nvidia card that they cannot boot.
i even left them running for hours and no boot.

i have switched to vfio-pci now. no more pci_stub:

~$ lsmod | grep vfio
vfio_pci 45056 0
vfio_virqfd 16384 1 vfio_pci
vfio_iommu_type1 20480 0
vfio 28672 2 vfio_iommu_type1,vfio_pci
irqbypass 16384 2 kvm,vfio_pci

still no dice.
is there another virtual machine that i can try to do the same thing.
willing to give it a shot.
p.s. i will step back through your tutorial and try again with mint 18.3.

bash64
Level 2
Level 2
Posts: 96
Joined: Mon Mar 23, 2015 12:19 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by bash64 » Sun Jun 17, 2018 10:18 am

i am stepping through your tutorial fresh.

i would like to get your help looking over my results to spot problems.

right now i am at the point of installing windows.

i am using seabios instead of uefi due to my laptop using nvidia optimus technology.

i cannot boot any iso image at all.
not windows 7 and not windows 10.
code 0005 error

or it just hangs after it tries to start booting
i also tried just a plain jane "-cdrom /pathtoiso"
no dice


i will wait to post long reports until i can at least get a fresh copy of windows installed.

i am not concerned about nvidia right now.
just need to get a test copy of any windows installed.
i am using all of your instructions.
i dont know what causes this bug but it is massively annoying.

p.s. using a compiled copy of qemu 2.12.50
any files it does not install come from the repo.


update: wrote w7 iso to usb thumb drive.
manually select thumb drive from seabios boot menu (hit esc).
no dice


#!/bin/bash

vmname="testvm"

if ps -A | grep -q $vmname; then
echo "$vmname is already running." &
exit 1

else

# use pulseaudio
export QEMU_AUDIO_DRV=pa
export QEMU_PA_SAMPLES=8192
export QEMU_AUDIO_TIMER_PERIOD=99
export QEMU_PA_SERVER=/run/user/1000/pulse/native

qemu-system-x86_64 \
-name $vmname,process=$vmname \
-machine type=q35,accel=kvm \
-cpu host,kvm=off \
-smp 8,sockets=1,cores=4,threads=2 \
-m 20G \
-bios /home/dad/seabios/out/bios.bin \
-mem-prealloc \
-rtc clock=host,base=localtime \
-serial none \
-parallel none \
-display gtk \
-soundhw hda \
-usb -device usb-host,hostbus=3,hostaddr=14 \
-device vfio-pci,host=01:00.0,multifunction=on \
-device vfio-pci,host=01:00.1 \
-boot order=dc \
-drive id=disk0,if=virtio,cache=none,format=raw,file=/home/dad/qemu/test7.img \
-drive file=/media/dad/QEMU-SSD/qemu-wd/w7u-sp1.iso,index=3,media=cdrom \
-drive file=/media/dad/QEMU-SSD/qemu-wd/virtio-win-0.1.149.iso,index=4,media=cdrom \
-netdev type=tap,id=net0,ifname=tap0,vhost=on \
-device virtio-net-pci,netdev=net0,mac=00:16:3e:00:01:01 \
-boot menu=on \
-boot order=dc

exit 0
fi

powerhouse
Level 6
Level 6
Posts: 1066
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse » Mon Jun 18, 2018 7:22 am

Hi bash64,

This does sound annoying, especially if you don't have a (full) backup.

One thing that sticks out is the qemu script:

Code: Select all

-smp 8,sockets=1,cores=4,threads=2 \
-m 20G \
It means that you pass 4 cores/8 threads to the VM, as well as 20 GB of memory. Seems rather a lot for a laptop and might prevent the guest from booting if you don't have that amount of memory.

If your PC is a 4 core machine with say 16GB memory, start with 2cores/4 threads and 8GB of memory. You can always add more and fine tune.

EDIT: I haven't tested / tried Ubuntu 18.04 / Linux Mint 19 yet, and will definitely wait until it is officially released. qemu / kvm is undergoing massive development and unfortunately they are changing their syntax on the way. Check the dmesg output. Also, make sure to run the script in a terminal window that allows you to follow the messages as they appear. Often they are the key to figuring out what's happening.

Before specifying the amount of memory for the VM, check how much PC memory you actually got:

Code: Select all

free
Recently I had two memory modules fail and it took me some head scratching until I actually figured out what was wrong and why my PC sometimes got "stuck".

Please note that my most up-to-date version of the tutorial is here: https://heiko-sieger.info/running-windo ... ssthrough/

At the end of the tutorial you'll find plenty of links and references to other tutorials and sites. Also don't forget to read your own posts here at this forum. If I remember correctly you posted some details on how you got it working with your Optimus laptop.
Asus Sabertooth X79, i7 3930K CPU, 8x4GB Kingston DDR3, Noctua NH-D14 CPU cooler, GTX 970 + Quadro 2000 GPU, Asus Xonar Essence STX, Sandisk 120GB + Samsung EVO 860 1TB SSD + 4 HDD, Corsair 500R, SeaSonic 660W Gold X PS https://heiko-sieger.info

bash64
Level 2
Level 2
Posts: 96
Joined: Mon Mar 23, 2015 12:19 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by bash64 » Mon Jun 18, 2018 8:10 am

thanks,

i upgraded the laptop to 32gb of ram.
when running fallout 4 i gave it 20gb because the game was a huge resource hog.
giving the machine all 4 cores with 8 threads was normal for me.
i can reduce it for now.
been trying endless seabios files.
no luck yet.
i'll wait a few weeks till i have tried quite a few things.
i am trying mint 18.3 as well as mint 19.
this is going to take some time.
i want to play fallout 76 online in november.
is there another gp passthrough virtual machine that i can try?

powerhouse
Level 6
Level 6
Posts: 1066
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse » Mon Jun 18, 2018 9:35 am

bash64 wrote:
Mon Jun 18, 2018 8:10 am
thanks,

i upgraded the laptop to 32gb of ram.
when running fallout 4 i gave it 20gb because the game was a huge resource hog.
giving the machine all 4 cores with 8 threads was normal for me.
i can reduce it for now.
been trying endless seabios files.
no luck yet.
i'll wait a few weeks till i have tried quite a few things.
i am trying mint 18.3 as well as mint 19.
this is going to take some time.
i want to play fallout 76 online in november.
is there another gp passthrough virtual machine that i can try?
OK, thanks for the answers. I read some of your earlier posts which may provide hints on how to get it to work. See viewtopic.php?f=231&t=212692&start=260#p1300634

When things don't work, start checking the basics. See if you installed all necessary packages including the seabios package. Start with an unmodified installation ISO, add the packages, make sure your PC is setup for passthrough (IOMMU enabled, BIOS settings, etc.). Update the packages, and install the latest kernel from the repository.

In your case, when using an Optimus laptop, you should probably stick to seabios, not the OVMF / UEFI method. This is slightly different as described in my tutorial.

With regards to other passthrough technologies or hypervisors, I don't think that switching horses will do much good. However, please check my links on my website for other tutorials that may be more relevant to your configuration.
Asus Sabertooth X79, i7 3930K CPU, 8x4GB Kingston DDR3, Noctua NH-D14 CPU cooler, GTX 970 + Quadro 2000 GPU, Asus Xonar Essence STX, Sandisk 120GB + Samsung EVO 860 1TB SSD + 4 HDD, Corsair 500R, SeaSonic 660W Gold X PS https://heiko-sieger.info

bash64
Level 2
Level 2
Posts: 96
Joined: Mon Mar 23, 2015 12:19 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by bash64 » Wed Jun 20, 2018 2:26 pm

Hey,

I have decided to work things out myself.
Sorry if panic mode got turned on.
I have worked on qemu for years, even providing how to docs here.
No need to panic now.

I will stick to helpful posts when I find helpful information.

I have determined that the issue is with seabios.
I was able to turn on debugging with seabios with this line:
-chardev stdio,id=seabios -device isa-debugcon,iobase=0x402,chardev=seabios \

here is the ouput of the important lines:
....
PCI: Enabling 01:00.0 for primary VGA

NOTE: 01.00.0 is my nvidia gtx 880m 8gb video card.....this line reads 00.01.0 when using the vga adapter...

PCI: Setting VGA enable on bridge 00:1c.0
handle_smp: apic_id=1
handle_smp: apic_id=5
handle_smp: apic_id=6
handle_smp: apic_id=2
handle_smp: apic_id=3
handle_smp: apic_id=7
handle_smp: apic_id=4
Found 8 cpu(s) max supported 8 cpu(s)
Copying PIR from 0x7ffbfca4 to 0x000f1710
Copying MPTABLE from 0x00006e0c/7ffa7230 to 0x000f15e0
Copying SMBIOS entry point from 0x00006e0c to 0x000f13f0
Scan for VGA option rom
Running option rom at c000:0003
Start SeaVGABIOS (version rel-1.11.0-0-g63451fca13-prebuilt.qemu-project.org)

You see that last line that says "Start SeaVGABIOS"?
that line never appears when using -vga none and getting a failed boot.

It only appears when I allow a vga adapter to be used and , thus, windows always boots...7 or 10. Doesn't matter which.

Running option rom is the last thing that happens on a failed boot.

I gotta go look this up, but I hope the seabios debug line is helpful.

bash64
Level 2
Level 2
Posts: 96
Joined: Mon Mar 23, 2015 12:19 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by bash64 » Thu Jun 21, 2018 12:26 pm

update:

kvm is definitely an issue.

seabios 1.7.x cannot boot my machines.
seabios 1.8.x does, but only with the vga adapter.
seabios 1.9.x and higher (up to 1.11.x) results in a kvm error.

make you wonder if kvm is the whole issue.
anyone else having problems?
strange that i can put back my system the way it was (mint 18.3, etc files...) and it wont boot without the vga adapter. windows 7/10 sees my gaming card perfectly. windows is not corrupted. seabios sees my gaming card. everything is right, but nothing boots.

p.s. got no proof this worked on mint 18.3 right before going to min 19.
i didn't exactly run it before the change.
i have both mint 18.3 and mint 19 installed at this time.
i am debugging mint 18.3.

an update could have come down right before the upgrade that broke it.
if that happened then mint 18.3 reinstalled has that update also.
i will try to revert to the earliest qemu packages in mint 18.3.

bash64
Level 2
Level 2
Posts: 96
Joined: Mon Mar 23, 2015 12:19 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by bash64 » Thu Jun 21, 2018 5:48 pm

VICTORY

sort of.

In the end I had to add nomodeset to my boot line.

Something is grabbing the video card that did not before.

I have tried:
1) modprobe.blacklist=nouveau
2) uninstall all nvidia software (only had to remove the dkms module before)

only nomodeset works which is a bummer. that means i have to operate in VESA mode in linux just so windows will boot.
no video acceleration. the linux side will be terrible at streaming videos.
ideas accepted.

bash64
Level 2
Level 2
Posts: 96
Joined: Mon Mar 23, 2015 12:19 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by bash64 » Thu Jun 21, 2018 9:03 pm

update:

the only driver loaded is i915.
but somehow it is grabbing the nvidia card.
i am using vfio-pci to grab the card on boot per the tutorial.
i can only think its the kms (kernel mode setting) kernel doing this.
i'd still rather not use nomodeset.

bash64
Level 2
Level 2
Posts: 96
Joined: Mon Mar 23, 2015 12:19 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by bash64 » Thu Jun 21, 2018 11:13 pm

TRIPLE VICTORY!!!

It not only works again in mint 18.3...I got it working in mint 19 beta.

The short answer on what broke....drumroll......i915.
The i915 driver causes the issue. Changes to this driver by Intel is causing issues with qemu with gpu passthrough for Optimus owners.

The quick fix is to blacklist i915.
On mint 18.3 add "nomodeset" to your boot line in /etc/default/grub and "sudo update-grub".

On mint 19 the i915 beast is harder to tame.
I made a blacklist file:
/etc/modprobe.d/blacklist-i915.conf
blacklist i915
install i915 /bin/false

I do not know if you need both lines...I used both and it worked.

The latest i915 driver will grab hold of you nvidia card if you have Optimus.
NOTE: One bad side effect is that any external monitors that the intel chip has access to are no longer available. I lost my external 15pin vga 26" monitor for the linux side. :-(


Victory #1 is mint 18.3 works once again.
Victory #2 is mint 19 also works.
Victory #3? I have always complained of it taking an excessively long time to boot windows. 4-8 minutes.
It now boots in exactly 60 seconds!

Perseverance has paid off again.
I am ready for you Fallout 76. Take my money please.

bash64
Level 2
Level 2
Posts: 96
Joined: Mon Mar 23, 2015 12:19 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by bash64 » Sun Jun 24, 2018 11:09 pm

UPDATE:
All you really need to get passthrough to work on mint 19 is nomodeset on your boot line and blacklist nouveau.

$ inxi -Gx
Graphics: Card-1: Intel 4th Gen Core Processor Integrated Graphics Controller bus-ID: 00:02.0
Card-2: NVIDIA GK104M [GeForce GTX 880M] bus-ID: 01:00.0
Display Server: x11 (X.Org 1.19.6 ) drivers: fbdev (unloaded: modesetting,vesa)
Resolution: 1920x1080@77.00hz
OpenGL: renderer: llvmpipe (LLVM 6.0, 256 bits) version: 3.3 Mesa 18.0.0-rc5 Direct Render: Yes


When it says fbdev, your there.

Since I use another kernel/ramdisk to boot with nvidia I had to remove the nvidia driver from dkms for the qemu kernel which was 4.15.0-22-generic.
$ dkms status
nvidia, 390.48, 4.15.0-23-generic, x86_64: installed
nvidia, 390.48, 4.15.0-22-generic, x86_64: installed

>dkms remove nvidia/390.48 -k 4.15.0-22-generic

So no nvidia driver for that kernel only, nomodeset, and blacklist nouveau. fbdev is the driver. qemu works.
Hope that is helpful.

I wanted to also add that I accepted all of the default qemu packages and even seabios 1.10.x and it all works.
I don't know why seabios 1.9.x and higher gave off kvm errors the first time I tried this.

After a reinstall of mint 19 it only took minutes to resetup everything and it all worked as expected.

powerhouse
Level 6
Level 6
Posts: 1066
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse » Mon Jun 25, 2018 8:57 am

@bash64: That's great news. Thanks for sharing all your steps. Perseverance certainly pays!

There is no way I could have helped you in this. I don't use the i915 driver (no internal GPU), no SeaBIOS either.

I wonder if nomodeset is really all that is needed. Do you mind posting your qemu script file? Any additional configurations?

Great work!
Asus Sabertooth X79, i7 3930K CPU, 8x4GB Kingston DDR3, Noctua NH-D14 CPU cooler, GTX 970 + Quadro 2000 GPU, Asus Xonar Essence STX, Sandisk 120GB + Samsung EVO 860 1TB SSD + 4 HDD, Corsair 500R, SeaSonic 660W Gold X PS https://heiko-sieger.info

User avatar
Adraido
Level 1
Level 1
Posts: 17
Joined: Mon Jun 25, 2018 9:17 pm

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by Adraido » Mon Jun 25, 2018 10:54 pm

This looks like a really advanced guide. I am thinking of going back to dual boot because of gaming issues and I looked at the first post. Seems pretty over my head in terms of doability but that's just me. It is great to know that this is possible and thank you for the guide.
"Hacking" abilities are limited to copy and pasting commands beyond that whatever you ask, most likely I'll have no clue and prefer to avoid inxi and what it's trying to fxz or what that even means.

bash64
Level 2
Level 2
Posts: 96
Joined: Mon Mar 23, 2015 12:19 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by bash64 » Thu Jun 28, 2018 11:48 pm

BTW: I am experimenting with Devuan linux (Debian 9, but you choose your init. SystemD is not forced on you).
It works fine there also with your tutorial.
I simply used nomodeset, blacklisted nouveau and then followed all of the steps in this tutorial.
nomodeset leaves my intel video running on the fbdev video driver.
fbdev is fine, it plays 1080p youtube videos just fine. i think it supports video acceleration.
It has been a hard lesson to learn that not only will the nvidia drivers keep passthrough from working, but so will any intel drivers on an nvidia optimus system.

script:
keep in mind that there are sections commented out for a reason.
also, at this time i have done so to the looking glass entries.
i dont' find looking glass very helpful.
even though pulseaudio is defined and can work, i no longer need it.
the 4k dell monitor i bought has internal speakers and the nvidia hdmi audio hardware passed off to windows works just fine.
the nbd section is for windows 7 only. i use a here-doc to block it out.
the vfio bind section is commented out. i now use vfio and not pci-stub. no need for this anymore.

MAN, I gotta lot of cleanup to do don't I?

#!/bin/bash

#lookingglass...

cd /media/dad/QEMU-SSD/qemu-wd

#comment out nbd section. may not work with windows 10...
<<NBDSECTION
echo 'Creating nbd device nodes...'
sudo modprobe nbd max_part=63
sleep 2
echo 'Attach hardfile to device node nbd0...'
qemu-nbd -f raw -c /dev/nbd0 windows7.img
echo 'Create folder to mount hardfile to...'
mkdir -p /media/w7
echo 'Mount nbd0p1 to the folder...'
mount /dev/nbd0p1 /media/w7
echo 'Display the contents of the mounted folder...'
ls /media/w7/Boot/
echo 'Replace BOOTSTAT.DAT with backup copy...'
cp BOOTSTAT.DAT /media/w7/Boot/
cp BCD.LOG /media/w7/Boot/
cp BCD.LOG1 /media/w7/Boot/
cp BCD.LOG2 /media/w7/Boot/
echo 'Wait 5 seconds...'
sleep 5
echo 'Unmount and remove folder...'
umount /media/w7
echo 'Disconnect nbd device node...'
qemu-nbd -d /dev/nbd0
NBDSECTION

configfile=/media/dad/QEMU-SSD/qemu-wd/vfio-pci1.cfg

#vfiobind function....
#vfiobind() {
# dev="$1"
# vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
# device=$(cat /sys/bus/pci/devices/$dev/device)
# echo $dev $vendor $device
# if [ -e /sys/bus/pci/devices/$dev/driver ]; then
# echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
# fi
# echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
#}

#modprobe vfio-pci

#cat $configfile | while read line;do
# echo $line | grep ^# >/dev/null 2>&1 && continue
# vfiobind $line
#done

# use pulseaudio
# use pulseaudio
export QEMU_AUDIO_DRV=pa
#export QEMU_PA_SAMPLES=8192
export QEMU_AUDIO_TIMER_PERIOD=99
export QEMU_PA_SERVER=/run/user/1000/pulse/native

#export QEMU_PA_BUFFER_SIZE_OUT=5000
#export QEMU_PA_BUFFER_SIZE_IN=5000
#export QEMU_PA_TLENGTH=5000
#export QEMU_PA_FRAGSIZE=5000

#use ALSA
#export QEMU_ALSA_DAC_BUFFER_SIZE=512
#export QEMU_ALSA_DAC_PERIOD_SIZE=170
#export QEMU_AUDIO_DRV=alsa

# display current time and expected boot time for Windows...
boottime1=$(date -d "$starttime today + 0 minutes" +'%I:%M:%S')
boottime2=$(date -d "$starttime today + 1 minutes + 13 seconds" +'%I:%M:%S')
echo "Windows expected to reach desktop at $boottime2..."

#launch windows 7 64bit using SEABIOS (non-uefi bios)....
#/home/dad/qemu/build/x86_64-softmmu/
#-chardev stdio,id=seabios -device isa-debugcon,iobase=0x402,chardev=seabios \
#-bios /media/dad/QEMU-SSD/qemu-wd/bios.bin-1.8.0 \
qemu-system-x86_64 -machine type=q35,accel=kvm -cpu host,kvm=off \
-smp 8,sockets=1,cores=4,threads=2 \
-bios /usr/share/seabios/bios.bin \
-chardev stdio,id=seabios -device isa-debugcon,iobase=0x402,chardev=seabios \
-m 20G \
-nodefaults \
-soundhw hda \
-mem-prealloc \
-rtc clock=host,base=localtime \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1 \
-device virtio-scsi-pci,id=scsi \
-drive id=disk0,if=virtio,cache=none,format=raw,file=/media/dad/QEMU-SSD/qemu-wd/windows10.img \
-drive file=/media/dad/QEMU-SSD/qemu-wd/virtio-win-0.1.126.iso,id=isocd,format=raw,if=none -device scsi-cd,drive=isocd \
-netdev type=tap,id=net0,ifname=tap0 \
-device virtio-net-pci,netdev=net0,mac=00:16:3e:00:01:01 \
-usb -device usb-host,vendorid=0x276d,productid=0x1119 \
-boot menu=on \
-boot order=c

exit 0

#lookingglass entries

#-object memory-backend-file,id=ivshmem,share=on,mem-path=/dev/shm/looking-glass,size=32M \
#-device ivshmem-plain,memdev=ivshmem \
#-spice port=5900,addr=127.0.0.1,disable-ticketing \

powerhouse
Level 6
Level 6
Posts: 1066
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse » Fri Jun 29, 2018 12:40 pm

Thanks bash64, this is very helpful. I have added your input to my tutorial here, with a link to your post.
Asus Sabertooth X79, i7 3930K CPU, 8x4GB Kingston DDR3, Noctua NH-D14 CPU cooler, GTX 970 + Quadro 2000 GPU, Asus Xonar Essence STX, Sandisk 120GB + Samsung EVO 860 1TB SSD + 4 HDD, Corsair 500R, SeaSonic 660W Gold X PS https://heiko-sieger.info

DoctorWho
Level 1
Level 1
Posts: 5
Joined: Wed Jun 27, 2018 11:34 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by DoctorWho » Wed Jul 04, 2018 6:50 pm

Hello everyone. I have been trying to get GPU passthru to work for quite some time. It would be the ultimate system
for me. But like many others I have been having troubles. I am hoping I can find a bit of guidance with this task.

First off my system, it is a bit old but still highly functional .... :)

System Specs:
MotherBoard: ASRock Z68 Pro Gen3
Bus Clock: 100 mhz
UEFI: American Megatrends Inc P1.30 08/15/2011
32 Gigs of system ram

CPU
Intel Core I7 2600
256 kb primary memory cache
1024 KB secondary memory cache
8192 KB tertiary memory cache
64-bit ready
Multi-Core (4 total)
Hyper-threaded: (8 total)

Video:
Onboard Intel 2000 HD graphics (primary GPU device)
NVIDIA GTX 980ti (GPU passed thru to windows VM)

So I have been running DUAL boot for quite awhile and have always wanted to get GPU passthru working. I have messed around with various hardwares to get to this point. So I finally have all the hardware in place and am attempting to GPU passthru to a Windows VM following this excellent guide.

First I have IO_MMU enabled:

[    0.000000] ACPI: DMAR 0x000000006EDC4AC8 0000E8 (v01 ALASKA A M I    00000001 INTL 00000001)
[    0.000000] DMAR: IOMMU enabled
[    0.018500] DMAR: Host address width 36
[    0.018501] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.018507] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a
[    0.018508] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.018511] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a
[    0.018511] DMAR: RMRR base: 0x0000006edd9000 end: 0x0000006edecfff
[    0.018512] DMAR: RMRR base: 0x0000006f800000 end: 0x0000007f9fffff
[    0.018514] DMAR-IR: IOAPIC id 0 under DRHD base  0xfed91000 IOMMU 1
[    0.018515] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.018516] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.018794] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    1.034203] DMAR: No ATSR found
[    1.034815] DMAR: dmar0: Using Queued invalidation
[    1.034821] DMAR: dmar1: Using Queued invalidation
[    1.035068] DMAR: Setting RMRR:
[    1.035109] DMAR: Setting identity map for device 0000:00:02.0 [0x6f800000 - 0x7f9fffff]
[    1.036268] DMAR: Setting identity map for device 0000:00:1a.0 [0x6edd9000 - 0x6edecfff]
[    1.036302] DMAR: Setting identity map for device 0000:00:1d.0 [0x6edd9000 - 0x6edecfff]
[    1.036310] DMAR: Prepare 0-16MiB unity mapping for LPC
[    1.036349] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    1.036617] DMAR: Intel(R) Virtualization Technology for Directed I/O
[    1.509917] [drm] DMAR active, disabling use of stolen memory

Next according to this guide, and where I am currently stuck, is to look at IO_MMU groups. According to this guide GPU passthru will not work unless my GPU and Sound card are in their own group. Of course on my system this is not the case. A bit of poking around shows me that Group one is where my Video card is (GTX 980ti).

My Groups:

/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/1/devices/0000:01:00.1
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/1/devices/0000:01:00.0

/sys/kernel/iommu_groups/10/devices/0000:00:1d.0
/sys/kernel/iommu_groups/11/devices/0000:00:1f.3
/sys/kernel/iommu_groups/11/devices/0000:00:1f.2
/sys/kernel/iommu_groups/11/devices/0000:00:1f.0
/sys/kernel/iommu_groups/12/devices/0000:03:00.0
/sys/kernel/iommu_groups/13/devices/0000:04:00.0
/sys/kernel/iommu_groups/14/devices/0000:05:00.0
/sys/kernel/iommu_groups/15/devices/0000:06:01.0
/sys/kernel/iommu_groups/16/devices/0000:06:04.0
/sys/kernel/iommu_groups/17/devices/0000:06:05.0
/sys/kernel/iommu_groups/18/devices/0000:06:06.0
/sys/kernel/iommu_groups/19/devices/0000:06:07.0
/sys/kernel/iommu_groups/2/devices/0000:00:02.0
/sys/kernel/iommu_groups/20/devices/0000:06:08.0
/sys/kernel/iommu_groups/21/devices/0000:06:09.0
/sys/kernel/iommu_groups/22/devices/0000:07:00.0
/sys/kernel/iommu_groups/23/devices/0000:08:00.0
/sys/kernel/iommu_groups/24/devices/0000:0a:00.0
/sys/kernel/iommu_groups/25/devices/0000:0b:00.0
/sys/kernel/iommu_groups/26/devices/0000:0c:00.0
/sys/kernel/iommu_groups/27/devices/0000:0d:00.0
/sys/kernel/iommu_groups/28/devices/0000:0e:00.1
/sys/kernel/iommu_groups/28/devices/0000:0e:00.0
/sys/kernel/iommu_groups/3/devices/0000:00:16.0
/sys/kernel/iommu_groups/4/devices/0000:00:1a.0
/sys/kernel/iommu_groups/5/devices/0000:00:1b.0
/sys/kernel/iommu_groups/6/devices/0000:00:1c.0
/sys/kernel/iommu_groups/7/devices/0000:00:1c.4
/sys/kernel/iommu_groups/8/devices/0000:00:1c.6
/sys/kernel/iommu_groups/9/devices/0000:00:1c.7

My Video Card is in Group One

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:1b06] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10ef] (rev a1)

Along with this device:

00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port [8086:01
01] (rev 09)

This guide says my video card MUST be in a separate group in order to work, but as you can see I have another device present in group one along with my Video card. It is a system Root Port. How do I get around this? I have tried moving the card to another PCI slot but it just adds that slot to Group one actually making things worse. This hardware combination SHOULD work but I still seem to have some configuration issues. Any ideas as to how I can isolate my video card in a separate group? Is it perhaps safe to ignore the Root device and proceed with the guide? I have been trying to get this to work FOREVER, so far this is the first time I have managed to get IO_MMU successfully enabled, but I digress…..

Any response is GREATLY APPRECIATED !

Thanx
Doc

powerhouse
Level 6
Level 6
Posts: 1066
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse » Fri Jul 06, 2018 4:19 am

Dear Doc,

00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port [8086:01
01] (rev 09)

As you have shown, 00:01.0 is a root port. You can pass through the graphics card and sound part, but don't pass through this root port. According to reports, this should work.

Let us know how it goes.
Asus Sabertooth X79, i7 3930K CPU, 8x4GB Kingston DDR3, Noctua NH-D14 CPU cooler, GTX 970 + Quadro 2000 GPU, Asus Xonar Essence STX, Sandisk 120GB + Samsung EVO 860 1TB SSD + 4 HDD, Corsair 500R, SeaSonic 660W Gold X PS https://heiko-sieger.info

DoctorWho
Level 1
Level 1
Posts: 5
Joined: Wed Jun 27, 2018 11:34 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by DoctorWho » Fri Jul 06, 2018 4:36 pm

Thanx for the Reply. I am making great progress with this now, I got to the
TURINO boot screen but then it tries to do some PXE boot, and dumps me
into the UEFI BIOS options. It does not seem to use the iso file i have specifed,
i will keep messing with it .... I suppose the fact that the TURINO boot screen
appeared at all indicates that the video card passed thru .... Woo Hoo!! 8)

Post Reply

Return to “Virtualization”