HOW-TO make dual-boot obsolete using kvm VGA passthrough

[powerhouse]

All this work to find it will not work with my laptop. And I went the extra mile and spent money on one that had a separate GPU. I've come to realize it just doesn't bother anyone anymore that even now more and more hardware manufacturers are locking us into the Microsoft machine. Maybe it's time to wave the white flag and just embrace the borg...

I do not recommend anyone try this with a laptop. I assume the laptop has nvidia synergy (nvidia/intel). That is not recommended by myself or by Powerhouse who owns this article.

It worked for me by using seabios rather than ovmf. I also use pci_stub instead of vfio-pci to reserve my video card for qemu. Perhaps I can help you get it working. I can even do a Teamviewer session if you have the time. Provide me a way to contact you if you want help. If you do a friend request to "Linux Mint Survival Guides" (I prefer this one) or "Roger Lee Lawhorn" on facebook we can use facebook chat. Sorry, no cell phone, at least not with any minutes left.

Here are some ebooks I have written at lulu.com:
http://www.lulu.com/spotlight/bash64

The mint 18 ebook contains a chapter on this with my own insights and how I made this work on a laptop with Synergy.

Hello. I want to make dual-boot obsolete on my laptop(Dell Alienware 17 R3 with Integrated Intel HD 530 HD + Nvidia GeForce GTX 970M 3GB) using this tutorial. So far i had no luck using other tutorials that was based on Xen Project HVM. I am wondering why you dont recommend to try it with a laptop. Please, explain that to me and give me advice if possible(about installing HVM on laptop, not buying PC ).

bash64 already gave a good answer.

I like to add: it depends. You need to have a careful look at your hardware. First check the CPU that it supports VT-d - the i7-6700HQ looks good. But the real problem is Nvidia Optimus. For an explanation on what that means and how the Linux community tries to answer that issue, see Bumblebee.

Bumblebee is the Linux equivalent of Optimus, a technology that allows switching GPU depending on the graphics workload. In theory it sounds nice, in practice it has some caveats.

Before you try anything, BACK UP your entire disk so that you can restore your Windows or Linux when necessary!!!

When running Windows in a VM, you want Linux to use the HD530 graphics while Windows uses the Nvidia GPU. As a first step you need to turn off Optimus in your BIOS (or turn off switchable graphics) and/or select the internal graphics as primary graphics (the one use to boot the system). Then boot into Linux and follow the tutorial. The very first thing you probably want to check is that you can bind the Nvidia GPU to vfio-pci (see part 4 of the tutorial).

If you want to use your wireless connection, skip the network bridge configuration and do not configure networking in the qemu command. Qemu should automatically create a routed interface. As an alternative or when this fails, follow the advice of bash64 above.

If the tutorial and specifically the UEFI boot into Windows do not work for you, you can still try bash64's SeaBIOS approach. However, with Windows 10 you should be able to use UEFI.

In summary, you do NOT want Optimus or Bumblebee to interfere with your GPUs. After you have disabled Optimus in the BIOS and/or selected the internal graphics as primary graphics, you may need to reinstall Linux Mint or somehow disable Bumblebee. But before you even think about reinstalling Linux Mint, try to find answers to the following questions: how is your hardware wired inside? Does the Nvidia GPU have an external port you could hook to an external display? Are both the internal GPU and the Nvidia GPU wired to the laptop screen, with a button to switch between them? The answers to these will determine whether or not you can use your hardware for VGA passthrough.

Hope you succeed. If yes, please share how.

[diabolusss]

Thank you for a great response. I have some result already, but i'm not satisfied with it: i received Code 43. From this point i have no clues what to do. And i would appreciate any help.
I've seen info about this error somewhere, but i can't find where. I found answer on page 14. And it seems that i should bet on Win10 or try seaBIOS solution...

UPDATE: Win10 doesnt make any difference, so seaBIOS is my last option, but i couldn't find any good tutorials about it. Please, point me in the right direction.

MACHINE SPECS

Code: Select all

DELL Alienware 17 R3, BIOS 1.3.6 08/05/2016
	- Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz
	- NVIDIA Corporation GM204M [GeForce GTX 970M]
	- Intel HD Graphics 530

What i did and results:
1. I have enabled virtualization in BIOS

2. Disabled nvidia driver and enabled iommu in grub

Code: Select all

> GRUB_CMDLINE_LINUX_DEFAULT="modprobe.blacklist=nvidia quiet splash intel_iommu=on"
		result:
		>[    0.759549] DMAR: Intel(R) Virtualization Technology for Directed I/O

3. Found video devices and sound(related devices i writed down, too)
However, i'm passing through only 3D controller for now.

Code: Select all

> lspci -nn		
		00:01.0 PCI bridge [0604]: Intel Corporation Sky Lake PCIe Controller (x16) [8086:1901] (rev 07)
		00:02.0 VGA compatible controller [0300]: Intel Corporation Skylake Integrated Graphics [8086:191b] (rev 06)
		00:1f.0 ISA bridge [0601]: Intel Corporation Sunrise Point-H LPC Controller [8086:a14e] (rev 31)
		00:1f.2 Memory controller [0580]: Intel Corporation Sunrise Point-H PMC [8086:a121] (rev 31)
		00:1f.3 Audio device [0403]: Intel Corporation Sunrise Point-H HD Audio [8086:a170] (rev 31)
		00:1f.4 SMBus [0c05]: Intel Corporation Sunrise Point-H SMBus [8086:a123] (rev 31)
		01:00.0 3D controller [0302]: NVIDIA Corporation GM204M [GeForce GTX 970M] [10de:13d8] (rev a1)

Code: Select all

> find /sys/kernel/iommu_groups/ -type l
		/sys/kernel/iommu_groups/1/devices/0000:00:01.0
		/sys/kernel/iommu_groups/1/devices/0000:01:00.0
		/sys/kernel/iommu_groups/2/devices/0000:00:02.0
		/sys/kernel/iommu_groups/9/devices/0000:00:1f.0
		/sys/kernel/iommu_groups/9/devices/0000:00:1f.2
		/sys/kernel/iommu_groups/9/devices/0000:00:1f.3
		/sys/kernel/iommu_groups/9/devices/0000:00:1f.4

Q1. At this point i see that PCI bridge is in one group with NVIDIA card. Do i need to passthrough it or not?
- I tried to send it VM, but it fails. And i suppose i shouldn't do that.

Q2. As i understand, i dont need to install drivers for NVIDIA card to passthrough it to VM. I need them to use it on host, right?

4. Found mouse to passthrough and created static network bridge, setuped hugepages, installed ovmf and got vfio drivers

5. In order to make the graphics card available to the Windows VM, i assigned a "dummy" driver as a place holder: vfio-pci.

Code: Select all

> /etc/modprobe.d/local.conf
		> options vfio-pci ids=8086:1901,10de:13d8

Code: Select all

> cat /etc/modprobe.d/kvm.conf
		> options kvm ignore_msrs=1

Code: Select all

> cat /etc/initramfs-tools/modules
		> vfio
		> vfio_iommu_type1
		> vfio_pci
		> vfio_virqfd
		> vhost-net

Code: Select all

dmesg at boot for vfio
		> [    1.613167] VFIO - User Level meta-driver version: 0.3
		> [    1.623338] vfio_pci: add [8086:1901[ffff:ffff]] class 0x000000/00000000
		> [    1.637120] vfio_pci: add [10de:13d8[ffff:ffff]] class 0x000000/00000000

6. FInal check

Code: Select all

> kvm-ok
		> INFO: /dev/kvm exists
		> KVM acceleration can be used

Code: Select all

> lsmod | grep kvm
		> kvm_intel             172032  0
		> kvm                   536576  1 kvm_intel
		> irqbypass              16384  2 kvm,vfio_pci

Code: Select all

> lsmod | grep vfio
		> vfio_pci               40960  0
		> irqbypass              16384  2 kvm,vfio_pci
		> vfio_virqfd            16384  1 vfio_pci
		> vfio_iommu_type1       20480  0
		> vfio                   28672  2 vfio_iommu_type1,vfio_pci

Code: Select all

> qemu-system-x86_64 --version
		> QEMU emulator version 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.14), Copyright (c) 2003-2008 Fabrice Bellard

vm_startup_script:

Code: Select all

qemu-system-x86_64 \
		  -name $vmname,process=$vmname \
		  -machine type=q35,accel=kvm \
		  -cpu host,kvm=off \
		  -smp 8,sockets=1,cores=4,threads=2 \
		  -enable-kvm \
		  -m 4G \
		  -mem-path /dev/hugepages \
		  -mem-prealloc \
		  -balloon none \
		  -rtc clock=host,base=localtime \
		  -serial none \
		  -parallel none \
		  -usb -usbdevice host:15d9:0a4c \
		  -device vfio-pci,host=01:00.0 \
		  -drive if=pflash,format=raw,readonly,file=/usr/share/OVMF/OVMF_CODE.fd \
		  -drive if=pflash,format=raw,file=/tmp/my_vars.fd \
		  -boot order=c \
		  -device virtio-scsi-pci,id=scsi \
		  -drive id=disk0,if=virtio,cache=none,format=raw,file=/media/colt/_DATA/win.img \
		  -netdev type=tap,id=net0,ifname=tap0,vhost=on \
		  -device virtio-net-pci,netdev=net0,mac=00:16:3e:00:01:01

7. Result
dmesg vfio

Code: Select all

> [ 7363.282162] vfio-pci 0000:01:00.0: enabling device (0400 -> 0403)
		> [ 7364.294350] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1e@0x258
		> [ 7364.294375] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900

- i managed to install Win8.1 and latest driver for NVIDIA card(it was detected by windows). But it doesn't work.
- After boot in windows it says that device is not working properly. -> Code 43 =\
- In windows device manager i tried to Disable\Enable device and it says that device is working properly, but i can't o\p to monitor.(However, i did it qemu monitor).

Code: Select all

> lspci -k | grep -i -A 3 3D
		> 01:00.0 3D controller: NVIDIA Corporation GM204M [GeForce GTX 970M] (rev a1)
		> Subsystem: Dell GM204M [GeForce GTX 970M]
		> Kernel driver in use: vfio-pci
		> Kernel modules: nvidiafb, nouveau, nvidia_375_drm, nvidia_375

From this point i have no clues what to do...

Code: Select all

=== VIDEO
== INTEGRATED: Intel HD Graphics 530
 	- uses default i915_bpo driver
	- has no o\p
	- sound card is binded to it
	- dmesg for pci
		[    0.282950] vgaarb: setting as boot device: PCI:0000:00:02.0
		[    0.282952] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none
		[    0.282954] vgaarb: loaded
		[    0.282955] vgaarb: bridge control possible 0000:00:02.0
		[    0.327609] pci 0000:00:02.0: Video device with shadowed ROM
		[    0.766205] fb0: EFI VGA frame buffer device
		[    0.943397] fb: switching to inteldrmfb from EFI VGA
		[    0.959093] vgaarb: device changed decodes: PCI:0000:00:02.0,olddecodes=io+mem,decodes=io+mem:owns=io+mem
		[    0.982854] ACPI: Video Device [GFX0] (multi-head: yes  rom: no  post: no)
		[    0.983073] input: Video Bus as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/LNXVIDEO:00/input/input6
		[    0.983169] ACPI Exception: AE_NOT_FOUND, Evaluating _DOD (20150930/video-1216)
		[    0.983171] ACPI: Video Device [PEGP] (multi-head: no  rom: yes  post: no)
		[    0.983195] input: Video Bus as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/device:12/LNXVIDEO:01/input/input7Result
		[    0.983391] [drm] Initialized i915_bpo 1.6.0 20160229 for 0000:00:02.0 on minor 0
		[    1.237608] i915_bpo 0000:00:02.0: fb0: inteldrmfb frame buffer device
		[    3.966241] snd_hda_intel 0000:00:1f.3: enabling device (0000 -> 0002)
		[    3.966273] snd_hda_intel 0000:00:1f.3: bound 0000:00:02.0 (ops i915_audio_component_bind_ops [i915_bpo])
		[    4.799698] input: HDA Intel PCH Front Line Out as /devices/pci0000:00/0000:00:1f.3/sound/card0/input10
		[    4.799786] input: HDA Intel PCH Line Out as /devices/pci0000:00/0000:00:1f.3/sound/card0/input11
		[    4.799883] input: HDA Intel PCH HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:1f.3/sound/card0/input12
		[    4.801268] input: HDA Intel PCH HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:1f.3/sound/card0/input13
		[    4.801362] input: HDA Intel PCH HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:1f.3/sound/card0/input14

== NVIDIA Corporation GM204M [GeForce GTX 970M]
 	- has hdmi port and integrated card has no access to it(i think so, because i managed to output smth to it only if 3D card is enabled)
	- by default uses noveau driver, but i installed nvidia driver to find out why hdmi port doesnt recognize external monitor(i thought it's broken, but it's not)

	- has bridge in group
		- o\p from dmesg
		[    0.327019] pci 0000:00:01.0: PCI bridge to [bus 01]
		[    0.327020] pci 0000:00:01.0:   bridge window [io  0xe000-0xefff]
		[    0.327022] pci 0000:00:01.0:   bridge window [mem 0xdc000000-0xdd0fffff]
		[    0.327024] pci 0000:00:01.0:   bridge window [mem 0xb0000000-0xc1ffffff 64bit pref]
		[    0.759598] iommu: Adding device 0000:00:01.0 to group 1
		[    0.759735] iommu: Adding device 0000:01:00.0 to group 1

[powerhouse]

@diabolusss: Below are some suggestions that might help.

1. Use -smp 6,sockets=1,cores=3,threads=2 \ in the qemu command. This way you won't assign all CPU resources to your VM. (I doubt this has any influence on code 43, but it's better to give the host enough resources to handle I/O etc.)

2. Try -cpu host,kvm=off,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_vendor_id=Nvidia43FIX

Then reboot and see if this makes a difference. Usually kvm=off should do the trick, but I've read suggestions that the above might also be necessary. The kvm documentation is unfortunately all over the place . See Alex' blog for more http://vfio.blogspot.co.il/2015/05/vfio ... -host.html.

3. Make sure your Nvidia graphics card supports UEFI. See https://wiki.archlinux.org/index.php/PC ... y_in_VBIOS. Check which model you have: https://www.techpowerup.com/vgabios/?ar ... ze=&since=.

If the current GPU BIOS doesn't support UEFI, you can perhaps find a modified BIOS that does support UEFI and specify the BIOS in the qemu command like that:

Code: Select all

-device vfio-pci,host=01:00.0,romfile=/path/to/your/gpu/bios.bin \

If you cannot get a UEFI BIOS for your graphics card, try SeaBIOS. See the script of bash64 further up. You will need to reinstall Windows.

I strongly suggest to get UEFI BIOS working, as with SeaBIOS you may also have to patch the kernel with the VGA arbiter patch. In general, UEFI is the preferred method.

4. If your Nvidia card BIOS supports UEFI and you tried all of the above without success, use MSI interrupts - see https://www.redhat.com/archives/vfio-us ... 00148.html

I like to clarify the current state:

a. You installed the Windows 8.1 VM?
b. Windows boots in UEFI mode (using the OVMF driver)?
c. Once booted into Windows, you installed the Nvidia driver?
d. When you boot Windows, the screen switches to the Windows screen? (Windows does not run in a qemu window?)
e. The Nvidia card uses the Windows VGA driver, instead of the Nvidia driver? (There is a code 43 error displayed in device manager?)


In answer to your questions:

Q1 - about IOMMU group 1: the first entry 0000:00:01.0 is a root port, which you do NOT pass through!

Q2 - install Nvidia drivers: You must install the Nvidia driver in the Windows VM !!! You do NOT use the Nvidia driver on the host. Install Windows and reboot into the Windows VM. Go to the Nvidia website and download the utility to identify your GPU. Alternatively, select the correct card in the menus. Download and install the driver. If needed, reboot.

[powerhouse]

Ryzen and VGA passthrough

Some of you might be interested in the following tutorial for AMD Ryzen systems: https://level1techs.com/article/ryzen-g ... ming-linux

It is written for the newest Fedora 26 using a 4.11 kernel (Ubuntu/Linux Mint is on 4.10). According to the author, Ryzen does require the newest kernel.

See also the Youtube video embedded in the post - it gives a lot of background information on VGA passthrough.

[procabiak]

Hey powerhouse, thanks for the guide! I've followed through most of it, and now stuck at the very end. Hoping to get some help with my issue. I got the VM booted up, everything works fine kinda, until I installed the AMD drivers. It would boot but after a few seconds from the win10 login screen, the vm hangs.

I have a Ryzen system so I thought there might be something hardware-wise that is blocking. But I watched & read through Wendell's Ryzen video/guide you just posted, it doesn't seem there is much Ryzen specific changes other than npt/avic that needs to be catered for.

My system:
CPU: Ryzen 1700
Mobo: X370 Taichi (2.40/Agesa 1006 bios)
GPU: two reference R9 290Xs (flashed with a modded 390x rom to enable UEFI, from here)

dmesg output tells me there's something wrong when it crashes (IOTLB_INV_TIMEOUT):

Code: Select all

[  130.788158] vfio-pci 0000:0f:00.0: enabling device (0002 -> 0003)
[  130.788329] vfio_ecap_init: 0000:0f:00.0 hiding ecap 0x19@0x270
[  130.788335] vfio_ecap_init: 0000:0f:00.0 hiding ecap 0x1b@0x2d0
[  130.808141] vfio-pci 0000:0f:00.1: enabling device (0000 -> 0002)
[  130.995095] AMD-Vi: Completion-Wait loop timed out
[  131.161264] AMD-Vi: Completion-Wait loop timed out
[  131.327455] AMD-Vi: Completion-Wait loop timed out
[  131.493567] AMD-Vi: Completion-Wait loop timed out
[  131.678161] AMD-Vi: Completion-Wait loop timed out
[  131.828168] AMD-Vi: Event logged [
[  131.828173] IOTLB_INV_TIMEOUT device=0f:00.0 address=0x0000000ff80929b0]
[  138.477778] usb 1-12: reset full-speed USB device number 6 using xhci_hcd
[  139.105908] usb 1-12: reset full-speed USB device number 6 using xhci_hcd
[  139.674300] usb 1-12: reset full-speed USB device number 6 using xhci_hcd
[  144.463830] usb 1-12: reset full-speed USB device number 6 using xhci_hcd
[  144.924143] usb 1-12: reset full-speed USB device number 6 using xhci_hcd
[  145.492103] usb 1-12: reset full-speed USB device number 6 using xhci_hcd
[  145.943791] usb 1-12: reset full-speed USB device number 6 using xhci_hcd
[  149.521527] usb 1-12: reset full-speed USB device number 6 using xhci_hcd
[  149.969389] usb 1-12: reset full-speed USB device number 6 using xhci_hcd
[  150.577306] usb 1-12: reset full-speed USB device number 6 using xhci_hcd
[  151.752811] usb 1-12: reset full-speed USB device number 6 using xhci_hcd
[  152.912391] usb 1-12: reset full-speed USB device number 6 using xhci_hcd
[  153.412070] usb 1-12: reset full-speed USB device number 6 using xhci_hcd
[  168.853714] do_IRQ: 0.41 No irq handler for vector
[  172.658029] do_IRQ: 0.41 No irq handler for vector
[  172.724770] do_IRQ: 0.41 No irq handler for vector

dmesg | grep AMD-Vi (before booting VM):

Code: Select all

[    1.305146] AMD-Vi: IOMMU performance counters supported
[    1.307326] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
[    1.307327] AMD-Vi: Extended features (0xf77ef22294ada):
[    1.307330] AMD-Vi: Interrupt remapping enabled
[    1.307330] AMD-Vi: virtual APIC enabled
[    1.307447] AMD-Vi: Lazy IO/TLB flushing enabled

/etc/default/grub:

Code: Select all

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash radeon.modeset=0 modprobe.blacklist=amdgpu iommu=1 amd_iommu=on"

my vm script:

Code: Select all

#!/bin/bash

vm_name=win10-mypc
mem_size=12G
mem_pages=/run/hugepages/kvm
pci_gpu=0f:00.0
pci_audio=0f:00.1
usb_kboard=046d:c52b
dir_os=/dev/nvme0n1p5
dir_bios=/usr/share/OVMF/OVMF_CODE.fd
dir_bios_cfg=/tmp/$vm_name.fd
dir_cd_install=/home/procabiak/KVM/ISOs/win10.iso
dir_cd_drivers=/home/procabiak/KVM/ISOs/virtio-win.iso

if ps -A | grep -q $vm_name; then
   echo "$vm_name is already running." &
   exit 1
fi

cp /usr/share/OVMF/OVMF_VARS.fd $dir_bios_cfg

taskset -c 0-7 qemu-system-x86_64 \
  -name $vmname,process=$vmname \
  -machine type=q35,accel=kvm \
  -cpu host \
  -smp 8,sockets=1,cores=8,threads=1 \
  -enable-kvm \
  -m $mem_size \
  -mem-path $mem_pages \
  -mem-prealloc \
  -balloon none \
  -rtc clock=host,base=localtime \
  -vga none \
  -nographic \
  -serial none \
  -parallel none \
  -usb \
  -usbdevice host:$usb_kboard \
  -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
  -device vfio-pci,host=$pci_gpu,bus=root.1,addr=00.0,multifunction=on \
  -device vfio-pci,host=$pci_audio,bus=pcie.0,multifunction=on \
  -drive if=pflash,format=raw,readonly,file=$dir_bios \
  -drive if=pflash,format=raw,file=$dir_bios_cfg \
  -boot order=dc \
  -device virtio-scsi-pci,id=scsi \
  -drive id=disk0,if=virtio,cache=none,format=raw,file=$dir_os \
  -drive file=$dir_cd_install,id=isocd,format=raw,if=none -device scsi-cd,drive=isocd \
  -drive file=$dir_cd_drivers,id=virtiocd,format=raw,if=none -device ide-cd,bus=ide.1,drive=virtiocd
  
exit 0

/etc/modprobe.d/kvm.conf, iirc npt & avic are disabled by default anyway:

Code: Select all

# options kvm ignore_msrs=1
options kvm_amd nested=0
options kvm_amd avic=0

[diabolusss]

@powerhouse, thank you, for response.

Current state is:

• a. You installed the Windows 8.1 VM?
  - First I installed Win8.1, but it doesnt worked out, so I have installed win10.
• b. Windows boots in UEFI mode (using the OVMF driver)?
  - yes, definitely.
• c. Once booted into Windows, you installed the Nvidia driver?
  - yes. Actually, windows have installed default driver for Nvidia, but I just updated it with newest one from nvidia site.
• d. When you boot Windows, the screen switches to the Windows screen? (Windows does not run in a qemu window?)
  - I installed Windows and nvidia driver without “vga none, nographic” option and after that I started VM with vga none option. I waited for 10 or more minutes, but nothing changed – screen is black. So, yes, I have only qemu window which is Microsoft Basic Display Adapter.
• e. The Nvidia card uses the Windows VGA driver, instead of the Nvidia driver? (There is a code 43 error displayed in device manager)?
  - yes, there is Code 43 error and NVIDIA card is disabled. But it should use newly installed nvidia driver(v384.76).

That is what I got from rom and it doesnt have EFI part =\:

Code: Select all

Valid ROM signature found @0h, PCIR offset 1a0h
	PCIR: type 0 (x86 PC-AT), vendor: 10de, device: 13d8, class: 030000
	PCIR: revision 3, vendor revision: 1
	Last image

But it’s strange that I managed to retrieve o\p only after I started VM with pass-through, before that I got(maybe, the reason is vfio-pci):

Code: Select all

[  352.653214] vfio-pci 0000:01:00.0: Invalid ROM contents
[  374.517640] vfio-pci 0000:01:00.0: Invalid ROM contents
[  386.371102] vfio-pci 0000:01:00.0: Invalid ROM contents

I have tried to boot with custom ROM that have UEFI, but still no success.

Q1(if I’ll solve the problem). If I start VM without “-vga none -nographic” flags, will my NVIDIA card be working in VM. I mean could I use both default and HDMI screens? Or if i’ll start VM with enabled VNC will NVIDIA card be working?

Q2. When I use custom ROM do I need to re-install card drivers? And how can I check that rom was loaded successfully?

Q3. Maybe I should try other UEFI enabled ROMs for my card?

[powerhouse]

@diabolusss: I'm afraid I have little to no experience with loading drivers. But back to the basics:

1. Without the "vga none, nographic" Windows sees another screen, this time a virtual screen and provides a simple VGA driver for it. It has nothing to do with the Nvidia card.

2. Make sure that you have the correct UEFI driver for your card. You need to find out your exact make and model (please share here), with the correct memory size. I've seen 6GB versions of the 980M that have UEFI drivers whereas the 3GB versions didn't support UEFI.

3. When you boot the VM without "vga none,..." and you get a qemu window, does the device manager show 2 separate graphics displays? A VGA graphics and the Nvidia graphics (with the error triangle)?

4. Are you using the laptop display or have you connected an external display?

5. When you boot the laptop and check the dmesg contents for vfio-pci messages, you get "invalid ROM contents"? What happens when you boot the VM? Is it correct that your started the VM and only then retrieved the ROM contents to check for UEFI capabilities?

In my opinion there are 2 possible scenarios that explain what you see:

I. Windows VM boots in UEFI mode and passthrough works. The only problem is that the output of the Nvidia GPU is not to the screen you use/have connected.
Try to hook up an external screen to the HDMI or whatever output(s). See if there are soft switches on the keyboard that allow you to switch between internal and external screen and try them.
However, the same phenomenon could also have to do with the laptop and the fact it uses Nvidia Optimus and there might not be a workaround.
The best option to try now is to not use UEFI boot, but the Seabios option. In that case you do not use the OVMF driver. See further up, I think bash64 posted an example for the qemu command. You can also follow the link to the Arch Linux forum that I posted in my how-to. You'll see that Seabios is even simpler. The possible caveat with Seabios is that you may have to patch your Linux kernel with the VGA arbiter patch. But let's go one step at a time.

II. The Windows VM hangs at the UEFI boot, probably because of the graphics card respectively the UEFI bios that doesn't work with your card. Try the Seabios option (see above).

In answer to your questions:

Q1: Yes. You should see two graphics devices in your Windows device manager - 1 VGA device (the emulated device shown in the qemu window) and 1 Nvidia GPU. When everything works as it should, Windows should switch to the Nvidia GPU.

Q2: I don't know, but possibly you have to reinstal the GPU driver. I don't have the answer to the second question, but please search the internet as this is indeed a crutial question.

Q3: Make sure you got the correct GPU BIOS. There are differences from manufacturer to manufacturer, and from model to model. If you can't find a 100% matching UEFI driver, go the Seabios way as suggested above.

[powerhouse]

@procabiac: Try to only install the AMD GPU driver, not the whole package. It has been reported that the control center or whatever it's called can cause problems.

If that doesn't help, see and try to follow the how-to in the link I posted. Of course you need to get the 4.11 kernel (I have 4.10). In the video he explains that the latest kernel is required. Go to the kernel sources and check for the changelog. You may find some notes related to AMD and perhaps Ryzen.

Tomorrow I hope to have a better look at your post, but now I need to get going. Good luck!

EDIT: Can you post the output of:

Code: Select all

#!/bin/bash
shopt -s nullglob
for d in /sys/kernel/iommu_groups/*/devices/*; do 
    n=${d#*/iommu_groups/*}; n=${n%%/*}
    printf 'IOMMU Group %s ' "$n"
    lspci -nns "${d##*/}"
done;

[diabolusss]

@powerhouse

• 2. What actually do you mean by "make and ..."?
  - I suppose model is by DELL, it's 3Gb and signed Bioses doesn't support UEFI, but unsigned does, so i'm trying them. Is it safe to check different bioses even if they aren't supposed for this card?
• 3. When you boot the VM without "vga none,..." and you get a qemu window, does the device manager show 2 separate graphics displays? A VGA graphics and the Nvidia graphics (with the error triangle)?
  - yes, exactly.
• 4. Are you using the laptop display or have you connected an external display?
  - I connected external display to HDMI port which is exclusively owned by nvidia card.
• 5. When you boot the laptop and check the dmesg contents for vfio-pci messages, you get "invalid ROM contents"? What happens when you boot the VM? Is it correct that your started the VM and only then retrieved the ROM contents to check for UEFI capabilities?
  - I receive “invalid ROM contents” only when I try to read it contents if it’s not loaded by VM. After I launch VM with VGA enabled(i.e. I have default qemu window) I can read it contents successfully. If I shutdown VM I receive this error again.

There exist soft switch on keyboard, but it dont work. When I hit it I receive errors in dmesg:

Code: Select all

[21981.795261] atkbd serio0: Unknown key pressed (translated set 2, code 0xba on isa0060/serio0).
[21981.795270] atkbd serio0: Use 'setkeycodes e03a <keycode>' to make it known.
[21981.805790] atkbd serio0: Unknown key released (translated set 2, code 0xba on isa0060/serio0).
[21981.805799] atkbd serio0: Use 'setkeycodes e03a <keycode>' to make it known.

Yeah, I already installed Win8.1 with seabios and when I tried to passthrough Nvidia card I received:

Code: Select all

qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: vfio error: 0000:01:00.0: failed getting region info for VGA region index 8: Invalid argument
device does not support requested feature x-vga

So , I suppose I need to patch kernel, but I don’t know how to do that.. I tried to patch my current kernel(Linux Mint 18 4.4.0-21-generic) using this tutorial:
http://kvm-exp.blogspot.com/2015/06/how ... biter.html, but It’s said nothing to patch. I suppose I need another kernel, i.e. like in tutorial linux-3.16.7?

I patched Nvidia driver using your tutorial, but it doesnt helped neither.

[powerhouse]

@diabolusss:

Before patching, check if you can upgrade to a newer kernel. I'm running Linux Mint 18.1 with kernel 4.10.0-26-generic x86_64.

Click the Update Manager, then select View --> Linux kernels. Do you see any 4.8 and 4.10 kernels? If yes, select a recent 4.10 kernel and install. Reboot and see if that helps.

Also, post the IOMMU groups using the small script from my post above.


EDIT: I might be wrong, but I believe it has to do with the UEFI BIOS not matching your actual video card.

The steps I'd do now, in the following order, are:

1. Upgrade kernel to the latest 4.10 kernel.
2. Check and recheck that you got the right UEFI enabled BIOS for YOUR graphics card.
3. To my knowledge, it's enough to load the UEFI BIOS at VM boot, but I might be mistaken. Search and look for users with the same issue and how they solved it. If I'm wrong, you might need to flash your graphics card BIOS, something that should be done with utmost caution (and only after you backed up the original BIOS). If that is what's needed, then first try Seabios.
4. If none of the above helps, reinstall Windows using Seabios.

When you use Seabios, do NOT use the UEFI BIOS! Remove the BIOS location. Also, remove the lines:

Code: Select all

  -drive if=pflash,format=raw,readonly,file=/usr/share/OVMF/OVMF_CODE.fd \
  -drive if=pflash,format=raw,file=/tmp/my_vars.fd \

from the script.

Hope this helps.

[powerhouse]

@diabolusss: With "make" I mean the manufacturer of your graphics card. I posted a link to techpowerup that lists the known makes and models, together with the UEFI support. Under Linux, lspci -vv should should list all the information needed.

Where did you find the UEFI BIOS for your card?

[diabolusss]

I found UEFI capable BIOS at techpowerup under unverified uploads. (they should add BIOS filter parameter ). It seems that I’ll try them all one by one...
Ok, i’m doing full system upgrade now. I have installed the latest kernel which is by now 4.10.0-26.
Well, I have to believe in something and I share your belief, moreover it would work finally. But I don’t want to brick GPU, so I wouldn’t flash its BIOS… Meanwhile, I have installed Windows with Seabios on other image and I know now that I need to patch kernel, but I found, I think, working patch only for kernel 4.2.5, so I am downloading now Linux Mint 17.
I have deleted default qemu, which was v2.5, and installed new one from sources, which is v2.9, and I can’t passthrough usb mouse now. Everything else is working fine.

Code: Select all

qemu-system-x86_64 --machine help
Supported machines are:
pc                   Standard PC (i440FX + PIIX, 1996) (alias of pc-i440fx-2.9)
pc-i440fx-1.4-1.7...2.0-2.9        Standard PC (i440FX + PIIX, 1996) (default)
pc-1.0...1.3               Standard PC (i440FX + PIIX, 1996)
pc-0.10...0.15              Standard PC (i440FX + PIIX, 1996)
q35                  Standard PC (Q35 + ICH9, 2009) (alias of pc-q35-2.9)
pc-q35-2.4...2.9           Standard PC (Q35 + ICH9, 2009)
isapc                ISA-only PC
none                 empty machine

UPDATE2:
Well, I managed to patch latest linux kernel of version 4.10.17 using this tutorials:

• Getting sources and building kernel image:
• http://technote.thispage.me/index.php/2 ... t-setting/
  Patching steps:
• http://kvm-exp.blogspot.com/2015/06/how ... biter.html
  And I used patch from here:
• https://github.com/f4bio/linux-c0mbine/ ... iter.patch

Can’t find how to check if this kernel is working with x-vga, because I still receive this horrible error:

Code: Select all

qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: vfio error: 0000:01:00.0: failed getting region info for VGA region index 8: Invalid argument
device does not support requested feature x-vga

If someone can check it, I will post link to file on dropbox.

Code: Select all

dmesg|grep arb
[    0.000000] Linux version 4.10.17--i915vgaarb-kernel (root@colt-Alienware-17-R3) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4) ) #1 SMP Fri Jul 7 22:15:11 EEST 2017 (Ubuntu 4.10.0-26.30~16.04.1-generic 4.10.17)
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.10.17--i915vgaarb-kernel root=UUID=85b070c7-1807-426e-8f2b-de59d732e051 ro modprobe.blacklist=nvidia quiet splash intel_iommu=on i915.enable_hd_vgaarb=1 vt.handoff=7
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.10.17--i915vgaarb-kernel root=UUID=85b070c7-1807-426e-8f2b-de59d732e051 ro modprobe.blacklist=nvidia quiet splash intel_iommu=on i915.enable_hd_vgaarb=1 vt.handoff=7
[    0.294953] pci 0000:00:02.0: vgaarb: setting as boot VGA device
[    0.294955] pci 0000:00:02.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
[    0.294957] pci 0000:00:02.0: vgaarb: bridge control possible
[    0.294958] vgaarb: loaded
[    0.853716] usb usb1: Manufacturer: Linux 4.10.17--i915vgaarb-kernel xhci-hcd
[    0.862134] usb usb2: Manufacturer: Linux 4.10.17--i915vgaarb-kernel xhci-hcd
[    1.036676] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io:owns=io+mem

Argh, I am running of options =\
Q2. Any help will be appreciated.

UPDATE1:
While testing different rom's from techpowerup in VM i found in device driver info my GPU Subsystem id, which is 1028 0708.(Actually i could get it using command lspci -vvn, This IGD and 3D shares the same sub_id)
Q1. Loaded custom ROM should override this value or not? Or custom ROM SUB_ID must exactly match real GRU SUB_ID?(I believe DEV_ID should)

Output from script:

Code: Select all

IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation Sky Lake Host Bridge/DRAM Registers [8086:1910] (rev 07)
IOMMU Group 10 00:1c.6 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #7 [8086:a116] (rev f1)
IOMMU Group 11 00:1d.0 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #9 [8086:a118] (rev f1)
IOMMU Group 12 00:1f.0 ISA bridge [0601]: Intel Corporation Sunrise Point-H LPC Controller [8086:a14e] (rev 31)
IOMMU Group 12 00:1f.2 Memory controller [0580]: Intel Corporation Sunrise Point-H PMC [8086:a121] (rev 31)
IOMMU Group 12 00:1f.3 Audio device [0403]: Intel Corporation Sunrise Point-H HD Audio [8086:a170] (rev 31)
IOMMU Group 12 00:1f.4 SMBus [0c05]: Intel Corporation Sunrise Point-H SMBus [8086:a123] (rev 31)
IOMMU Group 13 3b:00.0 Ethernet controller [0200]: Qualcomm Atheros Killer E2400 Gigabit Ethernet Controller [1969:e0a1] (rev 10)
IOMMU Group 14 3c:00.0 Network controller [0280]: Qualcomm Atheros QCA6174 802.11ac Wireless Network Adapter [168c:003e] (rev 32)
IOMMU Group 15 3d:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTS5227 PCI Express Card Reader [10ec:5227] (rev 01)
IOMMU Group 16 3e:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller [144d:a802] (rev 01)
IOMMU Group 1 00:01.0 PCI bridge [0604]: Intel Corporation Sky Lake PCIe Controller (x16) [8086:1901] (rev 07)
IOMMU Group 1 01:00.0 3D controller [0302]: NVIDIA Corporation GM204M [GeForce GTX 970M] [10de:13d8] (rev a1)
IOMMU Group 2 00:02.0 VGA compatible controller [0300]: Intel Corporation Skylake Integrated Graphics [8086:191b] (rev 06)
IOMMU Group 3 00:04.0 Signal processing controller [1180]: Intel Corporation Skylake Processor Thermal Subsystem [8086:1903] (rev 07)
IOMMU Group 4 00:14.0 USB controller [0c03]: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller [8086:a12f] (rev 31)
IOMMU Group 4 00:14.2 Signal processing controller [1180]: Intel Corporation Sunrise Point-H Thermal subsystem [8086:a131] (rev 31)
IOMMU Group 5 00:16.0 Communication controller [0780]: Intel Corporation Sunrise Point-H CSME HECI #1 [8086:a13a] (rev 31)
IOMMU Group 6 00:17.0 RAID bus controller [0104]: Intel Corporation 82801 Mobile SATA Controller [RAID mode] [8086:282a] (rev 31)
IOMMU Group 7 00:1c.0 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #1 [8086:a110] (rev f1)
IOMMU Group 8 00:1c.4 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #5 [8086:a114] (rev f1)
IOMMU Group 9 00:1c.5 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #6 [8086:a115] (rev f1)

Code: Select all

01:00.0 3D controller: NVIDIA Corporation GM204M [GeForce GTX 970M] (rev a1)
	Subsystem: Dell Device 0708
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 143
	Region 0: Memory at dc000000 (32-bit, non-prefetchable) [size=16M]
	Region 1: Memory at b0000000 (64-bit, prefetchable) [size=256M]
	Region 3: Memory at c0000000 (64-bit, prefetchable) [size=32M]
	Region 5: I/O ports at e000 [size=128]
	[virtual] Expansion ROM at dd000000 [disabled] [size=512K]
	Capabilities: [60] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee004f8  Data: 0000
	Capabilities: [78] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us
			ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Via message
		AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		AtomicOpsCtl: ReqEn-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
	Capabilities: [258 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
			  PortCommonModeRestoreTime=255us PortTPowerOnTime=10us
		L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
			   T_CommonMode=0us LTR1.2_Threshold=0ns
		L1SubCtl2: T_PwrOn=10us
	Capabilities: [128 v1] Power Budgeting <?>
	Capabilities: [420 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
	Capabilities: [900 v1] #19
	Kernel driver in use: nvidia
	Kernel modules: nouveau, nvidia_drm, nvidia

Maybe, this information will be useful, too.

Code: Select all

00:01.0 PCI bridge: Intel Corporation Skylake PCIe Controller (x16) (rev 07) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 122
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
	I/O behind bridge: 0000e000-0000efff [size=4K]
	Memory behind bridge: dc000000-dd0fffff [size=17M]
	Prefetchable memory behind bridge: 00000000b0000000-00000000c1ffffff [size=288M]
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [88] Subsystem: Dell Device 0708
	Capabilities: [80] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
		Address: fee00238  Data: 0000
	Capabilities: [a0] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 256 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #2, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <256ns, L1 <8us
			ClockPM- Surprise- LLActRep- BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt+
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #1, PowerLimit 75.000W; Interlock- NoCompl+
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet+ LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via WAKE# ARIFwd-
		AtomicOpsCap: Routing- 32bit+ 64bit+ 128bitCAS+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Via WAKE# ARIFwd-
		AtomicOpsCtl: ReqEn- EgressBlck-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [100 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
	Capabilities: [140 v1] Root Complex Link
		Desc:	PortNumber=02 ComponentID=01 EltType=Config
		Link0:	Desc:	TargetPort=00 TargetComponent=01 AssocRCRB- LinkType=MemMapped LinkValid+
			Addr:	00000000fed19000
	Capabilities: [d94 v1] #19
	Kernel driver in use: pcieport
	Kernel modules: shpchp

Code: Select all

00:02.0 VGA compatible controller: Intel Corporation HD Graphics 530 (rev 06) (prog-if 00 [VGA controller])
	Subsystem: Dell Device 0708
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 135
	Region 0: Memory at db000000 (64-bit, non-prefetchable) [size=16M]
	Region 2: Memory at 70000000 (64-bit, prefetchable) [size=256M]
	Region 4: I/O ports at f000 [size=64]
	[virtual] Expansion ROM at 000c0000 [disabled] [size=128K]
	Capabilities: [40] Vendor Specific Information: Len=0c <?>
	Capabilities: [70] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		AtomicOpsCtl: ReqEn-
	Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable- 64bit-
		Address: fee00018  Data: 0000
	Capabilities: [d0] Power Management version 2
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Process Address Space ID (PASID)
		PASIDCap: Exec+ Priv-, Max PASID Width: 14
		PASIDCtl: Enable- Exec- Priv-
	Capabilities: [200 v1] Address Translation Service (ATS)
		ATSCap:	Invalidate Queue Depth: 00
		ATSCtl:	Enable-, Smallest Translation Unit: 00
	Capabilities: [300 v1] Page Request Interface (PRI)
		PRICtl: Enable- Reset-
		PRISta: RF- UPRGI- Stopped-
		Page Request Capacity: 00008000, Page Request Allocation: 00000000
	Kernel driver in use: i915
	Kernel modules: i915

[procabiak]

@procabiac: Try to only install the AMD GPU driver, not the whole package. It has been reported that the control center or whatever it's called can cause problems.

Hey powerhouse, thanks for getting back to me. I managed to redo the guide in Mint 18.2, using qemu 2.6.x from the Jacob ppa (a must for AMD [edit - I mean AMD CPUs], there's a bug in 2.5.0), and using the 4.12 Kernel (I was on 4.11.5 previously). I installed the GPU drivers via Windows Update this time (instead of a simple install or the Windows Driver Update method), and it worked!

Unfortunately, I don't know what the issue was. I also had other weird issues like Windows only picking up 1 cpu core (despite seeing 8 in device manager) and thought it was a license issue. The new install now sees all 8 virtual cores.

In case you still want this, the iommu groups you asked me to output:

Code: Select all

IOMMU Group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 10 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 59)
IOMMU Group 10 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 11 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1460]
IOMMU Group 11 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1461]
IOMMU Group 11 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1462]
IOMMU Group 11 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1463]
IOMMU Group 11 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1464]
IOMMU Group 11 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1465]
IOMMU Group 11 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1466]
IOMMU Group 11 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1467]
IOMMU Group 12 01:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd Device [144d:a804]
IOMMU Group 13 03:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b9] (rev 02)
IOMMU Group 13 03:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b5] (rev 02)
IOMMU Group 13 03:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b0] (rev 02)
IOMMU Group 13 04:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 13 04:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 13 04:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 13 04:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 13 06:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02)
IOMMU Group 13 07:00.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
IOMMU Group 13 08:01.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
IOMMU Group 13 08:03.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
IOMMU Group 13 08:05.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
IOMMU Group 13 08:07.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
IOMMU Group 13 09:00.0 Network controller [0280]: Intel Corporation Device [8086:24fb] (rev 10)
IOMMU Group 13 0b:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
IOMMU Group 14 0e:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii XT [Radeon R9 290X] [1002:67b0]
IOMMU Group 14 0e:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii HDMI Audio [1002:aac8]
IOMMU Group 15 0f:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii XT [Radeon R9 290X] [1002:67b0]
IOMMU Group 15 0f:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii HDMI Audio [1002:aac8]
IOMMU Group 1 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 2 00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 3 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 4 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 5 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 6 00:03.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 7 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 8 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 8 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1454]
IOMMU Group 8 11:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:145a]
IOMMU Group 8 11:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Device [1022:1456]
IOMMU Group 8 11:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:145c]
IOMMU Group 9 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 9 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1454]
IOMMU Group 9 12:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:1455]
IOMMU Group 9 12:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 9 12:00.3 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Device [1022:1457]

[powerhouse]

@procabiac: Thanks for your feedback. I don't have a AMD card at my disposal now, so I can't test. This is why your advice about the bug in Qemu 2.5 and the need to use a later version is so valuable. The moment I have some time I will update my tutorial.

Would you be able to run some benchmarks and post them in the dedicated thread (see end of my tutorial)? Thanks!

[powerhouse]

@diabolusss: Thanks for sharing all the information. To me it looks like a graphics BIOS issue. Since you have tried all options using UEFI boot, except for flashing the GPU BIOS (a thing I wouldn't do unless I were 100% sure of what I'm doing), there is one last option I believe you haven't tried: Seabios.

Instead of booting the OVMF code, use the traditional way as I pointed out (or see bash64). You'll need to reinstall Windows. Keep the latest kernel and qemu as this might solve some issues. I haven't checked if the latest kernel includes the VGA arbitration patch, but I doubt. But perhaps the problem has been solved on the Intel driver side (I need to read up on that, haven't had much time lately).
If you can't get it to work, check out the VGA arbiter patch.
Note: UEFI boot does not require the VGA arbitration patch.

[diabolusss]

@powerhouse Actually i have tested, but i can't x-vga get to work.

UPDATE2:
Well, I managed to patch latest linux kernel of version 4.10.17 using this tutorials:

• Getting sources and building kernel image:
• http://technote.thispage.me/index.php/2 ... t-setting/
  Patching steps:
• http://kvm-exp.blogspot.com/2015/06/how ... biter.html
  And I used patch from here:
• https://github.com/f4bio/linux-c0mbine/ ... iter.patch

Can’t find how to check if this kernel is working with x-vga, because I still receive this horrible error:

Code: Select all

qemu-system-x86_64: -device vfio-pci,host=01:00.0,x-vga=on: vfio error: 0000:01:00.0: failed getting region info for VGA region index 8: Invalid argument
device does not support requested feature x-vga

If someone can check it, I will post link to file on dropbox.

Code: Select all

dmesg|grep arb
[    0.000000] Linux version 4.10.17--i915vgaarb-kernel (root@colt-Alienware-17-R3) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4) ) #1 SMP Fri Jul 7 22:15:11 EEST 2017 (Ubuntu 4.10.0-26.30~16.04.1-generic 4.10.17)
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.10.17--i915vgaarb-kernel root=UUID=85b070c7-1807-426e-8f2b-de59d732e051 ro modprobe.blacklist=nvidia quiet splash intel_iommu=on i915.enable_hd_vgaarb=1 vt.handoff=7
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.10.17--i915vgaarb-kernel root=UUID=85b070c7-1807-426e-8f2b-de59d732e051 ro modprobe.blacklist=nvidia quiet splash intel_iommu=on i915.enable_hd_vgaarb=1 vt.handoff=7
[    0.294953] pci 0000:00:02.0: vgaarb: setting as boot VGA device
[    0.294955] pci 0000:00:02.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
[    0.294957] pci 0000:00:02.0: vgaarb: bridge control possible
[    0.294958] vgaarb: loaded
[    0.853716] usb usb1: Manufacturer: Linux 4.10.17--i915vgaarb-kernel xhci-hcd
[    0.862134] usb usb2: Manufacturer: Linux 4.10.17--i915vgaarb-kernel xhci-hcd
[    1.036676] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io:owns=io+mem

Argh, I am running of options =\
Q2. Any help will be appreciated.

UPDATE1:

[procabiak]

@procabiac: Thanks for your feedback. I don't have a AMD card at my disposal now, so I can't test. This is why your advice about the bug in Qemu 2.5 and the need to use a later version is so valuable. The moment I have some time I will update my tutorial.

Would you be able to run some benchmarks and post them in the dedicated thread (see end of my tutorial)? Thanks!

Sorry, about that line, I meant that AMD CPUs need Qemu 2.6.x. I've updated my post to indicate that. More specifically, this is the exact bug: https://bugzilla.redhat.com/show_bug.cgi?id=1326721

My Benchmarks posted. I still have work to do in order to improve the performance. There's a quirk with the Ryzen CPU - it's kinda like two 4-core CPUs glued together. I think I have to pin only the threads that all belong in one of those 4-cores to improve performance. RAM speed also affects Ryzen, I need to OC the RAM to its XMP profile eventually... Also don't know why my GPU benches 30% lower (seems quite a lot), but it might be bottlenecked by the CPU.

But for now it's very passable, enough to play the not-so-demanding games.

Please do update the tutorial I'm sure there's early Ryzen adopters looking forward to trying this.

[powerhouse]

@procabiac: Thanks for your feedback. I don't have a AMD card at my disposal now, so I can't test. This is why your advice about the bug in Qemu 2.5 and the need to use a later version is so valuable. The moment I have some time I will update my tutorial.

Would you be able to run some benchmarks and post them in the dedicated thread (see end of my tutorial)? Thanks!

Sorry, about that line, I meant that AMD CPUs need Qemu 2.6.x. I've updated my post to indicate that. More specifically, this is the exact bug: https://bugzilla.redhat.com/show_bug.cgi?id=1326721

My Benchmarks posted. I still have work to do in order to improve the performance. There's a quirk with the Ryzen CPU - it's kinda like two 4-core CPUs glued together. I think I have to pin only the threads that all belong in one of those 4-cores to improve performance. RAM speed also affects Ryzen, I need to OC the RAM to its XMP profile eventually... Also don't know why my GPU benches 30% lower (seems quite a lot), but it might be bottlenecked by the CPU.

But for now it's very passable, enough to play the not-so-demanding games.

Please do update the tutorial I'm sure there's early Ryzen adopters looking forward to trying this.

Thanks for the update / correction! About the benchmark: Could you post results from http://www.userbenchmark.com? It's free and easy to share.

I will add your input to the tutorial once I have a little more time. Unfortunately I have no access to a Ryzen system so I am completely relying on user input.

Can you post the qemu command line and options you use? This will be helpful in finding suggestions on how to improve performance.

[powerhouse]

@diabolusss: Perhaps I missed it, but can you please post the qemu command and options you use when you get the x-vga error?

[diabolusss]

@powerhouse my startup script is:

Code: Select all

#!/bin/bash

vm_name=win10-mypc
vm_machine_type=q35
mem_size=4G
mem_pages=/dev/hugepages
pci_gpu=01:00.0
usb_mouse=15d9:0a4c
dir_os=/media/colt/_DATA/seabios.img
dir_bios=/usr/share/seabios/bios.bin
dir_cd_install=/media/colt/_DATA/win8.1_x86-x64.iso
dir_cd_drivers=/media/colt/_DATA/virtio-win-0.1.139.iso
boot_order=dc
net_card_mac=00:16:3e:00:01:01

if ps -A | grep -q $vm_name; then
   echo "$vm_name is already running." &
   exit 1
fi

qemu-system-x86_64 \
-enable-kvm \
-M $vm_machine_type \
-m $mem_size \
-cpu host,kvm=off \
-smp 6,sockets=1,cores=3,threads=2 \
-bios $dir_bios \
-serial none \
-parallel none \
-vga none \
-device vfio-pci,host=$pci_gpu,x-vga=on \
-rtc clock=host,base=localtime \
  -device virtio-scsi-pci,id=scsi \
  -drive id=disk0,if=virtio,cache=none,format=raw,file=$dir_os \
  -drive file=$dir_cd_install,id=isocd,format=raw,if=none -device scsi-cd,drive=isocd \
  -drive file=$dir_cd_drivers,id=virtiocd,format=raw,if=none -device ide-cd,bus=ide.1,drive=virtiocd \
  -boot order=$boot_order \
  -boot menu=on \
  -netdev type=tap,id=net0,ifname=tap0,vhost=on \
  -device virtio-net-pci,netdev=net0,mac=$net_card_mac

exit 0