HOW-TO make dual-boot obsolete using kvm VGA passthrough

Questions about virtualization software
Forum rules
Before you post read how to get help. Topics in this forum are automatically closed 6 months after creation.
Locked
powerhouse
Level 6
Level 6
Posts: 1144
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

@Juju: Thanks for sharing your journey to get Windows 7 working with UEFI boot. I gave up long ago and upgraded to Windows 10. I am sure your input will help others.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
Foxybingo15

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by Foxybingo15 »

Hi all,

Firstly, thanks for the amazing tutorial! I'm very new to Linux and am currently trying to follow to get windows 10 working in a VM. Here are my specs:
  • R7 1700
    Asus X370 hero
    Vega 56
    16gb 3000Mhz
    750gb MX300
    Evga GS 650
    Linux mint 18.2 64 bit Cinnamon
This is my first post here so forgive me if I don't include all the stuff you need to help.

So far I've got up to the script to run the VM (part 7) . The only difference I've noticed so far is that my Hugepages is located in '/dev/hugepages' rather than '/run/hugepages/kvm'.

Here is my script to start the VM:

Code: Select all

#!/bin/bash

vmname="gaming"

if ps -A | grep -q $vmname; then
   echo "$vmname is already running." &
   exit 1

else

# use pulseaudio
export QEMU_AUDIO_DRV=pa
export QEMU_PA_SAMPLES=8192
export QEMU_AUDIO_TIMER_PERIOD=99
export QEMU_PA_SERVER=/run/user/1000/pulse/native

cp /usr/share/OVMF/OVMF_VARS.fd /tmp/my_vars.fd

qemu-system-x86_64 \
  -name $vmname,process=$vmname \
  -machine type=pc,accel=kvm \
  -cpu host \
  -smp 12,sockets=1,cores=6,threads=2 \
  -m 12G \
  -mem-path /dev/hugepages \
  -mem-prealloc \
  -balloon none \
  -rtc clock=host,base=localtime \
 # -vga none \
  -nographic \
  -serial none \
  -parallel none \
  -soundhw hda \
  -usb -usbdevice host:1af3:0001 -usbdevice host:195d:2030 \
  -device vfio-pci,host=2b:00.0,multifunction=on \
  -device vfio-pci,host=2b:00.1 \
  -drive if=pflash,format=raw,readonly,file=/usr/share/OVMF/OVMF_CODE.fd \
  -drive if=pflash,format=raw,file=/tmp/my_vars.fd \
  -boot order=dc \
  -drive id=disk0,if=virtio,cache=none,format=raw,file=/media/daniel/win.img \
  -drive file=/home/daniel/Downloads/win10.iso,index=3,media=cdrom \
  -drive file=/home/daniel/Downloads/virtio-win-0.1.141.iso,index=4,media=cdrom \
  -netdev type=tap,id=net0,ifname=tap0,vhost=on \
  -device virtio-net-pci,netdev=net0,mac=00:16:3e:00:01:01

   exit 0
fi
I'm getting a few issues. Firstly if I uncomment '-vga none' I get the error 'qemu-system-x86_64: Too many IDE buses defined (3 > 2)'. When it's commented as above I get a popup screen for the network boot, but it fails with 'Boot failed: Could not read from CDROM (code 0003)'. The last issue I've notice is I get the message 'qemu-system-x86_64: AMD CPU doesn't support hyperthreading. Please configure -smp options properly.' aswell.

I'm using QEMU 2.6.2 with the latest stable kernal 4.13.7-041307 and virtio-win-0.1.141.iso. The windows 10 iso was downloaded straight from the Microsoft website.

I followed the first trouble shooting step by using the command:

Code: Select all

lspci -k | grep -i -A 3 vga
which gave me:
2b:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Device 687f (rev c3)
Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 6b76
Kernel driver in use: vfio-pci
Kernel modules: amdgpu
--
2c:00.0 VGA compatible controller: NVIDIA Corporation Device 1c81 (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device 8c97
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_375_drm, nvidia_375
So it seems my vega 56 has been successfully bound.

One other thing that might be useful is that my Audio and GPU video are in separate iommu groups. The output to:

Code: Select all

for a in /sys/kernel/iommu_groups/*; do find $a -type l; done
is shortened to:
/sys/kernel/iommu_groups/11/devices/0000:1d:04.0
/sys/kernel/iommu_groups/12/devices/0000:29:00.0
/sys/kernel/iommu_groups/13/devices/0000:2a:00.0
/sys/kernel/iommu_groups/14/devices/0000:2b:00.0
/sys/kernel/iommu_groups/15/devices/0000:2b:00.1
/sys/kernel/iommu_groups/16/devices/0000:2c:00.1
/sys/kernel/iommu_groups/16/devices/0000:2c:00.0
/sys/kernel/iommu_groups/2/devices/0000:00:02.0
/sys/kernel/iommu_groups/3/devices/0000:00:03.0
The 2b devices in 14 and 15 are the VGA & audio for my vega 56.

I tried to give all the info I could think of that would be useful, please let me know if I missed anything. Any help would be greatly appreciated, thanks :)
powerhouse
Level 6
Level 6
Posts: 1144
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

@Foxybingo15 :

EDIT: In the start script, change the smp option to "smp 4" - you specified the architecture for my CPU, which supports hyperthreading, yours is different and doesn't support hyperthreading.

When you first run the script to install Windows, specify the boot order to have it boot from the Windows ISO image - see the tutorial. It happened to me that when booting the first time, I landed in an UEFI shell. You'll then need to navigate to the Windows image (or boot drive). You should be presented with 3 (or 2?) options - choose the one that points to the Windows ISO. If unsure, select one and see if it works. If not, try the other option.

Once Windows boots, you will be prompted to select a driver disk. Select the disk image and point to the storage driver as described in the tutorial. From there on the installation should go smooth.

Once Windows is installed, change the boot order in the script. Then you will want to install the graphics driver under Windows.

Regarding hugepages: Thanks for pointing it out. I need to check once I'm home at my PC.

Regarding the start script and the errors: check your drive definitions, perhaps the path is incorrect.

Please post the output of:

Code: Select all

dmesg | grep AMD-Vi
and:

Code: Select all

hugeadm --explain
Hopefully these hints will bring you closer to success.

EDIT: Check your /etc/modprobe.d/local.conf file and post its contents here.

You need to specify both the graphics part and the audio part.

Last not least, check section no. 12 on my own website: https://heiko-sieger.info/running-windo ... ssthrough/. I continue to respond here, but the latest updates to the tutorial are found on my website (because of restrictions or limitations on this site).
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
Foxybingo15

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by Foxybingo15 »

@powerhouse

Thanks for your help, it is much appreciated. I'm struggling to find time to work on this at the moment but will continue to reply as soon as I can.

I'm still having issues with the hyper threading thing. My AMD CPU (R7 1700) has Simultaneous multi threading which is AMD's version of hyperthreading. Does this have a different command in the script? I tried setting it to 'smp 4' as you suggested but it just complained that 2*6 threads is greater than the 4 threads specified.

Doesn't the line '-boot order=dc' in my startup script specify the boot order?

I double checked the paths and made sure they are correct. I have located both the windows ISO and

Output to:

Code: Select all

dmesg | grep AMD-Vi
is:
  • [ 1.227519] AMD-Vi: IOMMU performance counters supported
    [ 1.229930] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
    [ 1.229931] AMD-Vi: Extended features (0xf77ef22294ada):
    [ 1.229934] AMD-Vi: Interrupt remapping enabled
    [ 1.229934] AMD-Vi: virtual APIC enabled
    [ 1.230193] AMD-Vi: Lazy IO/TLB flushing enabled
Output to:

Code: Select all

hugeadm --explain
is:
  • Total System Memory: 16035 MB

    Mount Point Options
    /dev/hugepages rw,relatime,pagesize=2M

    Huge page pools:
    Size Minimum Current Maximum Default
    2097152 6500 6500 6500 *
    1073741824 0 0 0

    Huge page sizes with configured pools:
    2097152

    The /proc/sys/vm/min_free_kbytes of 67584 is too small. To maximiuse efficiency
    of fragmentation avoidance, there should be at least one huge page free per zone
    in the system which minimally requires a min_free_kbytes value of 112640

    The recommended shmmax for your currently allocated huge pages is 13631488000 bytes.
    To make shmmax settings persistent, add the following line to /etc/sysctl.conf:
    kernel.shmmax = 13631488000

    To make your hugetlb_shm_group settings persistent, add the following line to /etc/sysctl.conf:
    vm.hugetlb_shm_group = 0

    Note: Permanent swap space should be preferred when dynamic huge page pools are used.
The contents of /etc/modprobe.d/local.conf is:
  • options vfio-pci ids=1002:687f,1002:aaf8
These are my Video and Audio addresses of my vega 56 which I'd like passed through to windows.

Here is a picture of the screen I get when I run the script with VGA commented out as shown in my last post.

https://imgur.com/vStb3as
powerhouse
Level 6
Level 6
Posts: 1144
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

@Foxybingo15:

Could it be that your Windows 10 image file is damaged? Try download it again and use the new file then. Another user here had similar issues and it turned out it was because of a corrupted file.

If the above doesn't help, have a look here: https://forum.level1techs.com/t/ryzen-v ... ent/117390

You do not need to change your /etc/default/grub file (just make sure you ran update-grub). Don't use dracut, as mentioned there - your graphics card gets bound to the vfio driver so it should be fine.

Regarding the cpu definition:

They use some option I'm not familiar with:

Code: Select all

<cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='4' threads='2'/>
    <feature policy='disable' name='smep'/>
  </cpu>
The above is in xml format - in plain qemu this should be:

-cpu host,-smep \
-smp sockets=1,cores=4,threads=2 \


I think you can try without the smp configuration. See also https://wiki.archlinux.org/index.php/PC ... on_AMD_CPU.

If my method using the bash script and qemu commands doesn't work, you can always install libvirt and virt-manager and follow the suggestions in the links I provided. My experiences with virt-manager haven't been positive, but many users had success.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
odtech

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by odtech »

Foxybingo15 wrote:I'm getting a few issues. Firstly if I uncomment '-vga none' I get the error 'qemu-system-x86_64: Too many IDE buses defined (3 > 2)'. When it's commented as above I get a popup screen for the network boot, but it fails with 'Boot failed: Could not read from CDROM (code 0003)'.
In my experience if you comment a line with # the script is interrupted. Try removing the line completely.

Edit: Yes the comment is definitely your problem. I can recreate the (0003) cd error on my pc. Either remove the comment and focus on the ide busses error or remove the whole line.
powerhouse
Level 6
Level 6
Posts: 1144
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

Thanks odtech! I totally missed the comment.

You are right, the qemu command can't be commented. I usually copy the command option to the end of the script and the place a # in front. The bash script allows #, just not the qemu command.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
odtech

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by odtech »

I've read on several forums that the acs override patch has been added since kernel version 4.9. I installed kernel 4.13 the other day and it doesn't work.

Is this true or is it hit and miss depending on hardware?
powerhouse
Level 6
Level 6
Posts: 1144
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

odtech wrote:I've read on several forums that the acs override patch has been added since kernel version 4.9. I installed kernel 4.13 the other day and it doesn't work.

Is this true or is it hit and miss depending on hardware?
Just to be sure, you did modify your grub file to enable ACS and ran update-grub?

If the ACS patch is included, it certainly won't be activated.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
odtech

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by odtech »

powerhouse wrote:
odtech wrote:I've read on several forums that the acs override patch has been added since kernel version 4.9. I installed kernel 4.13 the other day and it doesn't work.

Is this true or is it hit and miss depending on hardware?
Just to be sure, you did modify your grub file to enable ACS and ran update-grub?

If the ACS patch is included, it certainly won't be activated.
I did.
powerhouse
Level 6
Level 6
Posts: 1144
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

I have no idea regarding 4.9+ kernel and ACS patch. You can check the kernel sources and look at the change log. It should be listed. See https://www.kernel.org/ and https://kernelnewbies.org/Linux_4.9.

Where did you read that kernel 4.9+ includes the ACS patch? I didn't see it, but haven't researched.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
odtech

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by odtech »

powerhouse wrote:I have no idea regarding 4.9+ kernel and ACS patch. You can check the kernel sources and look at the change log. It should be listed. See https://www.kernel.org/ and https://kernelnewbies.org/Linux_4.9.

Where did you read that kernel 4.9+ includes the ACS patch? I didn't see it, but haven't researched.
I guess i was mistaken about reading it on "several forum posts". When looking again i just found 1 post mentioning it.
powerhouse
Level 6
Level 6
Posts: 1144
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

odtech wrote:
powerhouse wrote:I have no idea regarding 4.9+ kernel and ACS patch. You can check the kernel sources and look at the change log. It should be listed. See https://www.kernel.org/ and https://kernelnewbies.org/Linux_4.9.

Where did you read that kernel 4.9+ includes the ACS patch? I didn't see it, but haven't researched.
I guess i was mistaken about reading it on "several forum posts". When looking again i just found 1 post mentioning it.
Don't get me wrong - I meant to say that I simply don't know. I'm just curious to know where you read it. Did you have a chance to search the change log?

The ACS patch is not something you would want to apply without a good reason, as it can open security holes. To my knowledge only Arch Linux has the ACS patch as part of a virtualization add-on package that's easy to install.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
Misairu_G

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by Misairu_G »

kvm VGA passthrough with an Optimus laptop is not impossible anymore. Just did that myself, with OVMF.

The Code 43 error bash64 once encounter might be the incorrect Subsystem ID. Which can actually be masquerade using some vfio-pci option.

Spent days working on a complete tutorial. Check my Github Gist here https://gist.github.com/Misairu-G/616f7 ... addc940b28
powerhouse
Level 6
Level 6
Posts: 1144
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

Misairu_G wrote:kvm VGA passthrough with an Optimus laptop is not impossible anymore. Just did that myself, with OVMF.

The Code 43 error bash64 once encounter might be the incorrect Subsystem ID. Which can actually be masquerade using some vfio-pci option.

Spent days working on a complete tutorial. Check my Github Gist here https://gist.github.com/Misairu-G/616f7 ... addc940b28
Many thanks for sharing the link here. I included it in the tutorial. Kudos for your guide to VGA passthrough on Optimus laptops!
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
bash64
Level 3
Level 3
Posts: 116
Joined: Mon Mar 23, 2015 12:19 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by bash64 »

Hello,

Powerhouse,
Have you tried using looking glass yet?
You have to have a quadro professional card, but it changes qemu so that you do not need an external monitor for the gaming card.
You see your VM in a window with full gaming.

https://looking-glass.hostfission.com/

I will never buy a quadro so will never use this.
odtech

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by odtech »

bash64 wrote:Hello,

Powerhouse,
Have you tried using looking glass yet?
You have to have a quadro professional card, but it changes qemu so that you do not need an external monitor for the gaming card.
You see your VM in a window with full gaming.

https://looking-glass.hostfission.com/

I will never buy a quadro so will never use this.
You don't need a quadro. Their demo rig uses 1080ti for guest and host uses R9-390X
https://youtu.be/okMGtwfiXMo

Other links
https://forum.level1techs.com/t/looking ... ort/122387
https://github.com/gnif/LookingGlass
https://looking-glass.hostfission.com/quickstart
bash64
Level 3
Level 3
Posts: 116
Joined: Mon Mar 23, 2015 12:19 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by bash64 »

ok, i was able to compile the client.
how do i integrate it with qemu and my existing config file?
bash64
Level 3
Level 3
Posts: 116
Joined: Mon Mar 23, 2015 12:19 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by bash64 »

bash64 wrote:ok, i was able to compile the client.
how do i integrate it with qemu and my existing config file?

found instructions to start an ivshem server and then install a redhat driver in qemu with windows.
working on this
bash64
Level 3
Level 3
Posts: 116
Joined: Mon Mar 23, 2015 12:19 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by bash64 »

bash64 wrote:
bash64 wrote:ok, i was able to compile the client.
how do i integrate it with qemu and my existing config file?

found instructions to start an ivshem server and then install a redhat driver in qemu with windows.
working on this
got computer meeting on 23rd. have all day to try looking glass.
merry christmas
Locked

Return to “Virtual Machines”