HOW-TO make dual-boot obsolete using kvm VGA passthrough

Questions about virtualization software
Forum rules
Before you post read how to get help. Topics in this forum are automatically closed 6 months after creation.
Locked
bash64
Level 3
Level 3
Posts: 116
Joined: Mon Mar 23, 2015 12:19 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by bash64 »

I have settled on the following simply because it works for me and due to unresolvable issues:

Seabios, Pci-stub, virtio for the hard drive, standard nic and usb emulation (no passthrough), Windows 7 64bit.

Only strange thing is if I leave vga on and use virtio I boot in 15 secs (versus the old 7 minutes without virtio).
If I add -vga none and uninstall the vga adapter I get booted with the nvidia card but it takes 4 minutes solid.

I am using virtio and I jump from 15 secs boot to 4 minutes???
Thanks for all the help and hints.
As I said if anyone else needs help they can just ask.

Reminder: I am using an MSI GT70-2PE laptop with nvidia/intel hybrid graphics.
krebin

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by krebin »

Does my 970m being detected as a 3d controller mean I can't use it to pass through?
I have a razer blade 2015 with a integrated intel gpu and a dedicated nvidia 970m.

00:02.0 VGA compatible controller: Intel Corporation 4th Gen Core Processor Integrated Graphics Controller (rev 06)
01:00.0 3D controller: NVIDIA Corporation GM204M [GeForce GTX 970M] (rev a1)

Nothing shows up when I do dmesg | grep vfio
Trees138

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by Trees138 »

I too have issues with nothing returning on vfio, but from the looks of what bash was saying he reverted to pci-stub to get passed this issue.

The other two that were in here recently asking appeared to have given up or found their answers elsewhere.

I'm going to branch out and see if I can get an answer somewhere else as this is my current hang up. After getting a beer with the SO.

If I find something I'll be sure to report back.

Thanks for all the help so far!
Trees138

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by Trees138 »

Alright, I'm calling it quits for the evening.

ONe thing I'm noticing, and this seem like a simple name discrepancy, is that my driver (for an AMD r9 Fury) appears to be 'amdgpu', so at this point that's what i'm pursuing, as nothing is named 'radeon'.

Code: Select all

$ lspci -nnk | grep -A 4 VGA
00:02.0 VGA compatible controller [0300]: Intel Corporation Sky Lake Integrated Graphics [8086:1912] (rev 06)
	DeviceName:  Onboard IGD
	Subsystem: ASUSTeK Computer Inc. Skylake Integrated Graphics [1043:8694]
	Kernel driver in use: i915_bpo
	Kernel modules: i915_bpo
--
01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Fiji [Radeon R9 FURY / NANO Series] [1002:7300] (rev cb)
	Subsystem: PC Partner Limited / Sapphire Technology Fiji [Radeon R9 FURY / NANO Series] [174b:e331]
	Kernel driver in use: amdgpu
	Kernel modules: amdgpu
01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aae8]
I've tired a few things, and my issue just appears to be that my attempts to modeset happens before my driver allows modesetting.

Code: Select all

$ dmesg | grep amdgpu
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.4.0-43-generic root=UUID=3bb32b89-b3b4-49f6-95cb-3db2bc654d24 ro amdgpu.modeset=0 quiet splash intel_iommu=on vt.handoff=7
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.4.0-43-generic root=UUID=3bb32b89-b3b4-49f6-95cb-3db2bc654d24 ro amdgpu.modeset=0 quiet splash intel_iommu=on vt.handoff=7
[    0.918414] amdgpu: unknown parameter 'modeset' ignored
[    0.918479] [drm] amdgpu kernel modesetting enabled.
[    0.923540] amdgpu 0000:01:00.0: enabling device (0000 -> 0003)
[    2.010847] amdgpu 0000:01:00.0: VRAM: 4096M 0x0000000000000000 - 0x00000000FFFFFFFF (4096M used)
[    2.010848] amdgpu 0000:01:00.0: GTT: 4096M 0x0000000100000000 - 0x00000001FFFFFFFF
[    2.011024] [drm] amdgpu: 4096M of VRAM memory ready
[    2.011025] [drm] amdgpu: 4096M of GTT memory ready.
[    2.013602] amdgpu 0000:01:00.0: amdgpu: using MSI.
[    2.013615] [drm] amdgpu: irq initialized.
[    2.017259] amdgpu: powerplay initialized
[    2.019313] amdgpu 0000:01:00.0: fence driver on ring 0 use gpu addr 0x0000000100000008, cpu addr 0xffff88085010d008
[    2.019524] amdgpu 0000:01:00.0: fence driver on ring 1 use gpu addr 0x0000000100000018, cpu addr 0xffff88085010d018
[    2.019707] amdgpu 0000:01:00.0: fence driver on ring 2 use gpu addr 0x0000000100000028, cpu addr 0xffff88085010d028
[    2.019911] amdgpu 0000:01:00.0: fence driver on ring 3 use gpu addr 0x0000000100000038, cpu addr 0xffff88085010d038
[    2.020128] amdgpu 0000:01:00.0: fence driver on ring 4 use gpu addr 0x0000000100000048, cpu addr 0xffff88085010d048
[    2.020304] amdgpu 0000:01:00.0: fence driver on ring 5 use gpu addr 0x0000000100000058, cpu addr 0xffff88085010d058
[    2.020483] amdgpu 0000:01:00.0: fence driver on ring 6 use gpu addr 0x0000000100000068, cpu addr 0xffff88085010d068
[    2.020664] amdgpu 0000:01:00.0: fence driver on ring 7 use gpu addr 0x0000000100000078, cpu addr 0xffff88085010d078
[    2.020855] amdgpu 0000:01:00.0: fence driver on ring 8 use gpu addr 0x0000000100000088, cpu addr 0xffff88085010d088
[    2.021074] amdgpu 0000:01:00.0: fence driver on ring 9 use gpu addr 0x0000000100000098, cpu addr 0xffff88085010d098
[    2.021200] amdgpu 0000:01:00.0: fence driver on ring 10 use gpu addr 0x00000001000000a8, cpu addr 0xffff88085010d0a8
[    2.021837] amdgpu 0000:01:00.0: fence driver on ring 11 use gpu addr 0x000000000088f7b0, cpu addr 0xffffc9000604e7b0
[    2.022018] amdgpu 0000:01:00.0: fence driver on ring 12 use gpu addr 0x00000001000000c8, cpu addr 0xffff88085010d0c8
[    2.022143] amdgpu 0000:01:00.0: fence driver on ring 13 use gpu addr 0x00000001000000d8, cpu addr 0xffff88085010d0d8
[    2.243831] amdgpu 0000:01:00.0: No connectors reported connected with modes
[    2.244986] amdgpu 0000:01:00.0: fb1: amdgpudrmfb frame buffer device
[    2.260821] [drm] Initialized amdgpu 3.1.0 20150101 for 0000:01:00.0 on minor 0
This is probably obvious, but I have no idea what I'm talking about with linux, so if I'm contradicting with complete misinformation, I'm completely open to that.

As far as the basics, "$ dmesg | grep vfio" returns a blank slate. Everything else has gone well up to this point. Virtualization is going through and all of that.

I'm stuck at Step 7, "dmesg | grep vfio"

FWIW, radeon appears nowhere in my lspci or dmesg, this is why I started pursuing amdgpu as the driver to disable.

Does it at least seem like I'm sniffing up the right tree with the order of modesetting?
powerhouse
Level 6
Level 6
Posts: 1144
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

@Trees138: You are absolutely correct - the driver you want to replace with vfio or pci-stub is amdgpu.

amdgpu.modeset=0 does not work, as you have shown.

Please redo step 4 of my tutorial! I have made some corrections and changes and hope that this solves the issues. See post below!
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
powerhouse
Level 6
Level 6
Posts: 1144
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

Important changes and corrections in step 4 of the tutorial

I have discovered a typo in my tutorial that will prevent loading the vfio-pci driver!!!


Under step 4, my original tutorial read:

Code: Select all

echo "install vfio-pci /sbin/vfio-pci-override-vga.sh" >> /etc/modprobe.d/local.cfg
The correct syntax is:

Code: Select all

echo "install vfio-pci /sbin/vfio-pci-override-vga.sh" >> /etc/modprobe.d/local.conf
Notice the local.conf versus my earlier local.cfg? That little mistake had serious consequences. I apologize for wasting your time trying all sorts of ways to make things work when it was a simple naming issue.

By the way, if you do not have identical graphics cards for both Linux and Windows, you can use the following line in /etc/modprobe.d/local.conf:

Code: Select all

options vfio-pci ids=10de:13c2,10de:0fbb
Use the PCI IDs you noted in step 3. When using the PCI IDs you don't need the /sbin/vfio-pci-override-vga.sh script.

The revised version of step 4 uses the modprobe.blacklist=name_of_module option for the grub command line. Just in case, I removed the "nomodeset" option since it prevents using Kernel Mode Settings, yet another way to prevent loading a driver.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
Trees138

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by Trees138 »

Well, so far that looks to have done the trick. I will probably do a fresh install to get rid of any mess that I introduced previously and determine which of these steps are totally necessary. I'll do this before I move forward with KVM.

Code: Select all

$ lspci -nnk | grep -A 4 VGA
libkmod: ERROR ../libkmod/libkmod-config.c:635 kmod_config_parse: /etc/modprobe.d/local.conf line 3: ignoring bad line starting with 'cat'
00:02.0 VGA compatible controller [0300]: Intel Corporation Sky Lake Integrated Graphics [8086:1912] (rev 06)
	DeviceName:  Onboard IGD
	Subsystem: ASUSTeK Computer Inc. Skylake Integrated Graphics [1043:8694]
	Kernel driver in use: i915_bpo
	Kernel modules: i915_bpo
--
01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Fiji [Radeon R9 FURY / NANO Series] [1002:7300] (rev cb)
	Subsystem: PC Partner Limited / Sapphire Technology Fiji [Radeon R9 FURY / NANO Series] [174b:e331]
	Kernel driver in use: vfio-pci
	Kernel modules: amdgpu
01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aae8]
One thing I had to do is below, I had tried this previously to varying success. It seemed like sometimes it worked, and sometimes it didn't, this is the part I really want to test before moving forward.

I looked into loading the amdgpu during the intitramfs by inserting that module into /etc/initramfs-tools/modules. After updating intitramifs it now appears that the driver is no longer being assigned. The thought here was that this would load the amdgpu driver earlier and allow modesetting to be enabled early enough to catch it. This appears to work, but I'm not sure I don't have some other process messing me up and I was trying all sorts of things. I should be able to weigh in on this better after a re-install.

Thank you powerhouse for circling back and finding that one small correction. It does seem to have done the trick. I'll provide better feedback tonight or this weekend. I really appreciate it!
powerhouse
Level 6
Level 6
Posts: 1144
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

krebin wrote:Does my 970m being detected as a 3d controller mean I can't use it to pass through?
I have a razer blade 2015 with a integrated intel gpu and a dedicated nvidia 970m.

00:02.0 VGA compatible controller: Intel Corporation 4th Gen Core Processor Integrated Graphics Controller (rev 06)
01:00.0 3D controller: NVIDIA Corporation GM204M [GeForce GTX 970M] (rev a1)

Nothing shows up when I do dmesg | grep vfio
Please note the correction I made to step 4 of the tutorial (see 2 posts above), which will solve the dmesg | grep vfio not showing anything.

The 3D controller should not influence compatibility.

Hybrid graphics are tricky. For one thing you may not be able to use UEFI boot. See also the posts of bash64.

My tutorial uses UEFI boot for the Windows VM which happens to solve another issue related to the Intel integrated GPU. If, for some reason, you cannot use UEFI boot, you can boot into a virtual VGA using SEABIOS, install the graphics driver for your actual graphics card in Windows, then reboot the VM. However, you may need to patch the kernel to fix the Intel i915 driver issue - see http://vfio.blogspot.com/2014/08/whats- ... ation.html for more.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
Trees138

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by Trees138 »

Alright, I'm back with limited success and more info.

I did a fresh install of Mint 18 Cinnamon, I updated before giving it another go. I'm still having some issues with vfio-pci assigning to the GPUs.

My main issue, it seems to me, is that I'm getting an error at bootup

/sbin/vfio-pci-override-vga.sh: not found

This doesn't make much sense to me because that command is in /sbin, and it's executable, I circled back and re-did chmod 755 as the superuser just to make sure I didn't mess it up. I also see where Alex Williamson says to do the exact same thing on his blog, so I don't see where the dependency is coming from. If I open that folder with elevated privileges and run the script, vfio-pci binds to the gpu and sound devices just fine, but it does so with the options in local.conf that you'd recommended. So, it appears that I can get to a complete binding, I just have to manually run that script every boot to create vfio-pci.

Now, you'd said that if I were assigning vfio-pci with device id's I wouldn't need the script in sbin, but in practice that doesn't seem to work either, as if that script is not executed vfio-pci is never installed as an option, no? I see references for the module vfio_pci, but nothing for vfio-pci outside of that one script.

Sorry if I sound confrontational, that is not my intent at all, I'm trying to reconcile what I'm seeing to provide the best feedback I can with a limited understanding. I really do appreciate your help, I'm learning quite a bit along the way, or at least I think I am.

The "/sbin/vfio-pci-override-vga.sh: not found" I just plain do not understand, as everything looks correct, It's like whatever is trying to invoke it from local.conf does not have permission to access /sbin.

Question, do we have to define vfio-pci in /sbin? ...or could we potentially define it elsewhere and still invoke it in the local.conf. Isn't that just a placeholder and not an actual driver that's associated with the vfio_pci module?
Last edited by Trees138 on Sat Oct 22, 2016 1:56 pm, edited 1 time in total.
ZMOT7S

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by ZMOT7S »

First I just want to say what a great guide this is. Now to my problem.... When I try to run the sh script in step 8, it tells me my vm is already running. Even I try to kill the process or reboot my PC, I still get the vm is already running. No splash screen or windows pop up. Anyone have a clue what is going on? I'm using a windows 7 iso for the vm, if that has anything to do with it.

Mint 18
MSI 970 GPU
i5 6600k
mcaron1234

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by mcaron1234 »

ZMOT7S wrote:First I just want to say what a great guide this is. Now to my problem.... When I try to run the sh script in step 8, it tells me my vm is already running. Even I try to kill the process or reboot my PC, I still get the vm is already running. No splash screen or windows pop up. Anyone have a clue what is going on? I'm using a windows 7 iso for the vm, if that has anything to do with it.

Mint 18
MSI 970 GPU
i5 6600k
Try changing the name of your vm so it's not the same as the startup script.
powerhouse
Level 6
Level 6
Posts: 1144
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

Trees138 wrote:Alright, I'm back with limited success and more info.

I did a fresh install of Mint 18 Cinnamon, I updated before giving it another go. I'm still having some issues with vfio-pci assigning to the GPUs.

My main issue, it seems to me, is that I'm getting an error at bootup

/sbin/vfio-pci-override-vga.sh: not found

This doesn't make much sense to me because that command is in /sbin, and it's executable, I circled back and re-did chmod 755 as the superuser just to make sure I didn't mess it up. I also see where Alex Williamson says to do the exact same thing on his blog, so I don't see where the dependency is coming from. If I open that folder with elevated privileges and run the script, vfio-pci binds to the gpu and sound devices just fine, but it does so with the options in local.conf that you'd recommended. So, it appears that I can get to a complete binding, I just have to manually run that script every boot to create vfio-pci.

Now, you'd said that if I were assigning vfio-pci with device id's I wouldn't need the script in sbin, but in practice that doesn't seem to work either, as if that script is not executed vfio-pci is never installed as an option, no? I see references for the module vfio_pci, but nothing for vfio-pci outside of that one script.

Sorry if I sound confrontational, that is not my intent at all, I'm trying to reconcile what I'm seeing to provide the best feedback I can with a limited understanding. I really do appreciate your help, I'm learning quite a bit along the way, or at least I think I am.

The "/sbin/vfio-pci-override-vga.sh: not found" I just plain do not understand, as everything looks correct, It's like whatever is trying to invoke it from local.conf does not have permission to access /sbin.

Question, do we have to define vfio-pci in /sbin? ...or could we potentially define it elsewhere and still invoke it in the local.conf. Isn't that just a placeholder and not an actual driver that's associated with the vfio_pci module?
Here my suggestions:

1. Make sure the path name in local.conf is spelled correct. As I have written above, I had the local.conf file name spelled wrong and nothing worked.
2. Make sure your /sbin/vfio-pci-override-vga.sh file name is spelled correct. Somewhere there is probably a typo.
3. Have you loaded the required kernel modules in initramfs? See my instructions in the how-to.
4. Check again that the script is executable.
5. Where is your local.conf file located? Make sure it is in the right directory.
6. If nothing of the above works, use the PCI IDs in local.conf as described a few posts above.
7. If you can't get it work, execute the vfio-pci-override-vga.sh script in the startup script before the qemu command.
8. If you still have problems, please post the content of all the relevant files here.

vfio_pci and vfio-pci are equivalent. Module names are translated automatically. Do an lsmod to see if the modules are loaded.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
powerhouse
Level 6
Level 6
Posts: 1144
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

ZMOT7S wrote:First I just want to say what a great guide this is. Now to my problem.... When I try to run the sh script in step 8, it tells me my vm is already running. Even I try to kill the process or reboot my PC, I still get the vm is already running. No splash screen or windows pop up. Anyone have a clue what is going on? I'm using a windows 7 iso for the vm, if that has anything to do with it.

Mint 18
MSI 970 GPU
i5 6600k
As macaron1234 already wrote, make sure your vmname and the name of the start script are not the same. Also choose a unique name for the vmname, a name that does not exist as a process.

The start script contains a safeguard to prevent running the script and the VM twice. It's not perfect and there may be better ways to test for a running VM, but it works on my machine and should work for you too, as long as the vmname is different from other processes and script names.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
ZMOT7S

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by ZMOT7S »

powerhouse wrote:
ZMOT7S wrote:First I just want to say what a great guide this is. Now to my problem.... When I try to run the sh script in step 8, it tells me my vm is already running. Even I try to kill the process or reboot my PC, I still get the vm is already running. No splash screen or windows pop up. Anyone have a clue what is going on? I'm using a windows 7 iso for the vm, if that has anything to do with it.

Mint 18
MSI 970 GPU
i5 6600k
As macaron1234 already wrote, make sure your vmname and the name of the start script are not the same. Also choose a unique name for the vmname, a name that does not exist as a process.

The start script contains a safeguard to prevent running the script and the VM twice. It's not perfect and there may be better ways to test for a running VM, but it works on my machine and should work for you too, as long as the vmname is different from other processes and script names.
That was the issue but now I'm getting more errors launching qemu. This is what it returns when I run the sh script.

Code: Select all

W: /etc/qemu-ifup: no bridge for guest interface found
QEMU 2.5.0 monitor - type 'help' for more information
(qemu) qemu-system-x86_64: unable to map backing store for hugepages: Cannot allocate memory
qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: vfio: error opening /dev/vfio/1: No such file or directory
qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: vfio: failed to get group 1
qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: Device initialization failed
So it seems vfio isn't grabbing my GPU. I've went back and double checked the vfio-pci-override-vga.sh file and I have the right bus numbers. Any idea why it won't grab the GPU? Maybe it has to do with the no such file or directory error... I'm a linux noob, sorry I can't figure out much by myself. I'd love to run linux full time over windows but gaming is too important to make a full switch right now. If I can get this working as expected though, I may make the switch!

I see you updated step 4 with a bridge guide. I'll give that a try and post back.

Alright, two new things to add. I couldn't get xen to show in the grub menu after adding the line to the grub file like in the tutorial. I also noticed between grub and the mint login screen, the left corner says "sh: /sbin/vfio-pci-override-vga.sh not found" The file is definitely there though and able to run as root.
Last edited by ZMOT7S on Sun Oct 23, 2016 4:18 pm, edited 1 time in total.
powerhouse
Level 6
Level 6
Posts: 1144
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

ZMOT7S wrote:...That was the issue but now I'm getting more errors launching qemu. This is what it returns when I run the sh script.

Code: Select all

W: /etc/qemu-ifup: no bridge for guest interface found
QEMU 2.5.0 monitor - type 'help' for more information
(qemu) qemu-system-x86_64: unable to map backing store for hugepages: Cannot allocate memory
qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: vfio: error opening /dev/vfio/1: No such file or directory
qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: vfio: failed to get group 1
qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: Device initialization failed
So it seems vfio isn't grabbing my GPU. I've went back and double checked the vfio-pci-override-vga.sh file and I have the right bus numbers. Any idea why it won't grab the GPU? Maybe it has to do with the no such file or directory error... I'm a linux noob, sorry I can't figure out much by myself. I'd love to run linux full time over windows but gaming is too important to make a full switch right now. If I can get this working as expected though, I may make the switch!
W: /etc/qemu-ifup: no bridge for guest interface found
Today I added a link to the network configuration. You need to create a network bridge. Install bridge-utils and then proceed as described in Adjust the network settings to use a bridge in my Xen tutorial.
(qemu) qemu-system-x86_64: unable to map backing store for hugepages: Cannot allocate memory
It seems that you didn't define hugepages, but the qemu command specifies hugepages. If you didn't configure hugepages, remove the following 2 lines from the script:

Code: Select all

  -mem-path /run/hugepages/kvm \
  -mem-prealloc \
qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: vfio: error opening /dev/vfio/1: No such file or directory
Check with

Code: Select all

dmesg | grep vfio
to see if vfio-pci binds to the graphics card. The output should be similar to:
  • [ 2.917908] vfio_pci: add [10de:13c2[ffff:ffff]] class 0x000000/00000000
    [ 2.933923] vfio_pci: add [10de:0fbb[ffff:ffff]] class 0x000000/00000000
    [ 8.593650] vfio-pci 0000:02:00.0: enabling device (0000 -> 0003)
In your case the PCI bus should be 0000:01:00.0. If you do not get this, see below!

Please note that I had a typo in my how-to which I fixed 2 days ago. The name of the config file in /etc/modprobe.d is "local.conf" (I erroneously specified local.cfg which prevents the execution of the /sbin/vfio-pci-override-vga.sh script). The correct syntax is:

Code: Select all

echo "install vfio-pci /sbin/vfio-pci-override-vga.sh" >> /etc/modprobe.d/local.conf
See part 4 of the tutorial.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
ZMOT7S

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by ZMOT7S »

powerhouse wrote:
ZMOT7S wrote:...That was the issue but now I'm getting more errors launching qemu. This is what it returns when I run the sh script.

Code: Select all

W: /etc/qemu-ifup: no bridge for guest interface found
QEMU 2.5.0 monitor - type 'help' for more information
(qemu) qemu-system-x86_64: unable to map backing store for hugepages: Cannot allocate memory
qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: vfio: error opening /dev/vfio/1: No such file or directory
qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: vfio: failed to get group 1
qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: Device initialization failed
So it seems vfio isn't grabbing my GPU. I've went back and double checked the vfio-pci-override-vga.sh file and I have the right bus numbers. Any idea why it won't grab the GPU? Maybe it has to do with the no such file or directory error... I'm a linux noob, sorry I can't figure out much by myself. I'd love to run linux full time over windows but gaming is too important to make a full switch right now. If I can get this working as expected though, I may make the switch!
W: /etc/qemu-ifup: no bridge for guest interface found
Today I added a link to the network configuration. You need to create a network bridge. Install bridge-utils and then proceed as described in Adjust the network settings to use a bridge in my Xen tutorial.
(qemu) qemu-system-x86_64: unable to map backing store for hugepages: Cannot allocate memory
It seems that you didn't define hugepages, but the qemu command specifies hugepages. If you didn't configure hugepages, remove the following 2 lines from the script:

Code: Select all

  -mem-path /run/hugepages/kvm \
  -mem-prealloc \
qemu-system-x86_64: -device vfio-pci,host=01:00.0,multifunction=on: vfio: error opening /dev/vfio/1: No such file or directory
Check with

Code: Select all

dmesg | grep vfio
to see if vfio-pci binds to the graphics card. The output should be similar to:
  • [ 2.917908] vfio_pci: add [10de:13c2[ffff:ffff]] class 0x000000/00000000
    [ 2.933923] vfio_pci: add [10de:0fbb[ffff:ffff]] class 0x000000/00000000
    [ 8.593650] vfio-pci 0000:02:00.0: enabling device (0000 -> 0003)
In your case the PCI bus should be 0000:01:00.0. If you do not get this, see below!

Please note that I had a typo in my how-to which I fixed 2 days ago. The name of the config file in /etc/modprobe.d is "local.conf" (I erroneously specified local.cfg which prevents the execution of the /sbin/vfio-pci-override-vga.sh script). The correct syntax is:

Code: Select all

echo "install vfio-pci /sbin/vfio-pci-override-vga.sh" >> /etc/modprobe.d/local.conf
See part 4 of the tutorial.
Which is weird. I have it setup but I think the problem is the original mount point didn't go away. Here is what my hugeadm --explain shows.

Code: Select all

Total System Memory: 15941 MB

Mount Point          Options
/dev/hugepages       rw,relatime
/run/hugepages/kvm   rw,relatime,mode=775,gid=132

Huge page pools:
      Size  Minimum  Current  Maximum  Default
   2097152     4500     4500     4500        *
1073741824        0        0        0         

Huge page sizes with configured pools:

The recommended shmmax for your currently allocated huge pages is 9437184000 bytes.
To make shmmax settings persistent, add the following line to /etc/sysctl.conf:
  kernel.shmmax = 9437184000

hugeadm:WARNING: User zmot (uid: 1000) is not a member of the hugetlb_shm_group root (gid: 0)!

Note: Permanent swap space should be preferred when dynamic huge page pools are used.
Now for the Xen part, I added the xen line to the grub file and rebooted but there was no option for xen in grub. I am using wired connection. I've also redid step 4 since you updated it. The file is there and able to run as root but on startup between grub and the linux mint login screen I see in the top corner "sh: vfio-pci /sbin/vfio-pci-override-vga.sh not found" or something similar to that. The file is there though and has full permissions.
powerhouse
Level 6
Level 6
Posts: 1144
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

Update
I have updated my how-to and added the following:
1. bridge-utils package (Installing Qemu / KVM, part 2)
2. Link to bridge configuration (Prepare for passthrough, part 4)
3. Change log.

It was high time to add a change log as I have been fixing mistakes and bugs in recent days. Please always read the change log at the beginning to see what's new and what has been fixed.

Please consider this tutorial as a work in progress. There is always the possibility for mistakes on my side, or place for improvements. Your feedback is most welcome.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
powerhouse
Level 6
Level 6
Posts: 1144
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

ZMOT7S wrote:Now for the Xen part, I added the xen line to the grub file and rebooted but there was no option for xen in grub. I am using wired connection. I've also redid step 4 since you updated it. The file is there and able to run as root but on startup between grub and the linux mint login screen I see in the top corner "sh: vfio-pci /sbin/vfio-pci-override-vga.sh not found" or something similar to that. The file is there though and has full permissions.
Please remove the xen line in grub and update-grub - you do not need a xen configuration!

What you do need is the network bridge, so only do the part that relates to the bridge configuration. You need to modify /etc/network/interfaces and configure the bridge with either dhcp or static IP. After that run brctl to add the bridge. I named the bridge xenbr0, but you can choose any other name.

About the file not found, make sure there is no spelling mistake in either the file name or the content of /etc/modprobe.d/local.conf

Else than that, I can't explain how come the script is not found / executed.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
Trees138

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by Trees138 »

powerhouse wrote:Here my suggestions:

1. Make sure the path name in local.conf is spelled correct. As I have written above, I had the local.conf file name spelled wrong and nothing worked.
2. Make sure your /sbin/vfio-pci-override-vga.sh file name is spelled correct. Somewhere there is probably a typo.
3. Have you loaded the required kernel modules in initramfs? See my instructions in the how-to.
4. Check again that the script is executable.
5. Where is your local.conf file located? Make sure it is in the right directory.
6. If nothing of the above works, use the PCI IDs in local.conf as described a few posts above.
7. If you can't get it work, execute the vfio-pci-override-vga.sh script in the startup script before the qemu command.
8. If you still have problems, please post the content of all the relevant files here.

vfio_pci and vfio-pci are equivalent. Module names are translated automatically. Do an lsmod to see if the modules are loaded.

OK, I did pursue configuring the startup script and I am no able to reliably get vfio-pci to assign to the graphics card without intervention in boot up of the machine.

I too was getting the return that the machine was already running, even though it was not. I'm still playing around in that area of the tutorial. So for now, forward progress, but nothing really to report. I'm including my trouble shooting steps below as reference only, in case you're interested. I'll update more next chance I get to move forward.

Thank you again for all your time powerhouse!

_____________________________________________________________________


I'll go through the list, here are my replies:

The following link is an album, that hopefully illustrates this.
http://imgur.com/a/K0cpm



1. & 2. I did check the path name, it looks identical, to ensure it was I copied from the parts of sbin and pasted into the local.conf file. I then updated intitramfs. I'm still getting the same error. "/sbin/vfio-pci-override-vga.sh: not found"


3. Yes, I followed it verbatim.

Code: Select all

# List of modules that you want to include in your initramfs.
# They will be loaded at boot time in the order below.
#
# Syntax:  module_name [args ...]
#
# You must run update-initramfs(8) to effect this change.
#
# Examples:
#
# raid1
# sd_mod
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
vhost-net
Proof:

Code: Select all

$ lsmod | grep vfio
vfio_pci               40960  0
irqbypass              16384  2 kvm,vfio_pci
vfio_virqfd            16384  1 vfio_pci
vfio_iommu_type1       20480  0
vfio                   28672  2 vfio_iommu_type1,vfio_pci
4. It is, I've ran the chmod 755 part twice as root. I've also checked for it under permissions, see screenshot in photo album.

5. it is /etc/modprobe.d/local.conf per your instruction, verified with screencap.

6. I am trying to do that, as you'll see in the screen caps, but they only work after I've executed /sbin/vfio-pci-override-vga.sh by opening it manually.

7. I'll now pursue this method, I'll leave the other parts as feedback even if it works.
powerhouse
Level 6
Level 6
Posts: 1144
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

@Trees138:

In the local.conf file, you must remove one of the two lines - it is either / or.

Since you get an error that the /sbin/vfio-pci-override-vga.sh file is not found, try the "option ..." method! In other words, remove the line with "install ...". Make sure the file does not contain any additional lines.

Is there any other reason why /sbin couldn't be found, perhaps something related to mounting? I actually cannot imagine that since sbin should be mounted under / right from start. It is very strange and I have no idea. Can you post a separate post in another forum spot just asking why /sbin is not found? Perhaps someone is able to figure it out.

About the machine already running, I had the same issue and changing the vmname solved it. It doesn't matter how you name it, as long as it's unique and not the same as the script name or another program or script. I hope to figure out a better way to test for an already running VM. If someone has a suggestion, please don't be shy to post it.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
Locked

Return to “Virtual Machines”