HOW-TO make dual-boot obsolete using kvm VGA passthrough

Questions about virtualization software
Forum rules
Before you post read how to get help. Topics in this forum are automatically closed 6 months after creation.
powerhouse
Level 6
Level 6
Posts: 1138
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

DoctorWho wrote: Fri Jul 06, 2018 4:36 pm Thanx for the Reply. I am making great progress with this now, I got to the
TURINO boot screen but then it tries to do some PXE boot, and dumps me
into the UEFI BIOS options. It does not seem to use the iso file i have specifed,
i will keep messing with it .... I suppose the fact that the TURINO boot screen
appeared at all indicates that the video card passed thru .... Woo Hoo!! 8)
I ran into similar situations in the past, but I honestly don't remember how I overcame them. Make sure your paths are OK, and double check the start script. You can also post it here and I or some others can look at it. By the way, what's your OS, both Linux and Windows?

Just for your information, I did some editing in my tutorial, specifically regarding the IOMMU groups. You find the updated version on my blog page: Running Windows 10 on Linux using KVM with VGA Passthrough.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
DoctorWho

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by DoctorWho »

I been messin with my hardware. Now that passthru seems like it will work I need a larger
SSD :) Just got a 1tb SSD from AMAZON good deal. Now I am going to start all over from scratch
This time using Linux MInt 19 (CInnamon). I was previously using 18.3 which seemed to pretty
work well. Anyone have any opinions about GPU passthru on Mint 19? Or perhaps I am trying to
be to cutting edge and should just stay with 18.3?

My System Specs:
MotherBoard: ASRock Z68 Pro Gen3
Bus Clock: 100 mhz
UEFI: American Megatrends Inc P1.30 08/15/2011
32 Gigs of system ram

CPU
Intel Core I7 2600
256 kb primary memory cache
1024 KB secondary memory cache
8192 KB tertiary memory cache
64-bit ready
Multi-Core (4 total)
Hyper-threaded: (8 total)

I am currently running Linux Mint 19, although I think I am going to go back to 18.3. I don't really
have any preferences as to my distro. I have used mint for a while and like it. Are there any other
distros that MAY be MORE appropriate for GPU passthru? RedHat?? Fedora??? OpenSuse?
DoctorWho

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by DoctorWho »

BTW I am passing thru to a Windows 10 guest. Host OS will be /IS Linux of some sort.
Most likely Mint 18.3 as 19 seems a bit to new and I don't want the hassle of messing
with it to get it to work as I like ..
powerhouse
Level 6
Level 6
Posts: 1138
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

DoctorWho wrote: Wed Jul 11, 2018 2:21 pm BTW I am passing thru to a Windows 10 guest. Host OS will be /IS Linux of some sort.
Most likely Mint 18.3 as 19 seems a bit to new and I don't want the hassle of messing
with it to get it to work as I like ..
I have rewritten much of the tutorial to reflect changes in Linux Mint 19. For the latest version, see https://heiko-sieger.info/running-windo ... ssthrough/.

I strongly suggest you go with the latest version of Linux Mint.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
powerhouse
Level 6
Level 6
Posts: 1138
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

Here is the UserBenchmark on my Linux Mint 19 based Windows 10 VM:

UserBenchmarks: Game 60%, Desk 76%, Work 67%
CPU: Intel Core i7-3930K - 85.3%
GPU: Nvidia GTX 970 - 58.7%
SSD: Red Hat VirtIO 140GB - 75.2%
HDD: Red Hat VirtIO 2.5TB - 86.9%
HDD: Red Hat VirtIO 2TB - 54.6%
RAM: QEMU 1x16GB - 76.5%
MBD: QEMU Standard PC (Q35 + ICH9, 2009)

Notes:
qemu ver. 2.11.1
kernel 4.15
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
DoctorWho

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by DoctorWho »

I have run into a bit of a snag getting GPU passthru to work and am
wondering if anyone can point me in the right direction to continue.
So I have been following ths guide and am currently stuck on the USB
section in the QEMU script. I have cut and pasted and modified the script
per my system specs. When I attempt to run the script I get this Error:

qemu-system-x86_64: -usbdevice host:046d:c049: '-usbdevice' is deprecated, please use '-device usb-...' instead

I interpret this to mean the syntax of the USB command has changed. I have
looked all over for information and have been unable to find any concrete examples
of the new syntax. I have tried many different varations but I get either (-USB unknown command) or some other error
related to bad command or file name. I have tried:

-usb -device usb-host,hostbus=2,hostaddr=06A3,vendor=8020

I just am not sure of what the new syntax is, and have been unable to find any
examples. The USB devices I am trying to pass thru are just my keyboard and mouse to
get going.

Bus 002 Device 004: ID 046d:c049 Logitech, Inc. G5 Laser Mouse
Bus 002 Device 003: ID 06a3:8020 Saitek PLC Eclipse Keyboard

As always ANY response is greatly appreciated!

System Specs:
________________________________________________________________________________________________________________

AsRock Z68 Gen3 mobo, i7 2600 CPU, 4x8GB Kingston DDR3 ram, Coolmaster CPU cooler, GEFORCE GTX 980ti, 1 TB SSD + 2x2TB Hitachi SATA HDs, Thermaltake 1200w GOLD PSU / Linux Mint 19 OS
powerhouse
Level 6
Level 6
Posts: 1138
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

DoctorWho wrote: Wed Jul 25, 2018 8:46 pm I have run into a bit of a snag getting GPU passthru to work and am
wondering if anyone can point me in the right direction to continue.
So I have been following ths guide and am currently stuck on the USB
section in the QEMU script. I have cut and pasted and modified the script
per my system specs. When I attempt to run the script I get this Error:

qemu-system-x86_64: -usbdevice host:046d:c049: '-usbdevice' is deprecated, please use '-device usb-...' instead

I interpret this to mean the syntax of the USB command has changed. I have
looked all over for information and have been unable to find any concrete examples
of the new syntax. I have tried many different varations but I get either (-USB unknown command) or some other error
related to bad command or file name. I have tried:

-usb -device usb-host,hostbus=2,hostaddr=06A3,vendor=8020

I just am not sure of what the new syntax is, and have been unable to find any
examples. The USB devices I am trying to pass thru are just my keyboard and mouse to
get going.

Bus 002 Device 004: ID 046d:c049 Logitech, Inc. G5 Laser Mouse
Bus 002 Device 003: ID 06a3:8020 Saitek PLC Eclipse Keyboard

As always ANY response is greatly appreciated!

System Specs:
________________________________________________________________________________________________________________

AsRock Z68 Gen3 mobo, i7 2600 CPU, 4x8GB Kingston DDR3 ram, Coolmaster CPU cooler, GEFORCE GTX 980ti, 1 TB SSD + 2x2TB Hitachi SATA HDs, Thermaltake 1200w GOLD PSU / Linux Mint 19 OS
Hello DoctorWho:

Yes, the syntax has changed. The new syntax is:

Code: Select all

-usb
to activate the USB driver.

Code: Select all

usb-host,hostbus=bus,hostaddr=addr
Pass through the host device identified by bus and addr

Code: Select all

usb-host,vendorid=vendor,productid=product
Pass through the host device identified by vendor and product ID.

The information can be found locally in this directory: [url]file:///usr/share/doc/qemu-system-common/qemu-doc.html#usb_005fdevices[/url]

In your case, this is either:

Code: Select all

usb-host,hostbus=002,hostaddr=004
usb-host,hostbus=002,hostaddr=003
or:

Code: Select all

usb-host,vendorid=046d,productid=c049
usb-host,vendorid=06a3,productid=8020
Either option should lead to success.

I'm trying to keep up with the changes. However, all changes will only be made on my own website (see link in signature). I simply can't keep up editing multiple posts, and the forum here is to limiting when posting such long and rather complex tutorials.

Good luck!
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
powerhouse
Level 6
Level 6
Posts: 1138
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

For the past 3 months I noticed a severe drop in 2D performance on my Windows VM. I posted benchmarks and more details in a separate thread here.

I rolled back Nvidia drivers - no difference.

I checked and reconfigured hugetables - nope.

I reinstalled Windows 1803 - no dice.

I changed qemu script settings - no improvements, at least what I tried so far.

I installed a Windows on a new SSD using virt-manager/libvirt, and adjusted the xml config file. No improvement.

I installed Windows directly on an SSD and booted into Windows (bare metal installation) - 2D problem gone!

If you have benchmarks to compare, preferably Passmark 8 (old) or 9, please share.

More details and benchmarks on my separate post (see link above).

EDIT: Its the Spectre protection under Windows. See link above for more information.
Last edited by powerhouse on Sat Aug 18, 2018 2:25 pm, edited 1 time in total.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
s2156945

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by s2156945 »

Thanks all for your posts here and elsewhere.

I am this guy: https://heiko-sieger.info/running-windo ... comment-27

After struggling with this for a while there seem to be 4 main ways to get this going (paraphrasing / winging it)
1) Traditional Desktop - passthrough the GPU card, the bios is in a predictable place, windows is happy - works with most desktops and some high end laptops
2) Misairu_G laptop - as for #1 but you need to do some voodoo to make sure the GPU is powered when you boot qemu - works with some laptops
3) jscinoz - as for #2 but you also need to figure out your laptops acpi table so that the guest OS can find the video bios - has only worked for a hacked linux guest
4) bash64 - let the laptop's bios initialize the cards and if you set up linux so it doesn't touch them before you run qemu it should all be good - most promising for me

@bash64 I'm after some more details on your setup to see if it's similar to mine

Questions:
- how do I tell if my GPU is powered on?
- what are the next things I should try?
- is there some difference between your working setup and mine which might be important?
- any linux commandline args I should get rid of / add / try?

I've installed windows 10 successfully using approach #2 but I get Code 43. Similarly I can boot a linux ISO within the same KVM setups.

cheers,
Woody

My setup now...

Running Ubuntu 18.04, not mint:

Code: Select all

Linux curtis 4.15.0-29-generic #31-Ubuntu SMP Tue Jul 17 15:39:52 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
P950ER Clevo / Metabox
2 VGA cards in PCI; seems that the Nvidia is connected to the 3 external ports (going by windows nvidia control panel), Intel to the LCD (muxless)
When I boot up (windows or linux) the external monitors do not show anything until the OS is booted (e.g. windows 10 login)

This is in contrast to my old laptop P650 Clevo which has the Nvidia card listed as a "3D" device not a VGA, and the grub menu shows in whatever monitors I have connected. It seems to be Muxed, nvidia seems to connect to all monitors as needed.


Code: Select all

woody@curtis:~$ lspci | grep VGA
00:02.0 VGA compatible controller: Intel Corporation Device 3e9b
01:00.0 VGA compatible controller: NVIDIA Corporation GP104M [GeForce GTX 1070 Mobile] (rev a1)
woody@curtis:~$ 
Kernel Commandline and IOMMU:

Code: Select all

woody@curtis:~$ dmesg | grep -i iommu
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.15.0-29-generic root=UUID=69e42b1f-6eb0-11e8-a77a-80fa5b5a9d89 ro blacklist=nouveau modprobe.blacklist=nouveau nouveau.blacklist=1 elevator=deadline drm.debug=0xe intel_iommu=on kvm.ignore_msrs=1 acpi_osi=! acpi_osi=Linux "acpi_osi=Windows 2009" default_hugepagesz=1G hugepagesz=1G hugepages=12 transparent_hugepage=never rd.driver.pre=vfio-pci systemd.log_level=debug systemd.journald.forward_to_console=2 console=tty1 nomodeset
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.15.0-29-generic root=UUID=69e42b1f-6eb0-11e8-a77a-80fa5b5a9d89 ro blacklist=nouveau modprobe.blacklist=nouveau nouveau.blacklist=1 elevator=deadline drm.debug=0xe intel_iommu=on kvm.ignore_msrs=1 acpi_osi=! acpi_osi=Linux "acpi_osi=Windows 2009" default_hugepagesz=1G hugepagesz=1G hugepages=12 transparent_hugepage=never rd.driver.pre=vfio-pci systemd.log_level=debug systemd.journald.forward_to_console=2 console=tty1 nomodeset
[    0.000000] DMAR: IOMMU enabled
[    0.004000] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    2.234880] iommu: Adding device 0000:00:00.0 to group 0
[    2.234894] iommu: Adding device 0000:00:01.0 to group 1
[    2.234903] iommu: Adding device 0000:00:02.0 to group 2
[    2.234913] iommu: Adding device 0000:00:08.0 to group 3
[    2.234925] iommu: Adding device 0000:00:12.0 to group 4
[    2.234951] iommu: Adding device 0000:00:14.0 to group 5
[    2.234959] iommu: Adding device 0000:00:14.2 to group 5
[    2.234976] iommu: Adding device 0000:00:14.3 to group 5
[    2.234987] iommu: Adding device 0000:00:16.0 to group 6
[    2.234997] iommu: Adding device 0000:00:17.0 to group 7
[    2.235018] iommu: Adding device 0000:00:1d.0 to group 8
[    2.235034] iommu: Adding device 0000:00:1d.6 to group 9
[    2.235046] iommu: Adding device 0000:00:1d.7 to group 10
[    2.235066] iommu: Adding device 0000:00:1f.0 to group 11
[    2.235076] iommu: Adding device 0000:00:1f.3 to group 11
[    2.235085] iommu: Adding device 0000:00:1f.4 to group 11
[    2.235097] iommu: Adding device 0000:00:1f.5 to group 11
[    2.235103] iommu: Adding device 0000:01:00.0 to group 1
[    2.235118] iommu: Adding device 0000:03:00.0 to group 12
[    2.235137] iommu: Adding device 0000:04:00.0 to group 13
woody@curtis:~$ 
So group 1 is the card + the bridge, but I'm only binding vfio-pci to the card and passing that to qemu:

Code: Select all

00:01.0 PCI bridge: Intel Corporation Skylake PCIe Controller (x16) (rev 07)
01:00.0 VGA compatible controller: NVIDIA Corporation GP104M [GeForce GTX 1070 Mobile] (rev a1)
Binding the card to vfio-pci seems to be working

Code: Select all

woody@curtis:~$ lspci -nnk -s 01:00.0
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104M [GeForce GTX 1070 Mobile] [10de:1ba1] (rev a1)
	Kernel driver in use: vfio-pci
	Kernel modules: nvidiafb, nouveau
woody@curtis:~$ 
Booting up with nomodeset seems to work fine in linux

Code: Select all

woody@curtis:~$ inxi -Gx
Graphics:  Card-1: Intel Device 3e9b bus-ID: 00:02.0
           Card-2: NVIDIA GP104M [GeForce GTX 1070 Mobile] bus-ID: 01:00.0
           Display Server: x11 (X.Org 1.19.6 ) drivers: fbdev (unloaded: modesetting,vesa)
           Resolution: 3840x2160@88.00hz
           OpenGL: renderer: llvmpipe (LLVM 6.0, 256 bits) version: 3.3 Mesa 18.0.5 Direct Render: Yes
woody@curtis:~$ 
Running my script: bash64 with a bit of acpi power on:

Code: Select all

#!/bin/bash

#lookingglass...

#cd /media/dad/QEMU-SSD/qemu-wd

#comment out nbd section. may not work with windows 10...
<<NBDSECTION
echo 'Creating nbd device nodes...'
sudo modprobe nbd max_part=63
sleep 2
echo 'Attach hardfile to device node nbd0...'
qemu-nbd -f raw -c /dev/nbd0 windows7.img
echo 'Create folder to mount hardfile to...'
mkdir -p /media/w7
echo 'Mount nbd0p1 to the folder...'
mount /dev/nbd0p1 /media/w7
echo 'Display the contents of the mounted folder...'
ls /media/w7/Boot/
echo 'Replace BOOTSTAT.DAT with backup copy...'
cp BOOTSTAT.DAT /media/w7/Boot/
cp BCD.LOG /media/w7/Boot/
cp BCD.LOG1 /media/w7/Boot/
cp BCD.LOG2 /media/w7/Boot/
echo 'Wait 5 seconds...'
sleep 5
echo 'Unmount and remove folder...'
umount /media/w7
echo 'Disconnect nbd device node...'
qemu-nbd -d /dev/nbd0
NBDSECTION

configfile=/media/dad/QEMU-SSD/qemu-wd/vfio-pci1.cfg

#vfiobind function....
#vfiobind() {
# dev="$1"
# vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
# device=$(cat /sys/bus/pci/devices/$dev/device)
# echo $dev $vendor $device
# if [ -e /sys/bus/pci/devices/$dev/driver ]; then
# echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
# fi
# echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
#}

#modprobe vfio-pci

#cat $configfile | while read line;do
# echo $line | grep ^# >/dev/null 2>&1 && continue
# vfiobind $line
#done

# use pulseaudio
# use pulseaudio
export QEMU_AUDIO_DRV=pa
#export QEMU_PA_SAMPLES=8192
export QEMU_AUDIO_TIMER_PERIOD=99
export QEMU_PA_SERVER=/run/user/1000/pulse/native

#export QEMU_PA_BUFFER_SIZE_OUT=5000
#export QEMU_PA_BUFFER_SIZE_IN=5000
#export QEMU_PA_TLENGTH=5000
#export QEMU_PA_FRAGSIZE=5000

#use ALSA
#export QEMU_ALSA_DAC_BUFFER_SIZE=512
#export QEMU_ALSA_DAC_PERIOD_SIZE=170
#export QEMU_AUDIO_DRV=alsa

# display current time and expected boot time for Windows...
boottime1=$(date -d "$starttime today + 0 minutes" +'%I:%M:%S')
boottime2=$(date -d "$starttime today + 1 minutes + 13 seconds" +'%I:%M:%S')
echo "Windows expected to reach desktop at $boottime2..."

echo turn on nvidia card
modprobe -v acpi_call
echo '\_SB_.PCI0.PEG0.PEGP._ON' > /proc/acpi/call
echo sleep 2...
sleep 2;
echo slept;
# bind it to vfio-pci
#modprobe vfio-pci
#echo "10de 1ba1" > /sys/bus/pci/drivers/vfio-pci/new_id

#launch windows 7 64bit using SEABIOS (non-uefi bios)....
#/home/dad/qemu/build/x86_64-softmmu/
#-chardev stdio,id=seabios -device isa-debugcon,iobase=0x402,chardev=seabios \
#-bios /media/dad/QEMU-SSD/qemu-wd/bios.bin-1.8.0 \
qemu-system-x86_64 \
-machine type=q35,accel=kvm \
-chardev stdio,id=seabios -device isa-debugcon,iobase=0x402,chardev=seabios \
-bios /usr/share/seabios/bios.bin \
-cpu host,kvm=off \
-smp 1,sockets=1,cores=1,threads=1 \
-m 12G \
-mem-path /dev/hugepages \
-nodefaults \
-soundhw hda \
-mem-prealloc \
-rtc clock=host,base=localtime \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0 \
-boot menu=on \
-boot order=dc \
-drive id=disk0,if=virtio,cache=none,format=raw,file=/home/woody/win.img \
-drive file=/home/woody/Downloads/virtio-win-0.1.141.iso,index=2,media=cdrom \
-spice port=5900,addr=127.0.0.1,disable-ticketing \
#-device vfio-pci,host=01:00.1 \
#-device virtio-scsi-pci,id=scsi \
#-drive id=disk0,if=virtio,cache=none,format=raw,file=/media/dad/QEMU-SSD/qemu-wd/windows10.img \
#-drive file=/media/dad/QEMU-SSD/qemu-wd/virtio-win-0.1.126.iso,id=isocd,format=raw,if=none -device scsi-cd,drive=isocd \
#-netdev type=tap,id=net0,ifname=tap0 \
#-device virtio-net-pci,netdev=net0,mac=00:16:3e:00:01:01 \
#-usb -device usb-host,vendorid=0x276d,productid=0x1119 \

# unbind + unload vfio-pci
#echo "0000:01:00.0" > /sys/bus/pci/drivers/vfio-pci/0000:01:00.0/driver/unbind
#modprobe -r vfio-pci

# turn off nvidia card
#echo '\_SB_.PCI0.PEG0.PEGP._OFF' > /proc/acpi/call
#modprobe -r acpi_call

exit 0

#lookingglass entries

#-object memory-backend-file,id=ivshmem,share=on,mem-path=/dev/shm/looking-glass,size=32M \
#-device ivshmem-plain,memdev=ivshmem \
#-spice port=5900,addr=127.0.0.1,disable-ticketing \
It looks OK but still hangs at "Running option rom at c000:0003"

Code: Select all

woody@curtis:~$ sudo ./bash64_mod.sh 
[sudo] password for woody: 
Windows expected to reach desktop at 12:16:28...
turn on nvidia card
sleep 2...
slept
SeaBIOS (version 1.10.2-1ubuntu1)
BUILD: gcc: (Ubuntu 6.3.0-16ubuntu6) 6.3.0 20170506 binutils: (GNU Binutils for Ubuntu) 2.28
Running on QEMU (q35)
RamSize: 0x80000000 [cmos]
Relocating init from 0x000e3160 to 0x7ffb1d20 (size 57920)
Found QEMU fw_cfg
QEMU fw_cfg DMA interface supported
RamBlock: addr 0x0000000000000000 len 0x0000000080000000 [e820]
RamBlock: addr 0x0000000100000000 len 0x0000000280000000 [e820]
Moving pm_base to 0x600
=== PCI bus & bridge init ===
PCI: pci_bios_init_bus_rec bus = 0x0
PCI: pci_bios_init_bus_rec bdf = 0xe0
PCI: primary bus = 0x0
PCI: secondary bus = 0xff -> 0x1
PCI: pci_bios_init_bus_rec bus = 0x1
PCI: subordinate bus = 0x0 -> 0x1
=== PCI device probing ===
Found 8 PCI devices (max PCI bus is 01)
=== PCI new allocation pass #1 ===
PCI: check devices
PCI: secondary bus 1 size 00001000 type io
PCI: secondary bus 1 size 02000000 type mem
PCI: secondary bus 1 size 20000000 type prefmem
=== PCI new allocation pass #2 ===
PCI: IO: c000 - d09f
PCI: 32: 00000000c0000000 - 00000000fec00000
PCI: map device bdf=00:02.0  bar 0, addr 0000d000, size 00000040 [io]
PCI: map device bdf=00:1f.3  bar 4, addr 0000d040, size 00000040 [io]
PCI: map device bdf=00:1f.2  bar 4, addr 0000d080, size 00000020 [io]
PCI: map device bdf=00:01.0  bar 0, addr fe000000, size 00004000 [mem]
PCI: map device bdf=00:02.0  bar 1, addr fe004000, size 00001000 [mem]
PCI: map device bdf=00:1f.2  bar 5, addr fe005000, size 00001000 [mem]
PCI: map device bdf=00:02.0  bar 4, addr f0000000, size 00004000 [prefmem]
PCI: map device bdf=01:00.0  bar 5, addr 0000c000, size 00000080 [io]
PCI: map device bdf=01:00.0  bar 0, addr fc000000, size 01000000 [mem]
PCI: map device bdf=01:00.0  bar 6, addr fd000000, size 00080000 [mem]
PCI: map device bdf=01:00.0  bar 1, addr d0000000, size 10000000 [prefmem]
PCI: map device bdf=01:00.0  bar 3, addr e0000000, size 02000000 [prefmem]
PCI: init bdf=00:00.0 id=8086:29c0
PCI: init bdf=00:01.0 id=8086:2668
PCI: init bdf=00:02.0 id=1af4:1001
PCI: init bdf=00:1c.0 id=8086:3420
PCI: init bdf=00:1f.0 id=8086:2918
Q35 LPC init: elcr=00 0c
Using pmtimer, ioport 0x608
PCI: init bdf=00:1f.2 id=8086:2922
PCI: init bdf=00:1f.3 id=8086:2930
PCI: init bdf=01:00.0 id=10de:1ba1
PCI: Enabling 01:00.0 for primary VGA
PCI: Setting VGA enable on bridge 00:1c.0
Found 1 cpu(s) max supported 1 cpu(s)
Copying PIR from 0x7ffbfca0 to 0x000f1320
Copying MPTABLE from 0x00006e90/7ffa8930 to 0x000f1230
Copying SMBIOS entry point from 0x00006e90 to 0x000f1070
Scan for VGA option rom
Running option rom at c000:0003

powerhouse
Level 6
Level 6
Posts: 1138
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

@s2156945

Good to see you here.

EDIT: You might find this guide helpful: https://gitlab.com/YuriAlek/vfio

About your post / VM configuration, could you provide some additional details?

Qemu version:

Code: Select all

qemu-system-x86_64 --version
vfio kernel modules loaded?

Code: Select all

lsmod | grep -i vfio
How do you bind your graphics card to vfio-pci? Note that you commented out the following line:
#echo "10de 1ba1" > /sys/bus/pci/drivers/vfio-pci/new_id

Where is the Windows installation ISO - I see it commented out. Is Windows already installed? If yes, how? E.g. via qemu or native installation on bare metal?

Have you managed to boot /home/woody/win.img before?

Could you post a link to the original script you are using? You mention it's by bash64. I remember he posted here before, but can't find this script.

I'm not familiar with lookingglass nor single GPU passthrough, but will be happy to try and help. As always, first check the obvious.

Also, is hugetbls mounted to /dev/hugepages?

Another thing: smp 1 is a little on the low side. If the CPU uses hyperthreading and you got 4 cores, try

Code: Select all

-smp 2,sockets=1,cores=1,threads=2 \
Last edited by powerhouse on Sat Aug 18, 2018 10:09 am, edited 1 time in total.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
powerhouse
Level 6
Level 6
Posts: 1138
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

powerhouse wrote: Thu Aug 16, 2018 11:53 am For the past 3 months I noticed a severe drop in 2D performance on my Windows VM. I posted benchmarks and more details in a separate thread here.

I rolled back Nvidia drivers - no difference.

I checked and reconfigured hugetables - nope.

I reinstalled Windows 1803 - no dice.

I changed qemu script settings - no improvements, at least what I tried so far.

I installed a Windows on a new SSD using virt-manager/libvirt, and adjusted the xml config file. No improvement.

I installed Windows directly on an SSD and booted into Windows (bare metal installation) - 2D problem gone!

If you have benchmarks to compare, preferably Passmark 8 (old) or 9, please share.

More details and benchmarks on my separate post (see link above).
I found the problem: it's the Sceptre protection in Windows. For more on that, see viewtopic.php?f=231&t=275744.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
bash64
Level 3
Level 3
Posts: 116
Joined: Mon Mar 23, 2015 12:19 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by bash64 »

UPDATE:
For those reading I have a laptop with nvidia optimus.
I have good news.
the i915 driver is no longer preventing qemu from booting.
Sorry, I do not know if an update to the driver or qemu fixed the issue, but I have removed nomodeset from my boot line , the i915 driver loads, and windows10 comes up fine.
This also allows the linux side to finally control the vga port and I can use my external 26" monitor instead of the laptop lcd to view linux.
With nomodeset on your boot line you boot without any video drivers (well, you use fbdev) and it cannot access external vga ports. At least not on my laptop.

Cheers.
powerhouse
Level 6
Level 6
Posts: 1138
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

bash64 wrote: Sun Aug 19, 2018 2:13 am UPDATE:
For those reading I have a laptop with nvidia optimus.
I have good news.
the i915 driver is no longer preventing qemu from booting.
Sorry, I do not know if an update to the driver or qemu fixed the issue, but I have removed nomodeset from my boot line , the i915 driver loads, and windows10 comes up fine.
This also allows the linux side to finally control the vga port and I can use my external 26" monitor instead of the laptop lcd to view linux.
With nomodeset on your boot line you boot without any video drivers (well, you use fbdev) and it cannot access external vga ports. At least not on my laptop.

Cheers.
Thanks bash64 - I somewhere read that nomodeset might interfere. Which Linux version are you running now? Also, can you share the kernel version and the qemu version?

Another poster above referred to your start script, but somehow I can't find it. Care to share?

Thanks!
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
s2156945

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by s2156945 »

@powerhouse this is the post I copied the script from
viewtopic.php?f=231&t=212692&start=460#p1486237

@bash64 what HW have you got?
s2156945

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by s2156945 »

powerhouse wrote: Sat Aug 18, 2018 9:59 am About your post / VM configuration, could you provide some additional details?

Qemu version:

Code: Select all

qemu-system-x86_64 --version

Code: Select all

woody@curtis:~$ qemu-system-x86_64 --version
QEMU emulator version 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.4)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
woody@curtis:~$ 
powerhouse wrote: Sat Aug 18, 2018 9:59 am vfio kernel modules loaded?

Code: Select all

lsmod | grep -i vfio

Code: Select all

woody@curtis:~$ lsmod | grep -i vfio
vfio_pci               45056  0
vfio_virqfd            16384  1 vfio_pci
irqbypass              16384  2 kvm,vfio_pci
vfio_iommu_type1       24576  0
vfio                   28672  2 vfio_iommu_type1,vfio_pci
woody@curtis:~$ 
powerhouse wrote: Sat Aug 18, 2018 9:59 am How do you bind your graphics card to vfio-pci? Note that you commented out the following line:
#echo "10de 1ba1" > /sys/bus/pci/drivers/vfio-pci/new_id

Code: Select all

woody@curtis:~$ cat /etc/modprobe.d/vfio.conf
# GTX1070
options vfio-pci ids=10de:1ba1
options vfio-pci disable_vga=1

blacklist acpi-call
blacklist acpi_call
woody@curtis:~$ 
which works on boot:

Code: Select all

woody@curtis:~$ dmesg | grep vfio
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.15.0-29-generic root=UUID=69e42b1f-6eb0-11e8-a77a-80fa5b5a9d89 ro blacklist=nouveau modprobe.blacklist=nouveau nouveau.blacklist=1 elevator=deadline drm.debug=0xe intel_iommu=on kvm.ignore_msrs=1 acpi_osi=! acpi_osi=Linux "acpi_osi=Windows 2009" default_hugepagesz=1G hugepagesz=1G hugepages=12 transparent_hugepage=never rd.driver.pre=vfio-pci systemd.log_level=debug systemd.journald.forward_to_console=2 console=tty1 nomodeset
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.15.0-29-generic root=UUID=69e42b1f-6eb0-11e8-a77a-80fa5b5a9d89 ro blacklist=nouveau modprobe.blacklist=nouveau nouveau.blacklist=1 elevator=deadline drm.debug=0xe intel_iommu=on kvm.ignore_msrs=1 acpi_osi=! acpi_osi=Linux "acpi_osi=Windows 2009" default_hugepagesz=1G hugepagesz=1G hugepages=12 transparent_hugepage=never rd.driver.pre=vfio-pci systemd.log_level=debug systemd.journald.forward_to_console=2 console=tty1 nomodeset
[    4.818983] vfio-pci 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[    4.844137] vfio_pci: add [10de:1ba1[ffff:ffff]] class 0x000000/00000000
The ffff:ffff and 0x000000/00000000 look worryingly plain
powerhouse wrote: Sat Aug 18, 2018 9:59 am Where is the Windows installation ISO - I see it commented out. Is Windows already installed? If yes, how? E.g. via qemu or native installation on bare metal?

Have you managed to boot /home/woody/win.img before?
This image was installed with a version of your script, but passthrough was not possible (Card detected and drivers installed but Error 43).
With bash64's script it looks like SeaBIOS is not booting.
powerhouse wrote: Sat Aug 18, 2018 9:59 am Could you post a link to the original script you are using? You mention it's by bash64. I remember he posted here before, but can't find this script.
viewtopic.php?f=231&t=212692&start=460#p1486237
powerhouse wrote: Sat Aug 18, 2018 9:59 am I'm not familiar with lookingglass nor single GPU passthrough, but will be happy to try and help. As always, first check the obvious.
Not trying lookingglass yet, just left it commented out from bash64's script.
powerhouse wrote: Sat Aug 18, 2018 9:59 am Also, is hugetbls mounted to /dev/hugepages?
Yep, and it works with other VM scripts, but I don't think it's getting that far.

Code: Select all

woody@curtis:~$ hugeadm --list-all-mounts --pool-list
Mount Point          Options
/dev/hugepages       rw,relatime,pagesize=1024M
      Size  Minimum  Current  Maximum  Default
1073741824       12       12       12        *
woody@curtis:~$ 
powerhouse wrote: Sat Aug 18, 2018 9:59 am Another thing: smp 1 is a little on the low side. If the CPU uses hyperthreading and you got 4 cores, try

Code: Select all

-smp 2,sockets=1,cores=1,threads=2 \
Sorry, was changed in an attempt to strace qemu to see where it was hanging, have changed back to something more usual:

Code: Select all

-smp 4,sockets=1,cores=2,threads=2 \
Happy to try anything!

It would be of interest to me to see fullish logs of a working system, possibly annotated to show the key bits, see where mine is going wrong.

My main suspicion is that the graphics card bios is hidden somewhere in system bios and the windows driver is intent in looking for it there, so I'd need to lie to the windows driver (or trick it somehow - .ini entry?) so it looked for the bios is a more usual spot.
powerhouse
Level 6
Level 6
Posts: 1138
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

@s2156945

Remove the following from your /etc/modprobe.d/vfio.conf file:

Code: Select all

options vfio-pci disable_vga=1
Then run:

Code: Select all

sudo update-initramfs -u
options vfio-pci disable_vga=1 is for OVMF (UEFI) boot only! You are using SeaBIOS.

What is blacklist acpi-call ? Never saw it. Leave it in the file if unsure.

The dmesg vfio output is just fine - no need to worry about the fffffffffff and 0000000000.

EDIT:
woody@curtis:~$ hugeadm --list-all-mounts --pool-list
Mount Point Options
/dev/hugepages rw,relatime,pagesize=1024M
Size Minimum Current Maximum Default
1073741824 12 12 12 *
woody@curtis:~$
This doesn't look good - you are giving your Windows VM 12GB, but in pages of 1GB each. Try to define the standard size of 2MB.

Here is my setup for 16GB (for Windows) and spare change.

Code: Select all

 hugeadm --list-all-mounts --pool-list
Mount Point          Options
/dev/hugepages       rw,relatime,gid=129,mode=1770,pagesize=2M
      Size  Minimum  Current  Maximum  Default
   2097152     8400     8400     8400        *
1073741824        0        0        0 
EDIT2: If you got problems with the hugepages settings, just revert the whole setup process and delete them. This is the last thing you should worry about and the least important.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
s2156945

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by s2156945 »

powerhouse wrote: Mon Aug 20, 2018 3:53 pm @s2156945

Remove the following from your /etc/modprobe.d/vfio.conf file:

Code: Select all

options vfio-pci disable_vga=1
Then run:

Code: Select all

sudo update-initramfs -u
options vfio-pci disable_vga=1 is for OVMF (UEFI) boot only! You are using SeaBIOS.

What is blacklist acpi-call ? Never saw it. Leave it in the file if unsure.

The dmesg vfio output is just fine - no need to worry about the fffffffffff and 0000000000.
OK, did that and rebooted:

Code: Select all

woody@curtis:~$ dmesg | grep vfio
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.15.0-29-generic root=UUID=69e42b1f-6eb0-11e8-a77a-80fa5b5a9d89 ro blacklist=nouveau modprobe.blacklist=nouveau nouveau.blacklist=1 elevator=deadline drm.debug=0xe intel_iommu=on kvm.ignore_msrs=1 acpi_osi=! acpi_osi=Linux "acpi_osi=Windows 2009" default_hugepagesz=1G hugepagesz=1G hugepages=12 transparent_hugepage=never rd.driver.pre=vfio-pci systemd.log_level=debug systemd.journald.forward_to_console=2 console=tty1 nomodeset
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.15.0-29-generic root=UUID=69e42b1f-6eb0-11e8-a77a-80fa5b5a9d89 ro blacklist=nouveau modprobe.blacklist=nouveau nouveau.blacklist=1 elevator=deadline drm.debug=0xe intel_iommu=on kvm.ignore_msrs=1 acpi_osi=! acpi_osi=Linux "acpi_osi=Windows 2009" default_hugepagesz=1G hugepagesz=1G hugepages=12 transparent_hugepage=never rd.driver.pre=vfio-pci systemd.log_level=debug systemd.journald.forward_to_console=2 console=tty1 nomodeset
[    4.638699] vfio-pci 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
[    4.660099] vfio_pci: add [10de:1ba1[ffff:ffff]] class 0x000000/00000000
[   12.633252] vfio-pci 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
[   34.137765] vfio-pci 0000:01:00.0: enabling device (0002 -> 0003)
[   35.164129] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1e@0x258
[   35.164139] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
woody@curtis:~$ 
Still hangs :-/

Code: Select all

woody@curtis:~$ sudo ./bash64_mod.sh 
[sudo] password for woody: 
Windows expected to reach desktop at 07:30:58...
turn on nvidia card
sleep 2...
slept
SeaBIOS (version 1.10.2-1ubuntu1)
BUILD: gcc: (Ubuntu 6.3.0-16ubuntu6) 6.3.0 20170506 binutils: (GNU Binutils for Ubuntu) 2.28
Running on QEMU (q35)
RamSize: 0x80000000 [cmos]
Relocating init from 0x000e3160 to 0x7ffb1d20 (size 57920)
Found QEMU fw_cfg
QEMU fw_cfg DMA interface supported
RamBlock: addr 0x0000000000000000 len 0x0000000080000000 [e820]
RamBlock: addr 0x0000000100000000 len 0x0000000280000000 [e820]
Moving pm_base to 0x600
=== PCI bus & bridge init ===
PCI: pci_bios_init_bus_rec bus = 0x0
PCI: pci_bios_init_bus_rec bdf = 0xe0
PCI: primary bus = 0x0
PCI: secondary bus = 0xff -> 0x1
PCI: pci_bios_init_bus_rec bus = 0x1
PCI: subordinate bus = 0x0 -> 0x1
=== PCI device probing ===
Found 8 PCI devices (max PCI bus is 01)
=== PCI new allocation pass #1 ===
PCI: check devices
PCI: secondary bus 1 size 00001000 type io
PCI: secondary bus 1 size 02000000 type mem
PCI: secondary bus 1 size 20000000 type prefmem
=== PCI new allocation pass #2 ===
PCI: IO: c000 - d09f
PCI: 32: 00000000c0000000 - 00000000fec00000
PCI: map device bdf=00:02.0  bar 0, addr 0000d000, size 00000040 [io]
PCI: map device bdf=00:1f.3  bar 4, addr 0000d040, size 00000040 [io]
PCI: map device bdf=00:1f.2  bar 4, addr 0000d080, size 00000020 [io]
PCI: map device bdf=00:01.0  bar 0, addr fe000000, size 00004000 [mem]
PCI: map device bdf=00:02.0  bar 1, addr fe004000, size 00001000 [mem]
PCI: map device bdf=00:1f.2  bar 5, addr fe005000, size 00001000 [mem]
PCI: map device bdf=00:02.0  bar 4, addr f0000000, size 00004000 [prefmem]
PCI: map device bdf=01:00.0  bar 5, addr 0000c000, size 00000080 [io]
PCI: map device bdf=01:00.0  bar 0, addr fc000000, size 01000000 [mem]
PCI: map device bdf=01:00.0  bar 6, addr fd000000, size 00080000 [mem]
PCI: map device bdf=01:00.0  bar 1, addr d0000000, size 10000000 [prefmem]
PCI: map device bdf=01:00.0  bar 3, addr e0000000, size 02000000 [prefmem]
PCI: init bdf=00:00.0 id=8086:29c0
PCI: init bdf=00:01.0 id=8086:2668
PCI: init bdf=00:02.0 id=1af4:1001
PCI: init bdf=00:1c.0 id=8086:3420
PCI: init bdf=00:1f.0 id=8086:2918
Q35 LPC init: elcr=00 0c
Using pmtimer, ioport 0x608
PCI: init bdf=00:1f.2 id=8086:2922
PCI: init bdf=00:1f.3 id=8086:2930
PCI: init bdf=01:00.0 id=10de:1ba1
PCI: Enabling 01:00.0 for primary VGA
PCI: Setting VGA enable on bridge 00:1c.0
handle_smp: apic_id=0x2
handle_smp: apic_id=0x3
handle_smp: apic_id=0x1
Found 4 cpu(s) max supported 4 cpu(s)
Copying PIR from 0x7ffbfca0 to 0x000f1320
Copying MPTABLE from 0x00006e90/7ffa8930 to 0x000f1230
Copying SMBIOS entry point from 0x00006e90 to 0x000f1070
Scan for VGA option rom
Running option rom at c000:0003

powerhouse wrote: Mon Aug 20, 2018 3:53 pm EDIT:
woody@curtis:~$ hugeadm --list-all-mounts --pool-list
Mount Point Options
/dev/hugepages rw,relatime,pagesize=1024M
Size Minimum Current Maximum Default
1073741824 12 12 12 *
woody@curtis:~$
This doesn't look good - you are giving your Windows VM 12GB, but in pages of 1GB each. Try to define the standard size of 2MB.

Here is my setup for 16GB (for Windows) and spare change.

Code: Select all

 hugeadm --list-all-mounts --pool-list
Mount Point          Options
/dev/hugepages       rw,relatime,gid=129,mode=1770,pagesize=2M
      Size  Minimum  Current  Maximum  Default
   2097152     8400     8400     8400        *
1073741824        0        0        0 
EDIT2: If you got problems with the hugepages settings, just revert the whole setup process and delete them. This is the last thing you should worry about and the least important.
OK, I think that bit is working for my other kvm experiments, seasbios boots without -vga none
powerhouse
Level 6
Level 6
Posts: 1138
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

@s2156945: It might be a stupid question, but have you changed your machine type from fx440 to q35? If that is the case, the system won't boot (vice versa is the same).

Please post the output of:

Code: Select all

lspci -kn | grep -A 2 01:00
Could you post the /etc/default/grub file? I see you got tons of options buried in there.

Is nomodeset needed? If not, remove it and

Code: Select all

sudo update-grub
Reboot and check.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
s2156945

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by s2156945 »

powerhouse wrote: Tue Aug 21, 2018 8:13 am @s2156945: It might be a stupid question, but have you changed your machine type from fx440 to q35? If that is the case, the system won't boot (vice versa is the same).
I’m pretty sure I’ve only done q35 vms, no fx440.

Do you mean windows doesn’t boot?

Does seabios need fx440?
powerhouse wrote: Tue Aug 21, 2018 8:13 am Please post the output of:

Code: Select all

lspci -kn | grep -A 2 01:00
Could you post the /etc/default/grub file? I see you got tons of options buried in there.

Is nomodeset needed? If not, remove it and

Code: Select all

sudo update-grub
Reboot and check.
lspci -kn I’ll post tomorrow, but it says at least vfio_pci driver.

I tried without nomodeset, same same.

I tried unblacklisting i915 but still get fbdev as the x driver.

/etc/default/grub I’ll post as well - I have a mix of things in there to avoid nouveau hangs and lspci hangs. They should be in dmesg | grep vfio output in my previous posts.
powerhouse
Level 6
Level 6
Posts: 1138
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse »

@s2156945

Ok, no need for the grub file. What does the entries below do?

Code: Select all

acpi_osi=! acpi_osi=Linux "acpi_osi=Windows 2009"
They seem to contradict each other.

If you've only done q35, no need to worry. But some people change their machine type and wonder why Windows doesn't boot anymore.

Now I wonder, how did you install Windows in the first place? What was the configuration that worked? Or did you install and afterwards it doesn't boot again?

Seabios: No, it doesn't need fx440. Should work likewise with q35. However, some people swear that fx440 works better in some cases. With the new qemu and kernel releases q35 should not be an issue. Even libvirt/virt-manager should be able to produce sensible settings. If you don't mind re-installing Windows, you might give it a try. However, I would do that as a last recourse, and only after a complete backup of your Windows installation.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
Locked

Return to “Virtual Machines”