HOW-TO make dual-boot obsolete using kvm VGA passthrough

Questions about virtualization software
Forum rules
Before you post please read how to get help
vitriolicex
Level 1
Level 1
Posts: 1
Joined: Thu Feb 08, 2018 12:48 am
Location: Texas, USA

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by vitriolicex » Thu Feb 08, 2018 10:05 pm

Hello,
I'm having an issue with getting this to work on my computer - I get the output:

Code: Select all

collin@surveillance:~/vm$ sudo ./run.sh
qemu-system-x86_64: -device vfio-pci,host=03:00.0,multifunction=on: VFIO 0000:03:00.0 BAR 3 mmap unsupported. Performance may be slow
whenever I try to run my bash script. I get no output from the GPU passed through.

I am using an AMD Radeon R5 235X and an NVIDIA GTX 1060 3GB, the AMD as the "host" and the NVIDIA as the one being passed to the vm. I'm forced to use my AMD in the last PCI slot (05:00.0) because I plan to pass through the 1060 3GB as well as a 1060 6GB. Currently the 1060 3GB is in the PCI slot 03:00.0. I'm on Ubuntu (cinnamon would crash on linuxmint) - results from cat /proc/version:

Code: Select all

Linux version 4.13.0-32-generic (buildd@lgw01-amd64-004) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.5)) #35~16.04.1-Ubuntu SMP Thu Jan 25 10:13:43 UTC 2018

First, to prevent the NVIDIA from being defaulted to render the host, I have been using this xorg.conf file in /etc/X11/xorg.conf:

Code: Select all

Section "Device"
        Identifier  "AMD GPU"
        Driver      "radeon"
        BusID       "PCI:5:0:0"
EndSection

Section "Monitor"
   Identifier  "ASUS MONITOR"
   HorizSync   30-94
   VertRefresh 48-85
EndSection

Section "Screen"
   Identifier  "SCREEN 1"
   Device      "AMD GPU"
   Monitor     "ASUS MONITOR"
   DefaultDepth    24
   SubSection "Display"
       Depth       24
       Modes       "1920x1200"
   EndSubSection
EndSection

Section "ServerLayout"
    Identifier  "Default Layout"
    Screen  0   "SCREEN 1"
EndSection
Second, I blacklisted nouveau so that the NVIDIA card can use pci-stub by adding this to /etc/modprobe.d/blacklist-nouveau.conf:

Code: Select all

blacklist nouveau
options nouveau modeset=0
I then followed your guide and noticed that my output for dmesg | grep vfio is different(the entries above [ 16.071934 ] were added when I tried running the script later):

Code: Select all

[   12.792120] vfio-pci 0000:03:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[   12.836044] vfio_pci: add [10de:1c02[ffff:ffff]] class 0x000000/00000000
[   12.864128] vfio_pci: add [10de:10f1[ffff:ffff]] class 0x000000/00000000
[   16.071934] vfio-pci 0000:03:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[   60.844184] vfio_ecap_init: 0000:03:00.0 hiding ecap 0x19@0x900
[   60.845603] vfio-pci 0000:03:00.0: BAR 3: can't reserve [mem 0xb0000000-0xb1ffffff 64bit pref]
[   63.553983] vfio-pci 0000:03:00.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0xffff
[   63.553994] vfio-pci 0000:03:00.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0xffff
[  881.640205] vfio_ecap_init: 0000:03:00.0 hiding ecap 0x19@0x900
[  881.641764] vfio-pci 0000:03:00.0: BAR 3: can't reserve [mem 0xb0000000-0xb1ffffff 64bit pref]
Here's the bash script I use:

Code: Select all

#!/bin/bash

vmname="windows10vm"

if ps -A | grep -q $vmname; then
   echo "$vmname is already running." &
   exit 1

else

# use pulseaudio
#export QEMU_AUDIO_DRV=pa
#export QEMU_PA_SAMPLES=8192
#export QEMU_AUDIO_TIMER_PERIOD=99
#export QEMU_PA_SERVER=/run/user/1000/pulse/native

#alsa/ac97 test 1/8/26 9:55am
export QEMU_AUDIO_DRV=alsa
export QEMU_ALSA_ADC_BUFFER_SIZE=1024 QEMU_ALSA_ADC_PERIOD_SIZE=256
export QEMU_ALSA_DAC_BUFFER_SIZE=1024 QEMU_ALSA_DAC_PERIOD_SIZE=256
export QEMU_AUDIO_DAC_FIXED_SETTINGS=1
export QEMU_AUDIO_DAC_FIXED_FREQ=44100 QEMU_AUDIO_DAC_FIXED_FMT=S16 QEMU_AUDIO_ADC_FIXED_FREQ=44100 QEMU_AUDIO_ADC_FIXED_FMT=S16
export QEMU_AUDIO_DAC_TRY_POLL=1 QEMU_AUDIO_ADC_TRY_POLL=1
export QEMU_AUDIO_TIMER_PERIOD=50

cp /usr/share/OVMF/OVMF_VARS.fd /tmp/my_vars.fd

qemu-system-x86_64 \
  -name $vmname,process=$vmname \
  -machine type=q35,accel=kvm \
  -cpu host,kvm=off \
  -smp 4,sockets=1,cores=2,threads=2 \
  -m 8G \
  -mem-prealloc \
  -balloon none \
  -rtc clock=host,base=localtime \
  -vga none \
  -serial none \
  -parallel none \
  -soundhw ac97 \
  -usb -usbdevice host:04f2:0939 -usbdevice host:04d9:a0cd -usbdevice host:28de:1142 \
  -device vfio-pci,host=03:00.0,multifunction=on \
  -device vfio-pci,host=03:00.1, \
  -drive if=pflash,format=raw,readonly,file=/usr/share/OVMF/OVMF_CODE.fd \
  -drive if=pflash,format=raw,file=/tmp/my_vars.fd \
  -boot order=dc \
  -drive id=disk0,if=virtio,cache=none,format=raw,file=/home/collin/vm/win.img \
  -drive file=/home/collin/vm/windows.iso,index=3,media=cdrom \
  -drive file=/home/collin/vm/virtio.iso,index=4,media=cdrom \
  -net user,net=10.0.0.0/8,host=10.0.0.1,hostfwd=tcp:127.0.0.1:2222-10.0.0.2:22 \
  -net nic 
   exit 0
fi
And for good measure, here's the Xorg.0.log:

Code: Select all

[    16.027] 
X.Org X Server 1.19.5
Release Date: 2017-10-12
[    16.028] X Protocol Version 11, Revision 0
[    16.028] Build Operating System: Linux 4.4.0-101-generic x86_64 Ubuntu
[    16.028] Current Operating System: Linux surveillance 4.13.0-32-generic #35~16.04.1-Ubuntu SMP Thu Jan 25 10:13:43 UTC 2018 x86_64
[    16.028] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.13.0-32-generic root=UUID=578c959d-72df-4509-b468-c5501bc7ca99 ro intel_iommu=on
[    16.028] Build Date: 24 November 2017  09:44:25AM
[    16.028] xorg-server 2:1.19.5-0ubuntu2~16.04.1 (For technical support please see http://www.ubuntu.com/support) 
[    16.028] Current version of pixman: 0.33.6
[    16.028] 	Before reporting problems, check http://wiki.x.org
	to make sure that you have the latest version.
[    16.028] Markers: (--) probed, (**) from config file, (==) default setting,
	(++) from command line, (!!) notice, (II) informational,
	(WW) warning, (EE) error, (NI) not implemented, (??) unknown.
[    16.028] (==) Log file: "/var/log/Xorg.0.log", Time: Thu Feb  8 19:18:56 2018
[    16.028] (==) Using config file: "/etc/X11/xorg.conf"
[    16.028] (==) Using system config directory "/usr/share/X11/xorg.conf.d"
[    16.028] (==) ServerLayout "Default Layout"
[    16.028] (**) |-->Screen "SCREEN 1" (0)
[    16.028] (**) |   |-->Monitor "ASUS MONITOR"
[    16.028] (**) |   |-->Device "AMD GPU"
[    16.028] (==) Automatically adding devices
[    16.028] (==) Automatically enabling devices
[    16.028] (==) Automatically adding GPU devices
[    16.028] (==) Automatically binding GPU devices
[    16.028] (==) Max clients allowed: 256, resource mask: 0x1fffff
[    16.029] (WW) The directory "/usr/share/fonts/X11/cyrillic" does not exist.
[    16.029] 	Entry deleted from font path.
[    16.029] (WW) The directory "/usr/share/fonts/X11/100dpi/" does not exist.
[    16.029] 	Entry deleted from font path.
[    16.029] (WW) The directory "/usr/share/fonts/X11/75dpi/" does not exist.
[    16.029] 	Entry deleted from font path.
[    16.029] (WW) The directory "/usr/share/fonts/X11/100dpi" does not exist.
[    16.029] 	Entry deleted from font path.
[    16.029] (WW) The directory "/usr/share/fonts/X11/75dpi" does not exist.
[    16.029] 	Entry deleted from font path.
[    16.029] (==) FontPath set to:
	/usr/share/fonts/X11/misc,
	/usr/share/fonts/X11/Type1,
	built-ins
[    16.029] (==) ModulePath set to "/usr/lib/x86_64-linux-gnu/xorg/extra-modules,/usr/lib/xorg/extra-modules,/usr/lib/xorg/modules"
[    16.029] (II) The server relies on udev to provide the list of input devices.
	If no devices become available, reconfigure udev or disable AutoAddDevices.
[    16.029] (II) Loader magic: 0x55a100290e00
[    16.029] (II) Module ABI versions:
[    16.029] 	X.Org ANSI C Emulation: 0.4
[    16.029] 	X.Org Video Driver: 23.0
[    16.029] 	X.Org XInput driver : 24.1
[    16.029] 	X.Org Server Extension : 10.0
[    16.030] (++) using VT number 7

[    16.030] (II) systemd-logind: logind integration requires -keeptty and -keeptty was not provided, disabling logind integration
[    16.030] (II) xfree86: Adding drm device (/dev/dri/card0)
[    16.032] (--) PCI:*(0:3:0:0) 10de:1c02:1462:8c95 rev 161, Mem @ 0xfa000000/16777216, 0xa0000000/268435456, 0xb0000000/33554432, I/O @ 0x0000e000/128, BIOS @ 0x????????/131072
[    16.032] (--) PCI: (0:5:0:0) 1002:6771:1028:2120 rev 0, Mem @ 0xc0000000/268435456, 0xfb320000/131072, I/O @ 0x0000d000/256, BIOS @ 0x????????/131072
[    16.032] (II) LoadModule: "glx"
[    16.033] (II) Loading /usr/lib/xorg/modules/extensions/libglx.so
[    16.039] (II) Module glx: vendor="X.Org Foundation"
[    16.039] 	compiled for 1.19.5, module version = 1.0.0
[    16.039] 	ABI class: X.Org Server Extension, version 10.0
[    16.039] (II) LoadModule: "radeon"
[    16.039] (II) Loading /usr/lib/xorg/modules/drivers/radeon_drv.so
[    16.041] (II) Module radeon: vendor="X.Org Foundation"
[    16.041] 	compiled for 1.19.3, module version = 7.9.0
[    16.041] 	Module class: X.Org Video Driver
[    16.041] 	ABI class: X.Org Video Driver, version 23.0
[    16.041] (II) RADEON: Driver for ATI/AMD Radeon chipsets: [ cut down for character limit ]
[    16.071] (II) [KMS] Kernel modesetting enabled.
[    16.071] (**) RADEON(0): Depth 24, (--) framebuffer bpp 32
[    16.071] (II) RADEON(0): Pixel depth = 24 bits stored in 4 bytes (32 bpp pixmaps)
That's not the whole log, but I couldn't put much more because of the character limit.

Here's lspci -vnn:

Code: Select all

03:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:1c02] (rev a1) (prog-if 00 [VGA controller])
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:8c95]
	Physical Slot: 6
	Flags: fast devsel, IRQ 27
	Memory at fa000000 (32-bit, non-prefetchable) [size=16M]
	Memory at a0000000 (64-bit, prefetchable) [size=256M]
	Memory at b0000000 (64-bit, prefetchable) [size=32M]
	I/O ports at e000 [size=128]
	Expansion ROM at 000c0000 [disabled] [size=128K]
	Capabilities: <access denied>
	Kernel driver in use: vfio-pci
	Kernel modules: nvidiafb, nouveau

03:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10f1] (rev a1)
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:8c95]
	Physical Slot: 6
	Flags: fast devsel, IRQ 55
	Memory at fb080000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel

05:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Caicos XTX [Radeon HD 8490 / R5 235X OEM] [1002:6771] (prog-if 00 [VGA controller])
	Subsystem: Dell Caicos XTX [Radeon HD 8490 / R5 235X OEM] [1028:2120]
	Physical Slot: 2-1
	Flags: bus master, fast devsel, latency 0, IRQ 50
	Memory at c0000000 (64-bit, prefetchable) [size=256M]
	Memory at fb320000 (64-bit, non-prefetchable) [size=128K]
	I/O ports at d000 [size=256]
	Expansion ROM at fb300000 [disabled] [size=128K]
	Capabilities: <access denied>
	Kernel driver in use: radeon
	Kernel modules: radeon

05:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Caicos HDMI Audio [Radeon HD 6400 Series] [1002:aa98]
	Subsystem: Dell Caicos HDMI Audio [Radeon HD 6400 Series] [1028:aa98]
	Physical Slot: 2-1
	Flags: bus master, fast devsel, latency 0, IRQ 54
	Memory at fb340000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel
I have an Intel CPU and I know that this can work on the NVIDIA cards because I've used this guide before, having one NVIDIA as the host and the other as the guest GPU. Really, I just bought this AMD card to run linux while I use the windows VM with both NVIDIA card passed through.

If anyone needs more information, I can give it to them.
Thanks!

powerhouse
Level 6
Level 6
Posts: 1010
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse » Sat Feb 17, 2018 6:03 am

@vitriolicex

First, fix the bash script:

Change

Code: Select all

 -device vfio-pci,host=03:00.1, \
to

Code: Select all

 -device vfio-pci,host=03:00.1 \
Second, you may need to add this to your grub file /etc/default/grub:

Code: Select all

GRUB_CMDLINE_LINUX="nomodeset vga=normal video=vesafb:off"
If the above doesn't help: You have two nearly identical GPUs, which may cause a problem. Try to remove one of them, and see if it works.

You mentioned that you use pci-stub? My tutorial uses vfio, so you need to check that you follow either this tutorial or another, but not both!

Your script is missing the part where I bind the graphics card:

Code: Select all

configfile=/etc/vfio-pci.cfg
vmname="win10vm"

vfiobind() {
	dev="$1"
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
   
}


if ps -A | grep -q $vmname; then
	zenity --info --window-icon=info --timeout=15 --text="$vmname is already running." &
	exit 1

else

#modprobe vfio-pci

	cat $configfile | while read line;do
	echo $line | grep ^# >/dev/null 2>&1 && continue
		vfiobind $line
	done
Of course, if you use pci-stub, you will have a different way of configuration. Make sure the vfio-pci configuration lines in the script are compatible.

See also https://wiki.archlinux.org/index.php/PC ... ng_the_GPU
Asus Sabertooth X79, i7 3930K CPU, 8x4GB Kingston DDR3 RAM, Noctua NH-D14 CPU cooler, Gigabyte GTX 970 + PNY Quadro 2000 GPU, Asus Xonar Essence STX, Sandisk Extreme 120GB + Samsung EVO 850 250GB SSD + 4 HDD, Corsair 500R case, SeaSonic 660W Gold X PS

jockos
Level 1
Level 1
Posts: 1
Joined: Sun Feb 18, 2018 6:32 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by jockos » Sun Feb 18, 2018 6:49 am

Hi all!
I had a working configuration using this guide with one Nvidia card (1060) and the builtin Intel GPU (skylake). However, I recently got a new monitor which I wanted to use with Linux as well, so I added a second Nvidia GPU (1030), as the Intel one didn't output 4K.

Now I've done a fresh Mint (xfce) install (actually four already in trying to make it work), but I'm stuck..

I'm trying to go with this guide in order to not have to blacklist the Nvidia driver:
viewtopic.php?f=231&t=212692&start=40#p1174032

At boot I'm getting:
bash: /sbin/vfio-pci-override-vga.sh not found or something similar (it flashes by in a second).
And the /sys/bus/pci/devices/GPU AND GPUAUDIO/driver_override files contain null instead of vfio-pci as expected.

I'm quite new to Linux, so I really have no clue on how to fix this. I've tried running the /sbin/vfio-pci-override-vga.sh, which makes the driver_override files fill up correctly, but since it doesn't run at boot it's useless..



I've wasted about one day already trying to get the 1030 display 4k on HDMI, turns out the "High Speed" HDMI cable supplied with the monitor (AOC U3277PWQU) was not up to spec..
Noveau cannot detect this monitor on HDMI, so I'm using the Nvidia driver now. Will this be a problem as well?

ur_gf_irl
Level 1
Level 1
Posts: 1
Joined: Fri Feb 23, 2018 12:54 pm

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by ur_gf_irl » Fri Feb 23, 2018 1:16 pm

EDIT: Turns out all I had to do was to run update-initramfs -u again after editing /etc/modprobe.d/local.conf . I'll leave this up in case it helps someone else.

Hello everyone!

I followed this guide successfully up through step 4. However, once I began running Linux off of my integrated Intel graphics rather than my Nvidia GPU, I immediately noticed intermittent, extreme screen tearing. Over the past few days I've been unable to eliminate this tearing and want to "revert" my system so I can use my Nvidia GPU for Linux again. I made a naive attempt to reverse what I have done by following this guide (removing the "modprobe.blacklist=nvidia" command from my grub config, removing the "options vfio-pci ids=10de:13c2,10de:0fbb" line from /etc/modprobe.d/local.conf). However, when I try to boot into Linux using my Nvidia card I just see a black screen, and upon booting using my Intel chip lspci reveals that my Nvidia card is still using the vfio-pci driver, so clearly these changes aren't sufficient.

How can I undo step 4, so that my Nvidia card is usable by my Linux system?

Details:
I'm running Linux Mint 18.3 Cinnamon.
Hardware details via inxi -F:

Code: Select all

System:    Host: bucephalo Kernel: 4.13.0-32-generic x86_64 (64 bit) Desktop: Cinnamon 3.6.7
           Distro: Linux Mint 18.3 Sylvia
Machine:   Mobo: ASRock model: Z170 Pro4S Bios: American Megatrends v: P2.70 date: 11/04/2015
CPU:       Quad core Intel Core i7-6700 (-HT-MCP-) cache: 8192 KB 
           clock speeds: max: 4000 MHz 1: 3400 MHz 2: 3400 MHz 3: 3400 MHz 4: 3400 MHz 5: 3400 MHz 6: 3400 MHz
           7: 3400 MHz 8: 3400 MHz
Graphics:  Card-1: Intel Sky Lake Integrated Graphics
           Card-2: NVIDIA GM204 [GeForce GTX 970]
           Display Server: X.Org 1.18.4 drivers: (unloaded: fbdev,vesa) Resolution: 1920x1080@60.00hz
           GLX Renderer: Mesa DRI Intel HD Graphics 530 (Skylake GT2) GLX Version: 3.0 Mesa 17.3.2 - padoka PPA
Audio:     Card-1 NVIDIA GM204 High Definition Audio Controller driver: vfio-pci
           Card-2 Intel Sunrise Point-H HD Audio driver: snd_hda_intel
           Sound: Advanced Linux Sound Architecture v: k4.13.0-32-generic
Network:   Card: Intel Ethernet Connection (2) I219-V driver: e1000e
           IF: enp0s31f6 state: up speed: 1000 Mbps duplex: full mac: d0:50:99:9c:b1:49
Drives:    HDD Total Size: 2256.5GB (2.6% used) ID-1: /dev/sda model: HGST_HTS721010A9 size: 1000.2GB
           ID-2: /dev/sdb model: WDC_WD10EZEX size: 1000.2GB ID-3: /dev/sdc model: Samsung_SSD_850 size: 256.1GB
Partition: ID-1: / size: 61G used: 25G (44%) fs: ext4 dev: /dev/sdc5
           ID-2: swap-1 size: 34.31GB used: 0.00GB (0%) fs: swap dev: /dev/sdc6
RAID:      No RAID devices: /proc/mdstat, md_mod kernel module present
Sensors:   System Temperatures: cpu: 36.0C mobo: N/A
           Fan Speeds (in rpm): cpu: N/A
Info:      Processes: 249 Uptime: 21 min Memory: 2148.9/31924.1MB Client: Shell (bash) inxi: 2.2.35 
Output of lspci -knn is

Code: Select all

00:00.0 Host bridge [0600]: Intel Corporation Sky Lake Host Bridge/DRAM Registers [8086:191f] (rev 07)
	Subsystem: ASRock Incorporation Skylake Host Bridge/DRAM Registers [1849:191f]
	Kernel driver in use: skl_uncore
00:01.0 PCI bridge [0604]: Intel Corporation Sky Lake PCIe Controller (x16) [8086:1901] (rev 07)
	Kernel driver in use: pcieport
	Kernel modules: shpchp
00:02.0 VGA compatible controller [0300]: Intel Corporation Sky Lake Integrated Graphics [8086:1912] (rev 06)
	Subsystem: ASRock Incorporation Skylake Integrated Graphics [1849:1912]
	Kernel driver in use: i915
	Kernel modules: i915
00:14.0 USB controller [0c03]: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller [8086:a12f] (rev 31)
	Subsystem: ASRock Incorporation Sunrise Point-H USB 3.0 xHCI Controller [1849:a12f]
	Kernel driver in use: xhci_hcd
00:14.2 Signal processing controller [1180]: Intel Corporation Sunrise Point-H Thermal subsystem [8086:a131] (rev 31)
	Subsystem: ASRock Incorporation Sunrise Point-H Thermal subsystem [1849:a131]
	Kernel driver in use: intel_pch_thermal
	Kernel modules: intel_pch_thermal
00:16.0 Communication controller [0780]: Intel Corporation Sunrise Point-H CSME HECI #1 [8086:a13a] (rev 31)
	Subsystem: ASRock Incorporation Sunrise Point-H CSME HECI [1849:a13a]
	Kernel driver in use: mei_me
	Kernel modules: mei_me
00:17.0 SATA controller [0106]: Intel Corporation Sunrise Point-H SATA controller [AHCI mode] [8086:a102] (rev 31)
	Subsystem: ASRock Incorporation Sunrise Point-H SATA controller [AHCI mode] [1849:a102]
	Kernel driver in use: ahci
	Kernel modules: ahci
00:1b.0 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Root Port #17 [8086:a167] (rev f1)
	Kernel driver in use: pcieport
	Kernel modules: shpchp
00:1c.0 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #1 [8086:a110] (rev f1)
	Kernel driver in use: pcieport
	Kernel modules: shpchp
00:1d.0 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #9 [8086:a118] (rev f1)
	Kernel driver in use: pcieport
	Kernel modules: shpchp
00:1f.0 ISA bridge [0601]: Intel Corporation Sunrise Point-H LPC Controller [8086:a145] (rev 31)
	Subsystem: ASRock Incorporation Sunrise Point-H LPC Controller [1849:a145]
00:1f.2 Memory controller [0580]: Intel Corporation Sunrise Point-H PMC [8086:a121] (rev 31)
	Subsystem: ASRock Incorporation Sunrise Point-H PMC [1849:a121]
00:1f.3 Audio device [0403]: Intel Corporation Sunrise Point-H HD Audio [8086:a170] (rev 31)
	Subsystem: ASRock Incorporation Sunrise Point-H HD Audio [1849:5892]
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel
00:1f.4 SMBus [0c05]: Intel Corporation Sunrise Point-H SMBus [8086:a123] (rev 31)
	Subsystem: ASRock Incorporation Sunrise Point-H SMBus [1849:a123]
	Kernel modules: i2c_i801
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (2) I219-V [8086:15b8] (rev 31)
	Subsystem: ASRock Incorporation Ethernet Connection (2) I219-V [1849:15b8]
	Kernel driver in use: e1000e
	Kernel modules: e1000e
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
	Subsystem: eVga.com. Corp. GM204 [GeForce GTX 970] [3842:2974]
	Kernel driver in use: vfio-pci
	Kernel modules: nvidiafb, nouveau, nvidia_390, nvidia_390_drm
01:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)
	Subsystem: eVga.com. Corp. GM204 High Definition Audio Controller [3842:2974]
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel

bash64
Level 2
Level 2
Posts: 73
Joined: Mon Mar 23, 2015 12:19 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by bash64 » Sun Feb 25, 2018 2:45 pm

I am having issues accessing an audio cd with qemu.
I added "-cdrom /dev/sr0" to my script.
The cdrom drive is visible in windows.
I insert the music cd and then boot but get an error about unreadable format.
If i eject it and boot and then insert windows complains of an unreadable format.
I have an iPod and am running iTunes.
I wish to import the music cd into my iPod.

All help appreciated.

User avatar
davidem
Level 1
Level 1
Posts: 13
Joined: Sun Jan 22, 2017 4:40 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by davidem » Fri Mar 16, 2018 8:45 am

Hi guys, it's been a while since I contributed to this thread (last comment was on page 13 8) ). Ever since I got my Windows 10 VM working, I've been a happy camper. And recently I've figured out how to create a Linux VM with GPU passthrough. Linux is a bit more picky when it comes to emulated hardware than Windows, so it took some time to figure out the config. But in the end it appeared to be easy and I wondered why it took me so long to figure it out.

Code: Select all

/usr/local/qemu/bin/qemu-system-x86_64 \
  -name $VMname,process=$VMname \
  -machine q35,accel=kvm \
  -enable-kvm \
  -cpu host \
  -smp 4,sockets=1,cores=2,threads=2 \
  -m 8G \
  -mem-path /dev/hugepages \
  -mem-prealloc \
  -balloon none \
  -monitor unix:/usr/local/VMs/monitors/${VMname}.monitor,server,nowait \
  -monitor telnet:127.0.0.1:${Inst_port},server,nowait \
  -rtc clock=host,base=localtime \
  -vnc none \
  -vga qxl \
  -soundhw pcspk \
  -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/OVMF.fd \
  -drive if=pflash,format=raw,file=/tmp/OVMF_VARS_${VMname}.fd \
  -device vfio-pci,host=01:00.0,multifunction=on \
  -boot order=dc \
  -device virtio-scsi-pci,id=scsi \
  -drive id=disk0,if=virtio,cache=none,format=qcow2,file=/T1/images/Linux-00_boot-02_snap-01.qcow2 \
  -netdev type=tap,id=net0,ifname=tap${Instance},vhost=on \
  -device virtio-net-pci,netdev=net0,mac=00:16:3e:00:01:${Inst_MAC}
My windows setup uses a 'pc' machine type, whereas I was now forced to use a 'q35' machine. Another big difference is that I don't forward the audio-component of my GPU. That, and not using an ioh3420 bus, gave me a stable Linux system with dedicated GPU :D

Another peculiar thing about Linux and GPU-passthrough is that after shutdown of the VM, the GPU is more than often left in a failed state. With Windows this is a rare event, with Linux it seems common. 'Lspci -vnn' gives a bit more insight:

Expected output:

Code: Select all

01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:67df] (rev cf) (prog-if 00 [VGA controller])
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:3411]
	Flags: bus master, fast devsel, latency 0, IRQ 145
	Memory at c0000000 (64-bit, prefetchable) [size=256M]
	Memory at d0000000 (64-bit, prefetchable) [size=2M]
	I/O ports at e000 [size=256]
	Memory at df200000 (32-bit, non-prefetchable) [size=256K]
	Expansion ROM at df240000 [disabled] [size=128K]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
	Capabilities: [58] Express Legacy Endpoint, MSI 00
	Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150] Advanced Error Reporting
	Capabilities: [200] #15
	Capabilities: [270] #19
	Capabilities: [2b0] Address Translation Service (ATS)
	Capabilities: [2c0] #13
	Capabilities: [2d0] #1b
	Capabilities: [320] Latency Tolerance Reporting
	Capabilities: [328] Alternative Routing-ID Interpretation (ARI)
	Capabilities: [370] L1 PM Substates
	Kernel driver in use: vfio-pci
Output when failed:

Code: Select all

01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:67df] (rev ff) (prog-if ff)
	!!! Unknown header type 7f
	Kernel driver in use: vfio-pci
I haven't found a way to fix this online, the common suggestions found on the web (to remove and rescan) do not work in my case. I need to reboot each time I notice this. What I did do is I've added an extra check in the script, so the VM doesn't boot with a crooked GPU. I've added the check in the vfiobind-loop:

Code: Select all

while read line
do
   echo $line | grep ^# >/dev/null 2>&1 && continue
   # check if device is still sane:
   if [[ `lspci -vnn | awk '/'"${line:5}"'/' RS="" | awk '{if (/!!!/)e=$1};{if (/Subsystem/)e=$1}END{print e}'` = "!!!" ]];then
	echo " Device $line has gone insane. Check 'lspci -vnn' for more info "
	exit 1
   else
 
      vfiobind $line
   fi
done < <(cat $configfile)
Perhaps this might help some of you.

Cheers,

David
Linux Mint 18.3 with Windows Gaming VM:
Intel i7 6700, MSI Z170a SLI Plus, 16 GB DDR4 2133MHz, 256 GB Intel 600P PCIe M.2 SSD, 512 GB Samsung 850 EVO SSD, Radeon RX 470 Gaming x 4G (VM), Asus Xonar DG-X 5.1 (VM)

User avatar
davidem
Level 1
Level 1
Posts: 13
Joined: Sun Jan 22, 2017 4:40 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by davidem » Fri Mar 16, 2018 9:03 am

ersatzhero wrote:
Thu Dec 28, 2017 2:14 pm
Hello,

I would like to give a big thank you to powerhouse and everyone who helped to create this guide.
I love my setup with the Windows VM. It just works :>

But still, I got a networking problem when I have installed Docker following the official guide.
The VM can't reach the internet anymore after Docker has been installed.

I can't figure out what the problem is:
Is it the iptables rules installed by Docker or
is the problem the two added bridge networks or
something completely else?

I don't have any idea how to look for that, so I hope, someone can point me in the right direction.

Regards
Hi ersatzhero, I also have docker running next to the VMs with briding, and also stumbled on this issue a while back. Docker sets up iptables with rules which might block other traffic, in our case our bridged traffic. I added extra rules to allow our bridge and made it so that it loads during boot.

I created a simple script in /etc/init.d:

Code: Select all

# cat /etc/init.d/iptables_loader.sh 
#!/bin/bash
/sbin/iptables-restore /root/kvm_docker.rules
As you can see, it 'restores' a list of rules. In my case its this:

Code: Select all

# Generated by iptables-save v1.6.0 on Mon Aug 21 01:24:28 2017
*nat
:PREROUTING ACCEPT [1235280:483868344]
:INPUT ACCEPT [25633:5257573]
:OUTPUT ACCEPT [4583:346840]
:POSTROUTING ACCEPT [4583:346840]
:DOCKER - [0:0]
:KVM - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 192.168.178.0/24 ! -o kvmbr0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A KVM -i kvmbr0 -j RETURN
COMMIT
# Completed on Mon Aug 21 01:24:28 2017
# Generated by iptables-save v1.6.0 on Mon Aug 21 01:24:28 2017
*filter
:INPUT ACCEPT [4183134:512584179]
:FORWARD DROP [325751:46125460]
:OUTPUT ACCEPT [2291462:22348027355]
:DOCKER - [0:0]
:DOCKER-ISOLATION - [0:0]
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION -j RETURN
:KVM - [0:0]
:KVM-ISOLATION - [0:0]
-A FORWARD -j KVM-ISOLATION
-A FORWARD -o kvmbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o kvmbr0 -j KVM
-A FORWARD -i kvmbr0 ! -o kvmbr0 -j ACCEPT
-A FORWARD -i kvmbr0 -o kvmbr0 -j ACCEPT
-A KVM-ISOLATION -j RETURN
COMMIT

# Completed on Mon Aug 21 01:24:28 2017
One set to make Docker happy, and an identical set to make my other bridge happy. :wink:
Linux Mint 18.3 with Windows Gaming VM:
Intel i7 6700, MSI Z170a SLI Plus, 16 GB DDR4 2133MHz, 256 GB Intel 600P PCIe M.2 SSD, 512 GB Samsung 850 EVO SSD, Radeon RX 470 Gaming x 4G (VM), Asus Xonar DG-X 5.1 (VM)

powerhouse
Level 6
Level 6
Posts: 1010
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse » Sat Mar 24, 2018 8:48 am

bash64 wrote:
Sun Feb 25, 2018 2:45 pm
I am having issues accessing an audio cd with qemu.
I added "-cdrom /dev/sr0" to my script.
The cdrom drive is visible in windows.
I insert the music cd and then boot but get an error about unreadable format.
If i eject it and boot and then insert windows complains of an unreadable format.
I have an iPod and am running iTunes.
I wish to import the music cd into my iPod.

All help appreciated.
I have tried it now too, without any success. The only possible solution I found is this: https://bbs.archlinux.org/viewtopic.php?id=186945

It means sharing the CD Rom via Samba.
Asus Sabertooth X79, i7 3930K CPU, 8x4GB Kingston DDR3 RAM, Noctua NH-D14 CPU cooler, Gigabyte GTX 970 + PNY Quadro 2000 GPU, Asus Xonar Essence STX, Sandisk Extreme 120GB + Samsung EVO 850 250GB SSD + 4 HDD, Corsair 500R case, SeaSonic 660W Gold X PS

powerhouse
Level 6
Level 6
Posts: 1010
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse » Sat Apr 07, 2018 8:37 am

jockos wrote:
Sun Feb 18, 2018 6:49 am
Hi all!
I had a working configuration using this guide with one Nvidia card (1060) and the builtin Intel GPU (skylake). However, I recently got a new monitor which I wanted to use with Linux as well, so I added a second Nvidia GPU (1030), as the Intel one didn't output 4K.

Now I've done a fresh Mint (xfce) install (actually four already in trying to make it work), but I'm stuck..

I'm trying to go with this guide in order to not have to blacklist the Nvidia driver:
viewtopic.php?f=231&t=212692&start=40#p1174032

At boot I'm getting:
bash: /sbin/vfio-pci-override-vga.sh not found or something similar (it flashes by in a second).
And the /sys/bus/pci/devices/GPU AND GPUAUDIO/driver_override files contain null instead of vfio-pci as expected.

I'm quite new to Linux, so I really have no clue on how to fix this. I've tried running the /sbin/vfio-pci-override-vga.sh, which makes the driver_override files fill up correctly, but since it doesn't run at boot it's useless..



I've wasted about one day already trying to get the 1030 display 4k on HDMI, turns out the "High Speed" HDMI cable supplied with the monitor (AOC U3277PWQU) was not up to spec..
Noveau cannot detect this monitor on HDMI, so I'm using the Nvidia driver now. Will this be a problem as well?
Sorry for the late reply - have been very busy.

There is no need to follow the instructions for identical graphics cards - your graphics cards aren't identical.

With the Nvidia 1030 installed, get the proprietary Nvidia driver and install it. On Linux Mint you can do that conveniently via Control Center -> Driver Manager.

Make sure that the Nvidia 1060 has the nouveau driver (open source driver) selected, and select and install the latest or recommended proprietary Nvidia driver for the Nvidia 1030.

Open /etc/modprobe.d/local.conf and replace the content with the following:

Code: Select all

options vfio-pci ids=10de:13c2,10de:0fbb
Important: change the PCI IDs to match the ones you got for your Nvidia 1060 !!!
Asus Sabertooth X79, i7 3930K CPU, 8x4GB Kingston DDR3 RAM, Noctua NH-D14 CPU cooler, Gigabyte GTX 970 + PNY Quadro 2000 GPU, Asus Xonar Essence STX, Sandisk Extreme 120GB + Samsung EVO 850 250GB SSD + 4 HDD, Corsair 500R case, SeaSonic 660W Gold X PS

powerhouse
Level 6
Level 6
Posts: 1010
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse » Sat Apr 07, 2018 8:52 am

Update on my configuration. I'm currently running the following versions:

Linux Mint Mate 18.3 Sylvia
Kernel: 4.15.0-13-generic x86_64
QEMU emulator version 2.6.2
Nvidia Quadro 2000 with nvidia-384 binary Nvidia driver for the Linux host
intel-microcode
OVMF version 0~20160408.ffea0a2c-2

Almost needless to say that everything runs smooth :) .
Asus Sabertooth X79, i7 3930K CPU, 8x4GB Kingston DDR3 RAM, Noctua NH-D14 CPU cooler, Gigabyte GTX 970 + PNY Quadro 2000 GPU, Asus Xonar Essence STX, Sandisk Extreme 120GB + Samsung EVO 850 250GB SSD + 4 HDD, Corsair 500R case, SeaSonic 660W Gold X PS

bash64
Level 2
Level 2
Posts: 73
Joined: Mon Mar 23, 2015 12:19 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by bash64 » Sun Apr 08, 2018 2:27 pm

powerhouse wrote:
Sat Apr 07, 2018 8:52 am
Update on my configuration. I'm currently running the following versions:

Linux Mint Mate 18.3 Sylvia
Kernel: 4.15.0-13-generic x86_64
QEMU emulator version 2.6.2
Nvidia Quadro 2000 with nvidia-384 binary Nvidia driver for the Linux host
intel-microcode
OVMF version 0~20160408.ffea0a2c-2

Almost needless to say that everything runs smooth :) .
Laptop MSI GT70-2PE Intel i7-4940mx @3.1ghz (3.8ghz turbo), 32GB ram
(2) 4TB HD (raid 1 mirror) NOTE: case is modded to allow for the 2nd 12mm drive.
usb3 1TB Intel 540 series SSD (qemu hard files)
Logitech G213 RGB keyboard
Dell S2817Q 28" 4k monitor

Linux Mint Mate 18.3 Sylvia
Kernel: 4.15.13-041513-generic
QEMU emulator version 2.11.50 (custom build)
Nvidia GTX 880M 8GB with nvidia-384
Intel 4th Gen Core Processor Integrated Graphics Controller
SEABIOS for both windows7 and windows 10
20GB of ram to qemu

bash64
Level 2
Level 2
Posts: 73
Joined: Mon Mar 23, 2015 12:19 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by bash64 » Sun Apr 08, 2018 2:30 pm

I shared my latest findings on performance and linux with the facebook "linux mint cinnamon" page I admin so I'd like to just repeat the information here. This is very important and affects everyone:

If you have a high performance and/or gaming PC you really need to set the default temperature used by Thermal Daemon to something higher than 55C. The example temperature in the config file is 75C, but some brilliant soul decided that the default for all PCs should be one temp of 55C. When processing videos with Handbrake I can hit 89C with all 8 threads at 100%. When you reach 55C thermald starts throttling your PC which means it is going to take that much longer to get anything done. The hotter you get the more aggressive the method it uses to slow you down. I have had it literally do a hard lock up of my desktop it became so aggressive. The config file is in /etc/thermald/thermal-conf.xml. Look for the first <Temperature> line. Change 55000 (55C) to something higher like 75000 (75C). I personally set it to 90000 (90C). In all of my days as a computer user and repairman I have never seen a cpu be damaged by heat. Basically at 100C your cpu locks up and or in Windows you get a blue screen. Unless your overclocking I just don't see the need for the thermald at all. However, uninstalling it doesn't fix the issue as your kernel has a built in module to do the same thing. Raising the default temp is the best course if you want your PC to operate at its best performance.

odtech
Level 2
Level 2
Posts: 97
Joined: Sat Jan 21, 2017 9:25 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by odtech » Wed Apr 11, 2018 3:06 pm

Hi all.

I've not been using qemu kvm for a while for reason. I'm getting back into it now and unfortunately for me with a change in hardware for the worse.
I'm using Intel graphics for the host and a Radeon RX 470 for the windows 10 guest.

And naturaly the way my luck goes i have the issue where Windows will go into a crash loop mid way through installing the radeon drivers.
Things i tried:
Add a pci bus then attach the gpu to that in the qemu script.
Chang the machine type to pc
Upgraded qemu to 2.6
Tried Linux Mint Mate and Cinnamon
Upgrade to the last v13.xx kernel.

I am solidly stuck. Can someone suggest a solution or maybe a known working radeon driver.

odtech
Level 2
Level 2
Posts: 97
Joined: Sat Jan 21, 2017 9:25 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by odtech » Wed Apr 11, 2018 5:02 pm

Nevermind.

I figured it out.
Haswell has a issue where hdmi sound is broken when iommu is enabled in grub. To fix it i put "intel_iommu=on,igfx_off". I did not realise at the time that this will in turn break pci passthrough.
I got passthrough to work up to the point where the driver is installing and no further until i removed the "igfx_off" setting.

I'm using machine type pc with the pci bus added to the startup scipt. On the windows side i have the latest AMD driver installed.

powerhouse
Level 6
Level 6
Posts: 1010
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse » Mon Apr 16, 2018 3:36 am

bash64 wrote:
Sun Apr 08, 2018 2:30 pm
I shared my latest findings on performance and linux with the facebook "linux mint cinnamon" page I admin so I'd like to just repeat the information here. This is very important and affects everyone:

If you have a high performance and/or gaming PC you really need to set the default temperature used by Thermal Daemon to something higher than 55C. The example temperature in the config file is 75C, but some brilliant soul decided that the default for all PCs should be one temp of 55C. When processing videos with Handbrake I can hit 89C with all 8 threads at 100%. When you reach 55C thermald starts throttling your PC which means it is going to take that much longer to get anything done. The hotter you get the more aggressive the method it uses to slow you down. I have had it literally do a hard lock up of my desktop it became so aggressive. The config file is in /etc/thermald/thermal-conf.xml. Look for the first <Temperature> line. Change 55000 (55C) to something higher like 75000 (75C). I personally set it to 90000 (90C). In all of my days as a computer user and repairman I have never seen a cpu be damaged by heat. Basically at 100C your cpu locks up and or in Windows you get a blue screen. Unless your overclocking I just don't see the need for the thermald at all. However, uninstalling it doesn't fix the issue as your kernel has a built in module to do the same thing. Raising the default temp is the best course if you want your PC to operate at its best performance.
Thanks bash64 - I never would have looked there.
Asus Sabertooth X79, i7 3930K CPU, 8x4GB Kingston DDR3 RAM, Noctua NH-D14 CPU cooler, Gigabyte GTX 970 + PNY Quadro 2000 GPU, Asus Xonar Essence STX, Sandisk Extreme 120GB + Samsung EVO 850 250GB SSD + 4 HDD, Corsair 500R case, SeaSonic 660W Gold X PS

powerhouse
Level 6
Level 6
Posts: 1010
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse » Wed Apr 18, 2018 3:34 am

odtech wrote:
Wed Apr 11, 2018 5:02 pm
Nevermind.

I figured it out.
Haswell has a issue where hdmi sound is broken when iommu is enabled in grub. To fix it i put "intel_iommu=on,igfx_off". I did not realise at the time that this will in turn break pci passthrough.
I got passthrough to work up to the point where the driver is installing and no further until i removed the "igfx_off" setting.

I'm using machine type pc with the pci bus added to the startup scipt. On the windows side i have the latest AMD driver installed.
Good that you solved it. I have heard of issues with the Catalyst / Radeon driver software and that it's advisable to first install only the driver, not the entire Catalyst package. Other issues that were reported are that some Radeon cards don't seem to reset properly after exiting the Windows VM - Function Level Reset or FLR. As a result, when restarting the VM the PC might freeze, or other issues may occur. BUT, all these issues are hopefully a thing of the past.
Asus Sabertooth X79, i7 3930K CPU, 8x4GB Kingston DDR3 RAM, Noctua NH-D14 CPU cooler, Gigabyte GTX 970 + PNY Quadro 2000 GPU, Asus Xonar Essence STX, Sandisk Extreme 120GB + Samsung EVO 850 250GB SSD + 4 HDD, Corsair 500R case, SeaSonic 660W Gold X PS

Post Reply

Return to “Virtualization”