HOW-TO make dual-boot obsolete using kvm VGA passthrough

Questions about virtualization software
Forum rules
Before you post please read how to get help
driz
Level 1
Level 1
Posts: 24
Joined: Mon May 12, 2014 3:38 pm

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by driz » Tue Jan 17, 2017 9:51 pm

powerhouse wrote:@driz: thin provisioned disk

Did you format the disk BEFORE you installed the WIndows VM?

As a rule, do NOT format any drive or media that is to hold the VM. If you plan to use /dev/sdd or the partition /dev/sdd1 to hold your Windows VM, make sure it is not formated. If necessary, delete the partition and recreate it without specifying a format.

When you install the Windows VM, Windows should take care of the partitioning and formating.
i did as you posted, same for any virtualization afaik (i'm relatively new to kvm, but have worked in esxi for a long time)

powerhouse
Level 6
Level 6
Posts: 1085
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse » Thu Jan 19, 2017 3:43 am

@driz: I guess I'm out of answers. The only thing that comes to mind is some option during Windows installation. Sorry, can't help.

But I gathered that Windows recognized the disk as a SSD?
Asus Sabertooth X79, i7 3930K CPU, 8x4GB Kingston DDR3, Noctua NH-D14 CPU cooler, GTX 970 + Quadro 2000 GPU, Asus Xonar Essence STX, Sandisk 120GB + Samsung EVO 860 1TB SSD + 4 HDD, Corsair 500R, SeaSonic 660W Gold X PS https://heiko-sieger.info

driz
Level 1
Level 1
Posts: 24
Joined: Mon May 12, 2014 3:38 pm

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by driz » Thu Jan 19, 2017 10:22 pm

powerhouse wrote:@driz: I guess I'm out of answers. The only thing that comes to mind is some option during Windows installation. Sorry, can't help.

But I gathered that Windows recognized the disk as a SSD?

it doesn't, but i did notice that windows won't perform optimizations on a thin provisioned disk either. In terms of disk life, I feel much more comfortable.

powerhouse
Level 6
Level 6
Posts: 1085
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse » Fri Jan 20, 2017 1:00 am

driz wrote:
powerhouse wrote:@driz: I guess I'm out of answers. The only thing that comes to mind is some option during Windows installation. Sorry, can't help.

But I gathered that Windows recognized the disk as a SSD?

it doesn't, but i did notice that windows won't perform optimizations on a thin provisioned disk either. In terms of disk life, I feel much more comfortable.
Good to know. I hope one day someone will figure out why.

As to the setup you described in a previous post, I noticed a small mistake:
I include the radeon driver in /etc/initramfs-tools/modules
I blacklist the radeon driver in /etc/modprobe.d/blacklist.conf
Did you mean the vfio driver in the first line above?

Thanks again for your valuable input.
Asus Sabertooth X79, i7 3930K CPU, 8x4GB Kingston DDR3, Noctua NH-D14 CPU cooler, GTX 970 + Quadro 2000 GPU, Asus Xonar Essence STX, Sandisk 120GB + Samsung EVO 860 1TB SSD + 4 HDD, Corsair 500R, SeaSonic 660W Gold X PS https://heiko-sieger.info

driz
Level 1
Level 1
Posts: 24
Joined: Mon May 12, 2014 3:38 pm

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by driz » Fri Jan 20, 2017 10:35 pm

powerhouse wrote:
driz wrote:
powerhouse wrote:@driz: I guess I'm out of answers. The only thing that comes to mind is some option during Windows installation. Sorry, can't help.

But I gathered that Windows recognized the disk as a SSD?

it doesn't, but i did notice that windows won't perform optimizations on a thin provisioned disk either. In terms of disk life, I feel much more comfortable.
Good to know. I hope one day someone will figure out why.

As to the setup you described in a previous post, I noticed a small mistake:
I include the radeon driver in /etc/initramfs-tools/modules
I blacklist the radeon driver in /etc/modprobe.d/blacklist.conf
Did you mean the vfio driver in the first line above?

Thanks again for your valuable input.
Well, i didn't typo, but obviously that makes no sense! lol luckily it had no ill effect.

I'm still working on some minor issues such as the audio portion of the video card not functioning, windows just reports an error.. with that being the only issue, honestly, i don't care enough to mess with it. kids can deal with crackly usb audio :D

odtech
Level 2
Level 2
Posts: 95
Joined: Sat Jan 21, 2017 9:25 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by odtech » Sat Jan 21, 2017 8:15 pm

This is just a victory post all thanks to your guide.

I had this working with seabios using kvm previously on ubuntu 14.04 as that is what the first guide i followed used but after being comfortable with the setup i installed Mint 17 which is my OS of choice these days. I would have gone with Mint 18 but i need the ACS override patch which it seems is only out for kernel 3.18.

My main issue was to get the VM to run with a uefi bios, because the pci-e card i am passing to the vm can only boot with uefi and not bios. Starting the vm with a uefi bios would just result in a black screen but if i turned the qemu display on i could see that the vm is running. I initialy gave up trying to flash my card with the available uefi supported update because none of the Gigabyte tools worked. I briefly tried with the romfile option and a bios from techpowerup but that didn't work at all. I visited techpowerup again earlier to check if i missed something and sure enough i saw they have a NVFlash tool which successfully flashed my GTX670 bios to a uefi enabled one. So of i went to edit the script...again...heart sank a bit when it wouldn't boot with the pflash lines and the redhat git version of ovmf. I deleted the uefi bios lines then just added the normal bios option and renamed the mint repo's ovmf.fd file to bios.bin and voila the vm booted and i got display output on the GTX card.

The only thing left to do is recompile Kernel 3.18 with only the ACS override patch. The 915 arbitration patch is pretty crippling to the driver for the intel graphics so it would be nice to have that working properly again.

Edit: The pci-e card i'm referring to is my sata controller that i have my Windows 10 and storage disks attached to. The vm does not boot from the windows disk attached to the pci-e sata controller if i used seabios and the gpu would not display if the vm was set to uefi so i was forced to use seabios and a virtio disk for windows with the physical windows drive attached to the onboard sata controller.

Second edit fixed my first edit. I was half alseep when i wrote the post. :lol:

powerhouse
Level 6
Level 6
Posts: 1085
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse » Wed Jan 25, 2017 6:28 am

@odtech: Thanks for sharing your solution. However, some important pieces of information are missing. Could you share your hardware (motherboard, CPU, graphics card, SATA controller, etc.)? This will help others to see if your solution is applicable to them. Thanks!
Asus Sabertooth X79, i7 3930K CPU, 8x4GB Kingston DDR3, Noctua NH-D14 CPU cooler, GTX 970 + Quadro 2000 GPU, Asus Xonar Essence STX, Sandisk 120GB + Samsung EVO 860 1TB SSD + 4 HDD, Corsair 500R, SeaSonic 660W Gold X PS https://heiko-sieger.info

odtech
Level 2
Level 2
Posts: 95
Joined: Sat Jan 21, 2017 9:25 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by odtech » Thu Jan 26, 2017 5:09 am

powerhouse wrote:@odtech: Thanks for sharing your solution. However, some important pieces of information are missing. Could you share your hardware (motherboard, CPU, graphics card, SATA controller, etc.)? This will help others to see if your solution is applicable to them. Thanks!
Sure no problem.

My hardware. I have since changed motherboard and cpu but i had the same issue with the new setup.
  • MSI Z97A Gaming 9 ACK Motherboard
    Previously Asus H81M-D Plus Motherboard
    Intel i7 4790 CPU
    Previously I5-4690 CPU
    Gigabyte GTX670 OC 2G GDDR5 GPU (GV-N670OC-2GD)
    Sunix SATA2600 PCIe SATA Controller
Kernel Issues

After i upgraded my hardware i installed Linux Mint 18 because i read that on skylake system the ACS patch is no longer needed with kernel 4.7+ so i figured what the heck i will try it for my haswell setup.

Code: Select all

https://mintguide.org/system/639-install-linux-kernel-4-7-stable-on-linux-mint.html
My GPU shared a IOMMU group with a PCI Bridge and i did not need to patch the kernel with the ACS override patch to get passthrough working. I did however end up going back to kernel 4.4+ and patching it on Linux Mint 18 because my PCIe sata controller was grouped with a onboard USB3.0 controller so vfio could not claim the sata controller.

Below are two patches. You need them if your IOMMU groups are not ideal and you have your gpu grouped in with another device like a USB or SATA controller. Like i stated above though if you are running kernel 4.7 and your gpu is grouped with a PCI Bridge then passthrough seems to work ok so you don't need to patch right away. Try it first without patching.

Below code is the patch that was ported for the Ubuntu/Mint kernel 4.4+. Used in Linux Mint 18.

Code: Select all

diff -rupN ubuntu-xenial/drivers/pci/quirks.c ubuntu-xenial\ \(copy\)/drivers/pci/quirks.c 
--- ubuntu-xenial/drivers/pci/quirks.c	2016-07-16 16:16:02.857625599 +0800
+++ "ubuntu-xenial (copy)/drivers/pci/quirks.c"	2016-07-16 16:31:57.808130198 +0800
@@ -3635,6 +3635,9 @@ DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_T
 DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_JMICRON,
 			 PCI_DEVICE_ID_JMICRON_JMB388_ESD,
 			 quirk_dma_func1_alias);
+DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_LITE_ON,
+			 PCI_DEVICE_ID_PLEXTOR_M6E,
+			 quirk_dma_func1_alias);
 
 /*
  * Some devices DMA with the wrong devfn, not just the wrong function.
@@ -3780,6 +3783,107 @@ static void quirk_chelsio_T5_disable_roo
 DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_CHELSIO, PCI_ANY_ID,
 			 quirk_chelsio_T5_disable_root_port_attributes);
 
+static bool acs_on_downstream;
+static bool acs_on_multifunction;
+
+#define NUM_ACS_IDS 16
+struct acs_on_id {
+	unsigned short vendor;
+	unsigned short device;
+};
+static struct acs_on_id acs_on_ids[NUM_ACS_IDS];
+static u8 max_acs_id;
+
+static __init int pcie_acs_override_setup(char *p)
+{
+	if (!p)
+		return -EINVAL;
+
+	while (*p) {
+		if (!strncmp(p, "downstream", 10))
+			acs_on_downstream = true;
+		if (!strncmp(p, "multifunction", 13))
+			acs_on_multifunction = true;
+		if (!strncmp(p, "id:", 3)) {
+			char opt[5];
+			int ret;
+			long val;
+
+			if (max_acs_id >= NUM_ACS_IDS - 1) {
+				pr_warn("Out of PCIe ACS override slots (%d)\n",
+					NUM_ACS_IDS);
+				goto next;
+			}
+
+			p += 3;
+			snprintf(opt, 5, "%s", p);
+			ret = kstrtol(opt, 16, &val);
+			if (ret) {
+				pr_warn("PCIe ACS ID parse error %d\n", ret);
+				goto next;
+			}
+			acs_on_ids[max_acs_id].vendor = val;
+
+			p += strcspn(p, ":");
+			if (*p != ':') {
+				pr_warn("PCIe ACS invalid ID\n");
+				goto next;
+			}
+
+			p++;
+			snprintf(opt, 5, "%s", p);
+			ret = kstrtol(opt, 16, &val);
+			if (ret) {
+				pr_warn("PCIe ACS ID parse error %d\n", ret);
+				goto next;
+			}
+			acs_on_ids[max_acs_id].device = val;
+			max_acs_id++;
+		}
+next:
+		p += strcspn(p, ",");
+		if (*p == ',')
+			p++;
+	}
+
+	if (acs_on_downstream || acs_on_multifunction || max_acs_id)
+		pr_warn("Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA\n");
+
+	return 0;
+}
+early_param("pcie_acs_override", pcie_acs_override_setup);
+
+static int pcie_acs_overrides(struct pci_dev *dev, u16 acs_flags)
+{
+	int i;
+
+	/* Never override ACS for legacy devices or devices with ACS caps */
+	if (!pci_is_pcie(dev) ||
+	    pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ACS))
+		return -ENOTTY;
+
+	for (i = 0; i < max_acs_id; i++)
+		if (acs_on_ids[i].vendor == dev->vendor &&
+		    acs_on_ids[i].device == dev->device)
+			return 1;
+
+	switch (pci_pcie_type(dev)) {
+	case PCI_EXP_TYPE_DOWNSTREAM:
+	case PCI_EXP_TYPE_ROOT_PORT:
+		if (acs_on_downstream)
+			return 1;
+		break;
+	case PCI_EXP_TYPE_ENDPOINT:
+	case PCI_EXP_TYPE_UPSTREAM:
+	case PCI_EXP_TYPE_LEG_END:
+	case PCI_EXP_TYPE_RC_END:
+		if (acs_on_multifunction && dev->multifunction)
+			return 1;
+	}
+
+	return -ENOTTY;
+}
+
 /*
  * AMD has indicated that the devices below do not support peer-to-peer
  * in any system where they are found in the southbridge with an AMD
@@ -3981,6 +4085,7 @@ static const struct pci_dev_acs_enabled
 	{ PCI_VENDOR_ID_INTEL, PCI_ANY_ID, pci_quirk_intel_pch_acs },
 	{ 0x19a2, 0x710, pci_quirk_mf_endpoint_acs }, /* Emulex BE3-R */
 	{ 0x10df, 0x720, pci_quirk_mf_endpoint_acs }, /* Emulex Skyhawk-R */
+	{ PCI_ANY_ID, PCI_ANY_ID, pcie_acs_overrides },
 	{ 0 }
 };

diff -rupN ubuntu-xenial/Documentation/kernel-parameters.txt ubuntu-xenial\ \(copy\)/Documentation/kernel-parameters.txt 
--- ubuntu-xenial/Documentation/kernel-parameters.txt	2016-07-16 16:15:59.133680197 +0800
+++ "ubuntu-xenial (copy)/Documentation/kernel-parameters.txt"	2016-07-16 16:19:37.418213416 +0800
@@ -2914,6 +2914,16 @@ bytes respectively. Such letter suffixes
 		nomsi	Do not use MSI for native PCIe PME signaling (this makes
 			all PCIe root ports use INTx for all services).
 
+	pcie_acs_override =
+			[PCIE] Override missing PCIe ACS support for:
+		downstream
+			All downstream ports - full ACS capabilties
+		multifunction
+			All multifunction devices - multifunction ACS subset
+		id:nnnn:nnnn
+			Specfic device - full ACS capabilities
+			Specified as vid:did (vendor/device ID) in hex
+
 	pcmv=		[HW,PCMCIA] BadgePAD 4
 
 	pd_ignore_unused

diff -rupN ubuntu-xenial/include/linux/pci_ids.h ubuntu-xenial\ \(copy\)/include/linux/pci_ids.h 
--- ubuntu-xenial/include/linux/pci_ids.h	2016-07-16 16:16:04.157606496 +0800
+++ "ubuntu-xenial (copy)/include/linux/pci_ids.h"	2016-07-16 16:37:01.391560538 +0800
@@ -3004,4 +3004,7 @@
 
 #define PCI_VENDOR_ID_OCZ		0x1b85
 
+#define PCI_VENDOR_ID_LITE_ON 0x1c28
+#define PCI_DEVICE_ID_PLEXTOR_M6E 0x0122
+
 #endif /* _LINUX_PCI_IDS_H */
Below code is the patch for the Ubuntu/Mint kernel 3.18+. Used in Linux Mint 17.

Code: Select all

diff -rupN linux-3.15.old/Documentation/kernel-parameters.txt linux-3.15/Documentation/kernel-parameters.txt
--- linux-3.15.old/Documentation/kernel-parameters.txt	2014-06-08 15:19:54.000000000 -0300
+++ linux-3.15/Documentation/kernel-parameters.txt	2014-06-09 18:05:31.744055580 -0300
@@ -2554,6 +2554,16 @@ bytes respectively. Such letter suffixes
 		nomsi	Do not use MSI for native PCIe PME signaling (this makes
 			all PCIe root ports use INTx for all services).
 
+	pcie_acs_override =
+			[PCIE] Override missing PCIe ACS support for:
+		downstream
+			All downstream ports - full ACS capabilties
+		multifunction
+			All multifunction devices - multifunction ACS subset
+		id:nnnn:nnnn
+			Specfic device - full ACS capabilities
+			Specified as vid:did (vendor/device ID) in hex
+
 	pcmv=		[HW,PCMCIA] BadgePAD 4
 
 	pd_ignore_unused
diff -rupN linux-3.15.old/drivers/pci/quirks.c linux-3.15/drivers/pci/quirks.c
--- linux-3.15.old/drivers/pci/quirks.c	2014-06-08 15:19:54.000000000 -0300
+++ linux-3.15/drivers/pci/quirks.c	2014-06-09 18:06:36.688743183 -0300
@@ -3384,6 +3384,107 @@ struct pci_dev *pci_get_dma_source(struc
 	return pci_dev_get(dev);
 }
 
+static bool acs_on_downstream;
+static bool acs_on_multifunction;
+
+#define NUM_ACS_IDS 16
+struct acs_on_id {
+	unsigned short vendor;
+	unsigned short device;
+};
+static struct acs_on_id acs_on_ids[NUM_ACS_IDS];
+static u8 max_acs_id;
+
+static __init int pcie_acs_override_setup(char *p)
+{
+	if (!p)
+		return -EINVAL;
+
+	while (*p) {
+		if (!strncmp(p, "downstream", 10))
+			acs_on_downstream = true;
+		if (!strncmp(p, "multifunction", 13))
+			acs_on_multifunction = true;
+		if (!strncmp(p, "id:", 3)) {
+			char opt[5];
+			int ret;
+			long val;
+
+			if (max_acs_id >= NUM_ACS_IDS - 1) {
+				pr_warn("Out of PCIe ACS override slots (%d)\n",
+					NUM_ACS_IDS);
+				goto next;
+			}
+
+			p += 3;
+			snprintf(opt, 5, "%s", p);
+			ret = kstrtol(opt, 16, &val);
+			if (ret) {
+				pr_warn("PCIe ACS ID parse error %d\n", ret);
+				goto next;
+			}
+			acs_on_ids[max_acs_id].vendor = val;
+
+			p += strcspn(p, ":");
+			if (*p != ':') {
+				pr_warn("PCIe ACS invalid ID\n");
+				goto next;
+			}
+
+			p++;
+			snprintf(opt, 5, "%s", p);
+			ret = kstrtol(opt, 16, &val);
+			if (ret) {
+				pr_warn("PCIe ACS ID parse error %d\n", ret);
+				goto next;
+			}
+			acs_on_ids[max_acs_id].device = val;
+			max_acs_id++;
+		}
+next:
+		p += strcspn(p, ",");
+		if (*p == ',')
+			p++;
+	}
+
+	if (acs_on_downstream || acs_on_multifunction || max_acs_id)
+		pr_warn("Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA\n");
+
+	return 0;
+}
+early_param("pcie_acs_override", pcie_acs_override_setup);
+
+static int pcie_acs_overrides(struct pci_dev *dev, u16 acs_flags)
+{
+	int i;
+
+	/* Never override ACS for legacy devices or devices with ACS caps */
+	if (!pci_is_pcie(dev) ||
+	    pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ACS))
+		return -ENOTTY;
+
+	for (i = 0; i < max_acs_id; i++)
+		if (acs_on_ids[i].vendor == dev->vendor &&
+		    acs_on_ids[i].device == dev->device)
+			return 1;
+
+	switch (pci_pcie_type(dev)) {
+	case PCI_EXP_TYPE_DOWNSTREAM:
+	case PCI_EXP_TYPE_ROOT_PORT:
+		if (acs_on_downstream)
+			return 1;
+		break;
+	case PCI_EXP_TYPE_ENDPOINT:
+	case PCI_EXP_TYPE_UPSTREAM:
+	case PCI_EXP_TYPE_LEG_END:
+	case PCI_EXP_TYPE_RC_END:
+		if (acs_on_multifunction && dev->multifunction)
+			return 1;
+	}
+
+	return -ENOTTY;
+}
+
 /*
  * AMD has indicated that the devices below do not support peer-to-peer
  * in any system where they are found in the southbridge with an AMD
@@ -3483,6 +3584,7 @@ static int pci_quirk_intel_pch_acs(struc
 	return acs_flags & ~flags ? 0 : 1;
 }
 
+
 static const struct pci_dev_acs_enabled {
 	u16 vendor;
 	u16 device;
@@ -3495,6 +3597,7 @@ static const struct pci_dev_acs_enabled
 	{ PCI_VENDOR_ID_ATI, 0x4384, pci_quirk_amd_sb_acs },
 	{ PCI_VENDOR_ID_ATI, 0x4399, pci_quirk_amd_sb_acs },
 	{ PCI_VENDOR_ID_INTEL, PCI_ANY_ID, pci_quirk_intel_pch_acs },
+	{ PCI_ANY_ID, PCI_ANY_ID, pcie_acs_overrides },
 	{ 0 }
 };
 
Download the kernel from below. Get longterm: 4.4.45 or longterm: 3.18.47, click on tar.xz to down the appropriate file.

Code: Select all

https://www.kernel.org/
Step by step howto on patching compiling and installing a kernel. I take no credit for the video, that is not my channel.

Code: Select all

sudo apt-get update && sudo apt-get install build-essential kernel-package libncurses5-dev bzip2 fakeroot git git-core git-gui
In Mint 17 "kernel-package" is bugged so you have to download it in deb form and install it. Link in the comments of below bug report.

Code: Select all

https://bugs.launchpad.net/ubuntu/+source/kernel-package/+bug/1308183

Code: Select all

https://youtu.be/2EnR4fDPi8w
When you are done patching and installing the kernel edit the grub file and add "pcie_acs_override=downstream" to the appropriate line as shown below.

Code: Select all

sudo nano /etc/default/grub

Code: Select all

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on pcie_acs_override=downstream"
Comment out the menu timeout as below example so grub will always show the advanced boot menu. This is just insurance again you installing a broken kernel and not being able to boot it. You can just select the old kernel if the new one won't boot.

Code: Select all

#GRUB_HIDDEN_TIMEOUT=0
When done editing press "Ctrl+x", "y" then "enter" to exit and save. Nano isn't the most user friendly but it's what i prefer, you can use whatever editor you like.

Code: Select all

sudo update-grub

Code: Select all

sudo reboot


GPU Issues

My main issue with my gpu is that it is old and did not support UEFI out of the box which forced me to run seabios for my VM. PCI Passthrough does not work if you are using ovmf with a non uefi gpu but i needed to use ovmf to get my sata controller working. Running seabios caused the windows installation to not be able to install to a drive connected to my Sunix SATA2600 controller so i had to flash my gpu to make it uefi compatible and fix the SATA controller issue.

http://vfio.blogspot.com.au/2014/08/doe ... t-efi.html
I followed above link that was also posted by Powerhouse earlier in the guide, i just had to make one small change to be able to get a copy of the rom file from the gpu.

Code: Select all

cd /sys/bus/pci/devices/0000:01:00.0/
sudo su
echo 1 > rom
cat rom > /tmp/image.rom
echo 0 > rom
exit
Without entering sudo su i would get permission denied.

If it turns out your gpu bios does not support uefi then head over to the manufacturer website and check if they have a uefi bios available. Gigabyte had one for my card so i downloaded it.
Their flash tool did not work however so i used one from techpowerup.com
For ATI

Code: Select all

https://www.techpowerup.com/downloads/Utilities/BIOS_Flashing/ATI/
For NVIDIA

Code: Select all

https://www.techpowerup.com/downloads/Utilities/BIOS_Flashing/NVIDIA/
Be very careful when flashing your GPU, you can brick it if you use the wrong bios or if something goes wrong. Read your GPU manufacturers notes on flashing.

I think i covered all the bumps i had, if i think of something i will update my post though.

Edit: Added a bit more info and my old board/cpu and fixed list of dependencies for compiling a kernel.
Last edited by odtech on Thu Jan 26, 2017 12:02 pm, edited 2 times in total.

powerhouse
Level 6
Level 6
Posts: 1085
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse » Thu Jan 26, 2017 11:20 am

@odtech: You're awesome - thanks for sharing your experience and solution. I have taken the liberty to link to your post and your instructions on how to patch the kernel with the ACS patch are great. Do you have a link to where you took the ACS patch from?

I've been digging a little on my own. Below is a port of the ACS override patch to Ubuntu 16.04, tested on 4.4.0-31-generic(from official ubuntu git source):
https://drive.google.com/file/d/0B2vXTG ... sp=sharing
See here for more!

Here another patch for the 4.4 kernel in 16.04: https://drive.google.com/open?id=0B8hkQ ... mRyQ1l6djg (might be the same as above, but I haven't compared).

Following this same thread on the Ubuntu forum, there is a patch for kernel 4.8 too. See here for more.
Asus Sabertooth X79, i7 3930K CPU, 8x4GB Kingston DDR3, Noctua NH-D14 CPU cooler, GTX 970 + Quadro 2000 GPU, Asus Xonar Essence STX, Sandisk 120GB + Samsung EVO 860 1TB SSD + 4 HDD, Corsair 500R, SeaSonic 660W Gold X PS https://heiko-sieger.info

odtech
Level 2
Level 2
Posts: 95
Joined: Sat Jan 21, 2017 9:25 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by odtech » Thu Jan 26, 2017 11:43 am

powerhouse wrote:@odtech: You're awesome - thanks for sharing your experience and solution. I have taken the liberty to link to your post and your instructions on how to patch the kernel with the ACS patch are great. Do you have a link to where you took the ACS patch from?

I've been digging a little on my own. Below is a port of the ACS override patch to Ubuntu 16.04, tested on 4.4.0-31-generic(from official ubuntu git source):
https://drive.google.com/file/d/0B2vXTG ... sp=sharing
See here for more!

Here another patch for the 4.4 kernel in 16.04: https://drive.google.com/open?id=0B8hkQ ... mRyQ1l6djg (might be the same as above, but I haven't compared).

Following this same thread on the Ubuntu forum, there is a patch for kernel 4.8 too. See here for more.
The 4.4 patch i posted is from the first google drive link you posted and i found it in the ubuntu forum post you linked just below it.
I'm gonna try out the 4.8 patch tomorrow, i installed Mint in legacy mode and not efi so i need to redo it.

To anyone that is going to compile kernels. Do yourself a favour and keep the compiled ones, they takea bout 45 minutes to finish so it's a pita to redo. That way if you need to reinstall Mint then you can just install .deb files to get your patched kernel up and running again.

odtech
Level 2
Level 2
Posts: 95
Joined: Sat Jan 21, 2017 9:25 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by odtech » Fri Jan 27, 2017 1:22 pm

I reloaded my system yesterday with EFI booting since i derped the first time and managed to load it in legacy mode.

I left the kernel patching till last and first tried to start the VM with the stock kernel. To my surprise i got a clean Qemu console with no errors and the VM booted with all my specified devices passed through.

The devices in bold is what i passed through and you can see they share a group with PCI Bridges so it seems you can safely ignore them. I don't know if this is the case for everyone and on all kernels but it's worth keeping in mind.
### Group 0 ###
00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06)
### Group 1 ###
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
00:01.2 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x4 Controller (rev 06)
02:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 670] (rev a1)
02:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1)

### Group 2 ###
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06)
### Group 3 ###
00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06)
### Group 4 ###
00:14.0 USB controller: Intel Corporation 9 Series Chipset Family USB xHCI Controller
### Group 5 ###
00:16.0 Communication controller: Intel Corporation 9 Series Chipset Family ME Interface #1
### Group 6 ###
00:1a.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2
### Group 7 ###
00:1b.0 Audio device: Intel Corporation 9 Series Chipset Family HD Audio Controller
### Group 8 ###
00:1c.0 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 1 (rev d0)
### Group 9 ###
00:1c.2 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 3 (rev d0)
### Group 10 ###
00:1c.3 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 4 (rev d0)
### Group 11 ###
00:1c.4 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 5 (rev d0)
### Group 12 ###
00:1c.6 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 7 (rev d0)
### Group 13 ###
00:1c.7 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 8 (rev d0)
### Group 14 ###
00:1d.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1
### Group 15 ###
00:1f.0 ISA bridge: Intel Corporation 9 Series Chipset Family Z97 LPC Controller
00:1f.2 SATA controller: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode]
00:1f.3 SMBus: Intel Corporation 9 Series Chipset Family SMBus Controller
### Group 16 ###
04:00.0 Network controller: Qualcomm Atheros QCA6174 802.11ac Wireless Network Adapter (rev 20)
### Group 17 ###
05:00.0 Ethernet controller: Qualcomm Atheros Killer E220x Gigabit Ethernet Controller (rev 13)
### Group 18 ###
06:00.0 USB controller: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller
### Group 19 ###
07:00.0 PCI bridge: ASMedia Technology Inc. Device 1184
### Group 20 ###
08:01.0 PCI bridge: ASMedia Technology Inc. Device 1184
09:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9128 PCIe SATA 6 Gb/s RAID controller with HyperDuo (rev 11)
### Group 21 ###
08:03.0 PCI bridge: ASMedia Technology Inc. Device 1184
### Group 22 ###
08:05.0 PCI bridge: ASMedia Technology Inc. Device 1184
0b:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)
### Group 23 ###
08:07.0 PCI bridge: ASMedia Technology Inc. Device 1184
### Group 24 ###
0d:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)

User avatar
davidem
Level 1
Level 1
Posts: 13
Joined: Sun Jan 22, 2017 4:40 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by davidem » Fri Jan 27, 2017 5:57 pm

Powerhouse asked me to post my settings as well as tell a bit about the things I ran into when creating my VM, so here I go... My apologies upfront, it turned out te be a long read 8)

Hardware:
Core i7 6700
MSI Z170a SLI Plus
16 GB memory DDR4 2133MHz
256 GB Intel 600P PCIe M.2 SSD
Graphics: IGD Intel HD 530
Graphics: PCIe Radeon RX 470 Gaming x 4G
One Display: Samsung 24 inch with FreeSync, 1 x VGA and 1 x HDMI as input

The Plan:
1 main Linux installation fit for general use as well as hosting several virtual machines. At least one linux VM running as backup MX for my mail server and one Windows VM capable of running games.

- Host: will use the internal intel graphics and is connected to the display with VGA.
- The backupMX VM will be a simple VM running Debian + Postfix, no tricky stuff (left out if this short novel)
- The Windows VM should use the Radeon card and is connected to the display with HDMI

Although we use VMware a lot at work, I wanted to go the open source way. I read about the pro’s and con’s of Xen and KVM and decided to go for Xen at first. Regarding the OS, as I’m already running a Debian VPS, my brain decided that Ubuntu was the most logical choice. So, here’s how the story went:

Attempt 1: Ubuntu 16.04 LTS with Xen
I am amazed about how blazingly fast the M.2 SSD is… In the past I could have easily made a cup of tea whilst installing an OS, but on this device it was barely a matter of minutes. The only problem is the cumbersome support of it. I’m used to customize filesystems (/tmp, /var, /opt, /whatever), but on this device I ran into funny issues like grub refusing to install and LVM failing… I ended up with 3 manually created partitions: one for EFI, one for root and one for swap. So although the installation was blazingly fast, after a few times it gets annoying, despite the speed.

But in the end, I had a nice installation of Ubuntu. Next up: Xen… But right after the installation of Xen, and the subsequent reboot, the SSD was left unbootable*. I reinstalled and retried several times, each time with the same result.

* “No bootable device. Press the any key”

Attempt 2: Ubuntu 16.04 with KVM, both IGD + Radeon explicitly enabled in BIOS, IGD primary
So, no Xen on this M.2 device… but I had better luck with KVM. At least it installed without any issues.

In the BIOS/UEFI of MSI, I can either select the IGD as primary (no mention at all of the PCIe card) or I can select the PCIe card as primary, with the option to also enable the IGD. Wanting both cards enabled, I chose the latter option.

But for some reason Ubuntu only uses the primary graphics card. As soon as I configured Ubuntu to blacklist and ‘pci-stub’ the PCIe card, Ubuntu failed to load the GUI properly. It simply left me staring at the message it had successfully blacklisted. So…

Attempt 3: Ubuntu 16.04 with KVM, IGD enabled in BIOS
… I selected IGD as the primary in the BIOS. The install of Ubuntu was screwed up, but by this time I was used to reinstalling it :)

Following the blogpost of davideyat.es (for a big part based on Powerhouse’s forum post(s)), I used the PCI-Stub method , but I couldn’t get it to work properly. Each time I started a VM using PCIe graphics, the VGA screen went black.

Attempt 4: Mint 18.1 with KVM, both IGD + Radeon explicitly enabled in BIOS, Radeon primary
After almost giving up and going for the plain old dual boot, I decided to give it one more shot with this forum-post.

Starting from scratch I again enabled the PCIe card in the BIOS, but that brought me the same issues as with Ubuntu. Mint also binds to the primary card (PCIe in this case), but I couldn’t make it bind to the secondary… So…

Attempt 5: Mint 18.1 with KVM, only IGD enabled in BIOS
… I selected the IGD as the primary again. As already mentioned, when you do that, MSI removes all options for the PCIe card. So fingers crossed and going through all the steps.

But… another failure: removing Network-Manager also removes Cinnamon!? And that left me again with a crippled OS not able to boot… So…

Attempt 5.1: Mint 18.1 with KVM, only IGD enabled in BIOS, disabling Network-Manager
… another fresh install, this time only disabling Network-Manager. But afterwards, I finally have some good results: as I boot the VM both VGA (from the IGD) and HDMI (form the PCIe card) send a signal to the display at the same time! So I can finally start to tinker with the VM settings!

Attempt 5.2: Mint 18.1 with KVM, only IGD enabled in BIOS, q35
I started with the startup-script provided by Powerhouse and simply changed the paths to the images, iso’s and OVMF.fd. The VM booted fine from the Windows ISO and the installation of Windows went well. I was presented with an 800x600 resolution, but installing system updates (Radeon drivers included) failed and I was left with a reboot-loop.

Also manually downloading Radeon drivers and installing them resulted in a reboot-loop.

Attempt 5.3: Mint 18.1 with KVM, only IGD enabled in BIOS, q35 + ioh3420 PCIe hub
Added the ioh3420 line and adjusted the vfio-pci lines (like this:)

Code: Select all

  -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
  -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
  -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
but still the same reboot-loop in the end. I fiddled around with some settings, nothing seemed to work.

Attempt 5.4: Mint 18.1 with KVM, only IGD enabled in BIOS, i440fx
I decided to ‘switch’ to i440fx. And after I also removed the ioh342 line and adjusted the vfio lines again, like this:

Code: Select all

  -device vfio-pci,host=01:00.0,multifunction=on \
  -device vfio-pci,host=01:00.1 \
a tiny miracle happened: the manually downloaded Radeon drivers were installed! I had a 1980x1600 resolution! Yippee! It still failed to complete the installation properly (and rebooted, once), but at least I was on the right path. At this point I first tried Passmark to benchmark my Windows 10 VM and it came back with the odd message I was running a build of Windows 10 which did not support Direct-X 12. So…

Attempt 6: Mint 18.1 with KVM, only IGD enabled in BIOS, i440fx, fresh download of Windows 10 iso
…I downloaded a fresh copy of Windows 10. And this one worked fabulously! The Radeon drivers installed completely as well as Windows updates. No issues whatsoever. In one word: awesome!

I retried 5.2 and 5.3 with the new ISO though, but that still resulted in a reboot-loop. So for me only i440fx works fine. There’s only one thing nagging me, and that the crippled audio. What ‘ve read about it, it is not an easy fix without using an additional sound-card or usb-audio-thingy. I ordered a sound-card and it should arrive any day now, after which I’ll update this post.

Oh, one more thing: I still use one set of mouse/keyboard. I have yet to look into that part...

For the time being, here are my current settings. My performance results can be seen here: viewtopic.php?f=225&t=153482&start=40#p1268895

Final Specs:
Linux Host:
Core i7 6700
MSI Z170a SLI Plus
16 GB memory DDR4 2133MHz
256 GB Intel 600P PCIe M.2 SSD
OS: Linux Mint 18.1 Cinnamon
Graphics: Intel HD 530 via VGI to the monitor
Virtualization: KVM - qemu

Windows Guest:
4 vCPU cores
i440fx v2.5
8 GB memory, with hugepages
45GB bootdisk (RAW image + qcow2 snapshot)
OS: Windows 10
Graphics: Radeon RX 470 Gaming x 4G via HDMI to the monitor
Virtual IO drivers: 0.1.126 (stable)

qemu-part from the startup scrip (the rest is still the same as provided by Powerhouse):

Code: Select all

qemu-system-x86_64 \
  -name $vmname,process=$vmname \
  -machine type=pc,accel=kvm \
  -cpu host \
  -smp 4,sockets=1,cores=2,threads=2 \
  -enable-kvm \
  -m 8G \
  -mem-path /dev/hugepages \
  -mem-prealloc \
  -balloon none \
  -rtc clock=host,base=localtime \
  -vga none \
  -nographic \
  -serial none \
  -parallel none \
  -soundhw hda \
  -usb -usbdevice host:03f0:0024 -usbdevice host:0461:4d51 \
  -device vfio-pci,host=01:00.0,multifunction=on \
  -device vfio-pci,host=01:00.1 \
  -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/OVMF.fd \
  -drive if=pflash,format=raw,file=/tmp/my_vars.fd \
  -boot order=dc \
  -device virtio-scsi-pci,id=scsi \
  -drive id=disk0,if=virtio,cache=none,format=qcow2,file=/media/images/new_win10up_snap.qcow2 \
  -drive id=disk1,if=virtio,cache=none,format=raw,file=/media/images/win_steam.img \
  -drive file=/home/david/ISOs/Win10_1607_EnglishInternational_x64.iso,id=isocd,format=raw,if=none -device scsi-cd,drive=isocd \
  -drive file=/home/david/ISOs/virtio-win-0.1.126.iso,id=virtiocd,format=raw,if=none -device ide-cd,bus=ide.1,drive=virtiocd \
  -netdev type=tap,id=net0,ifname=tap0,vhost=on \
  -device virtio-net-pci,netdev=net0,mac=00:16:3e:00:01:01

   exit 0
fi
I hope this might help someone :)

Cheers,

David
Linux Mint 18.3 with Windows Gaming VM:
Intel i7 6700, MSI Z170a SLI Plus, 16 GB DDR4 2133MHz, 256 GB Intel 600P PCIe M.2 SSD, 512 GB Samsung 850 EVO SSD, Radeon RX 470 Gaming x 4G (VM), Asus Xonar DG-X 5.1 (VM)

powerhouse
Level 6
Level 6
Posts: 1085
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse » Sat Jan 28, 2017 9:58 am

Thanks David, this is awesome! I admire your persistence.

About the sound: Hope the sound card works out for you.

Keyboard & mouse: Synergy seems to be the best option. In the past I used a KVM (keyboard, video, mouse) switch but the one I bought would sometimes hang. A few months ago it finally stopped working. The cheap KVM switches are often garbage, the good ones are expensive. I tested Synergy in the past and it worked well. It's now a commercial solution and costs a little money, but I think this is money well spent.
Asus Sabertooth X79, i7 3930K CPU, 8x4GB Kingston DDR3, Noctua NH-D14 CPU cooler, GTX 970 + Quadro 2000 GPU, Asus Xonar Essence STX, Sandisk 120GB + Samsung EVO 860 1TB SSD + 4 HDD, Corsair 500R, SeaSonic 660W Gold X PS https://heiko-sieger.info

driz
Level 1
Level 1
Posts: 24
Joined: Mon May 12, 2014 3:38 pm

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by driz » Mon Jan 30, 2017 9:38 pm

Hey all, i wanted to add some info since i made a few changes. first, without hugepages, but passing the full ssd through my windows score was 6.4. Assigned a 6G hugepage to the vm increased this to 6.5.

That being said, i was still having that crackly audio, so i purchased an hdmi monitor for one of the girls and it has a headphone out jack. Well, i quickly discovered that the HD Audio device in windows wasn't actually loading. I did a full reinstall of windows and noted that upon initial boot, i had a code 43 on my amd radeon r7 240.

the relevent video portion resulting in a code 43 is below

Code: Select all

  -device vfio-pci,host=04:00.0,romfile=/srv/rom2.rom \
  -device vfio-pci,host=04:00.1 \
no ioh3420 specified in the above scenario. While the video card had a code43, the hd audio bus worked fine. I loaded in the drivers for the video card and immediately began bootlooping upon driver loading.

loading video a different way would result in video working but no audio bus, see below:

Code: Select all

  -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
  -device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,romfile=/srv/rom2.rom \
  -device vfio-pci,host=04:00.1,bus=root.1 \
I started playing around with some things and finally decided to try changing the machine type from q35 to PC. I booted up and had working sound and video... i assumed ok, i just swapped out new-shiny q35 for some old stuff, let's check the windows score. i have a 6.6 right now... with hugepages turned off. IMO a 6.6 is pretty stellar considering this is a shared I3 processor, 6G of ram and an r7 240 video card. I still plan to play around a little and see what i can do...

I put a post on MSI's forums to get the latest uefi capable vbios for my card and I'll try and see if anything different happens.. such strange issues..

also, since i see some of you discussing KVMs, i used to use this (ATEN 2-Port Dual Video Dual Link DVI KVMP Switch with Audio Support and Cables CS1642). supports dual dvi monitors and up to 5 usb (including m/k) and supports hot swap buttons. I'm actually selling it on clist right now pretty cheaply... :)


sadly, i started getting graphic distortion, this could be due to my system_reset continuously from the qemu command line while trying different settings. going to reinstall windows and see how it goes. turns out in this configuration, i can't seem to boot from the windows 10 install media.. very interesting.

all right, got it resolved. I'm back to using Q35 which eliminated the glitches I was having with type=pc. I am incapable of changing only 1 thing, so i made a few changes

Code: Select all

  -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
  -device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,romfile=/srv/rom2.rom \
  -device vfio-pci,host=04:00.1,bus=pcie.0 \
note that i changed the bus for the audio portion of the device! this corrected my issue fully for audio.

I also made some changes to my disks to try and squeeze out some more performance.

Code: Select all

  -object iothread,id=iothread0 \
  -device virtio-scsi-pci,id=scsi,iothread=iothread0 \
  -drive file=/dev/sdd,id=disk0,if=none,format=raw,cache=none,aio=native \
iothreads are supposed to be completely badass in all my research. I actually forgot to change my cache from none to directsync, but i plan to go back and change this soon. I may go back and setup hugepages for each vm again, but honestly, im just not sure on the best practice for hugepages on 2 guests.

powerhouse
Level 6
Level 6
Posts: 1085
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by powerhouse » Tue Jan 31, 2017 4:38 pm

@driz: Thanks for sharing your experience - this is great!

About hugepages: I never compared hugepages vs. without hugepages, but simply took it as a performance optimizer. From what I read, however, the impact of hugepages is rather small. I wouldn't worry about it.

Using iothread is the recommended way to improve I/O that is disk performance at the moment (the KVM syntax seems to undergo frequent changes). I wanted to include it, but for some strange reason I get an error when I include iothread. I will need to research that.

In order to get a complete picture, could you post your entire start script? I take it that you solved your audio problem. This is some great input!

While looking for additional info on the ioh3420 device I ran across the following tutorial by David Yates: https://davidyat.es/2016/09/08/gpu-passthrough/. It often refers to my tutorial, but expands on some points and offers solutions that aren't covered here. David solves the audio issue using Spice. But your solution looks much neater - in fact, I have to try it (my audio kinda works but produces distortions - not an issue for photo editing, but with movies or games it's annoying).
Asus Sabertooth X79, i7 3930K CPU, 8x4GB Kingston DDR3, Noctua NH-D14 CPU cooler, GTX 970 + Quadro 2000 GPU, Asus Xonar Essence STX, Sandisk 120GB + Samsung EVO 860 1TB SSD + 4 HDD, Corsair 500R, SeaSonic 660W Gold X PS https://heiko-sieger.info

driz
Level 1
Level 1
Posts: 24
Joined: Mon May 12, 2014 3:38 pm

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by driz » Tue Jan 31, 2017 5:58 pm

powerhouse wrote:@driz: Thanks for sharing your experience - this is great!

About hugepages: I never compared hugepages vs. without hugepages, but simply took it as a performance optimizer. From what I read, however, the impact of hugepages is rather small. I wouldn't worry about it.

Using iothread is the recommended way to improve I/O that is disk performance at the moment (the KVM syntax seems to undergo frequent changes). I wanted to include it, but for some strange reason I get an error when I include iothread. I will need to research that.

In order to get a complete picture, could you post your entire start script? I take it that you solved your audio problem. This is some great input!

While looking for additional info on the ioh3420 device I ran across the following tutorial by David Yates: https://davidyat.es/2016/09/08/gpu-passthrough/. It often refers to my tutorial, but expands on some points and offers solutions that aren't covered here. David solves the audio issue using Spice. But your solution looks much neater - in fact, I have to try it (my audio kinda works but produces distortions - not an issue for photo editing, but with movies or games it's annoying).
to be fair, i haven't tested the audio for distortions or anything yet. The issue i was resolving was that the device itself (hd audio bus) wasn't loading. it now loads properly, i'm planning to test the rest soon(tm) :)

David's solutions won't really work for me as he's playing audio through the spice channel on the host. I have 2 girls playing concurrently with different audio streams. With this being the case, i need the guest to present audio to their headsets or speakers as the case may be which requires the audio to play via the guest. hdmi or a usb passthrough are the only solutions I've found thus far. My girls use the USB right now and while they don't complain, the crackly audio drives me nuts. Surprisingly, the passed through usb microphones are crystal clear, it's only playback that presents an issue (and as I understand from other reading, an issue that only presents in a windows guest environment) I'm hoping using HDMI will get rid of the crackle.

code as requested.

Code: Select all

#!/bin/bash

configfile=/etc/vfio-pci2.cfg
vmname="kid1"

vfiobind() {
   dev="$1"
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id

}


if ps -A | grep -q $vmname; then
   echo "$vmname is already running."
   exit 1

else

   cat $configfile | while read line;do
   echo $line | grep ^# >/dev/null 2>&1 && continue
      vfiobind $line
   done


if [ ! -f /tmp/system2.fd ]; then
  cp /srv/ovmf-x64/OVMF_VARS-pure-efi.fd /tmp/system2.fd
fi

qemu-system-x86_64 \
  -name $vmname,process=$vmname \
  -serial none \
  -parallel none \
  -machine type=q35,accel=kvm \
  -cpu host,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time \
  -smp 4,sockets=1,cores=2,threads=2 \
  -enable-kvm \
  -m 6G \
  -mem-prealloc \
  -balloon none \
  -vga none \
  -nographic \
  -rtc clock=host,base=localtime \
  -usb -usbdevice host:04f2:0939 -usbdevice host:04f2:0833 -device usb-host,hostbus=1,hostaddr=5 \
  -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
  -device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,romfile=/srv/rom2.rom \
  -device vfio-pci,host=04:00.1,bus=pcie.0 \
  -drive if=pflash,format=raw,readonly,file=/srv/ovmf-x64/OVMF_CODE-pure-efi.fd \
  -drive if=pflash,format=raw,file=/tmp/system2.fd \
  -boot order=dc \
  -object iothread,id=iothread0 \
  -device virtio-scsi-pci,id=scsi,iothread=iothread0 \
  -drive file=/dev/sdd,id=disk0,if=none,format=raw,cache=directsync,aio=native \
  -device scsi-hd,drive=disk0,bootindex=2 \
  -netdev type=tap,id=net0,ifname=tap1,vhost=on \
  -device virtio-net-pci,netdev=net0

#  -drive file=/srv/win10.iso,id=isocd,format=raw,if=none -device scsi-cd,drive=isocd,bootindex=1 \
#  -drive file=/srv/virtio-win.iso,id=virtiocd,format=raw,if=none -device ide-cd,bus=ide.1,drive=virtiocd \
#  -mem-path /dev/hugepages \

exit 0
fi

MikeF
Level 1
Level 1
Posts: 3
Joined: Wed Feb 01, 2017 6:13 pm

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by MikeF » Wed Feb 01, 2017 6:40 pm

First thank you for writing this how-to, it has been extremely helpful.

I was able to create a Windows 10 VM, passthrough my Nvidia card, install the latest drivers, and then run OpenGL 4.4 based apps.

I then did the same thing with a Linux Mint VM.

I just copied and edited the windows10vm.sh to linuxmintvm.sh, used a different .img file and installation .iso image
I just edited two lines as follows
-drive id=disk0,if=virtio,cache=none,format=raw,file=/media/user/mint.img
-drive file=/home/user/download/linuxmint18.iso,id=isocd,format=raw,if=none -device scsi-cd,drive=isocd

The Mint install went perfectly fine, as did the nvidia driver install, and test apps. I even rebooted it (without exiting the qemu session) and it restarted without issue.

The problem is after I powered down the linux VM I cannot restart it. The windows 10 VM starts fine but I get the following error when I try and start the linux VM

Boot Failed: EFI DVD/CDROM
Boot Failed: EFI misc device


It then went into the network boot sequence.

Any advice on what is going wrong? The virtual disk (linuxmint18.iso) is not being recognized as a valid boot device even though the linux mint installer recognized it as a valid device/drive to install into.

Did I miss needing to install additional device drivers?

Do you have a .sh file that will boot a linux VM as a starting point? I am hoping you replicated your Windows 10 install with a linux install.

I've tried to get a 'valid' .sh file for a linux VM as a starting point, but as you point out the vast majority of docs out there use virt-manager with the info buried in xml files. I tried to tease out a working .sh file but so for no luck.

Thanks

odtech
Level 2
Level 2
Posts: 95
Joined: Sat Jan 21, 2017 9:25 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by odtech » Wed Feb 01, 2017 7:31 pm

MikeF wrote:First thank you for writing this how-to, it has been extremely helpful.

I was able to create a Windows 10 VM, passthrough my Nvidia card, install the latest drivers, and then run OpenGL 4.4 based apps.

I then did the same thing with a Linux Mint VM.

I just copied and edited the windows10vm.sh to linuxmintvm.sh, used a different .img file and installation .iso image
I just edited two lines as follows
-drive id=disk0,if=virtio,cache=none,format=raw,file=/media/user/mint.img
-drive file=/home/user/download/linuxmint18.iso,id=isocd,format=raw,if=none -device scsi-cd,drive=isocd

The Mint install went perfectly fine, as did the nvidia driver install, and test apps. I even rebooted it (without exiting the qemu session) and it restarted without issue.

The problem is after I powered down the linux VM I cannot restart it. The windows 10 VM starts fine but I get the following error when I try and start the linux VM

Boot Failed: EFI DVD/CDROM
Boot Failed: EFI misc device


It then went into the network boot sequence.

Any advice on what is going wrong? The virtual disk (linuxmint18.iso) is not being recognized as a valid boot device even though the linux mint installer recognized it as a valid device/drive to install into.

Did I miss needing to install additional device drivers?

Do you have a .sh file that will boot a linux VM as a starting point? I am hoping you replicated your Windows 10 install with a linux install.

I've tried to get a 'valid' .sh file for a linux VM as a starting point, but as you point out the vast majority of docs out there use virt-manager with the info buried in xml files. I tried to tease out a working .sh file but so for no luck.

Thanks
The problem might be that the installer is installing Mint in legacy bios mode and not uefi which could cause OVMF then to not recognize the disk as bootable. The easiest way to confirm if this is the case is to restart the live disc then use the "Disks" tool to check the partitions on the Mint disk. The first partition should be a "EFI Partition", if it is not then i'm pretty sure this is the problem.. The solution is to switch your startup script to seabios or use a version of OVMF with Seabios CSM to see if the VM boots then.

Source about OVMF and Seabios CSM
http://www.tianocore.org/ovmf/
Link to the Redhat RPM's as seen in above link containing the OVMF bios versions.
https://www.kraxel.org/repos/jenkins/edk2/

If you would like to stick with OVMF then get the following file and point your Mint startup script's bios lines to the new OVMF file.
Download this file.
  • edk2.git-ovmf-x64-0-20170127.b2441.g7fcb735.noarch.rpm
Use this file from the rpm package for your bios in the Mint startup script.
  • OVMF_VARS-with-csm.fd
Seabios should already be installed as it's a dependency for KVM or Qemu if i'm not mistaken. So you just have to comment out the two OVMF bios lines in your script and add the following.

Code: Select all

-bios /usr/share/qemu/bios.bin
or

Code: Select all

-bios /usr/share/seabios/bios.bin
Incase you get a bios.bin not found error. There's a bug in one of the repo version of seabios that doesn't create the needed sym links.

MikeF
Level 1
Level 1
Posts: 3
Joined: Wed Feb 01, 2017 6:13 pm

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by MikeF » Wed Feb 01, 2017 9:39 pm

Thanks for the suggestion.

I booted using the linux mint install disk and kept the drive pointing at the mint.img

Once Mint boots the Disks utility sees the drive and its first partition is EFI

Model QEMU HARDDISK (2.5+)
Size 54 GB
Serial Number QM00001
Volumes

Device /dev/sda1
Partition Type EFI System
Contents FAT (32-bit version)

Device /dev/sda2
Partition Type Linux Filesystem

Device /dev/sda3
Partition Type Linux Swap

I went ahead and tried the seabios anyway and that does not recognize the mint.img file as a bootable filesystem either.

My .sh file is below. It is the same file I use to boot the Windows 10 vm, I just point it ar a different -drive .img file.

It boots the linux mint CD fine and the Disks utility sees the mint.img disk (as described above).

If I take out the -drive line for the CD it fails to boot from the mint.img

Code: Select all

#!/bin/bash

vmname="linuxmint181vm"

# use pulseaudio
export QEMU_AUDIO_DRV=pa
export QEMU_PA_SAMPLES=8192
export QEMU_AUDIO_TIMER_PERIOD=99
export QEMU_PA_SERVER=/run/user/1000/pulse/native

cp /usr/share/OVMF/OVMF_VARS.fd /tmp/my_vars.fd

qemu-system-x86_64 \
  -name $vmname,process=$vmname \
  -machine type=q35,accel=kvm \
  -cpu host,kvm=off \
  -smp 4,sockets=1,cores=2,threads=2 \
  -enable-kvm \
  -m 8G \
  -mem-path /run/hugepages/kvm \
  -mem-prealloc \
  -balloon none \
  -rtc clock=host,base=localtime \
  -vga none \
  -nographic \
  -serial none \
  -parallel none \
  -soundhw hda \
  -usb -usbdevice host:413c:2105 -usbdevice host:046d:c401 \
  -device vfio-pci,host=01:00.0,multifunction=on \
  -device vfio-pci,host=01:00.1 \
  -drive if=pflash,format=raw,readonly,file=/usr/share/OVMF/OVMF_CODE.fd \
  -drive if=pflash,format=raw,file=/tmp/my_vars.fd \
  -boot order=dc \
  -drive id=disk0,cache=none,format=raw,file=/media/user/linuxmint181.img \
  -drive file=/home/mednum/linuxmint-18.1-mate-64bit.iso,id=isocd,format=raw,if=none -device ide-cd,bus=i
de.1,drive=isocd \
  -netdev type=tap,id=net0,ifname=tap0,vhost=on \
  -device virtio-net-pci,netdev=net0,mac=00:16:3e:00:01:01

odtech
Level 2
Level 2
Posts: 95
Joined: Sat Jan 21, 2017 9:25 am

Re: HOW-TO make dual-boot obsolete using kvm VGA passthrough

Post by odtech » Wed Feb 01, 2017 11:20 pm

MikeF wrote:Thanks for the suggestion.

I booted using the linux mint install disk and kept the drive pointing at the mint.img

Once Mint boots the Disks utility sees the drive and its first partition is EFI

Model QEMU HARDDISK (2.5+)
Size 54 GB
Serial Number QM00001
Volumes

Device /dev/sda1
Partition Type EFI System
Contents FAT (32-bit version)

Device /dev/sda2
Partition Type Linux Filesystem

Device /dev/sda3
Partition Type Linux Swap

I went ahead and tried the seabios anyway and that does not recognize the mint.img file as a bootable filesystem either.

My .sh file is below. It is the same file I use to boot the Windows 10 vm, I just point it ar a different -drive .img file.

It boots the linux mint CD fine and the Disks utility sees the mint.img disk (as described above).

If I take out the -drive line for the CD it fails to boot from the mint.img

Code: Select all

#!/bin/bash

vmname="linuxmint181vm"

# use pulseaudio
export QEMU_AUDIO_DRV=pa
export QEMU_PA_SAMPLES=8192
export QEMU_AUDIO_TIMER_PERIOD=99
export QEMU_PA_SERVER=/run/user/1000/pulse/native

cp /usr/share/OVMF/OVMF_VARS.fd /tmp/my_vars.fd

qemu-system-x86_64 \
  -name $vmname,process=$vmname \
  -machine type=q35,accel=kvm \
  -cpu host,kvm=off \
  -smp 4,sockets=1,cores=2,threads=2 \
  -enable-kvm \
  -m 8G \
  -mem-path /run/hugepages/kvm \
  -mem-prealloc \
  -balloon none \
  -rtc clock=host,base=localtime \
  -vga none \
  -nographic \
  -serial none \
  -parallel none \
  -soundhw hda \
  -usb -usbdevice host:413c:2105 -usbdevice host:046d:c401 \
  -device vfio-pci,host=01:00.0,multifunction=on \
  -device vfio-pci,host=01:00.1 \
  -drive if=pflash,format=raw,readonly,file=/usr/share/OVMF/OVMF_CODE.fd \
  -drive if=pflash,format=raw,file=/tmp/my_vars.fd \
  -boot order=dc \
  -drive id=disk0,cache=none,format=raw,file=/media/user/linuxmint181.img \
  -drive file=/home/mednum/linuxmint-18.1-mate-64bit.iso,id=isocd,format=raw,if=none -device ide-cd,bus=i
de.1,drive=isocd \
  -netdev type=tap,id=net0,ifname=tap0,vhost=on \
  -device virtio-net-pci,netdev=net0,mac=00:16:3e:00:01:01
Powerhouse mentions in his guide having to use the "pure-efi" OVMF file from the same link i posted when he tested a Mint VM so maybe the repo version of OVMF does not play nice with Mint in a VM. I suggest still downloading that rpm package and try with below version.
  • OVMF-pure-efi.fd
If it doesn't work then maybe try doing a fresh install of Mint but after that i'd suspect a grub issue. I'm not sure how you would check if grub was installed and working after the fact though.

Edit: Remember to edit both the copy command and your actualy bios lines in the Mint script. The copy part especialy will overwrite the my_vars.fd file in the temp folder which might bork you windows vm. I did a reinstall of Mint host when i installed my ssd and using exactly the same packages as before i had to redo my Windows guest.

Post Reply

Return to “Virtualization”