16.12.2014 - removed call for virtualization forum, added link to KVM Ubuntu tutorial
05.07.2014 - added test for IOMMU support and possible solutions to step 1
05.05.2014 - notes on Xen versus KVM
25.04.2014 - remarks on Ubuntu 14.04 final and disappointing VGA passthrough result with AMD HD 7770
14.04.2014 - new website for signed GPLPV drivers with latest drivers
11.04.2014 - remark on Ubuntu 14.04 / Linux Mint 17 and improvements in VGA passthrough
10.04.2014 - link to tutorial on soft-modding Nvidia GeForce cards to Quadro models
08.04.2014 - update on AMD GPU compatibility, link to work around if X doesn't start with Xen
04.04.2014 - update on GPLPV signed drivers
03.04.2014 - updated network setup instructions
14.03.2014 - link to post on passthrough tests with AMD / ATI Radeon 7770 and different kernel/Xen releases, using xl and xm toolstack
12.03.2014 - instructions for switching to xl toolstack (step 22 b), clarifications on Windows 8.1 and xl in Part 2 and step 3
11.03.2014 - note on using xl toolstack and external link to tutorial, note on successful Windows 8.1 installation with xl toolstack
10.03.2014 - link to explanation on why to use secondary passthrough (under Glossary), tidied up Hardware requirements
03.03.2014 - link to post on another way to blacklist "radeon" driver (step 15)
27.02.2014 - reference Ubuntu, add link to Fedora guide; BIOS settings for VT-d/IOMMU; reminder to select to boot with IGP, changed order of steps 1 and 2
25.02.2014 - added audio section (step 26)
14.02.2014 - added /etc/modeprobe.d/blacklist.conf option to blacklist video driver under step 17
05.02.2014 - shortened version of http://forums.linuxmint.com/viewtopic.php?f=42&t=112013&start=360#p818716 - for LM 16 only! Truncated change log
Abstract: This how-to is about running Microsoft Windows concurrently with Linux Mint on a Xen hypervisor, with FULL graphics acceleration for the Windows guest system.
There are special hardware requirements for this to work. Please make sure your hardware meets these requirements.
Running Windows as a guest in Linux is nothing new. But how about native graphics and near-native CPU performance? Here a little teaser - this is my Windows Experience Index when running Windows on top of Linux / Xen hypervisor, while Linux is fully functional at the same time:
WEI using GPLPV block device driver and VGA graphics passthrough!
For a long time I wanted to get rid of dual-booting Linux and Windows. But my photo editing software would only run on Windows (or Mac). So I started to look into virtualisation. I needed true graphics acceleration from within Windows, as well as direct access to some USB ports, something VirtualBox cannot provide as of this writing!
I chose Xen since at the time it was more mature. Today I would consider KVM as well, as in some respects it is easier to implement. More on that below.
The following is for people who want to run Linux and Windows at the same time - without the need for dual-boot - with both Linux and Windows having native or near native graphics acceleration and performance.
Xen employs two related concepts that make it work: PCI passthrough and VGA passthrough.
Here a few terms used in the how-to:
domU: This is Xen terminology for a Virtual Machine or VM. In our case it will be Windows.
dom0: This is a special Virtual Machine with administrative privileges. In our case this will be Linux Mint. All other VMs (or domUs in Xen talk) are started and controlled from the dom0.
PCI passthrough (or passthru): This is a technique which allows the dom0 to pass through a PCI device to the guest domU. The guest OS - in our case Windows - then uses its own hardware driver to access the device directly.
VGA passthrough: This is a specialized form of PCI passthrough for graphics cards / GPU. These devices are more complex and so is the interaction with them. Fortunately, using the right hardware together with Xen should take care of that. There are two forms of VGA passthrough:
VGA passthrough to GPU as primary adapter: The guest system (domU) boots and uses the passed through graphics card as its primary or only graphics adapter.
VGA passthrough to GPU as secondary adapter: The guest system first boots using the Qemu-dm virtual Cirrus adapter as the primary graphics adapter (this is where you see the BIOS screen and the Windows boot logo). After installation of the graphics driver under Windows and subsequent reboot of the domU, the system will switch to the secondary graphics adapter at some stage of it's boot process. This is the technique described here in this How-To. For an explanation on why, see here: http://wiki.xenproject.org/wiki/GSoC_2013#gpu-passthrough.
First of all you need to check the hardware. This will ONLY work when the hardware requirements below are met, if not, forget this tutorial !!!
- • CPU with VT-d support (Intel terminology) - see http://ark.intel.com/search/advanced/?s=t&VTD=true. When using an AMD CPU, it must support AMD-Vi, also known as IOMMU - the following chipsets support IOMMU: SR5690/SR5670 server chipsets, AMD 890FX, 990FX, 990X and 970 desktop chipsets. See http://wiki.xen.org/wiki/VTdHowTo for more information.
- • Motherboard and BIOS version with option to enable VT-d or IOMMU (beware, some vendors have buggy BIOS versions) - see http://wiki.xen.org/wiki/VTd_HowTo and http://www.overclock.net/t/1338063/vt-d-compatible-motherboards for compatible motherboards!
- • A dedicated graphics card for your Windows guest that supports Xen VGA passthrough - see http://wiki.xen.org/wiki/Xen_VGA_Passthrough_Tested_Adapters and http://www.overclock.net/t/1307834/xen-vga-passthrough-compatible-graphics-adapters for compatible graphics cards! Until now (10.4.2014) I recommended AMD cards for better compatibility with VGA passthrough. Recent changes in the Linux kernel and/or Xen, however, had a negative impact on VGA passthrough with AMD cards - more about that below.
Regarding AMD cards, read my post here. There is one acronym that describes it well: YMMV (your mileage may vary). Some of the recent changes in either Xen and/or the Linux kernel seem to affect VGA passthrough adversely, at least for some AMD cards. I have tested both the AMD HD 7770 and the AMD HD 6450 (see here) and only Linux Mint 13 with the very old 3.2...-23 kernel and Xen 4.1.2 would work. Newer kernels and/or Xen releases would report "error 22" for the xm toolstack, or libxl errors for the xl toolstack. These phenomena might be limited to the above mentioned cards.
As for Nvidia cards, only the "Multi-OS" Nvidia cards are suitable for VGA passthrough - see http://www.nvidia.com/object/sli_multi_os.html. This means only the outrageously expensive Quadro 2000, 4000, ... cards and some even more expensive models are suitable. But, read on.
Some Nvidia GeForce cards can be soft-modded into Quadro cards - for more on that, see http://www.altechnative.net. See also further below on hard-modding Nvidia cards.
- • Enough RAM - the more the better, though 8GB should do in most cases
- • Linux Mint 16 or later (or Ubuntu 13.10 or later), 64bit version
- • Windows 7, Windows 8 (see note on Windows 8.1 below), or Windows XP
Check the above links! If you have a compatible CPU, motherboard (and BIOS), and graphics card you can skip to the how-to below.
Note: Some Asus motherboards - specifically those for AMD CPUs with AMD 990FX and 970 chipsets - have a faulty BIOS. This has been reported here http://www.gossamer-threads.com/lists/xen/users/283968?do=post_view_threaded#283968. A workaround is to install an older version of Xen using the "force" option in Synaptic. For more see here http://forums.linuxmint.com/viewtopic.php?f=42&t=112013&start=200#p764039.
EDIT: Asus has fixed it for at least two boards - see http://xen.1045712.n5.nabble.com/Xen-IOMMU-disabled-due-to-IVRS-table-Blah-blah-blah-td5716461i20.html and http://forums.linuxmint.com/viewtopic.php?f=42&t=112013&start=200#p764822 (thanks GizmoChicken!)
EDIT 2: Asus has released a new BIOS version v2005 that reportedly fixes the issues with some of its AMD CPU boards. For more details, see GizmoChicken's post here: http://forums.linuxmint.com/viewtopic.php?f=42&t=112013&start=220#p782159
Ivybridge CPUs on Asus Rampage and Sabertooth X79 boards: Asus Rampage and Sabertooth X79 boards currently don't support VT-d when used with the new generation Ivybridge CPUs. The good news is that Asus R&D is working on it and has released a beta BIOS to fix VT-d in one of their Rampage boards. A BIOS release is expected for the Asus Sabertooth X79 that is supposed to fix the issue. See here: http://rog.asus.com/forum/showthread.php?40309-No-VT-d-support-for-IVB-E-on-R4E-boards.
Here is my hardware list for reference:
- i7 3930K CPU with C2 stepping! (If you buy new, it should be with C2 stepping, but check for spec code = SR0KY!)
- Asus Sabertooth X79 motherboard with BIOS version 4608 (latest BIOS as of 09.01.2014)
- AMD 7770 graphics card as primary graphics adapter for Linux dom0 (before that I had an AMD 6450)
- Nvidia Quadro 2000 graphics card with "Multi-OS" feature (Nvidia terminology) as secondary graphics adapter for Windows domU - see below for more on that!
- USB KVM switch to connect keyboard/mouse to two different USB hosts and easily switch between them
- A graphics display with two DVI or HDMI ports and the ability to switch between the ports (e.g. "input" button)
- Enough RAM (I have 32GB) and disk space (I use a 120GB SSD and several 2TB hard drives, less will do)
Most Nvidia graphics cards are tricky to pass through to the guest OS (Windows domU), only the "Multi-OS" versions are designed to work out of the box. You could try to patch and rebuild the Linux kernel and Xen, and there are some instructions on how to do it, but this is way more demanding than getting my above hardware to work. I have failed with a Nvidia Quadro 600 card, though it may be possible. Most AMD (ATI) graphics cards are easier to get working with VGA passthrough. If in doubt, get one of the tested AMD or Nvidia graphics adapters that are listed under the above mentioned links!
For the very adventurous, there is a way to modify Nvidia graphics adapters to register as professional Nvidia cards and thus enable VGA passthrough. See here: http://www.eevblog.com/forum/projects/hacking-nvidia-cards-into-their-professional-counterparts/ and http://www.eevblog.com/forum/projects/hacking-nvidia-cards-into-their-professional-counterparts/msg207550/#msg207550. Warning: If you do modify your hardware, you will loose all manufacturer warranty and take a big chance of turning your expensive VGA card into a brick! I won't take any responsibility for that. You have been warned.
If you plan on building a new PC, this is the time to check for compatible hardware!
Ubuntu 14.04 and Linux Mint 17
I tested Ubuntu 14.04 Beta2 with Xen 4.4, which will form the basis for Linux Mint 17. The new Xen release together with a 3.13 kernel appear to bring some improvements for VGA passthrough, in particular with regard to AMD graphic cards. I managed to pass through an AMD 6450 using the xl toolstack with no real issues, but testing the AMD HD 7770 with the official Ubuntu 14.04 release showed that the guest reboot problem has not been fixed! Sadly this means that as for AMD graphics cards passthrough your milage may vary.
Although written for Linux Mint 16, the following how-to should work equally well with Ubuntu 13.10 and probably other Ubuntu-based distributions. If you prefer a RedHat-based distribution like Fedora, have a look here: http://www.overclock.net/t/1205216/guide-create-a-gaming-virtual-machine.
Xen versus KVM
KVM has come a long way, even or especially with regard to VGA passthrough. I've tested KVM on my rig, passing through a AMD Radeon HD 7770 to a Windows 7 guest and although KVM throws in a lot more configuration options, it actually worked. More surprising was that KVM provides primary passthrough, without the need to boot with a Cirrus virtual VGA adapter.
But there are as well downsides to KVM. The most notable is the fact that benchmarking software such as Passmark PerformanceTest and PCmark complain about missing CPU features, and Passmark would only run in "SAFEMODE". Actually, running Passmark PerformanceTest in the regular mode crashes the Windows guest (BSOD). As a result, Passmark will only provide a summary result without the CPU tests, not something to compare with other PCs or VMs. This happens with PerformanceTest 8 as well as with the old release 7. In KVM I tried both the cpu=host as well as the cpu=SandyBridge settings, neither prevented Passmark from BSODing. Still, I can't rule out user error, as I'm not yet familiar with KVM.
Xen doesn't have these issues with benchmarking software, at least not in my setup. Another huge advantage to Xen is its documentation. With KVM even the simplest configuration task turns into a research project. Most KVM documentation recommends you to use virt-manager, virt-install, or libvirt, but those tools are far from usable for VGA passthrough (although there might be some tricks to make them work). virt-manager hides away the configuration from the user (I never found a config file), other tools use XML files to make a simple task more complicated. I'm sure those who developed these tools had a reason for doing what they did, but it certainly takes a little getting used to the way KVM is configured and monitored. Lack of useful management tools combined with cryptic configuration parameters and the general void of useful and up-to-date documentation can make KVM more challenging. But once these hurdles are overcome the setup of a Windows guest with VGA passthrough is easy.
For those interested in trying KVM, here is a Ubuntu-based KVM tutorial I ran across (I didn't try it, though): http://www.pugetsystems.com/labs/articles/Multiheaded-NVIDIA-Gaming-using-Ubuntu-14-04-KVM-585/
I will do some more testing with KVM and hopefully write a how-to for Linux Mint. For the time being, below is the Xen how-to.
To be continued...