Xen VGA passthrough how-to for Linux Mint 17.3 - release 3.4
02.10.2016 - rewrite for Linux Mint 17.3, some fixes
03.12.2015 - installed Linux Mint 17.3 (fresh install) with Xen 4.4
07.09.2015 - added link to another KVM tutorial; domU (VM) memory limit of 6G for some systems
02.09.2015 - added warning regarding removal of network-manager
30.05.2015 - added link to thread on Ubuntu forum for AMD CPU configuration options (see step 12), updated info on Windows PV drivers
01.04.2015 - added instructions for Linux Mint 17.1 under Step 2 of the how-to
01.04.2015 - KVM versus Xen - removed pros and cons and added link to tutorial on Archlinux forum
16.12.2014 - removed call for virtualization forum, added link to KVM Ubuntu tutorial
05.07.2014 - added test for IOMMU support and possible solutions to step 1
05.05.2014 - notes on Xen versus KVM
25.04.2014 - remarks on Ubuntu 14.04 final and disappointing VGA passthrough result with AMD HD 7770
14.04.2014 - new website for signed GPLPV drivers with latest drivers
11.04.2014 - remark on Ubuntu 14.04 / Linux Mint 17 and improvements in VGA passthrough
10.04.2014 - link to tutorial on soft-modding Nvidia GeForce cards to Quadro models
08.04.2014 - update on AMD GPU compatibility, link to work around if X doesn't start with Xen
04.04.2014 - update on GPLPV signed drivers
03.04.2014 - updated network setup instructions
14.03.2014 - link to post on passthrough tests with AMD / ATI Radeon 7770 and different kernel/Xen releases, using xl and xm toolstack
12.03.2014 - instructions for switching to xl toolstack (step 22 b), clarifications on Windows 8.1 and xl in Part 2 and step 3
11.03.2014 - note on using xl toolstack and external link to tutorial, note on successful Windows 8.1 installation with xl toolstack
10.03.2014 - link to explanation on why to use secondary passthrough (under Glossary), tidied up Hardware requirements
03.03.2014 - link to post on another way to blacklist "radeon" driver (step 15)
27.02.2014 - reference Ubuntu, add link to Fedora guide; BIOS settings for VT-d/IOMMU; reminder to select to boot with IGP, changed order of steps 1 and 2
25.02.2014 - added audio section (step 26)
14.02.2014 - added /etc/modeprobe.d/blacklist.conf option to blacklist video driver under step 17
05.02.2014 - shortened version of http://forums.linuxmint.com/viewtopic.php?f=42&t=112013&start=360#p818716 - for LM 16 only! Truncated change log
Abstract: This how-to is about running Microsoft Windows concurrently with Linux Mint on a Xen hypervisor, with FULL graphics acceleration for the Windows guest system.
There are special hardware requirements for this to work. Please make sure your hardware meets these requirements.
Running Windows as a guest in Linux is nothing new. But how about native graphics and near-native CPU performance? Here a little teaser - this is my Windows Experience Index when running Windows on top of Linux / Xen hypervisor, while Linux is fully functional at the same time:
WEI using GPLPV block device driver and VGA graphics passthrough!
For a long time I wanted to get rid of dual-booting Linux and Windows. But my photo editing software would only run on Windows (or Mac). So I started to look into virtualisation. I needed true graphics acceleration from within Windows, as well as direct access to some USB ports, something VirtualBox cannot provide as of this writing!
I chose Xen since at the time it was more mature. Today I would consider KVM as well, as in some respects it is easier to implement. More on that below.
The following is for people who want to run Linux and Windows at the same time - without the need for dual-boot - with both Linux and Windows having native or near native graphics acceleration and performance.
Xen employs two related concepts that make it work: PCI passthrough and VGA passthrough.
Here a few terms used in the how-to:
domU: This is Xen terminology for a Virtual Machine or VM. In our case it will be Windows.
dom0: This is a special Virtual Machine with administrative privileges. In our case this will be Linux Mint. All other VMs (or domUs in Xen talk) are started and controlled from the dom0.
PCI passthrough (or passthru): This is a technique which allows the dom0 to pass through a PCI device to the guest domU. The guest OS - in our case Windows - then uses its own hardware driver to access the device directly.
VGA passthrough: This is a specialized form of PCI passthrough for graphics cards / GPU. These devices are more complex and so is the interaction with them. Fortunately, using the right hardware together with Xen should take care of that. There are two forms of VGA passthrough:
VGA passthrough to GPU as primary adapter: The guest system (domU) boots and uses the passed through graphics card as its primary or only graphics adapter.
VGA passthrough to GPU as secondary adapter: The guest system first boots using the Qemu-dm virtual Cirrus adapter as the primary graphics adapter (this is where you see the BIOS screen and the Windows boot logo). After installation of the graphics driver under Windows and subsequent reboot of the domU, the system will switch to the secondary graphics adapter at some stage of it's boot process. This is the technique described here in this How-To. For an explanation on why, see here: http://wiki.xenproject.org/wiki/GSoC_2013#gpu-passthrough.
First of all you need to check the hardware. This will ONLY work when the hardware requirements below are met, if not, forget this tutorial !!!
- • CPU with VT-d support (Intel terminology) - see http://ark.intel.com/search/advanced/?s=t&VTD=true. When using an AMD CPU, it must support AMD-Vi, also known as IOMMU - the following chipsets support IOMMU: SR5690/SR5670 server chipsets, AMD 890FX, 990FX, 990X and 970 desktop chipsets. See http://wiki.xen.org/wiki/VTdHowTo for more information.
- • Motherboard and BIOS version with option to enable VT-d or IOMMU (beware, some vendors have buggy BIOS versions) - see http://wiki.xen.org/wiki/VTd_HowTo and http://www.overclock.net/t/1338063/vt-d-compatible-motherboards for compatible motherboards!
- • A dedicated graphics card for your Windows guest that supports Xen VGA passthrough - see http://wiki.xen.org/wiki/Xen_VGA_Passthrough_Tested_Adapters and http://www.overclock.net/t/1307834/xen-vga-passthrough-compatible-graphics-adapters for compatible graphics cards! Until now (10.4.2014) I recommended AMD cards for better compatibility with VGA passthrough. Recent changes in the Linux kernel and/or Xen, however, had a negative impact on VGA passthrough with AMD cards - more about that below.
Regarding AMD cards, read my post here. There is one acronym that describes it well: YMMV (your mileage may vary). Some of the recent changes in either Xen and/or the Linux kernel seem to affect VGA passthrough adversely, at least for some AMD cards. I have tested both the AMD HD 7770 and the AMD HD 6450 (see here) and only Linux Mint 13 with the very old 3.2...-23 kernel and Xen 4.1.2 would work. Newer kernels and/or Xen releases would report "error 22" for the xm toolstack, or libxl errors for the xl toolstack. These phenomena might be limited to the above mentioned cards.
As for Nvidia cards, only the "Multi-OS" Nvidia cards are suitable for VGA passthrough - see http://www.nvidia.com/object/sli_multi_os.html. This means only the outrageously expensive Quadro 2000, 4000, ... cards and some even more expensive models are suitable. But, read on.
Some Nvidia GeForce cards can be soft-modded into Quadro cards - for more on that, see http://www.altechnative.net. See also further below on hard-modding Nvidia cards.
- • Enough RAM - the more the better, though 8GB should do in most cases
- • Linux Mint 17.3 or later (or Ubuntu 14.04 or later), 64bit version
- • Windows 7, Windows 8, or Windows 10
Check the above links! If you have a compatible CPU, motherboard (and BIOS), and graphics card you can skip to the how-to below.
Here is my hardware list for reference:
- i7 3930K CPU with C2 stepping! (If you buy new, it should be with C2 stepping, but check for spec code = SR0KY!)
- Asus Sabertooth X79 motherboard with BIOS version 4608 (latest BIOS as of 09.01.2014)
- AMD 7770 graphics card as primary graphics adapter for Linux dom0 (before that I had an AMD 6450)
- Nvidia Quadro 2000 graphics card with "Multi-OS" feature (Nvidia terminology) as secondary graphics adapter for Windows domU - see below for more on that!
- USB KVM switch to connect keyboard/mouse to two different USB hosts and easily switch between them
- A graphics display with two DVI or HDMI ports and the ability to switch between the ports (e.g. "input" button)
- Enough RAM (I have 32GB) and disk space (I use a 120GB SSD and several 2TB hard drives, less will do)
Most Nvidia graphics cards are tricky to pass through to the guest OS (Windows domU), only the "Multi-OS" versions are designed to work out of the box. You could try to patch and rebuild the Linux kernel and Xen, and there are some instructions on how to do it, but this is way more demanding than getting my above hardware to work. I have failed with a Nvidia Quadro 600 card, though it may be possible. Most AMD (ATI) graphics cards are easier to get working with VGA passthrough. If in doubt, get one of the tested AMD or Nvidia graphics adapters that are listed under the above mentioned links!
For the very adventurous, there is a way to modify Nvidia graphics adapters to register as professional Nvidia cards and thus enable VGA passthrough. See here: http://www.eevblog.com/forum/projects/hacking-nvidia-cards-into-their-professional-counterparts/ and http://www.eevblog.com/forum/projects/hacking-nvidia-cards-into-their-professional-counterparts/msg207550/#msg207550. Warning: If you do modify your hardware, you will loose all manufacturer warranty and take a big chance of turning your expensive VGA card into a brick! I won't take any responsibility for that. You have been warned.
If you plan on building a new PC, this is the time to check for compatible hardware!
Xen versus KVM
KVM and Xen have become equal contestants. In some way it seems that kvm is making quick progress. I posted my own kvm tutorial here. While it seems easier to support some graphics card under KVM, in particular Nvidia cards, performance tweaking can be a real nightmare. Xen presents its own challenges, but at least once you get it working, performance should not be an issue.
Triggered by a hardware upgrade where I replaced an AMD graphics card with an Nvidia GTX 970, I have recently changed my system from Xen to KVM and run now Windows 10 using KVM. Each system has its advantages and disadvantages.
To be continued...