HOW-TO make dual-boot obsolete using XEN VGA passthrough

Questions about virtualization software
Forum rules
Before you post read how to get help. Topics in this forum are automatically closed 6 months after creation.
Locked
powerhouse
Level 6
Level 6
Posts: 1138
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by powerhouse »

05.01.2016: Please note: I've posted a new how-to on VGA passthrough using KVM - see here: http://forums.linuxmint.com/viewtopic.p ... 1&t=212692

Xen VGA passthrough how-to for Linux Mint 17.3 - release 3.5
Change log:
13.07.2017 - added link to latest Windows PV drivers
02.10.2016 - rewrite for Linux Mint 17.3, some fixes
03.12.2015 - installed Linux Mint 17.3 (fresh install) with Xen 4.4
07.09.2015 - added link to another KVM tutorial; domU (VM) memory limit of 6G for some systems
02.09.2015 - added warning regarding removal of network-manager
30.05.2015 - added link to thread on Ubuntu forum for AMD CPU configuration options (see step 12), updated info on Windows PV drivers
01.04.2015 - added instructions for Linux Mint 17.1 under Step 2 of the how-to
01.04.2015 - KVM versus Xen - removed pros and cons and added link to tutorial on Archlinux forum
16.12.2014 - removed call for virtualization forum, added link to KVM Ubuntu tutorial
05.07.2014 - added test for IOMMU support and possible solutions to step 1
05.05.2014 - notes on Xen versus KVM
25.04.2014 - remarks on Ubuntu 14.04 final and disappointing VGA passthrough result with AMD HD 7770
14.04.2014 - new website for signed GPLPV drivers with latest drivers
11.04.2014 - remark on Ubuntu 14.04 / Linux Mint 17 and improvements in VGA passthrough
10.04.2014 - link to tutorial on soft-modding Nvidia GeForce cards to Quadro models
08.04.2014 - update on AMD GPU compatibility, link to work around if X doesn't start with Xen
04.04.2014 - update on GPLPV signed drivers
03.04.2014 - updated network setup instructions
14.03.2014 - link to post on passthrough tests with AMD / ATI Radeon 7770 and different kernel/Xen releases, using xl and xm toolstack
12.03.2014 - instructions for switching to xl toolstack (step 22 b), clarifications on Windows 8.1 and xl in Part 2 and step 3
11.03.2014 - note on using xl toolstack and external link to tutorial, note on successful Windows 8.1 installation with xl toolstack
10.03.2014 - link to explanation on why to use secondary passthrough (under Glossary), tidied up Hardware requirements
03.03.2014 - link to post on another way to blacklist "radeon" driver (step 15)
27.02.2014 - reference Ubuntu, add link to Fedora guide; BIOS settings for VT-d/IOMMU; reminder to select to boot with IGP, changed order of steps 1 and 2
25.02.2014 - added audio section (step 26)
14.02.2014 - added /etc/modeprobe.d/blacklist.conf option to blacklist video driver under step 17
05.02.2014 - shortened version of http://forums.linuxmint.com/viewtopic.p ... 60#p818716 - for LM 16 only! Truncated change log


Abstract: This how-to is about running Microsoft Windows concurrently with Linux Mint on a Xen hypervisor, with FULL graphics acceleration for the Windows guest system.

There are special hardware requirements for this to work. Please make sure your hardware meets these requirements.

Running Windows as a guest in Linux is nothing new. But how about native graphics and near-native CPU performance? Here a little teaser - this is my Windows Experience Index when running Windows on top of Linux / Xen hypervisor, while Linux is fully functional at the same time:
Win7_VM_on_Xen.jpg
WEI using GPLPV block device driver and VGA graphics passthrough!


Introduction

For a long time I wanted to get rid of dual-booting Linux and Windows. But my photo editing software would only run on Windows (or Mac). So I started to look into virtualisation. I needed true graphics acceleration from within Windows, as well as direct access to some USB ports, something VirtualBox cannot provide as of this writing!

I chose Xen since at the time it was more mature. Today I would consider KVM as well, as in some respects it is easier to implement. More on that below.

The following is for people who want to run Linux and Windows at the same time - without the need for dual-boot - with both Linux and Windows having native or near native graphics acceleration and performance.

Xen employs two related concepts that make it work: PCI passthrough and VGA passthrough.


Glossary

Here a few terms used in the how-to:
domU: This is Xen terminology for a Virtual Machine or VM. In our case it will be Windows.
dom0: This is a special Virtual Machine with administrative privileges. In our case this will be Linux Mint. All other VMs (or domUs in Xen talk) are started and controlled from the dom0.
PCI passthrough (or passthru): This is a technique which allows the dom0 to pass through a PCI device to the guest domU. The guest OS - in our case Windows - then uses its own hardware driver to access the device directly.
VGA passthrough: This is a specialized form of PCI passthrough for graphics cards / GPU. These devices are more complex and so is the interaction with them. Fortunately, using the right hardware together with Xen should take care of that. There are two forms of VGA passthrough:
VGA passthrough to GPU as primary adapter: The guest system (domU) boots and uses the passed through graphics card as its primary or only graphics adapter.
VGA passthrough to GPU as secondary adapter: The guest system first boots using the Qemu-dm virtual Cirrus adapter as the primary graphics adapter (this is where you see the BIOS screen and the Windows boot logo). After installation of the graphics driver under Windows and subsequent reboot of the domU, the system will switch to the secondary graphics adapter at some stage of it's boot process. This is the technique described here in this How-To. For an explanation on why, see here: http://wiki.xenproject.org/wiki/GSoC_20 ... assthrough.


Hardware requirements

First of all you need to check the hardware. This will ONLY work when the hardware requirements below are met, if not, forget this tutorial !!!

Hardware requirements:

  1. • CPU with VT-d support (Intel terminology) - see http://ark.intel.com/search/advanced/?s=t&VTD=true. When using an AMD CPU, it must support AMD-Vi, also known as IOMMU - the following chipsets support IOMMU: SR5690/SR5670 server chipsets, AMD 890FX, 990FX, 990X and 970 desktop chipsets. See http://wiki.xen.org/wiki/VTdHowTo for more information.
  • • A dedicated graphics card for your Windows guest that supports Xen VGA passthrough - see http://wiki.xen.org/wiki/Xen_VGA_Passth ... d_Adapters and http://www.overclock.net/t/1307834/xen- ... s-adapters for compatible graphics cards! Until now (10.4.2014) I recommended AMD cards for better compatibility with VGA passthrough. Recent changes in the Linux kernel and/or Xen, however, had a negative impact on VGA passthrough with AMD cards - more about that below.
    :arrow: Regarding AMD cards, read my post here. There is one acronym that describes it well: YMMV (your mileage may vary). Some of the recent changes in either Xen and/or the Linux kernel seem to affect VGA passthrough adversely, at least for some AMD cards. I have tested both the AMD HD 7770 and the AMD HD 6450 (see here) and only Linux Mint 13 with the very old 3.2...-23 kernel and Xen 4.1.2 would work. Newer kernels and/or Xen releases would report "error 22" for the xm toolstack, or libxl errors for the xl toolstack. These phenomena might be limited to the above mentioned cards.
    As for Nvidia cards, only the "Multi-OS" Nvidia cards are suitable for VGA passthrough - see http://www.nvidia.com/object/sli_multi_os.html. This means only the outrageously expensive Quadro 2000, 4000, ... cards and some even more expensive models are suitable. But, read on.
    :arrow: Some Nvidia GeForce cards can be soft-modded into Quadro cards - for more on that, see http://www.altechnative.net. See also further below on hard-modding Nvidia cards.
  • • Enough RAM - the more the better, though 8GB should do in most cases
  • • Linux Mint 17.3 or later (or Ubuntu 14.04 or later), 64bit version
  • • Windows 7, Windows 8, or Windows 10

Check the above links! If you have a compatible CPU, motherboard (and BIOS), and graphics card you can skip to the how-to below.

Here is my hardware list for reference:

- i7 3930K CPU with C2 stepping! (If you buy new, it should be with C2 stepping, but check for spec code = SR0KY!)
- Asus Sabertooth X79 motherboard with BIOS version 4608 (latest BIOS as of 09.01.2014)
- AMD 7770 graphics card as primary graphics adapter for Linux dom0 (before that I had an AMD 6450)
- Nvidia Quadro 2000 graphics card with "Multi-OS" feature (Nvidia terminology) as secondary graphics adapter for Windows domU - see below for more on that!
- USB KVM switch to connect keyboard/mouse to two different USB hosts and easily switch between them
- A graphics display with two DVI or HDMI ports and the ability to switch between the ports (e.g. "input" button)
- Enough RAM (I have 32GB) and disk space (I use a 120GB SSD and several 2TB hard drives, less will do)

Most Nvidia graphics cards are tricky to pass through to the guest OS (Windows domU), only the "Multi-OS" versions are designed to work out of the box. You could try to patch and rebuild the Linux kernel and Xen, and there are some instructions on how to do it, but this is way more demanding than getting my above hardware to work. I have failed with a Nvidia Quadro 600 card, though it may be possible. Most AMD (ATI) graphics cards are easier to get working with VGA passthrough. If in doubt, get one of the tested AMD or Nvidia graphics adapters that are listed under the above mentioned links!

For the very adventurous, there is a way to modify Nvidia graphics adapters to register as professional Nvidia cards and thus enable VGA passthrough. See here: http://www.eevblog.com/forum/projects/h ... nterparts/ and http://www.eevblog.com/forum/projects/h ... #msg207550. Warning: If you do modify your hardware, you will loose all manufacturer warranty and take a big chance of turning your expensive VGA card into a brick! I won't take any responsibility for that. You have been warned.

If you plan on building a new PC, this is the time to check for compatible hardware!


Xen versus KVM

KVM and Xen have become equal contestants. In some way it seems that kvm is making quick progress. I posted my own kvm tutorial here. While it seems easier to support some graphics card under KVM, in particular Nvidia cards, performance tweaking can be a real nightmare. Xen presents its own challenges, but at least once you get it working, performance should not be an issue.

Triggered by a hardware upgrade where I replaced an AMD graphics card with an Nvidia GTX 970, I have recently changed my system from Xen to KVM and run now Windows 10 using KVM. Each system has its advantages and disadvantages.

To be continued...
Last edited by powerhouse on Thu Jul 13, 2017 2:35 pm, edited 127 times in total.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
powerhouse
Level 6
Level 6
Posts: 1138
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by powerhouse »

Part 2 - Prerequisites and Installation of Xen Hypervisor

If you are running Linux Mint 13 or 14, see my old how-to here: http://forums.linuxmint.com/viewtopic.p ... 60#p818716


1. Enable VT-d / IOMMU in BIOS

Boot/Reboot your PC and enter your BIOS setup. Look for an IOMMU or VT-d option (check the manual of your motherboard) and enable it. VT-d (or AMD-Vi for AMD) is an essential prerequisite to make this work!

Note: Most motherboard BIOSes have VT-d disabled by default.

:arrow: If you plan on having Linux use the IGP (integrated graphics processor) of your CPU, make sure to select it in the BIOS!

Once booted into Linux Mint, open a terminal window and check the following:
On AMD machines use:

Code: Select all

dmesg | grep AMD-Vi
The output should be similar to this:
...
AMD-Vi: Enabling IOMMU at 0000:00:00.2 cap 0x40
AMD-Vi: Lazy IO/TLB flushing enabled
AMD-Vi: Initialized for Passthrough Mode
...

On Intel machines use:

Code: Select all

dmesg | grep "Virtualization Technology for Directed I/O"
The output should this:
DMAR: Intel(R) Virtualization Technology for Directed I/O

If you do not get this output, then VT-d or AMD-Vi is not working - you need to fix that before you continue. See point 3 here for some suggestions. Also, see this post by Boggler for some suggestions regarding AMD boards and Marvell controllers.


2. Install Linux Mint 17.3 64 bit with LVM / (root) and /home partitions

To install Linux Mint 17.3 on LVM, see my how-to here: http://forums.linuxmint.com/viewtopic.p ... 42#p793203.


3. Install Xen hypervisor and utilities

After reboot into the installed Linux Mint system, update the packages using the "Update Manager". Then open a terminal window:

Code: Select all

sudo apt-get install xen-system-amd64 qemu-common qemu-keymaps bridge-utils

4. Add boot options to grub

Edit the /etc/default/grub file as root and add or change the following:

Code: Select all

GRUB_CMDLINE_XEN="iommu=1 dom0_mem=2G"
The above Xen command line tells the hypervisor to use IOMMU and gives the dom0 (Linux Mint) 2048 MB of RAM. Change the amount of RAM you give to your Linux Mint dom0 according to your needs.

While at it, comment out the GRUB_HIDDEN_TIMEOUT=0 and GRUB_HIDDEN_TIMEOUT_QUIET=true so that it looks like this:

Code: Select all

GRUB_DEFAULT=3
GRUB_TIMEOUT_STYLE=countdown
GRUB_TIMEOUT=0
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="ipv6.disable=1 quiet nomodeset"
GRUB_CMDLINE_LINUX=""
GRUB_CMDLINE_XEN="iommu=1 dom0_mem=2G"
Note: I used GRUB_DEFAULT=3 to load Xen automatically at boot. With the GRUB_HIDDEN... options commented out you will have a grub menu to select your regular Linux kernel if things go bad, or to manually select either Xen or regular boot.

Open a terminal window and run

Code: Select all

sudo update-grub

5. Reboot into the Xen hypervisor

Select the appropriate grub menu option.

:arrow: Note: If X doesn't start and you can't get a graphical login screen/desktop, see step 7 here. Another option is to deinstall the second GPU (the one you want to pass through to the guest) and go ahead with the configuration up until you need to reboot again. After shutdown and power off, install the second graphics card, turn on the power and boot into Xen.
Last edited by powerhouse on Sun Oct 02, 2016 12:54 pm, edited 50 times in total.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
powerhouse
Level 6
Level 6
Posts: 1138
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by powerhouse »

Part 3

Now that you booted into the Xen hypervisor / dom0, it's time for some configurations.


6. Adjust the network settings to use a bridge

The following terminal commands require root permission so we make it stick. Open a terminal window and enter:

Code: Select all

sudo -i
Enter:

Code: Select all

stop network-manager
to stop the network manager so we can configure the bridge and enable it.

You need to decide if you want to use a dynamic IP address for your Linux Mint dom0, or a static address. I recommend using a static IP address (the second option below), as this will allow you to easily access your Linux dom0 from another PC in your network, or from within the Windows guest.

If you want to use DHCP to automatically assign an IP address etc. to your network card, replace the content of your /etc/network/interfaces file with this:

Code: Select all

auto lo
iface lo inet loopback

auto xenbr0
iface xenbr0 inet dhcp
    bridge_ports eth0

# auto eth0
For a static IP address, use this configuration in /etc/network/interfaces:

Code: Select all

auto lo
iface lo inet loopback

auto xenbr0

iface xenbr0 inet static
# IP of Linux Mint dom0:
address 192.168.0.120
broadcast 192.168.0.255
netmask 255.255.255.0
# IP of router or modem:
gateway 192.168.0.1
# IP of the DNS name server (if you don't know, use the Google DNS server below):
dns-nameservers 8.8.8.8
bridge_ports eth0
# disable Spanning Tree Protocol - optional:
bridge_stp off
# bridge_waitport 0	# no delay before a port becomes available - optional
# bridge_fd 0	 # no forwarding delay - optional
Adopt the IP settings above to your requirements.

Now we create the bridge:

Code: Select all

brctl addbr xenbr0
Enable networking:

Code: Select all

service networking restart
and check your network configuration:

Code: Select all

ifconfig
Then test your network configuration - open Firefox and see if you have Internet access.

If all is fine, remove the "network-manager" package as this is a major pain in the neck (to be polite):

Code: Select all

apt-get remove network-manager
Warning: The step above has been reported to cause problems. See http://forums.linuxmint.com/viewtopic.p ... 1#p1000916.


7. Create missing links

Linux Mint 17.3 requires us to add some links for the xen tools to work properly:

Code: Select all

ln -s /usr/lib /usr/lib64
ln -s /usr/lib/xen-4.4 /usr/lib/xen-default

8. Determine the PCI IDs of the devices you want to pass through to the domU (guest)

Enter the following command to list your graphics adapters' PCI IDs:

Code: Select all

lspci | grep VGA
You should get something like this:
  • 01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Caicos [Radeon HD 6450]
    02:00.0 VGA compatible controller: NVIDIA Corporation GF106GL [Quadro 2000] (rev a1)
I have two graphics adapters:
  • 01:00.0 is the primary graphics adapter installed in the first PCIe 16 slot, in my case an AMD Radeon HD 6450 graphics card. This is the GPU currently used by Linux dom0.

    02:00.0 is the secondary graphics adapter installed in the second PCIe 16 slot, in my case a Nvidia Quadro 2000 card. This is the one we want to pass through to the guest system. This card MUST support VGA passthrough, or else this whole procedure will fail !!! Read the hardware requirements above at the beginning very carefully.
Make a note of the PCI ID you want to pass through. Most likely your graphics card will have a second PCI ID for the onboard audio controller. This is typically 02:00.1 (for the card with PCI ID 02:00.0). You need to pass that through as well!

To check, enter the following and scroll through the output:

Code: Select all

lspci | grep 02:00.
You will see something like this:
  • 02:00.0 VGA compatible controller: NVIDIA Corporation GF106GL [Quadro 2000] (rev a1)
    02:00.1 Audio device: NVIDIA Corporation GF106 High Definition Audio Controller (rev a1)
If you don't get the second line, there is no onboard audio controller you need to pass through, so use only the 02:00.0 PCI ID.

Note: You may get different PCI IDs altogether, so use the ones appropriate to your system.

Next are the USB devices (see also the "Alternative method" below):

Code: Select all

lspci | grep USB
The output on my machine is this:
  • 00:1a.0 USB controller: Intel Corporation X79 series chipset USB2 Enhanced Host Controller #2 (rev 05)
    00:1d.0 USB controller: Intel Corporation X79 series chipset USB2 Enhanced Host Controller #1 (rev 05)
    06:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
    07:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
    08:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
I have 5 host controllers, and each of mine has 2 or more ports (IIRC).

I decided to pass through the second Intel USB controller for mouse/keyboard, since it's a USB-2 host controller which may give less troubles than USB-3. I also have a USB KVM switch that allows me to switch between two USB ports on the PC - one cable is connected to the passed-through USB port, the other to a USB-2 port that hasn't been passed through (for Linux). Here is my setup:
00.1a.0 is passed through to the domU (Windows 7) guest and one cable of the KVM is connected to this port (see below on how to determine that).
00.1d.0 is used with dom0 (Linux) and the second cable of the KVM is connected to that port.

To find out the PCI ID of the host controller(s) where your keyboard and mouse is currently connected, use:

Code: Select all

usb-devices
and scroll down to your mouse and keyboard entries. The PCI ID can be found on the "SerialNumber=" line of the corresponding Host Controller. In the example below my USB Multimedia Keyboard is connected to the EHCI Host Controller with the PCI ID 0000:00:1a.0:
  • T: Bus=01 Lev=00 Prnt=00 Port=00 Cnt=00 Dev#= 1 Spd=480 MxCh= 2
    D: Ver= 2.00 Cls=09(hub ) Sub=00 Prot=00 MxPS=64 #Cfgs= 1
    P: Vendor=1d6b ProdID=0002 Rev=03.11
    S: Manufacturer=Linux 3.11.0-12-generic ehci_hcd
    S: Product=EHCI Host Controller
    S: SerialNumber=0000:00:1a.0
    C: #Ifs= 1 Cfg#= 1 Atr=e0 MxPwr=0mA
    I: If#= 0 Alt= 0 #EPs= 1 Cls=09(hub ) Sub=00 Prot=00 Driver=hub

    T: Bus=01 Lev=01 Prnt=01 Port=00 Cnt=01 Dev#= 2 Spd=480 MxCh= 6
    D: Ver= 2.00 Cls=09(hub ) Sub=00 Prot=01 MxPS=64 #Cfgs= 1
    P: Vendor=8087 ProdID=0024 Rev=00.00
    C: #Ifs= 1 Cfg#= 1 Atr=e0 MxPwr=0mA
    I: If#= 0 Alt= 0 #EPs= 1 Cls=09(hub ) Sub=00 Prot=00 Driver=hub

    T: Bus=01 Lev=02 Prnt=02 Port=04 Cnt=01 Dev#= 3 Spd=12 MxCh= 0
    D: Ver= 1.10 Cls=00(>ifc ) Sub=00 Prot=00 MxPS= 8 #Cfgs= 1
    P: Vendor=058f ProdID=9410 Rev=01.22
    S: Product=USB Multimedia Keyboard
    S: SerialNumber=USB Multimedia Keyboard
    C: #Ifs= 3 Cfg#= 1 Atr=a0 MxPwr=50mA
    I: If#= 0 Alt= 0 #EPs= 1 Cls=03(HID ) Sub=01 Prot=01 Driver=usbhid
    I: If#= 1 Alt= 0 #EPs= 1 Cls=03(HID ) Sub=00 Prot=00 Driver=usbhid
    I: If#= 2 Alt= 0 #EPs= 1 Cls=03(HID ) Sub=01 Prot=02 Driver=usbhid
    ...
:arrow: Tip: A detailed description on identifying USB ports and their corresponding PCI IDs can be found here: http://forums.linuxmint.com/viewtopic.p ... 78#p795878.

My KVM has two ports to connect both keyboard and mouse to it. If you don't use a KVM switch, it's best to have both keyboard and mouse on the same controller, so you may need to replug them and check again.

Alternative method: You can use "synergy" to share one mouse and keyboard between your Linux dom0 and your Windows guest, in which case you do not need to pass through USB hosts. See here: http://synergy-foss.org/. To install "synergy", enter:

Code: Select all

apt-get install synergy libcrypto++9 quicksynergy
For more information on using synergy, see http://forums.linuxmint.com/viewtopic.p ... 80#p706770. Thanks gordon.cooke !


9. Create pciback.conf file

:arrow: If you do NOT pass through USB ports, skip to step 11.

Once you determined the PCI IDs for your graphics adapter and USB port(s) you want to pass through to Windows, create a config file named pciback.conf in /etc/xen with the PCI ID(s) of the USB host(s) you wish to pass through, for example:

Code: Select all

# This file contains the list of pci devices the pciback driver should bind
# to at system startup.
# Please provide the devices in long BDF notation.
# Example:
# 0000:08:00.0
# NO EMPTY LINES!
#
0000:00:1a.0
Note: Replace the PCI ID in this file with the one(s) you determined before! You must enter the long PCI ID notation, with a preceding 0000: for the domain! This configuration file is used by the pciback shell script below.


10. Install and activate pciback script

We are using the xen-pciback "hide" command to hide the second graphics card and bind it to xen-pciback (see above). However, if you wish to pass through USB hosts to your Windows guest, you can use the script below for convenience.

Save the following pciback script named pciback to /etc/init.d:

Code: Select all

#!/bin/sh
#
### BEGIN INIT INFO
# Provides: pciback
# Required-Start: $local_fs
# Required-Stop: $local_fs
# Should-Start:
# Should-Stop:
# X-Start-Before: xend
# X-Stop-After:
# Default-Start: 2 3 4 5
# Default-Stop:
### END INIT INFO

. /lib/lsb/init-functions
scriptname=/etc/init.d/pciback
configfile=/etc/xen/pciback.conf

unbind() {
# Unbind a PCI function from its driver as necessary
[ ! -e /sys/bus/pci/devices/$1/driver/unbind ] || \
echo -n $1 > /sys/bus/pci/devices/$1/driver/unbind
}

bind() {
# Add a new slot to the PCI Backend’s list
echo -n $1 > /sys/bus/pci/drivers/pciback/new_slot
# Now that the backend is watching for the slot, bind to it
echo -n $1 > /sys/bus/pci/drivers/pciback/bind
}

case $1 in

start)
cat $configfile | while read line;do
echo $line | grep ^# >/dev/null 2>&1 && continue
unbind $line
bind $line
done
;;

stop|status|restart|force-reload)
# As we don’t know which driver was bound before, there is not much we can do here
;;

*)
echo “Usage: $scriptname start” >&2
exit 3
;;

esac
Make the pciback shell script executable

Code: Select all

chmod +x /etc/init.d/pciback
and run

Code: Select all

update-rc.d pciback defaults 21 19
The number 21 means that it will run AFTER xend is started (xend has 20 in the init.d), and stopped before xend will stop.

Note: For some reason, the command insserv didn't work for me. You may try it instead of the above command. insserv doesn't need arguments since it should read the header and determine when to run the script.


11. Load xen-pciback and pass through graphics card

We use the initramfs method to pass through the graphics card:

Code: Select all

echo "xen-pciback passthrough=1 hide=(02:00.0)(02:00.1)" >> /etc/initramfs-tools/modules
Note: You have to change the PCI IDs to the ones you determined above!
The above command loads the xen-pciback module from initramfs and binds the graphics card to xen-pciback, before the graphics driver gets loaded.

Then we update the initramfs by issuing:

Code: Select all

update-initramfs -k all -c

12. Create the Windows 7 guest configuration file

:arrow: If you get the following error:

Code: Select all

Error: Errors were found at line 14 while processing /etc/xen/win7.cfg:
	None
remove the comments in your config file.[/size]
:arrow: Some AMD graphics cards have issues with large guest memory sizes and/or overclocking. In that case, keep the memory size below 6Gb.

Copy/paste the following configuration file into /etc/xen and name it win7.cfg (or whatever name you chose):

Code: Select all

builder='hvm'
memory = 4096
name = 'win7'
vcpus=6
pae=1
acpi=1
apic=1
on_xend_stop='shutdown'
vif = [ 'mac=00:16:3e:68:e1:01,type=vif,bridge=xenbr0' ]
disk = [ '/dev/mapper/guest-win7,raw,hda,rw'  , 'file:/media/install/Win7.iso,raw,hdd,devtype=cdrom' ]
device_model_version = 'qemu-xen-traditional'
boot='dc'
sdl=0
vnc=1
vncpasswd=''
serial='pty'
tsc_mode=0
viridian=1
usb=1
usbdevice='tablet'
gfx_passthru=0
pci=[ '02:00.0', '02:00.1' , '00:1a.0' ]
localtime=1
xen_platform_pci=1
pci_power_mgmt=1
Important: You must edit the file and adjust it to your requirements !!! It most likely won't work for your hardware/setup.

I have commented the configuration file to explain the different configuration options. Here some further explanations about the options you need to change/adopt for your hardware:

memory = # Enter the amount of memory (RAM) in Mbyte your guest will get. For example, if you have 8GB RAM, you could give the guest (Windows) a value of 4096 which is 4GB, leaving the rest for dom0 (Linux) and other Xen guests. If you have a problem booting the guest, reduce the memory size to 6GB or less. Some AMD cards have issues with large guest memory.

vcpus= # Enter the number of virtual CPUs your guest will get. I have a 6-core CPU with 12 threads and I decided to give Windows 10 of the 12 available threads (virtual CPUs). As long as I run only Linux, it will get all CPUs / VCPUs. Once I start Windows I give it 10 VCPUs, leaving 2 VCPUs for Linux. It's best to leave some CPU resources to the Linux dom0 since Linux still needs to handle I/O, networking, etc. In my case, having a 3930K CPU, 2 VCPUs for dom0 should be enough to handle that.

vif = [ 'vifname=win7,type=ioemu,mac=00:16:3e:68:e1:01,bridge=xenbr0' ] # This defines the networking part for the guest system. vifname=win7 gives a name to the interface (this feature is buggy, so I removed it). iommu means it uses the VT-d functions of the memory controller. mac=... assigns a static MAC address to the bridge so Windows won't complain about "illegal copy" etc. when the MAC changes for each boot. 00:16:3e:68: is reserved for Xen and you should use it too! Change the last two e1:01 to your liking. Just watch out that you don't have the same exact MAC addresses in your LAN or when running two guests simultaneously.

disk = [ 'phy:/dev/mapper/guest-win7,hda,w' , 'file:/media/install/Win7.iso,hdc:cdrom,r' ] # This is the disk configuration for the Windows guest, i.e. as Windows will see it. See below for creating the Windows partition on LVM. The 'file:/media...' option points to the Windows ISO image on your drive, reading it as CDROM.

boot='dc' # is the boot order for Windows. This entry will first boot from d: then c:. Once Windows is installed you need to edit this file and remove the "d" so it reads: boot='c'.

pci=[ '02:00.0', '02:00.1' , '00:1a.0' ] # These are the PCI IDs of the PCI devices we want to pass through (the ones we specified in the /etc/xen/pciback.conf file or in the /etc/initramfs-tools/modules file). If you have problems with the USB ports and power management, and the global "pci_power_mgmt=1" option doesn't work, try this:
pci=[ '02:00.0', '02:00.1' , '00:1a.0,power_mgmt=1' ]
See http://www.overclock.net/t/1205216/guid ... t_19572876.

The rest should be left as is for now. If, and only if your Windows doesn't boot or you don't get a screen or mouse/keyboard don't work, you may need to change the following, depending on the problem:

# nographic=1 # Uncomment if the Nvidia card doesn't work, or perhaps for Windows 8.

gfx_passthru=0 # Set to 1 may help with some Nvidia cards. Search the Internet for AMD cards and recommendations, though AMD cards should work with 0. Setting gfx_passthru=1 will pass through the graphics card as primary (and only) display adapter, without first booting the virtual Cirrus Logic adapter. In most cases this requires a patched Xen hypervisor, but in some cases it may work, depending on the graphics adapter and perhaps the Windows version in use.

usb=1 # Set to "0" if you pass through your keyboard/mouse (this may also be necessary after installing "Synergy")

usbdevice='tablet' # When usb=1 (see above), this option improves mouse tracking

Documentation on the above guest configuration options can be found here: http://xen.org/files/Support/XenConfigu ... etails.pdf

For AMD CPUs, see this very helpful post on the Ubuntu forum for configuration options: http://ubuntuforums.org/showthread.php?t=2219436


13. Create a logical volume to install Windows

To make this easier, I assume you are installing on a separate disk named /dev/sdb. Of course you can also use a partition.

Note: The disk (here sdb) MUST be flagged for LVM. You can use gparted for it. Don't format it!

Code: Select all

pvcreate /dev/sdb
Let's check:

Code: Select all

pvdisplay
This shows you the physical LVM volumes (PV). Your newly created PV should show up as empty.

Now we create a logical volume (LV) named "win7", size 100GB for Windows in volume group (VG) "guest":

Code: Select all

vgcreate guest /dev/sdb # assign the PV we created to volume group "guest"
lvcreate -L 100G -n win7 guest # create the logical volume named "win7", size 100G in VG "guest"
Let's check the results:

Code: Select all

lvdisplay
See if logical volume "win7" appears.


14. Copy the Windows installation DVD to disk

Make an installation folder for Windows:

Code: Select all

mkdir /media/install
Insert your Windows DVD into your DVD drive and enter:

Code: Select all

dd if=/dev/cdrom of=/media/install/Win7.iso
Note: You will need a Windows retail license to use Windows inside a virtual machine.


15. Reboot the computer into Xen

Important note: After reboot with the PCI devices being detached I had problems with the nouveau driver (Nvidia) for my secondary graphics card (the one to passthrough). My PC got stuck right after login. Blacklisting the nouveau driver worked. See http://community.linuxmint.com/tutorial/view/176 for how to blacklist the nouveau driver.
Comment: In my case the nouveau driver wasn't required, as my Nvidia card is only used by Windows.
EDIT: Similar problems are reported with AMD graphics cards, in which case the solution is to blacklist the radeon driver using the radeon.blacklist=1 grub entry like this:

Code: Select all

GRUB_CMDLINE_LINUX_DEFAULT="radeon.blacklist=1 quiet splash nomodeset"
If the above doesn't work, try the solution described here: http://forums.linuxmint.com/viewtopic.p ... 60#p827407. Thanks wollymammoth!

However, if you choose to use 2 AMD or 2 Nvidia graphics cards, you would need the respective driver for your dom0. In that case the PCI devices should be detached before the driver starts. You may need to run update-rc.d pciback defaults 21 19 with different numbers, or find another way to run this script before the graphics drivers are loaded.

See also earlboy's post here: http://forums.linuxmint.com/viewtopic.p ... 60#p705871.
Last edited by powerhouse on Sun Oct 02, 2016 12:50 pm, edited 50 times in total.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
powerhouse
Level 6
Level 6
Posts: 1138
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by powerhouse »

Part 4

We are almost there.


16. Install vncviewer

We need a VNC viewer for the installation of Windows. In the terminal window:

Code: Select all

sudo apt-get install vncviewer

17. Verify that pciback works

First check that everything is setup correctly:

Code: Select all

sudo xl pci-assignable-list
should give the list of detached devices that can be assigned (bound) to the Windows guest. My list looks like that:
  • 0000:02:00.0
    0000:02:00.1
    0000:00:1a.0
All devices I configured are ready for passthrough.

Note: If some PCI IDs are missing - namely the PCI IDs for the graphics card - check steps 9 to 11 to make sure you didn't miss anything (also verify the IDs). In some configurations the graphics driver gets loaded and attached to the graphics card despite having xen-pciback specified in step 11. To verify that, check the following file:

Code: Select all

cat /sys/bus/pci/devices/0000:02:00.0/uevent
Note: Change the 0000:02:00.0 to the PCI ID of your graphics card.

The first line should read DRIVER=pciback. If the first line reads DRIVER=radeon or DRIVER=nouveau, try the following:

Code: Select all

echo 'blacklist <driver>' >> /etc/modprobe.d/blacklist.conf
where <driver> is the name of the graphics card driver that we want to prevent from loading, for example "fglrx" or "radeon" for AMD.
If you use 2 AMD cards, you may also try the following instead of the above method:
a) Open "Control Center" from the Menu and select "Driver Manager"
b) Install the proprietary AMD driver "fglrx" or "fglrx-updates" (I used the latter and it works fine)
c) Reboot, then check with: sudo xm pci-list-assignable-devices

Other CLI commands to remember:

Code: Select all

sudo xm list
lists the current domains. At the moment only domain0 should be listed.

For debugging purposes this command can come handy:

Code: Select all

sudo xm dmesg

18. Create the Windows domU using its configuration file

Code: Select all

sudo xl create /etc/xen/win7.cfg
and connect to it via VNC:

Code: Select all

sudo vncviewer localhost:0
Note: It may work without sudo.


19. Install Windows guest

Unless there is some configuration error (check your path to the /media/install/Win7.iso file or whatever you name it), the Windows installer should boot now and you should see the screen output in the console (enlarge it if necessary).

Install Windows as you normally would. At some point Windows will reboot the guest domU (not Linux).


20. Access Windows after automatic reboot

After Windows has rebooted, it should come back to the VNC console. If not, close VNCviewer and run again.

If Windows doesn't start and you get a qemu shell screen, it might have passed through your mouse/keyboard and/or your display adapter. Alternatively, you might need to change your /etc/xen/win7.cfg file and specify boot="c". Before you edit the win7.cfg file, shutdown the domU (via xm shutdown win7). Edit your win7.cfg file and run xm delete win7, followed by

a) sudo xm create /etc/xen/win7.cfg, and connect via VNC

Note: If, after your Windows guest reboots, you get a message like "parallel0 console" on your VNCviewer screen, see here: http://forums.linuxmint.com/viewtopic.p ... 27#p786784.


21. Install Windows graphics driver

After the second Windows 7 VM reboot, Windows will tell that "Setup is preparing your computer for first use...". Once booted, still in the VNC console screen, you'll need to install and enable the graphics adapter driver for your display card. Inside Windows under Device Manager, check for graphics adapters. It should list two graphics adapters, one is for the emulated Cirrus VGA adapter that you are currently using, the other one is the real graphics adapter (in my case it had a yellow triangle as the driver was missing).

Before going any further, check the following hardware requirements:

a) A separate screen connected to the secondary graphics adapter to be used by Windows; or a second cable connection to your current screen; or a KVM switch for keyboard, mouse, and optionally display. I use my KVM switch only for keyboard/mouse and connected my secondary graphics adapter via DVI cable to the second DVI port on my screen. Some screens will have VGA ports, or HDMI. Both should work, though I don't care much about VGA because of it's low quality compared to DVI or HDMI. The screen should have a button to select the input source, if using only one screen.
b) If you want to pass through your keyboard/mouse, the best choice is a USB KVM switch. Else you need to unplug and replug your keyboard/mouse to the appropriate USB ports.
c) You could also use your keyboard and mouse without passing the PCI device through (using the USB=1 and/or tablet=1 settings in the win7.cfg file). Frankly I have no experience with this.
Or you could use a second mouse/keyboard just for Windows when using PCI passthrough on the USB port.

Now install the driver for your graphics adapter. In my case, I went to the Nvidia support website and ran the Java script to determine the correct driver, downloaded and installed it. The more common option would be to let Windows find the suitable display driver.

Once the installation has finished:

* the screen should reset and switch to the graphics adapter. If at that point you get a black screen, switch your screen to the other display port. If you see the Windows screen - open a bottle of Champagne - you've done it.
* alternatively Windows may ask you to reboot. Do so and watch the boot process in the VNC screen. After loading the graphics driver, and before the login screen, Windows should switch the output to your installed graphics adapter and you should be greeted with a Windows screen on your display.

If you get a black screen when rebooting Windows, or at any point during installation, switch your display to the second input. Do you see a Windows screen - if yes, it's a reason to celebrate.


22. What if VGA passthrough doesn't work?

If you can't get your VGA adapter to pass through to Windows, there are some options you may want to check:

a) Make sure your graphics adapter is supported !!! See my lengthy introduction to hardware requirements. Unless you have a Nvidia "multi-OS" graphics adapter, AMD will be a better choice. Have a look at the following threads/blogs for more information on VGA passthrough and adapters:

http://www.overclock.net/t/1307834/xen- ... s-adapters

http://www.overclock.net/t/1205216/guid ... al-machine - New updated instructions for Fedora 20

https://dl.dropboxusercontent.com/u/225 ... hrough.pdf

http://gro.solexiv.de/2012/08/pci-passthrough-howto/

http://wiki.xen.org/wiki/Xen_VGA_Passthrough

http://wiki.xen.org/wiki/Xen_VGA_Passth ... d_Adapters
This page seems a bit outdated, but probably the safest reference when purchasing a graphics card.

http://www.phoronix.com/scan.php?page=n ... px=MTA3NjA - this is the complicated way to get Xen VGA passthrough with Nvidia cards (see link inside the text). This procedure should work with some Nvidia cards that are NOT "multi-OS".

http://www.davidgis.fr/blog/index.php?2 ... ss-through - this is currently THE resource for instructions and patches when you want or need to compile the Xen hypervisor. This is specifically for Nvidia cards that don't have native support for VGA passthrough.


b) If you have installed Windows 8.1, chances are the "xm" toolstack we chose won't work. Luckily it's easy to switch to the "xl" toolstack. Here are the steps:

(1) Edit your /etc/xen/win7.cfg file (replace the file name with yours). Comment out

Code: Select all

#device_model = '/usr/lib/xen-default/bin/qemu-dm'
and add

Code: Select all

device_model_version = 'qemu-xen-traditional'
(2) In a terminal window, enter the following command to switch to the "xl" toolstack:

Code: Select all

sudo sed -i 's/TOOLSTACK=.*\+/TOOLSTACK="xl"/' /etc/default/xen
(3) Reboot your computer.

(4) Boot your Windows guest with

Code: Select all

xl create /etc/xen/win7.cfg
(5) If this solves booting into Windows 8.1 with VGA passthrough, you may have to install some utility and scripts to "eject" your graphics card. More information and the scripts can be found here: http://blog.ktz.me/?p=219. Before installing the scripts, test your Windows VM by shutting down and rebooting it several times in sequence. I had no need to use the scripts, and in fact, they didn't work in my configuration (perhaps because I use an Nvidia card for VGA passthrough?).


23. Change Windows boot order in guest config file

After Windows is installed, edit the /etc/xen/win7.cfg file and change the "boot=dc" option to boot="c". This is equivalent to ejecting the installation disk from the DVD drive. If the installation ISO is still required, but you want/need to change the boot order, specify boot="cd".


24. Booting the Windows guest

After everything is installed and VGA / PCI passthrough is working properly, you can use the following command to start your Windows domU:

Code: Select all

sudo xm create /etc/xen/win7.cfg

25. Install PV drivers under Windows

:arrow: Update: There has been some development on the driver side. Previously one had the option to install the GPL PV drivers under Windows. Now there is a new set of drivers developed by the Windows PV drivers team. For more information, see http://wiki.xen.org/wiki/Xen_FAQ_Drivers,_Windows.

Inside Windows, install either of the two driver packages for improved disk I/O, network, and general performance:

Windows PV drivers: http://xenproject.org/downloads/windows-pv-drivers.html

For the latest Windows PV drivers, see here: https://www.xenproject.org/developers/t ... ivers.html

Signed GPLPV drivers: http://wiki.univention.de/index.php?tit ... PV-drivers

Note: The following website mentioned here earlier is down as of May 2015: http://www.ejbdigital.com.au/gplpv/

I currently use the gplpv_Vista2008x64_0.11.0.373.msi drivers. If you upgrade from an older version (<=.400), first uninstall the GPLPV drivers!

Important: When using the GPLPV network driver, you need to modify the win7.cfg and remove the "type=ioemu" option from the vif=... line. The same may be true for the Windows PV driver, but I haven't tested it.


26. Audio under Windows

Ideally the graphics card used for Windows has onboard audio support and your screen supports audio output via built-in speakers or headphone jack. In this case you are able to use the graphics card audio device in Windows to get sound output.

However, if your screen doesn't have any sound output option (speakers or headphone jack), there are at least 3 options available to get sound under Windows:
  1. 1. Using "jack" in both Windows and Linux - a free software only solution:
    Forum user wollymammoth has provided us with this how-to: http://forums.linuxmint.com/viewtopic.php?f=42&t=160527.
    Many thanks, mate!
  • 2. Using a discrete PCI(e) sound card such as this:
    asus-xonar-sound-card.jpg
    If you are going to buy a sound card, do yourself a big favor and check the ALSA compatibility list for a Linux compatible sound card: http://www.alsa-project.org/main/index.php/Matrix:Main. Of course Windows doesn't care about this list, and any Windows 7/8 compatible sound card will do, but perhaps one day you wish to use that hardware under Linux.
    You will need to pass through that sound card to Windows, as described before.
  • 3. Using a USB sound card such as this:
    USB sound card.jpg
    All USB sound cards that are standards compliant will work both with Windows and Linux. I chose a USB stick similar to the one shown here - it cost me $6. Of course, more money can buy you better sound quality - see for example here: http://www.stereophile.com/content/audi ... -converter
    You plug this USB sound card into one of the passed through USB ports used by Windows. Then you connect the headphone jack of the USB sound card with the "line in" jack of your on-board sound card (usually found on the back panel, check your motherboards user manual if unsure).

27. Post Passmark results

Once your Windows VM with VGA passthrough is up and running, please post your Passmark results here: http://forums.linuxmint.com/viewtopic.p ... 5&t=153482.

Feel free to add other benchmark or performance results (WEI, etc.), particularly benchmarks using the same hardware with Windows installed on "bare metal". Or simply post your opinion about running Windows and Windows applications in a virtual machine.

Thanks !!!

END

Note: Once you successfully installed Windows on a Xen hypervisor, you may like the backup and restore scripts I've written. See http://forums.linuxmint.com/viewtopic.php?f=42&t=133089. Enjoy!
Last edited by powerhouse on Thu Jul 13, 2017 2:34 pm, edited 36 times in total.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
powerhouse
Level 6
Level 6
Posts: 1138
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO create the ultimate gaming/graphics machine with

Post by powerhouse »

Edit: New startup script for Windows guest

Once you got everything running and you can boot and shutdown and reboot your Windows guest, with Windows using the graphics card, it's time for a startup script.

Here are the steps to create and launch a startup script without the need to give a root password:

1. If it doesn't exist, create a folder named .bin in your home directory. This is a hidden folder so it won't show up unless you select -> View -> Show hidden files in your file browser.

2. Create a file named "windows.sh" inside /home/myusername/.bin and paste the following shell script into that file:

Code: Select all

#!/bin/sh

if xm list | grep -q win7; then
	zenity --warning --timeout=15 --text="Windows is already running." &
	exit 1

else

	xm create /etc/xen/win7.cfg &> /dev/null
	zenity --info --timeout=15 --text="Starting Windows, please wait." &
	exit 0

fi
Check that your Windows guest config file is /etc/xen/win7.cfg - if not, replace the win7.cfg name in the script above.

In a terminal window, enter the following:

Code: Select all

cd ~/.bin
sudo chown root windows.sh
sudo chmod 500 windows.sh
This changes the ownership of the file to root and allows only root to read and execute the script. This is an important security measure. Don't skip it, since the next steps will allow you to run this script without a password!

3. I'm using "zenity" in my script to display screen messages. Check if /usr/bin/zenity exists. If yes, continue with the step 4.
If not and you are using Mate instead of Gnome, the mate-dialogs package should be installed. To make it work under Mate, you can use either of the two methods below:
a) Enter in a terminal window:

Code: Select all

sudo ln -s matedialog /usr/bin/zenity
OR

b) Install the mate-dialogs-gnome package.

4. Warning:
This step is potentially dangerous. It changes the way sudo works. Entering incorrect values here can render the sudo command nonfunctional. Instead of creating the /etc/sudoers.d/permissions file, you may also use sudo visudo and make the changes / additions directly in the /etc/sudoers file, where visudo will check the syntax and prevent syntax errors. You have been warned.

Create a file named permissions in the /etc/sudoers.d folder (you need to be root to do that), and paste the following into the file:

Code: Select all

# Host alias specification
Host_Alias HOST = hostname
Host_Alias LAN = 192.168.1.0/255.255.255.0
Host_Alias HOME = HOST,LAN

# User alias specification
User_Alias WIN_USERS = myusername

# Cmnd alias specification
Cmnd_Alias WIN = /home/myusername/.bin/windows.sh

# User privilege specification
WIN_USERS HOME=(root) NOPASSWD:WIN
Important ! Replace the following in the /etc/sudoers.d/permissions file before saving it:

hostname: Replace with the hostname of your PC. If you don't know, open a terminal window and type:

Code: Select all

cat /etc/hostname
192.168.1.0: Replace with the IP of your network. To find out, use:

Code: Select all

ifconfig | grep -A 1 xenbr0
The second line reads something like "inet addr:192.168.1.145 Bcast:192.168.1.255 Mask:255.255.255.0". "192.168.1.145" in this example is the IP address of xenbr0 (your virtual network device), so the network IP would be 192.168.1.0. Adjust to what you find.

myusername: Replace with your username. If you forgot your user name, type whoami in a terminal window (isn't it amazing, your PC can tell you who you are).

After you replaced the hostname, IP, and myusername, save the file.

5. Open a terminal window and enter the following:

Code: Select all

cd /etc/sudoers.d/
sudo chmod 440 permissions
to change the file permissions to read-only for root and group.

6. To create a launcher, right-click on the desktop and fill in the prompt. For the Command: prompt, enter

Code: Select all

sudo /home/myusername/.bin/windows.sh
Replace "myusername" with your user name.


The above steps will allow you to run the windows.sh start script from your PC (host) as well as from your home network (LAN). Remove "LAN" from the "Host_Alias HOME = HOST,LAN" line of the permissions file if you do not want to grant password-free access from the LAN (as user myusername). The script also checks if Windows is already running (make sure the domain name win7 in the script matches the name specified in your /etc/xen/win7.cfg file).
Last edited by powerhouse on Mon May 13, 2013 10:38 am, edited 6 times in total.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
GizmoChicken

Re: HOW-TO create the ultimate gaming/graphics machine with

Post by GizmoChicken »

Thanks for the great tutorial! I imagine that I'll be referring to your post many times in the coming weeks. :)
powerhouse
Level 6
Level 6
Posts: 1138
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO create the ultimate gaming/graphics machine with

Post by powerhouse »

GizmoChicken wrote:Thanks for the great tutorial! I imagine that I'll be referring to your post many times in the coming weeks. :)
Thanks GizmoChicken. Have you tried it, or are you going to try it?

I had to write some parts of it from memory, though I've done these steps several times when trying to get everything working. Still, there could be steps I've missed in this howto. That's why feedback is most welcome.

One thing I know I've left out was my nouveau driver issue after enabling passthrough. Linux Mint had installed the nouveau driver for my secondary (Nvidia) graphics card and used to boot with the driver. The result when detaching the Nvidia card was that the boot process would stop after user sign in and the PC froze. So I blacklisted the nouveau driver and all is fine now.

EDIT: I updated my how-to and mentioned this problem. Still need to find a generic way to avoid this graphics driver issue altogether (I suspect this problem to occur with any graphics driver bound to the passed-through device). Ideas are welcome.
Last edited by powerhouse on Tue Sep 18, 2012 6:07 am, edited 1 time in total.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
GizmoChicken

Re: HOW-TO create the ultimate gaming/graphics machine with

Post by GizmoChicken »

powerhouse wrote:Have you tried it, or are you going to try it?
I haven't tried VGA passthrough yet. My processor (an AMD Phenom II X4) supports AMD-V, so I'm good with many aspects of Xen. But my motherboard (a Gigabyte GA-770TA-UD3) doesn't support IOMMU, so I'm not quite ready for VGA passthrough. When I get around to it, I'll upgrade my motherboard to a Gigabyte GA-990FXA-UD3, which reportedly supports IOMMU.

I currently have a few non-graphical servers running as virtual machines under XCP 1.6, which is an open source version of Citrix XenServer, but with a few more features. But XCP doesn't provide any sort of desktop environment, and so I've been experimenting with other options. For example, in one test configuration, I have my non-graphical servers running as virtual machines under Hyper-V that comes with Windows 8 Release Preview. In another test configuration, I have my non-graphical servers running as virtual machines under Xen with Ubuntu 12.10 Desktop as dom0. But ultimately, I'd like to run Ubuntu Server as a headless dom0 with a graphical desktop on a domU. And so that's why I have an interest in VGA passthrough.
Last edited by GizmoChicken on Tue Oct 02, 2012 6:19 pm, edited 1 time in total.
powerhouse
Level 6
Level 6
Posts: 1138
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO create the ultimate gaming/graphics machine with

Post by powerhouse »

GizmoChicken wrote:
powerhouse wrote:Have you tried it, or are you going to try it?
I haven't tried VGA passthrough yet. My processor (an AMD Phenom II X4) supports AMD-V, so I'm good with many aspects of Xen. But my motherboard (a Gigabyte GA-770TA-UD3) doesn't support IOMMU, so I'm not quite ready for VGA passthrough. When I get around to it, I'll upgrade my motherboard to a Gigabyte GA-990FXA-UD3, which reportedly supports IOMMU.

I currently have a few non-graphical servers running as virtual machines under XCP 1.5, which is an open source version of Citrix XenServer, but with a few more features. But XCP doesn't provide any sort of desktop environment, and so I've been experimenting with other options. For example, in one test configuration, I have my non-graphical servers running as virtual machines under Hyper-V that comes with Windows 8 Release Preview. In another test configuration, I have my non-graphical servers running as virtual machines under Xen with Ubuntu 12.10 Desktop as dom0. But ultimately, I'd like to run Ubuntu Server as a headless dom0 with a graphical desktop on a domU. And so that's why I have an interest in VGA passthrough.
IOMMU is a must for VGA passthrough on AMD boards. See http://wiki.xen.org/wiki/Xen_VGA_Passthrough and http://wiki.xen.org/wiki/VTdHowTo for more information on IOMMU support. It's probably best to check the motherboard manufacturers' websites for IOMMU support, as well as the release notes for their BIOS releases.

Given the right hardware, your goal should be achievable.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
powerhouse
Level 6
Level 6
Posts: 1138
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO create the ultimate gaming/graphics machine with

Post by powerhouse »

Just to add some information for those interested.

My Nvidia Quadro 2000 graphics card works under Windows domU as if I was running bare metal. I have passed through a USB host and I connected my NEC version of the EyeOne spectrometer to one of the passed-through ports under Windows and ran the NEC Spectraview screen calibration program. It works perfect and I calibrated my screen and uploaded the calibration (LUT) to the screen. So, the VGA passthrough works just as if I would have Windows installed on bare metal.

Today I needed to backup my photos on the Windows domU drive/partition (it's a LVM striped formatted logical volume) to an external backup drive. No problem. I just passed through my USB3 ASMedia 09:00.0 PCI ID to Windows, installed the Seagate backup program on Windows, moved the folders to the appropriate drive letters of my current Windows domU on my Seagate external drive (they were different since I had used that drive before on a different PC), installed the Seagate utility and backed up my disk. What can I say, it's just like running Windows on bare metal, perhaps even better.

Using the same USB port I also imported a nearly full 16GB CF card using a Kingston USB2 card reader into Lightroom 4.1 (on Windows). It went smooth, but I will buy a USB3 card reader for the additional speed.

Yesterday I converted ~750 RAW files into jpeg with some curve adjustments, sharpening, resize and color space conversion etc. under Capture NX2. It ttok a while but was at least 5 times as fast as using my old Intel Core2duo machine. I noticed that CaptureNX2 doesn't make use of most of the resources (CPU and memory) that I have. CPU load under Windows maxed at 66% and usually stayed somewhere at 35%. Memory wasn't challenged at all, perhaps 10-15% (I gave 24GB to my Windows domU). Next time I will use Lightroom to see what that does.

OK, I'm not a games freak, but this performance is great!

EDIT: I got myself a USB3 card reader which significantly boosts file transfer speed between memory cards (CF) and the disks.
Last edited by powerhouse on Sun Oct 14, 2012 4:31 am, edited 1 time in total.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
powerhouse
Level 6
Level 6
Posts: 1138
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO create the ultimate gaming/graphics machine with

Post by powerhouse »

Xen 4.2 has just been released (September 17, 2012) !!!

Here some more about it: http://wiki.xen.org/wiki/Xen_4.2_Release_Notes

Hope to find it in the repository soon.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
powerhouse
Level 6
Level 6
Posts: 1138
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO create the ultimate gaming/graphics machine with

Post by powerhouse »

Another performance update:

I was always wondering how Xen shares CPU time between dom0 (Linux) and domU (guest - Windows in my case). I gave Windows 10 out of 12 threads in the win7.cfg vcpu option, leaving only 2 vcpus to Linux / hypervisor.

Instead of taking some benchmarking tools, I used real live applications. Here is what I ran concurrently:

1. handbrake to rip a DVD using "normal" mode under Linux dom0, and writing to a separate hard drive /dev/sdd1
2. Copy a 1.9GB file from /dev/sdd1 to a remote Samba media streamer connected via router / Fast Ethernet
3. Lightroom 4.2 to convert RAW photos to jpeg under Windows domU, both file types on the same LVM stripped hard drives (similar to RAID0 using /dev/sdb3 and /dev/sdc3)

I used

Code: Select all

sudo xm top
to monitor processor usage of both dom0 and win7.

Results:

1. When I start handbrake without running Lightroom RAW photo conversion, I get above 400fps, while Windows is running. The moment I start Lightroom RAW conversion it goes down to around half that speed or 200fps. Compared to my old Core2Duo 2400 MHz PC running native Linux, this is still 5 times as fast!
2. 4.2MB/s is the average - there is definitely a bottleneck in networking. I have only 1 Ethernet port, though. Still, the result is the same as if nothing else was running, so this can't be attributed to CPU bottlenecks in dom0.
3. Lightroom converted the photos in around 1 photo per second. Both the RAW photos and the jpegs were on the same LVM stripped hard drives (no SSD), and watching the CPU load I guess this speed is limited to disk IO. There wasn't any noticeable slowdown due to handbrake running at the same time.

Once Lightroom had finished the speed under handbrake increased to above 400fps, which shows that Xen is able to utilize the full CPU power even while the guest is running but more or less idle.

The DVD I used was an old one, with scratches. From time to time the speed would go down, probably while trying to read scratched DVD areas at a lower speed and CRC correction. I've seen that with other DVDs when running a native Linux install, so this is expected.

xm top showed a 2:1 to 5:3 relation of CPU usage between domU:dom0. Total CPU usage was close to maximum most of the time. CPU usage went a little down when handbrake tried to read bad DVD sectors at lower speed, but Lightroom didn't use much (or any?) of that extra CPU power, probably due to IO constraints. Since xm top shows CPU usage in %, it was like 630% versus 330% at most times.

The results clearly show that Xen is able to manage the CPU resources in an efficient way, and easily recovers spare CPU time for dom0.

My /etc/xen/xend-config.sxp file has this entry:

Code: Select all

# In SMP system, dom0 will use dom0-cpus # of CPUS
# If dom0-cpus = 0, dom0 will take all cpus available
(dom0-cpus 0)
My domU Windows guest config file /etc/xen/win7.cfg has this entry:

Code: Select all

vcpus=10 # I gave my Win 7 10 virtual CPUs, out of 12 (6-core CPU)
So, when domU is idle, dom0 will use the full 12 vcpus. This is also how it shows in xm top: 12 vcpus for dom0, 10 vcpus for domU. domU can use up to 10 vcpus when needed. CPU time for both dom0 and domU is assigned dynamically within the specified minimum/maximum values.

I wonder how Xen 4.2 will perform. There have been some improvements in the CPU scheduler to better handle short interrupts / IO tasks from devices requiring realtime support (i.e. audio or perhaps disk?). It may also improve network speed, hopefully.

Next thing I need to do is get network speed on the bridged virtual network up on par with native Linux network speed.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
powerhouse
Level 6
Level 6
Posts: 1138
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO create the ultimate gaming/graphics machine with

Post by powerhouse »

While I'm still having a bit of a problem with the file transfer speed over the network to other PCs (Samba shares), file transfers between Windows domU and Linux dom0 work great. Yesterday I measured ~130MB/s transfer speed between Windows 7 domU and the Samba share on dom0 (Linux Mint) :D . The file transfer speed may have been limited by the disk read/write speed (the source was a SSD, the target a regular HDD, both formatted to LVM).

I'm using now LVM disks exclusively. Performance under Xen is top notch.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
Draucia

Re: HOW-TO create the ultimate gaming/graphics machine with

Post by Draucia »

I'm considering doing this, but I'm not sure if my motherboard supports it. I couldn't find anything on the manufacturer's website. I have the gigabyte GA-b75m-D3V. I also have an i3-2100 and a Radeon HD 6870. Do you think I should give this a go? Also, I enabled VT-D in my bios.
powerhouse
Level 6
Level 6
Posts: 1138
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO create the ultimate gaming/graphics machine with

Post by powerhouse »

Draucia wrote:I'm considering doing this, but I'm not sure if my motherboard supports it. I couldn't find anything on the manufacturer's website. I have the gigabyte GA-b75m-D3V. I also have an i3-2100 and a Radeon HD 6870. Do you think I should give this a go? Also, I enabled VT-D in my bios.

With regard to the CPU, have a look here: http://ark.intel.com/

I checked the i3-2100 and it doesn't support VT-d :( . See here http://ark.intel.com/products/53422/Int ... e-3_10-GHz.

In fact, the only i3 I could find that supports VT-d is the i3-2115C - see here http://ark.intel.com/products/68332/Int ... e-2_00-GHz.

To get a complete list of Intel CPUs with VT-d support see here: http://ark.intel.com/search/advanced?VTD=true

If you have a VT-d option in the BIOS, chances are your motherboard supports VT-d. Just make sure it's VT-d and NOT VT-x (VT-x should also be enabled, but it's not the same thing).

So unless you got the i3-2115C you won't be able to make it work, at least not with an i3 CPU. Sorry for the bad news.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
powerhouse
Level 6
Level 6
Posts: 1138
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Windows sound output

Post by powerhouse »

Another update on this ongoing project. This time it's about sound.

Up until recently I didn't have any sound support under Windows 7. Though I don't really need it, I wanted it to work.

I set up pulseaudio under Linux Mint and looked for a Windows 7 client only to find out that the "latest" Windows pulseaudio client is 10 years old :( . However, I found some instructions on the Internet that should allow sound forwarding from Windows to Linux (see here http://gleamynode.net/articles/2228/). Yet my experiments failed.

After installing a virtual sound card and some other utilities in Windows and Linux I tried to forward the sound but the driver/virtual sound card wasn't recognized by Windows. Here the Windows command line:

Code: Select all

linco.exe -B 16 -C 2 -R 44100 | plink mylinuxdom0 "cat - | pacat --playback"
Someone else made it work using the steps in the following post (see end of post): http://www.overclock.net/t/1205216/guid ... t_18525583.

In the end I grabbed a $8 USB sound card (these tiny USB sticks with headphone and mic jacks) and a $4 mini-RCA connection cable and plugged the USB sound stick into one of the passed-through USB2.0 ports I gave to Windows. The cable is connected to the headphone jack on the USB stick side, and to the "line in" port of the motherboard back panel. Not a neat solution (yet another wire), but it works fine for me.

Maybe one day I will give the software based sound solution another try.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
powerhouse
Level 6
Level 6
Posts: 1138
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Backing up the Windows 7 guest system

Post by powerhouse »

I am lazy so I wrote a little script that backs up my Windows 7 domU. The script also works while Windows is running, since it uses a LVM feature called "snapshot". Here the script:

Code: Select all

#!/bin/sh
mv /media/backup/guest-win7.img /media/backup/guest-win7.img.old
lvcreate --size 10G --snapshot -n win7-snap /dev/guest/win7
echo Creating backup copy of win7 domU...
pv /dev/guest/win7-snap | dd of=/media/backup/guest-win7.img
lvremove -f /dev/guest/win7-snap
echo Finished
Here a short explanation:
win7 - name of the Windows 7 domU guest system
win7-snap - name of the snapshot of win7 - I gave it 10GB which should be ample, as only the changes versus the win7 LV need to be stored
guest - name of the volume group where win7 resides
/media/backup - folder where the backup drive (LVM) is mounted
guest.win7.img - the image file (copy) of the Windows 7 domU guest

Notes:
1. The backup drive (in my case a virtual LVM volume) is already mounted at /media/backup.
2. The backup media must have at least twice the capacity of your entire Windows 7 domU partition or image file. If you gave your Windows 7 domU 100GB disk space, the backup drive or partition needs to have at least 200GB capacity.
3. The volume group (VG) "guest" must have at least 10GB unassigned capacity to hold the snapshot.
4. pv /dev/guest... | dd of=... uses dd to make an exact image copy of the Windows 7 guest, with the tweak that pv ... outputs a progress bar, time elapsed, and time to completion, giving you an indiciation on how long it will take.
5. I didn't use any compression, but it would be easy to implement (see next post).
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
powerhouse
Level 6
Level 6
Posts: 1138
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by powerhouse »

Except for the /boot partition, I'm using LVM for everything. Here is a little script for backing up my entire /, /boot, and /home partitions using gzip compression and tar:

Code: Select all

#!/bin/sh
mv /media/backup/lm13-root.tgz /media/backup/lm13-root.tgz.old
mv /media/backup/lm13-home.tgz /media/backup/lm13-home.tgz.old
mv /media/backup/boot.tgz /media/backup/boot.tgz.old
lvcreate --size 5G --snapshot -n root-snap /dev/lm13/root
lvcreate --size 10G --snapshot -n home-snap /dev/lm13/home

echo Creating backup copy of /boot...
tar -cf - /boot | pv -s $(du -sb /boot | awk '{print $1}') | gzip > /media/backup/boot.tgz

echo Creating backup copy of root...
mount /dev/lm13/root-snap /media/snapshot
tar -cf - /media/snapshot | pv -s $(du -sb /media/snapshot | awk '{print $1}') | gzip > /media/backup/lm13-root.tgz
umount /media/snapshot
lvremove -f /dev/lm13/root-snap

echo Creating backup copy of /home...
mount /dev/lm13/home-snap /media/snapshot
tar -cf - /media/snapshot | pv -s $(du -sb /media/snapshot | awk '{print $1}') | gzip > /media/backup/lm13-home.tgz
umount /media/snapshot
lvremove -f /dev/lm13/home-snap

echo Finished
The above script can be used on a running system, as it uses LVM snapshots to create a "snapshot" of the system (kind of an instant "freeze" of the file system and data). This snapshot is then compressed and copied to the backup media. Any file or file system changes after the snapshot are written to disk, but not included in the backup.

You need at least 15GB unassigned capacity under volume group "lm13" for the two snapshots. Note that /boot is NOT a LVM volume, so we can't take a snapshot of it. But it's highly unlikely that it gets modified during the backup.

5GB snapshot size for / is far more than you will need - you can make it much smaller, say 1GB. The snapshot size of the /home partition should be large enough to hold all changes/additions to that partition while backing it up.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
User avatar
Oscar799
Level 20
Level 20
Posts: 10393
Joined: Tue Aug 11, 2009 9:21 am
Location: United Kingdom

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by Oscar799 »

powerhouse,
Rather than keep changing post titles mid-thread,which makes the whole thread confusing,it would be better to post different topics as different threads
Image
powerhouse
Level 6
Level 6
Posts: 1138
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by powerhouse »

Oscar799 wrote:powerhouse,
Rather than keep changing post titles mid-thread,which makes the whole thread confusing,it would be better to post different topics as different threads
Sorry for the confusion. I didn't want to open new threads as the posts are all related to the original how-to.

It would be a lot easier if there was a "virtualization" section in the forum. If I open a new thread for every different topic (but all related to the one topic Xen VGA passthrough / virtualization), the information would be all over the place and hard to find or relate to.

P.S.: I will stop changing post titles mid-thread, if that causes confusion.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
Locked

Return to “Virtual Machines”