HOW-TO make dual-boot obsolete using XEN VGA passthrough

[ozieboy]

Hi Powerhouse,

I didn't get a chance to do any further testing in the last couple of weeks. I have now blown away the install from mSATA and installed ubuntu server 14.04 on the SATA SSD instead. See how that goes.
Fedora install was from Dizzy's guide, so Fedora 20 with Xen 4.3 .
If this didnt work, I am probably going to install windows 8.1 for the moment and try again some time later.

Best Regards,

[FastRealm]

I finally found out why Windows 8.1 reboot doesn't work for my PC. I have tried with "qemu-xen-traditional" and "qemu-xen-dm" they both give me same results and it is a 95% work. For my case instead of putting the script "gpedit.msc" startup i need to start the script in task scheduler. This way i can guarantee a 95% work. Why 95% you might be asking? On some rare occasion the "gpedit.msc" shutdown fail to work (haven't figure out why it fail) or some software cause it to fail. Most of the time it is the software.

On Windows 8.1, it is best not to install GPLPV driver, xen_platform_pci=0 and msi_translate=0. By default they are "1". If i install the GPLVP driver what will happen is every other restart/shutdown, the next boot up Win8 will hang at find device. I will need to destroy VM and restart the VM then only it can start like normal. Another thing i notice is Synergy doesn't play well with Win8. It will cause the mouse to move very slowly every now and then. Imagine using a 5kg mouse.

As for win7 with xen 4.3 XL, the script does work but unlike win8 where i can enable the ATI straight away, win7 needs to reboot. So if i want to restart/shutdown without restarting the Host, 1st i need to run the GPU_off script then restart/shutdown. If you don't do this the next startup on win7 will say that there is an error with the ATI and you will need to restart the HOST to clear this error. Once win7 bootup run the GPU_on script and reboot again. Only then the GPU will be enable. It is a bit troublesome but better than restart HOST every time. (If anyone got a way around this please share) If I'm going to shutdown the HOST, i just shutdown Win7 without script.

As for Ubuntu 14.04 with xen 4.4, other than the memory bug where you can't put more than 4GB, i can say it is very fast compare to LM16 with xen 4.3. But until someone find a way around the memory bug, xen 4.4 is no go.

[powerhouse]

Hi Powerhouse,

I didn't get a chance to do any further testing in the last couple of weeks. I have now blown away the install from mSATA and installed ubuntu server 14.04 on the SATA SSD instead. See how that goes.
Fedora install was from Dizzy's guide, so Fedora 20 with Xen 4.3 .
If this didnt work, I am probably going to install windows 8.1 for the moment and try again some time later.

Best Regards,

What is the CPU in your NUC? Could it be that you are giving your Windows guest too many VCPUs? Try less vcpus in the guest config file.

If you got it working under Fedora, there must be a way to get it working under Ubuntu. Did you use virt-manager to configure? Can you paste the working Fedora configuration (there should be a way to export the configuration in virt-manager)?

[bonzi]

Hi, has anyone tried this yet with Mint 17?

[Nesousx]

Hello all,

I have an hardware question for you. I plan to upgrade my scren to a 27" WQHD. However, in order to attain this resolution both in Dom0 and domU I need to replace the Dom0 onboad IGP by a dedicated PCIe card. The problem is that my current motheboard (Asrock Z77 Pro 4 M) only boot from first PCI express card. This means that my Linux "working" rig has "low end" PCIe card at PCI 3.0 16x while my "gaming" rig has a "high end" card at PCI 2.0 1X (I can't use the 4X slot with my current case, space limitation). The best solution would be to use my "high end" card in PCIe slot 1 and "low end" card in PCIe slot 2 or 3. Which probably means changing motherboard.

I am looking for motherboard (supporting socket 1155 CPUs), that offers the possibiilty to boot from PCI express 1 or 2. Something like PEG1 or PEG2 into the bios. I know Asus does it, but it also seems that Asus doesn't offer VT-d. It is very hard to find accurate information about bios even from vendors themselves, that's why I am asking here directly.

Could you please let me know if your motherboard allows those functionnalities :

* ability to choose from PEG1 / PEG2 inbox Bios
* VT-D
* has to be micro ATX (or smaller)

Thanks in advance

I would hold on to your board. My board doesn't offer any VGA related option in the BIOS and, like you, I thought all hope is lost. Until I tried out Ubuntu 14.04. Both the Unity and the Gnome versions automatically detect both cards and the attached screen and configure a dual screen setup. At first it was annoying, until I saw I could easily disable any of the "screens" (actually the graphics card). In all my recent experiments with Ubuntu 14.04 and Xen as well as KVM I chose the dom0 and domU graphics cards via configuration, not by changing their PCIe slot.
Linux Mint 17 is just around the corner. I'm sure the same can be achieved with LM16, I just haven't tried.
For an experiment, replace the PCI IDs for your domU graphics card in the initramfs and make sure you got a graphics driver for it, then reboot.

Hi,
Just a small update to let you know that I've swicthed to Ubuntu 14.04 (server edition), and I am succesfully using VGA passthrough with KVM and vfio (I no longer use Xen). EDIT : I did run the Passmark test, and I'am doing 3559 points, while I was doing 3009 points with my Xen setup.

So far, it seems to be working quite well. I'll keep you guys updated, and I'll probably write some how to as well.

As soon as I have a second GPU, I'll do more tests in order to see if I can use 1st PCI card in the Win guest, and PCI 3 solt card in the Linux host. However, is should be ok thanks to the "screen" options of latest versions of desktop manager (using Cinnamon and XFCE4 atm).

[powerhouse]

Hi, has anyone tried this yet with Mint 17?

I've downloaded and installed LM17RC on my test drive, but haven't found the time to test Xen or kvm. I did try both Xen and kvm with Ubuntu 14.04 and both work but with different cards (haven't tried patching for kvm yet). See my previous posts on that. With kvm and my Sapphire HD 7770 (AMD) card I can even run primary passthrough with kvm and vfio.

@Nesousx: Thanks for the update. It would be great to have a how-to for kvm! So your m/board works after all .

[FastRealm]

@Nesousx

Can you send me the links where you read about KVM and how to setup on Ubuntu with VGA Passthrough. I'm interested to try the Pros and Cons compare to Xen.

[powerhouse]

@Nesousx

Can you send me the links where you read about KVM and how to setup on Ubuntu with VGA Passthrough. I'm interested to try the Pros and Cons compare to Xen.

The most comprehensive information on KVM and VGA passthrough I found so far is here: https://bbs.archlinux.org/viewtopic.php?id=162768. I managed to get primary pass through with Ubuntu 14.04 and a Sapphire HD 7770 (AMD) card, without any patches at all. YMMV.

[iwaru]

Hi - First thanks for the guide.

I have gotten this to work with passthrough for both cards in two different VMs simultaneously.

Hardware:
Mobo: asrock extreme 4
Cpu: 3770T
GPU1: Radeon 7850
GPU2: Radeon 5450

Software:
Ubuntu 14.04
Xen 4.4


However eventhough my windows rating has a minimum of 7 I still feel like it isn't quite up to par with the normal feel. It feels sluggish between window swapping and the cpu load seems excessively compared to the amount running on the VM.
Utorrent with two torrents is taking between 20-50% cpu load with 7 Vcpu, which shouldn't be possible.
Youtube 720p takes between 20-60% cpu and opening another window with nothing in it will spike up to 90% cpu.

Another thing is the fact that it takes - not kidding - about 15min to boot the VMs..... which quite frankly is too long and also makes me think something is wrong

So my main questions are:
1: anyone got trouble with boot times and knows a fix? also the windows installation took like 8 hours or so...
2: Cpu usage seems very high compared to actual work being done. A VM with 3 cores seems very sluggish and running small things takes a lot of CPU load.

TY for any help

Update: Changing to XL toolstack fixes the boot issue it seems, but the CPU load is still very bad.

[powerhouse]

@iwaru:

There is definitely something wrong. The boot time should be less than a minute even for HDD, and there shouldn't be any real performance hit inside the VM. Can you post your VM config file?

Are you running one VM at a time? Have you installed the GPLPV drivers?

A problem that has been reported is performance degradation after VM restart. To rule that out, shut down your PC and reboot. Then boot your VM and check the performance. Is it good now?

Another possible reason is not assigning enough VCPUs to dom0. Remember that your Xen host still has to handle all the I/O etc. (this is where the GPLPV drivers come in as well). So try with 2 VCPUs for dom0. You could also experiment with pinning VCPUs.

[iwaru]

Ty for fast reply

GPLPV installed, which changed the disk performance drastically.
VM has been restarted, powered down and such without any difference.
Still terrible CPU performance with 4Vcpu and 4 for dom0 and only using one VM.
Have not tried pinning will give it a go later.
Again XL Toolstack changed boot-time to normal, takes around 15-30sec to boot now.

Bios: All visualization options enabled and all sleep/speedstep functions disabled.


Content of my win7.cfg: Have passed through GPU, 2 usb and the onboard audio, since my monitor doesn't support hdmi and currently it's on XL.

kernel = '/usr/lib/xen-default/boot/hvmloader'
builder='hvm'
memory = 12096
name = 'win7'
vcpus=4
#pae=1
acpi=1
apic=1
on_xend_stop='shutdown'
vif = [ 'mac=00:16:3e:68:e1:01,bridge=xenbr0' ]
disk = [ 'phy:/dev/sdb,hda,w' ]
#device_model = '/usr/lib/xen-default/bin/qemu-dm'
device_model_version = 'qemu-xen-traditional'
boot='c'
sdl=0
vnc=1
vncpasswd=''
stdvga=0
serial='pty'
tsc_mode=0
viridian=1
usb=1
usbdevice='tablet'
gfx_passthru=0
pci=[ '01:00.0', '01:00.1' , '00:14.0' , '00:1d.0', '00:1b.0' ]
localtime=1
pci_power_mgmt=1

Update: Did some more testing.
Running OCCT (stresstesting cpu) and then monitoring CPU load on Dom0 shows that while DomU is at 100% cpu load Dom0 is only spiking to 11% on a single core and average is around 0-5% on all cores.


Update:
I'm almost sure pinning solves the problem. I have pinned 4Vcpu and now the Dom0 actually shows CPU utilization. Will report back later when i'm certain

Update:
I have limited Dom0 to 2Vcpu and given my VMs 4 and 2 VCPU each - all pinned.
CPU utilization is now more or less like a normal system
Freaking awesome - ty for this guide!
Will post passmark later when I'm done setting up.
Now I just have to figure out the reboot problem regarding the gpu and xl.

Thanks for the help in advance

[powerhouse]

@iwaru: Glad you found a solution.

About the VM reboot issue - it either works without issues, or it doesn't (I mean it may cause problems). Some disable/enable the graphics card in Windows while shutting down. There is a script for it (you may need to search). Citrix is going to offer its own Xen drivers (also open source), you may want to look for them as the GPLPV drivers don't work with the graphics unplug script I mentioned .

[Boggler]

Hi,

First of all, thanks for this fantastic How-To!
Unfortunately, it seems like IOMMU doesn't work on my setup, even though my motherboard and CPU should support it according to the lists of tested hardware in the first post.

My setup:
Gigabyte ga-990fxa-ud5 running the latest BIOS version; IOMMU is enabled
AMD FX-8120
16GB DDR3 RAM (2x8)
1st graphics card: Nvidia GeForce 7600GS running nouveau
2nd graphics card (to be passed through): ATI 7870 2GB Ghz edition running pciback
OS: Linux Mint 17 Cinnamon (kernel 3.13.11.2)

Everything works fine until step 18 when attempting to create the Windows domU:

Code: Select all

ga-990fxa-ud5 boot # xl create /etc/xen/win7.cfg
Parsing config from /etc/xen/win7.cfg
got a tsc mode string: "default"
libxl: error: libxl_pci.c:1045:libxl__device_pci_add: PCI device 0000:04:00.0 cannot be assigned - no IOMMU?
libxl: error: libxl_pci.c:1045:libxl__device_pci_add: PCI device 0000:04:00.1 cannot be assigned - no IOMMU?

"dmesg | grep -i AMD-Vi" has no output

iommu kernel options

Code: Select all

ga-990fxa-ud5 boot # zgrep -i iommu /boot/config-3.13.11.2
CONFIG_GART_IOMMU=y
CONFIG_CALGARY_IOMMU=y
CONFIG_CALGARY_IOMMU_ENABLED_BY_DEFAULT=y
CONFIG_IOMMU_HELPER=y
CONFIG_VFIO_IOMMU_TYPE1=m
CONFIG_IOMMU_API=y
CONFIG_IOMMU_SUPPORT=y
CONFIG_AMD_IOMMU=y
CONFIG_AMD_IOMMU_STATS=y
CONFIG_AMD_IOMMU_V2=m
CONFIG_INTEL_IOMMU=y
# CONFIG_INTEL_IOMMU_DEFAULT_ON is not set
CONFIG_INTEL_IOMMU_FLOPPY_WA=y
# CONFIG_IOMMU_DEBUG is not set
# CONFIG_IOMMU_STRESS is not set

my grub config

Code: Select all

GRUB_DEFAULT=3
#GRUB_HIDDEN_TIMEOUT=0
#GRUB_HIDDEN_TIMEOUT_QUIET=true
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="radeon.blacklist=1 iommu=1 kopt=root=/dev/mapper/vgmint-root"
GRUB_CMDLINE_LINUX=""
GRUB_CMDLINE_XEN="iommu=1 dom0_mem=8192M"

I assured that the options in fact get loaded:

Code: Select all

ga-990fxa-ud5 boot # dmesg | grep -i iommu
[    0.000000] Command line: placeholder root=/dev/mapper/vgmint-root ro iommu=1 radeon.blacklist=1 kopt=root=/dev/mapper/vgmint-root
[    4.306782] Kernel command line: placeholder root=/dev/mapper/vgmint-root ro iommu=1 radeon.blacklist=1 kopt=root=/dev/mapper/vgmint-root

I also tried the option "iommu=pt" on the recommendation of a friend but neither of them seem to have an effect.

Thanks for the help in advance!

[powerhouse]

@Boggler:

1. About IOMMU support/activation, check this: http://wiki.xen.org/wiki/Xen_Common_Pro ... pported.3F

2. My how-to describes VGA passthrough using the xm toolstack. With Linux Mint 17 you may want to try the xl toolstack. Pay attention to the guest.cfg configuration file - it needs to contain

Code: Select all

device_model_version = 'qemu-xen-traditional'

Here is an example of my configuration file, with some notes on Ubuntu 14.04 which are also valid for Linux Mint 17: http://forums.linuxmint.com/viewtopic.p ... 00#p846397.

3. Check that your graphics card has been bound to pciback and that it's assignable:

Code: Select all

sudo xl pci-assignable-list

You should get something like this:

Code: Select all

user@powerhouse ~ $ sudo xl pci-assignable-list
0000:02:00.0
0000:02:00.1

4. Some motherboards / CPUs require special settings, either in grub or in the guest config files. If non of the above helps or provides clues then the Xen wiki (see link above) is a good place to start looking. It might also be worthwhile signing up for the xenuser mailing list.

5. If nothing works, you may want to try kvm. Sometimes kvm works better than Xen, and vice versa. Unfortunately I haven't found the time to write a kvm how-to, in fact I have very little experience with kvm, though I did successfully pass through an AMD 7770 card using kvm with primary passthrough . Here is an excellent thread with lots of information: https://bbs.archlinux.org/viewtopic.php?id=162768.

Below is my kvm startup script (and qemu-system-x86_64 guest configuration):

Code: Select all

#!/bin/bash

configfile=/etc/vfio-pci.cfg

vfiobind() {
	dev="$1"
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
   
}

modprobe vfio-pci

cat $configfile | while read line;do
	echo $line | grep ^# >/dev/null 2>&1 && continue
	vfiobind $line
	done

sudo qemu-system-x86_64 \
-bios /usr/share/qemu/bios.bin -vga none \
-name win7 \
-cpu host \
-smp 6,sockets=1,cores=3,threads=2 \
-enable-kvm \
-m 8G \
-rtc clock=host \
-vga none \
-serial null \
-parallel null \
-monitor none \
-display none \
-k en-us \
-machine type=q35,accel=kvm \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-boot order=c \
-device piix4-ide,bus=pcie.0,id=piix4-ide \
-drive file=/media/user/win7kvm/windows.img,id=disk,format=raw -device ide-hd,bus=piix4-ide.0,drive=disk \
-drive file=/home/user/Win7.iso,id=isocd -device ide-cd,bus=piix4-ide.1,drive=isocd \
-drive file=/home/user/virtio-win-0.1-74.iso,id=virtiocd -device ide-cd,bus=ide.1,drive=virtiocd \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=pcie.0 \
-device vfio-pci,host=00:1a.0,bus=pcie.0 \
-net nic,model=virtio,macaddr=00:16:3e:00:01:01 -net bridge,br=virbr0

exit 0

The content of my /etc/vfio-pci.cfg, that is the PCI IDs I pass through, is:

Code: Select all

0000:02:00.0
0000:02:00.1
0000:00:1a.0

If Passmark produces a BSOD, enter the following in a terminal screen (as root):

Code: Select all

echo "options kvm ignore_msrs=1" >> /etc/modprobe.d/kvm.conf

Reboot the PC, start the VM and run Passmark again. This should do the trick. See also https://bbs.archlinux.org/viewtopic.php ... 9#p1413079 and the replies/posts following it (2 pages).

It was much easier to get the AMD 7770 work under kvm than under Xen, and I didn't apply any patches whatsoever. You can follow the instructions of nbhs in the Archlinux thread (see link above), but without applying the patches. Modify my startup script to suit your needs.

[Boggler]

Thanks for your time and input

1. It seems that IOMMU is not supported on my system:

Code: Select all

ga-990fxa-ud5 opt # xl dmesg | grep -i i/o
(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) I/O virtualisation disabled

2. I already switched to the xl toolstack as xen told me that xm isn't supported anymore (or something similar)
device_model_version = 'qemu-xen-traditional' is already in my win7.cfg as it was recommended when attempting to create the domU.
I also adapted a few other parameters as there were errors when attempting to create the domU. Now the config seems fine and no errors are thrown when attempting to create the domU (except the missing IOMMU, of course).

3. My ATI graphics card was using the pciback driver and was also marked as assignable (not at the moment though, as I am using it as primary till there is progress on the IOMMU issue - the Nvidia graphics card is really slow and passive, so I get artifacts after a few hours).

At this point there is a little flaw in your how to: In step 9 you recommend to skip to step 11 if one isn't going to passthrough usb controllers, but I actually had to do the steps 9 and 10 with my graphics cards's address in order to get it listed as assignable and use the pciback driver.

4./5. The issue has to be with my hardware, as IOMMU as such isn't supported (I checked with an arch live session, because my friend, who uses kvm/qemu at first said it was the fault of xen or mint).
This is very strange, as my motherboard as well as my CPU are known to support IOMMU and to work with xen.
I think it is something with my CPU because

Code: Select all

sudo cat /proc/cpuinfo | grep -i svm

has no output.

Thank you for your help, but it seems like I have to get myself another CPU. Or do you think it is possible to enable this flag? Have you got any other ideas perhaps?

best regards

[powerhouse]

@Boggler:

Code: Select all

sudo cat /proc/cpuinfo | grep -i svm

You need to boot into a non-Xen environment to see this flag. While in Xen you won't see the svm flag!

According to the specs I've seen it should support AMD-Vi.

[Boggler]

@powerhouse

When choosing the normal boot option in grub, all the features seem to be recognized. svm is now in the list of supported features, AMD-Vi as well and IOMMU somehow:

Code: Select all

ga-990fxa-ud5 boot # dmesg | grep -i AMD-Vi
[    2.017670] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
[    2.017673] AMD-Vi: Interrupt remapping enabled
[    2.030492] AMD-Vi: Lazy IO/TLB flushing enabled
[    3.482930] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000020440 flags=0x0070]
[    3.487582] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000020450 flags=0x0070]
[    3.492209] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000020440 flags=0x0070]
[    3.496795] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000020450 flags=0x0070]
[    3.672809] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000020440 flags=0x0070]
[    3.672819] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000020450 flags=0x0070]
[    3.830845] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000020420 flags=0x0070]
[    3.830850] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000020430 flags=0x0070]
[    3.830857] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x00000000000585b0 flags=0x0070]
[    3.830860] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x00000000000585c0 flags=0x0070]
[    3.830861] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000058600 flags=0x0070]
[    3.830863] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000058640 flags=0x0070]
[    3.830865] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000058680 flags=0x0070]
[    3.830868] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x00000000000586c0 flags=0x0070]
[    3.830871] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000058700 flags=0x0070]
[    3.830872] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000058740 flags=0x0070]
[    3.830874] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x00000000000203e0 flags=0x0070]
[    3.830876] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000058780 flags=0x0070]
[    3.830877] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x00000000000587a0 flags=0x0070]
[    3.830878] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x00000000000587b0 flags=0x0070]
[    8.835217] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000020440 flags=0x0070]
[    8.835274] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000020450 flags=0x0070]
[    9.162974] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000020440 flags=0x0070]
[    9.163094] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000020450 flags=0x0070]
[    9.321150] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000020420 flags=0x0070]
[    9.321153] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x00000000000595b0 flags=0x0070]
[    9.321156] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x00000000000203e0 flags=0x0070]
[    9.321158] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000020430 flags=0x0070]
[    9.321160] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x00000000000595c0 flags=0x0070]
[    9.321163] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000059680 flags=0x0070]
[    9.321165] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000059600 flags=0x0070]
[    9.321168] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x00000000000596c0 flags=0x0070]
[    9.321170] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000059640 flags=0x0070]
[    9.321172] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000059700 flags=0x0070]
[    9.321175] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000059740 flags=0x0070]
[    9.321177] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000059780 flags=0x0070]
[    9.321179] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x00000000000597a0 flags=0x0070]
[    9.321182] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x00000000000597b0 flags=0x0070]
[   14.325220] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000020440 flags=0x0070]
[   14.325309] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000020450 flags=0x0070]
[   14.653052] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000020440 flags=0x0070]
[   14.653175] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000020450 flags=0x0070]
ga-990fxa-ud5 boot # dmesg | grep -i iommu
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-3.13.11.2 root=/dev/mapper/vgmint-root ro iommu=1 radeon.blacklist=1 iommu=1 kopt=root=/dev/mapper/vgmint-root
[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.13.11.2 root=/dev/mapper/vgmint-root ro iommu=1 radeon.blacklist=1 iommu=1 kopt=root=/dev/mapper/vgmint-root
[    0.000000] Please enable the IOMMU option in the BIOS setup
[    2.017670] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40

Right at the end it says, that IOMMU should be enabled in the BIOS setup, but I double-checked: it is enabled. I toggled the option and rebooted several times and even reset the CMOS.
The events above that concern the my 2nd SATA controller (Marvell 88SE9172), that seems to stop working as soon as IOMMU is enabled in BIOS setup. But I think there is a patch for that, I will look into it as it could be the source of my problems. I also contacted the Gigabyte support in the matter of IOMMU and whether or how I can get an UEFI running on the GA-990FXA-UD5 Rev 1.0. Hopefully they can help me, but I doubt it as they don't even have Linux as an option in the OS field and probably will blame Linux in terms of the non-existing IOMMU functionality.

Thanks & best regards

[powerhouse]

@powerhouse

When choosing the normal boot option in grub, all the features seem to be recognized. svm is now in the list of supported features, AMD-Vi as well and IOMMU somehow:

Code: Select all

ga-990fxa-ud5 boot # dmesg | grep -i AMD-Vi
[    2.017670] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
[    2.017673] AMD-Vi: Interrupt remapping enabled
[    2.030492] AMD-Vi: Lazy IO/TLB flushing enabled
[    3.482930] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000020440 flags=0x0070]
[    3.487582] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000020450 flags=0x0070]
[    3.492209] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000020440 flags=0x0070]
[    3.496795] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000020450 flags=0x0070]
[    3.672809] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000020440 flags=0x0070]
[    3.672819] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000020450 flags=0x0070]
[    3.830845] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000020420 flags=0x0070]
[    3.830850] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000020430 flags=0x0070]
[    3.830857] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x00000000000585b0 flags=0x0070]
[    3.830860] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x00000000000585c0 flags=0x0070]
[    3.830861] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000058600 flags=0x0070]
[    3.830863] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000058640 flags=0x0070]
[    3.830865] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000058680 flags=0x0070]
[    3.830868] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x00000000000586c0 flags=0x0070]
[    3.830871] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000058700 flags=0x0070]
[    3.830872] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000058740 flags=0x0070]
[    3.830874] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x00000000000203e0 flags=0x0070]
[    3.830876] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000058780 flags=0x0070]
[    3.830877] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x00000000000587a0 flags=0x0070]
[    3.830878] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x00000000000587b0 flags=0x0070]
[    8.835217] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000020440 flags=0x0070]
[    8.835274] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000020450 flags=0x0070]
[    9.162974] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000020440 flags=0x0070]
[    9.163094] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000020450 flags=0x0070]
[    9.321150] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000020420 flags=0x0070]
[    9.321153] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x00000000000595b0 flags=0x0070]
[    9.321156] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x00000000000203e0 flags=0x0070]
[    9.321158] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000020430 flags=0x0070]
[    9.321160] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x00000000000595c0 flags=0x0070]
[    9.321163] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000059680 flags=0x0070]
[    9.321165] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000059600 flags=0x0070]
[    9.321168] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x00000000000596c0 flags=0x0070]
[    9.321170] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000059640 flags=0x0070]
[    9.321172] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000059700 flags=0x0070]
[    9.321175] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000059740 flags=0x0070]
[    9.321177] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000059780 flags=0x0070]
[    9.321179] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x00000000000597a0 flags=0x0070]
[    9.321182] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x00000000000597b0 flags=0x0070]
[   14.325220] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000020440 flags=0x0070]
[   14.325309] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000020450 flags=0x0070]
[   14.653052] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000020440 flags=0x0070]
[   14.653175] AMD-Vi: Event logged [IO_PAGE_FAULT device=07:00.1 domain=0x0000 address=0x0000000000020450 flags=0x0070]
ga-990fxa-ud5 boot # dmesg | grep -i iommu
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-3.13.11.2 root=/dev/mapper/vgmint-root ro iommu=1 radeon.blacklist=1 iommu=1 kopt=root=/dev/mapper/vgmint-root
[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.13.11.2 root=/dev/mapper/vgmint-root ro iommu=1 radeon.blacklist=1 iommu=1 kopt=root=/dev/mapper/vgmint-root
[    0.000000] Please enable the IOMMU option in the BIOS setup
[    2.017670] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40

Right at the end it says, that IOMMU should be enabled in the BIOS setup, but I double-checked: it is enabled. I toggled the option and rebooted several times and even reset the CMOS.
The events above that concern the my 2nd SATA controller (Marvell 88SE9172), that seems to stop working as soon as IOMMU is enabled in BIOS setup. But I think there is a patch for that, I will look into it as it could be the source of my problems. I also contacted the Gigabyte support in the matter of IOMMU and whether or how I can get an UEFI running on the GA-990FXA-UD5 Rev 1.0. Hopefully they can help me, but I doubt it as they don't even have Linux as an option in the OS field and probably will blame Linux in terms of the non-existing IOMMU functionality.

Thanks & best regards

The source of the problem is most likely the Marvell SATA controller - see for example here: https://bugzilla.kernel.org/show_bug.cgi?id=42679. Note that your controller has also been mentioned as problematic - see comment 18!

I also have a Marvell controller that doesn't work with Xen and just disabled it. Try that too and see if it works.

[FastRealm]

I have just tested LM17 with Xen 4.4 and here are my findings.
1) No error code 12 in Windows saying not enough resources to run the GPU. I get this error code if i use ubuntu 14.04 with xen 4.4. To solve this just use "device_model=xen-traditional"
2) XL GPU detached and attached bug. GPU cannot reattached after restart. Still need the script or reboot PC.
3) LM17 network speed is cap to 15MBps, this is due to "xen-traditional". On ubuntu if i use "qemu-dm" i will get 90-130MBps but my GPU cannot boot up due to not enough resources "error code 12", on the other hand if i use "xen-traditional" network speed is cap to 15MBps but GPU can boot up. You will see a screen lag and mouse lag when you attempt to transfer files over network or use synergy.
4) HDD speed Improvement when use with the latest GPLPV driver.
5) Windows 8.1 works but slow and doesn't play nice with GPLPV driver.

Conclusion, unless you don't share folder between Linux and Windows then Xen 4.4 is very fast. But if you transfer files between Linux, Windows and other Backup devices then Xen 4.4 is a not recommended. I have not found any fix/solution to up the network speed, anyone got any solution or fix please share.

Cheers all

[powerhouse]

I have just tested LM17 with Xen 4.4 and here are my findings.
1) No error code 12 in Windows saying not enough resources to run the GPU. I get this error code if i use ubuntu 14.04 with xen 4.4. To solve this just use
2) XL GPU detached and attached bug. GPU cannot reattached after restart. Still need the script or reboot PC.
3) LM17 network speed is cap to 15MBps, this is due to "xen-traditional". On ubuntu if i use "qemu-dm" i will get 90-130MBps but my GPU cannot boot up due to not enough resources "error code 12", on the other hand if i use "xen-traditional" network speed is cap to 15MBps but GPU can boot up. You will see a screen lag and mouse lag when you attempt to transfer files over network or use synergy.
4) HDD speed Improvement when use with the latest GPLPV driver.
5) Windows 8.1 works but slow and doesn't play nice with GPLPV driver.

Conclusion, unless you don't share folder between Linux and Windows then Xen 4.4 is very fast. But if you transfer files between Linux, Windows and other Backup devices then Xen 4.4 is a not recommended. I have not found any fix/solution to up the network speed, anyone got any solution or fix please share.

Cheers all

Hello FastRealm, thanks for the findings.

1) I haven't seen this issue with Ubuntu 14.04, nor with LM17 - probably different hardware. "device_model=xen-traditional" is needed for both Ubuntu and LM when using Xen 4.4, as the new qemu-xen still doesn't support VGA passthrough, so they say.

2) I have seen that issue when trying kvm and my Nvidia Quadro 2000 - somehow it won't reboot. Also some AMD cards have this issue. IIRC, my AMD 7770 doesn't work with XL and Xen 4.4 (i.e. doesn't allow VM reboot), whereas it works witk kvm and primary passthrough without a problem (multiple VM restarts and benchmarks) - go figure (see also my post here). Now the AMD 6450 does work with Xen 4.4 and secondary passthrough. My Nvidia Quadro 2000 cards works nicely under Xen 4.3 and 4.4, and both xm and xl toolstacks. I couldn't get the Nvidia Quadro 2000 to work under kvm yet, but that's due to my mistake (I found the solution . In general, kvm is supposed to handle Nvidia cards better than Xen (there are a number of success stories, though in most cases it requires some patches).
If I remember correctly, some kvm/qemu developer explained that Nvidia jeopardizes VGA passthrough on purpose in their "enthusiast" line of graphics cards. The Nvidia driver (under Windows) is able to detect if it's running in a virtual machine and then disables the card (or seizes to work). The professional Nvidia graphics cards (e.g. Quadro series from Quadro 2000 onwards), using the same driver, detects the Quadro card and carries on doing its job. This is why some/many people are "quadro-fying" their enthusiast cards, i.e. hardware-coding them into their Quadro-series equivalents - see here and here.

There have been some very revealing posts over on the Archlinux forum concerning Nvidia's disabling their Windows (and Linux) driver when encountering an "enthusiast" graphics card running inside a VM - see here (that post and some of the following ones) and here. When you try to pass through a regular Nvidia card (non-Quadro 2000 or higher) you will get an "error 43" in Windows after installing the Nvidia driver, and pass-through won't work. Nvidia seems to do this on purpose (to sell more expensive Quadro cards), or it may have been an oversight (???). The latest kvm (development tree) has an option to hide the hypervisor by adding "kvm=off" to the command line. I wonder when something similar is being implemented under Xen?

Back to your topic: This guest reboot issue is probably a result of the graphics card not being properly reset after Windows VM shutdown. I don't know why the xm toolstack and some old Xen releases don't show that issue, and why newer Xen releases and xl suffer from this problem. The GPLPV drivers might interfere with the Windows script (to detach the graphics card), as you already reported earlier. It should be mentioned that Citrix announced new open-source PV drivers that will be available via the Windows Update mechanism - see http://wiki.xenproject.org/wiki/Windows ... t_Proposal.

3) This isn't normal. I can't remember any network issues, but I'm not sure I did throughput tests. Can you post your Windows guest.cfg file? It could be a configuration issue.

4) Nice to hear - I need to update my GPLPV drivers and see if the new drivers also affect disk I/O speed on LM16/Xen 4.3.

5) What do you mean by slow? Do you have benchmarks to compare? Or are you referring to the slow network speed? I'm a little confused.

P.S.: I'm still running LM16 with Xen 4.3 and xl toolstack. All my tests were done using an external HDD drive to install LM17 / Ubuntu 14.04 etc. Actually, I would still run LM13 if it wasn't for this how-to. Just that right now I haven't got the time to upgrade to LM17 and update the how-to.