HOW-TO make dual-boot obsolete using XEN VGA passthrough

[Dez]

The 4.1.3-3ubuntu1 is OK, the 4.1.3-3ubuntu1.5 is broken.

The latest update version of Xen 4.2 (4.2.1-0ubuntu3.2) appears to be working ok.

Has Xen 4.1 had another update recently as well?

Do I understand you correctly that

1. You use the xm toolstack?

2. You are able to boot into Windows and it switches to the second graphics adapter?

3. Windows is using the proprietary (AMD or Nvidia) graphics driver and graphics performance is just like native (i.e. it performs as if Windows is running natively on hardware)?

If you answer all 3 questions affirmatively, I'll have to update my how-to .

The past few days I've been busy (moving apartments, etc.) so I didn't check for Xen 4.1 updates. Hope to be able to do that soon. I'll post the results.

1. No, I'm using the xl toolstack in Xen 4.2. (I used to use the xm toolstack in Xen 4.1, but have updated after updating to Mint 15).

2. Yes. With the latest 4.2.1-0ubuntu3.2 update, I boot into Windows 8 that uses a ATI 7750 and a seperate screen. It works fine.

3. Yes, Win 8 is using a proprietary AMD driver I installed seperately and the graphics performance appears to be as per native (under the performance index test).

I think your how-to is excellent for xm, but you may want to update it to reflect the xl toolstack eventually (if time permits). Using Mint 15 (LVM) and Xen 4.2 should simplify many things .

[powerhouse]

Hey, so I've been playing around with the system some more, getting thing to work just as I want them to. (Disabling the gui in dom0 and things like that) And everything seems fine sofar. But there are a few questions left.

* First of all, I noticed that the virtual machine only speaks to the graphics card when the drivers activate it. Which is after bootup, Which means that safemode and bootup questions are invisible unless you start a VNC viewer. How do you guys deal with that?

VNC viewer is what I use now (you could also use the VNC viewer of virt-manager).

* Is there a script possible on the client to disconnect and reconnect my usb pci device? So that I could use the keyboard/mouse on dom0 when windows isn't running in domU? I understand the graphics card doesn't take kindly to that sort of treatment, but I can imagine that a USB device could handle it.

In dom0, run:

Code: Select all

sudo virsh nodedev-reattach pci_0000_0b_00_0

with 0b_00_0 being the PCI device (note the underscore instead of the ":").
You can't do that with the graphics card, only USB hosts, etc.

* I'd like some way of administering xen outside the command line preferrably via the web. (Being able to start windows from my phone would be an awesome party trick) Have you guys found a solution that could help?

Try perhaps ssh -X ... - this allows remote access with GUI. You need to have a ssh server installed and running. But if you want to start Windows via iPhone there is a VNC viewer application for the iPhone. Just have a VNC server running on Xen. Haven't tried it though, but I don't see a reason why this shouldn't work.

* I have a working windows on sdb2 not wrapped in lvm, and bootable from grub is there a way I could import that thing into a VM?

"duet" seems to be the ticket - see here: [url]http://www.overclock.net/t/1205216/guide-create-a-gaming-virtual-machine/510#post_19096913[/url]. This is actually not an import but booting Windows from HDD into a Xen domU.
I personally wouldn't bother with it and prefer to do a clean install of Windows as a Xen domU. This way I know I got a clean copy without all the junk that accumulates over time. Then I install the software/apps I use and run a backup. See the following link for my domU backup and restore scripts: [url]http://forums.linuxmint.com/viewtopic.php?f=42&t=133089[/url].

[powerhouse]

...
I think your how-to is excellent for xm, but you may want to update it to reflect the xl toolstack eventually (if time permits). Using Mint 15 (LVM) and Xen 4.2 should simplify many things .

Thanks for the answers! I totally agree with you that Linux Mint 15 should simplify things a lot. Right now I'm in the middle of changing apartments, and I need both Linux and Windows to work - no time for experiments.

I hope to be able to update/modify the how-to sometime beginning of July.

[willow512]

Thanks for the answers Powerhouse. Toughest thing to deal with is that you're essentially blind whilst booting the vm, at least untill the graphics card is activated.

I think I'm all set now. I managed to play few games of chivalry inside the virtual machine. It's sort of perfect. But I do experience a more drastic framerate drop than I had hoped for. I went from 60+ (game is capped at 60) to 45, I'm guessing that's half what it could be. I'm not sure yet what causes this, I'm guessing it's a cpu thing rather than a gpu thing because the drop wasn't consistent (It mainly seemed to occur whenever more players were active on the same map). The Windows experience index showed all stats nice and high (7.9 ish) except for harddrive which was a poor 5, then again, I was copying files when doing the experience index so take that with a grain of salt, and even if IO is really that poor it should not affect my game once it is running.

I'll experiment with that a little more to see if I can find out anything, if nothing else I can always lower the render quality..

Good luck with the appartment change!

[powerhouse]

@willow512: I understand you deliberately don't run a GUI in dom0.

Though secondary passthrough is easier, you're "flying blind" until it switches to the "second" graphics card (the primary being the emulated Cirrus card). If your graphics card supports it, you may experiment with primary passthrough, but it usually requires patches and compiling. However, if there is no compelling need I would skip this.

So, sometimes after a Windows update you will not be able to start your VM in the first instance, but it should come up the second time you start it. This may be a little challenging when you don't have any GUI or screen in fact. Having at least a remote shell from another PC is helpful, or better ssh -X or VNC.

I'm not using my Windows VM for games so I can't tell if my graphics performance under Windows is what it's supposed to be, except that I don't have any issues accessing my hardware via Windows applications (I have software that uploads color adjustments to the screen using the DVI port). The lower frame rate could be a CPU limit. Haven't you got a 4-core 3770? You could perhaps check via System monitor if the CPU is reaching some limit.

Low I/O performance: have you tried the latest GPLPV drivers? Use the ones from [url]http://www.meadowcourt.org/downloads/[/url], the Vista drivers for Windows Vista/7/8 (not the debug). The drivers had my WEI jump from ~5.9 to 7.8 for an SSD-based installation. I hope you run your Windows VM on a LVM formatted RAW drive (the way I described).

EDIT: I/O is handled by dom0 (unless you passthrough a SATA controller). So if you don't give dom0 enough resources, or your I/O setup is suboptimal (drivers, whatever), you may experience performance problems in the VM as well. There is a xm command that gives you the stats for both dom0 and the guest(s): xm top.

[willow512]

@Powerhouse
Indeed, I wanted to setup the machine so it'd go from a linux mint loading screen straight into windows and from there work with linux via ssh. I love the CLI, xwindows and windows really don't make a big difference for me. I'd prefer a web interface to manage xen with. But if that fails then cli will do.


I also figured out the reason for the poor performance and now everything is (as far as I can tell) up to native speeds.. Turns out that if you give your virtual machine 10 virtual cpus when your cpu only has 8 hyper threads a lot of swapping is going to happen. (Who'd have guessed right? ) When I pinned the first 2 virtual cpus to the dom0 and the other 6 to the windows domU performance skyrocketed back to native. (Funny enough this made no difference for my windows experience index.)

Harddrive performance still isn't optimal. The dom0 has 4gb memory and 2 vcpus, so that should be enough. The harddrive is a green eco friendly drive, soI fear it may just not be capable of going faster. I haven't tested it native yet I was planning to do that tonight. I also haven't installed those GPLPV drives yet, I'll also do that tonight, that's a good tip. And yes, windows is installed in an lvm the way you described. (The dom0 partition isn't, but the windows virtual machine is)

[powerhouse]

@willow512:

So you copied my win7.cfg file without changing the VCPUs ? I guess I should change it to something more normal - not everyone has a 6-core/12 thread CPU.

With regard to the Windows Experience Index, the CPU "overbooking" for the Windows guest will perhaps only show in low I/O performance, and probably not even there. So when you fixed the VCPU setting in your Windows config file it won't influence the WEI, but most likely the real performance (because real applications use CPU, GPU and I/O at the same time, not consecutively as the WEI measures).

Try the GPLPV drivers, they should improve I/O (disk) performance dramatically. But, some people also reported that the GPLPV drivers may impede network performance. I for myself am using them and it's OK for me. Disk performance is definitely much improved.

The green HDDs aren't the fastest, but I would like to see the performance difference of your HDD when used natively under Windows and when used with a Windows guest using GPLPV drivers, if you can do that? I can't check Windows in a native installation.

[FastRealm]

Thanks to powerhouse detailed step by step guide, i(learning how to use linux) have manage to get Xen working.
Specs -
AMD A10 5800k (c/w ATI 7600)
Asrock FM2A85X-Mini ITX
16GB Ram
ATI Radeon 5770

There is one thing that i don't fully understand, you set the "dom0_mem=4G" does this mean that the dom0 will only have 4G of usable RAM?
If domU is not running will dom0 fully utilize all the 16GB RAM?


It is possible to set domU to always use cpu 2,3 rather than any random cpu that is available when domU is activated.

Once again thank you for the step by step guide.

[powerhouse]

Thanks to powerhouse detailed step by step guide, i(learning how to use linux) have manage to get Xen working.
Specs -
AMD A10 5800k (c/w ATI 7600)
Asrock FM2A85X-Mini ITX
16GB Ram
ATI Radeon 5770

There is one thing that i don't fully understand, you set the "dom0_mem=4G" does this mean that the dom0 will only have 4G of usable RAM?
If domU is not running will dom0 fully utilize all the 16GB RAM?


It is possible to set domU to always use cpu 2,3 rather than any random cpu that is available when domU is activated.

Once again thank you for the step by step guide.

Thanks for sharing your hardware specs. I understand that you run Windows and pass through your ATI Radeon 5770?

Now to your questions:

1. Yes, dom0 will "only" use 4G of RAM, unless you enable memory ballooning (see the guide). Memory ballooning is NOT recommended, however. It's best to adjust your dom0 memory to a reasonable amount - 4G is more than enough for almost any purpose (at least when you use Linux).

2. Yes, you can pin your vcpus. This is sometimes useful for dom0, where you want to ensure that dom0 gets enough CPU resources to handle its own requirements as well as the I/O from all VMs. I do not use vcpu pinning and have setup Xen so it will give dom0 all vcpus when Windows (domU) isn't running, and take as much CPU time it can get when Windows is running. In my case Windows gets 10 vcpus (out of 12), dom0 2 vcpus. However, Xen will allocate the resources according to demand, so Windows will only get the full 10 vcpus when it really requires them. See the how-to for more explanations.

[Hoerbsli]

Hi,

I also just wanted to get a Win7 HVM running. Also used the latest Xen driver 4.2.1 package for Mint 15.

What can i say: It worked fine without any problem for the first start with the XL toolstack. But with the XL toolstack I cannot reboot the domU. the GPU only works on the first run of the Domu. On all later runs the GPU passthrough doesn't work(yellow question marks in Windows).
It seems the xl toolstack cannot reset the card properly (btw the card doesn't support flreset).

When I instead use the XM toolstack, with the same configuration, I can reboot the domU as often as I like, the passthrough always works.
But with the XM toolstack, I can only pass the GPU itself. E.g. when I try to add the additional HDMI device of the GPU, I get an "Invalid argument" error on starting the VM.

BTW I'm using a Radeon 7750 on a Z77 Board (extreme 6) with a i7-3770 and 32GB RAM.

BTW2: Isn't it strange, that when the secondary GPU is added, the primary GPU is only used for startup, then it gets neglected by Win7. I always thought it would go into a dual GPU mode, with the qemu device and the radeon working together. Is it because the standard driver for the qemu GPU device doesn't support WDDM 1.1?

BTW3: How would one do a primary passthrough. As I get it, it should work for my GPU. But when I enable it, nothing happens...the screen remains blank...

[powerhouse]

@Hoerbsli: Thanks for sharing this info. Not being able to reboot the domU sounds much like what's been reported by KVM users. Some suggest to "eject" the graphics card in Windows before shutting Windows down. I haven't tried it yet, nor have I tried the xl toolstack or Mint 15.

About xm toolstack for Mint 15: Have you tried changing the order of the PCI devices you pass through? Not sure it helps, but somewhere I read about it.

I never thought about why secondary passthrough works with Windows as it does. Actually, I once tried VGA passthrough with Linux - it's a nightmare and I couldn't get it working.

Hope to be able to upgrade to Mint 15 soon, though I'm not sure I should. Mint 14 and the xm toolstack just works (well, as long as you force Mint to use an older version).

[Hoerbsli]

Some suggest to "eject" the graphics card in Windows before shutting Windows down.

I read something similar too. But when it works in the first run, then after ejecting the GPU, the screen just gets blank for a short time and the reappears and is again "mounted". And when running after a reboot, the device cannot be ejected, as it is in an error state anyways...
What I read, it should help, if there is some strange reduction in power after a reboot. All in all it seems, there's really a problem with the XL toolstack in regarding to reset the devices properly resulting in different problems.

@Dez: Don't you also have this reboot problem with your Xen 4.2.1 XL setup?

Have you tried changing the order of the PCI devices you pass through?

AFAIK this is a known problem with the newer versions. I even think it has been solved in the meantime, but then it hasn't yet propagated into the newest mint update.
It seems kinda strange to me, that especially the never versions have so much more problems than the older ones. One would think that they would be better tested before releasing new versions...
As I see it, it could be this problem:
[url]http://xen.crc.id.au/bugs/view.php?id=5[/url]
But how to easily apply this patch to Mint 15?

[willow512]

So you copied my win7.cfg file without changing the VCPUs ? I guess I should change it to something more normal - not everyone has a 6-core/12 thread CPU.

Yeeep, that's what I did. :/ I remember specifically thinking. "Hey I wonder how many cores my cpu has, let me come back to that later." And then promptly forgetting.

I looked up the WEI for you.
Native:
Proc: 7.7 Mem: 7.7 Desktop: 7.7 3d: 7.7 Disk: 7.9

And virtual:
Proc: 7.7 Mem: 7.9 Desktop: 7.9 Disk: 5.9

So disk is not improved much in spite of the GPLPV drivers. At least in WEI, I haven't checked actual throughput rates.

I'm using the virtual pc more often now, some other minor nuisances have come up. For example when I turn on my computer I have 2 or 3 failed boots that result in a kernel panic before I get into mint. From there everything is fine. Windows inside the virtual machine gives me license problems for some reason the machine isn't identified properly. I guess those guys didn't expect anyone running unity from a VM..

I hope they will release the proper version of xen for mint 15 soon so I may test if this solves the booting problem.

[powerhouse]

@willow512: somewhere I remember that you installed the Windows guest without LVM? That may explain the low WEI for disk I/O.

[FastRealm]

Just for sharing.

My Xen Server
AMD A10 5800k (c/w ATI 7600)
Asrock FM2A85X-Mini ITX
16GB Ram
ATI Radeon 5770
1x Corsair GT 3 SSD
6x WD Green HDD(ZFS Storage)

I decided to install Xubuntu 13.04 with the latest Xen 4.2.1 to try out and see how fares against the current LM14 with Xen 4.1.3.

4.2.1 uses XL toolstack. These are some of the changes required for XL toolstack to work.
- to check if PCI passthrough, xm-list-assignable-device --> xl pci-assignable-list
- kernel = "/usr/lib/xen-default/boot/hvmloader" --> kernel_override = "/usr/lib/xen-default/boot/hvmloader"
- 'phy:/dev/mapper/os-Windows,hda,w' --> '/dev/mapper/os-Windows,raw,hda,rw'
- device_model_override = '/usr/lib/xen-default/bin/qemu-dm'
- tsc_mode = 0 --> tsc_mode = "default"
everything else i follow the guide posted by Powerhouse. (Remember to thank powerhouse for his awesome guide)

Pros
- i can use AMD proprietary on Xen 4.2.1, where else if i install it on Xen 4.1.3 i will get a blank screen
- I don't have the numbers to proof this but i felt XL is faster than XM. Windows 7 installation from start until 1st reboot took 17mins on XM, XL took 14 mins.
- I tried borderlands 2 with the same setting, i didnt get any screen lag on XL during intense fight but i do on XM.
- Network speed is about the same +- 1-2 mb

Cons
- virt-manager doesn't work with XL toolstack. It is a bug in virt-manager and the developers are taking their own sweet time to solve it. So virt-manager is no go. i have to use VNCViewer instead.
- everytime when domU shutdown/restart, i cannot restart domU without rebooting the PC. What i notice is when i run "xl create /etc.../win.cfg" it will allocate the RAMs needed to run the domU. When i restart/shutdown domU and run "xl create /etc.../win.cfg" again the RAMs will not be allocated and domU witll not run. Worse case scenario, PC hang. So i need to reboot PC everytime domU shutdown.

Conclusion
I find 13.04 with xen 4.2.1 is a bit more responsive compare to 12.10 with 4.1.3. The annoying part is virt doesn't work and i don't know how to restart the domU without reboot the PC(still a linux idiot at the moment).

Thats all for my sharing. Hope you all find it useful.

[powerhouse]

@FastRealm: Thanks very much for sharing your hardware setup and changes using the xl toolstack, and most important for listing the pros and cons for the xl versus xm.

With regard to the Pros, I am running an AMD graphics card under dom0 using the proprietary driver. Others have reported problems getting it working. Perhaps the difference between my setup and that of most users is that I pass through a Nvidia card to domU, whereas most will have AMD cards for both dom0 and domU. Just a thought.

I'm glad to hear that the xl toolstack improves speed, as this is something to look forward to. If only there wasn't the issue with the domU restart and PCI/VGA passthrough that doesn't work (yet) .

Linux Mint 15 with xl toolstack should make things a lot easier, even if it's only the LVM option in the installer that was missing in previous releases. As to virt-manager, my opinion about it is mixed, so I don't take it as a great loss. Still I hope they'll fix it.

Once more thanks and I hope to be able to update my how-to soon. For the meantime I'll put in a link to your post for the changes in the domU config file.

P.S.: For a Linux newbie you are already doing what most would consider advanced stuff.

[easypeasy]

Cons
- virt-manager doesn't work with XL toolstack. It is a bug in virt-manager and the developers are taking their own sweet time to solve it. So virt-manager is no go. i have to use VNCViewer instead.
- everytime when domU shutdown/restart, i cannot restart domU without rebooting the PC. What i notice is when i run "xl create /etc.../win.cfg" it will allocate the RAMs needed to run the domU. When i restart/shutdown domU and run "xl create /etc.../win.cfg" again the RAMs will not be allocated and domU witll not run. Worse case scenario, PC hang. So i need to reboot PC everytime domU shutdown.

I had similar bug on Linux Mint 15 using XL toolstack it froze everything including dom0. I am running 4.3 with XM again as was still having issues with XL. Finally got passthrough to work reliably.

@powerhouse

Thanks for the guide, it was pretty much my introduction to all of this stuff and extremely helpful, thank you. I too would consider myself a newbie and wouldn't have made it this far without it. I ended up building xen-unstable to use as it was more stable than what was offered.

Not sure if this was covered in this thread already, but I put my dom0 on a small 32 GB SSD and booted one day to find it completely full. I tracked down the used space to /var/lib/xen/save . I had apparently crashed the hypervisor a few times and it was writing my domU's to there. It completely filled up my SSD.

I ended up just nulling out the saves, but you can direct them elsewhere. In /etc/default/xendomains

Code: Select all

## Type: string
## Default: /var/lib/xen/save
#
# Directory to save running domains to when the system (dom0) is
# shut down. Will also be used to restore domains from if # XENDOMAINS_RESTORE
# is set (see below). Leave empty to disable domain saving on shutdown 
# (e.g. because you rather shut domains down).
# If domain saving does succeed, SHUTDOWN will not be executed.
#XENDOMAINS_SAVE=/var/lib/xen/save
XENDOMAINS_SAVE=

[Hoerbsli]

Hmm, I tried something new (BTW I also have no experience in this stuff, just FYI).

Tried to compile and install the newest xen, in the hope the xm fix is included in there and works.
I did the following:

added the following two lines to /etc/apt/sources.list (so that build-dep works)

Code: Select all

deb-src http://archive.ubuntu.com/ubuntu/ raring main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu/ raring-updates main restricted universe multiverse

Then did the following as root(changing directories not included, as obvious):

Code: Select all

apt-get update
apt-get install git
git clone git://xenbits.xen.org/xen.git
apt-get install build-essential
apt-get build-dep xen
apt-get install yajl-tools libbz2-dev bison flex git-core texinfo debconf-utils debootstrap libpixman-1-dev
nano .config (make file with just PYTHON_PREFIX_ARG=--install-layout=deb in it)
./configure
make dist
make install
update-grub
/sbin/ldconfig

Two strange things then happened. The xen-tools services don't get started automatically anymore. So they have to be started manually:

Code: Select all

service xencommons start (for XL)
service xend start (additionally for xm)

And in the grub kernel line is a "placeholder" as argument (/boot/grub/grub.cfg). Don't know what this should be. I deleted it, as with this additional kernel option, the kernel took ages to start. Without it, it starts as usual.

I can start it with xm (now, the illegal argument bug isn't there anymore), but I always get the yellow exclamation mark on the GPU in Win7, and the GPU doesn't work.
When I try to start with xl, it always hangs on booting Win7 (on the screen with the animated Win Logo, it just remains there animated forever).

Any suggestions???

[powerhouse]

I'm glad you guys are keeping this thread alive and bringing in new ideas and questions as well. I am extremely busy now and didn't find any time playing with Xen.

Until I've tried LM 15 and Xen 4.2 with xl or xm toolstack I can't answer any questions on that. As it stands I have my doubts about "upgrading" to LM 15 - my system is running perfectly smooth on LM 14 with Xen 4.1.3-3ubuntu1 and xm (with the "force" option to ensure Xen isn't updated).

I did try all the Xen updates under LM 14 up to Xen 4.1.3-3ubuntu1.7, though. Unfortunately none works - the guest wouldn't start when using PCI passthrough.

Hope someone else can answer the questions.

[easypeasy]

Two strange things then happened. The xen-tools services don't get started automatically anymore. So they have to be started manually:

Code: Select all

service xencommons start (for XL)
service xend start (additionally for xm)

This happened to me as well. Once I got it up and running, I just launch these automatically at boot.

And in the grub kernel line is a "placeholder" as argument (/boot/grub/grub.cfg). Don't know what this should be. I deleted it, as with this additional kernel option, the kernel took ages to start. Without it, it starts as usual.

I never even noticed this. I did some Googling and only saw this: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=606483#10
According to that removing the placeholder is causing the next argument to be ignored.

I can start it with xm (now, the illegal argument bug isn't there anymore), but I always get the yellow exclamation mark on the GPU in Win7, and the GPU doesn't work.
When I try to start with xl, it always hangs on booting Win7 (on the screen with the animated Win Logo, it just remains there animated forever).

Any suggestions???

I looked back on your posts and see what type of system you are booting and it's almost an identical setup as mine (i7-3770, HD7750, I'm running Z77 Extreme4 board though). I was getting kernel panics (default - Linux Mint 15 install) when I tried to boot a domU with my HD7750. Updated kernel and that issue went away. Other than that, just passing through graphics card and a USB controller (I could never get USB devices to work right by themselves). I don't recall doing anything special to get it running after it finally booted into Windows. Just followed the advice in the original post.