HOW-TO make dual-boot obsolete using XEN VGA passthrough

Questions about virtualization software
Forum rules
Before you post read how to get help. Topics in this forum are automatically closed 6 months after creation.
Locked
Quich

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by Quich »

powerhouse wrote:@Quich: Thanks for the answers and clarifying the points.

Regarding networking: You can try to purge network-manager, but be aware that it can break networking and that is a real fiasco with LM16. In case you are prepared to reinstall LM16 if things go wrong, purge network-manager (while still using dhcp configured in /etc/network/interfaces), then reboot and change to static, then restart the network and cross your fingers. Sorry I can't be of more help, they keep messing up the networking part and it's a real shame.
Well, it is working right now, and I do not want to break anything so I will let it like that.
powerhouse wrote:Regarding sound: You could connect the headphone jack of your screen to the line-in on your rear panel. Then under Linux you would select the line-in to enable sound from Windows. The mic input works as usual under Linux, so I don't see a problem here.
Well, I tried that. But I am not able to get the input line and the mic working at the same time. It might be a configuration problem.
I tried again, and even with only the input line, the sound is horrible/not audible. Might be a cable problem.
powerhouse wrote:I agree that sound support is a tragedy under Xen. A software solution would be much neater. Have a look at "jack" here http://jackaudio.org/netjack and http://jackaudio.org/download for the Windows part, and install the jackd2 package under Linux.
I will look into that.
powerhouse
Level 6
Level 6
Posts: 1144
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by powerhouse »

Quich wrote:
powerhouse wrote:Regarding sound: You could connect the headphone jack of your screen to the line-in on your rear panel. Then under Linux you would select the line-in to enable sound from Windows. The mic input works as usual under Linux, so I don't see a problem here.
Well, I tried that. But I am not able to get the input line and the mic working at the same time. It might be a configuration problem.
I tried again, and even with only the input line, the sound is horrible/not audible. Might be a cable problem.
Re line input: If it's distortion, you may have to lower the output volume on Windows and see if that helps.

EDIT: Check out this thread, 3rd post: http://ubuntuforums.org/archive/index.p ... 55121.html
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
Quich

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by Quich »

powerhouse wrote: EDIT: Just followed the link you posted - it's a conversation between me and another Xen user I had totally forgotten :oops: . Never mind, it didn't work for me, and the other also reported that it's pretty useless.
Another thing is that this solution is based on PulseAudio, something I wouldn't run as it re-encodes on the fly every audio stream to 48kHz or so. It's good enough for games, video, and mp3 audio, but not for audiophile use.
Well, I am not an audiophile at all, so even if the quality is not very high, it might be ok for me. My Windows domU is intended to be here mainly for gaming, and the lag introduced by this technique is not acceptable for this use.
powerhouse wrote:Re line input: If it's distortion, you may have to lower the output volume on Windows and see if that helps.
EDIT: Check out this thread, 3rd post: http://ubuntuforums.org/archive/index.p ... 55121.html
Well, it did not seem to be distortion. It was weird. I will test again if I can.

Yesterday night, I tried to pass to the xl toolstack as xend (xm) is not anymore the default toolstack of Xen 4.3 and it does not support upstream-qemu but only the previous one, which does not support the hda emulated sound card that works under 7 x64.
I encountered a mysterious (xen or qemu) bug, the domU did not want to boot (I did not even get the seabios screen via vnc) if I passed the pci id of the graphic card. No problem with the pci id of the audio part of the graphic card, I was able to boot the Win7 domU and to get the emulated audio card (soundhw="hda") and the hdmi audio card, but no way to make it work with the pci-e graphic card attached. Exactly like this guy : http://xen.1045712.n5.nabble.com/VGA-Pa ... 20323.html which solved it by re-compiling qemu.

I'm currently trying to build and install Xen 4.4 unstable with an updated upstream-qemu.
Tz2001

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by Tz2001 »

I'm looking at upgrading my PC and looking for Vt-d compatible hardware so I can continue gaming without dual booting.

However, how feasible is this if you only have one GPU? Is it worthwhile or is it a lot simpler/better with two GPUs? Presumably a single GPU solves the problem of needing multiple inputs on the monitor.
Quich

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by Quich »

You need two graphic cards, because one has to be dedicated to the Windows guest.

I succeed to boot the Win7 domU with Xen-4.4 and the associated upstream-qemu with the emulated intel hda soundcard and the pci-e card passthrough (on a fresh Mint 16 Install).
However, I was not able to get any sound to my host (maybe the used qemu was not build with the alsa / pulseaudio driver), and I experienced random and weird behaviors.
At the begining, the ATI card was getting a "cannot find enough resources (code 12)". I reduced the allocated ram memory for the Guest from 4GB to 1GB (the host had 3GB, total 8GB, the graphic card is 2GB) and I was able to get the card working again after some reboot of the guest.
After that I was able to increase again the guest memory and it booted successfully. But I got strange behaviors when rebooting the guest, I sometimes had artifact on the windows desktop, and poor graphic performances (my first run at 3DMark11 was 1750pts, then I rebooted and get only 800pts, checked 2 times). The host system even freeze at a time (but before the 3DMark runs, when I was trying to get the card working).

I finally reinstall Mint another time, trying only to build upstream-qemu with the usefull parameters and to use it with Xen-4.3, but without success.

I think I will do another fresh install and try with kvm.
powerhouse
Level 6
Level 6
Posts: 1144
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by powerhouse »

Tz2001 wrote:I'm looking at upgrading my PC and looking for Vt-d compatible hardware so I can continue gaming without dual booting.

However, how feasible is this if you only have one GPU? Is it worthwhile or is it a lot simpler/better with two GPUs? Presumably a single GPU solves the problem of needing multiple inputs on the monitor.
The best way to go is with a CPU with integrated GPU (IGP), such as the Intel i5/i7 3770 or 4770 etc., or similar AMD CPUs. This way you can use the IGP for Linux and a dedicated graphics card for Windows. In any case, you need 2 physical GPUs to make it work, and you would need either 2 screens, or 2 ports on your current screen. Re CPU, check the links re hardware compatibility at the beginning. If you choose an Intel CPU, most "K" versions will not support VT-d. See the link to the Intel website.

As for graphics card, you will be much better of if you get an AMD graphics card, particularly one that is listed as compatible (again, see link at the beginning).

Pay close attention to the motherboard compatibility list: Don't buy any motherboard that doesn't explicitly say VT-d (or AMD-Vi/IOMMU) in it's specifications or BIOS release notes.

Some motherboards will not yet support VT-d when using the latest Intel Ivybridge or Haswell CPUs. Some manufacturers are planning to release new BIOS versions to support VT-d, but it most likely will not be their top priority.

Many users have reported success with ASrock motherboards. However, you would need to check specifically which CPU / motherboard / BIOS combination does support VT-d (or AMD-Vi). The best is to choose a combination that's been confirmed to work, or, if in doubt, contact the manufacturer.

Do you have any idea what kind of PC you are going to build? What are you going to use it for?
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
powerhouse
Level 6
Level 6
Posts: 1144
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by powerhouse »

Quich wrote:You need two graphic cards, because one has to be dedicated to the Windows guest.

I succeed to boot the Win7 domU with Xen-4.4 and the associated upstream-qemu with the emulated intel hda soundcard and the pci-e card passthrough (on a fresh Mint 16 Install).
...But I got strange behaviors when rebooting the guest, I sometimes had artifact on the windows desktop, and poor graphic performances (my first run at 3DMark11 was 1750pts, then I rebooted and get only 800pts, checked 2 times). The host system even freeze at a time (but before the 3DMark runs, when I was trying to get the card working).

I finally reinstall Mint another time, trying only to build upstream-qemu with the usefull parameters and to use it with Xen-4.3, but without success.

I think I will do another fresh install and try with kvm.
This isn't strange behavior, it's a documented bug related to the "xl" toolstack! Which is why I recommend "xm".

There is a work-around for this bug. It involves "ejecting" the graphics cards when you shutdown/reboot Windows - see here for a script to automate the process: http://blog.ktz.me/?p=219.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
Quich

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by Quich »

powerhouse wrote: This isn't strange behavior, it's a documented bug related to the "xl" toolstack! Which is why I recommend "xm".

There is a work-around for this bug. It involves "ejecting" the graphics cards when you shutdown/reboot Windows - see here for a script to automate the process: http://blog.ktz.me/?p=219.
Yes, I read your warning about that. I was hoping it to be fixed in 4.4.

I checked your link. Interesting, but it works with traditionnal qemu which does not support hda virtual sound card. Maybe I can adapt it.

Edit : Here is the inxi -Fx output. I just upgraded to 32GB and I am not currently running Xen.

Code: Select all

System:    Host: IdeaCentre-K450 Kernel: 3.11.0-12-generic x86_64 (64 bit, gcc: 4.8.1) 
           Desktop: N/A Distro: Linux Mint 16 Petra
Machine:   System: LENOVO product: 10120 version: IdeaCentre K450 serial: ES11638761 
           Mobo: LENOVO model: N/A version: 31900003 STD serial: ES11638761 Bios: LENOVO version: I1KT15AUS date: 03/28/2013
CPU:       Quad core Intel Core i7-4770 CPU (-HT-MCP-) cache: 8192 KB flags: (lm nx sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx) bmips: 27134.3 
           Clock Speeds: 1: 800.00 MHz 2: 800.00 MHz 3: 800.00 MHz 4: 800.00 MHz 5: 800.00 MHz 6: 800.00 MHz 7: 800.00 MHz 8: 3401.00 MHz
Graphics:  Card-1: Intel Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller bus-ID: 00:02.0 
           Card-2: Advanced Micro Devices [AMD/ATI] Oland [Radeon HD 8570] bus-ID: 01:00.0 
           X.org: 1.14.3 drivers: intel (unloaded: fbdev,vesa) tty size: 144x41 Advanced Data: N/A for root 
Audio:     Card-1: Intel Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller driver: snd_hda_intel bus-ID: 00:03.0
           Card-2: Intel 8 Series/C220 Series Chipset High Definition Audio Controller driver: snd_hda_intel bus-ID: 00:1b.0
           Card-3: Advanced Micro Devices [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] driver: snd_hda_intel bus-ID: 01:00.1
           Card-4: Trust driver: USB Audio usb-ID: 145f:0199
           Sound: Advanced Linux Sound Architecture ver: k3.11.0-12-generic
Network:   Card-1: Realtek RTL8188EE Wireless Network Adapter driver: rtl8188ee port: c000 bus-ID: 04:00.0
           IF: wlan0 state: down mac: 24:fd:52:a6:9f:17
           Card-2: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller 
           driver: r8169 ver: 2.3LK-NAPI port: d000 bus-ID: 03:00.0
           IF: eth0 state: up speed: 100 Mbps duplex: full mac: d4:3d:7e:aa:d4:a0
Drives:    HDD Total Size: 2008.3GB (0.8% used) 1: id: /dev/sda model: ST2000DM001 size: 2000.4GB temp: 30C 
           2: USB id: /dev/sdc model: STORAGE_DEVICE size: 7.9GB temp: 0C 
Partition: ID: / size: 30G used: 5.3G (19%) fs: ext4 ID: /boot size: 504M used: 44M (10%) fs: ext2 
           ID: /home size: 69G used: 9.2G (15%) fs: ext4 ID: swap-1 size: 5.37GB used: 0.00GB (0%) fs: swap 
RAID:      No RAID devices detected - /proc/mdstat and md_mod kernel raid module present
Sensors:   System Temperatures: cpu: 37.0C mobo: 29.8C 
           Fan Speeds (in rpm): cpu: N/A 
Info:      Processes: 222 Uptime: 18 min Memory: 1102.4/32052.7MB Runlevel: 2 Gcc sys: 4.8.1 Client: Shell inxi: 1.8.4 
Last edited by Quich on Mon Jan 13, 2014 10:41 am, edited 1 time in total.
powerhouse
Level 6
Level 6
Posts: 1144
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by powerhouse »

I finally got around to clean out some old stuff, namely the /etc/modules method of loading the xen-pciback module which I had forgotten to remove. It didn't do any harm, though.

The how-to now uses the initramfs method of loading xen-pciback, which should make sure that the module gets loaded before the graphics driver. In the same step we bind the graphics card PCI IDs to xen-pciback.

I hope this solves the issue where sudo xm pci-list-assignable-devices doesn't list the graphics card PCI IDs.

If that doesn't work, one may need to blacklist the graphics card driver. Depending on the hardware configuration, it's usually enough to blacklist the open source driver (after installing the proprietary driver for dom0) for the graphics card we pass through.

The last resource - if nothing works - is to compile xen-pciback into the kernel and add "xen-pciback.hide=(02:00.0)(02:00.1)" to the XEN_LINUX_CMDLINE in grub.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
Tz2001

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by Tz2001 »

powerhouse wrote:
Tz2001 wrote:I'm looking at upgrading my PC and looking for Vt-d compatible hardware so I can continue gaming without dual booting.

However, how feasible is this if you only have one GPU? Is it worthwhile or is it a lot simpler/better with two GPUs? Presumably a single GPU solves the problem of needing multiple inputs on the monitor.
The best way to go is with a CPU with integrated GPU (IGP), such as the Intel i5/i7 3770 or 4770 etc., or similar AMD CPUs. This way you can use the IGP for Linux and a dedicated graphics card for Windows. In any case, you need 2 physical GPUs to make it work, and you would need either 2 screens, or 2 ports on your current screen. Re CPU, check the links re hardware compatibility at the beginning. If you choose an Intel CPU, most "K" versions will not support VT-d. See the link to the Intel website.

As for graphics card, you will be much better of if you get an AMD graphics card, particularly one that is listed as compatible (again, see link at the beginning).

Pay close attention to the motherboard compatibility list: Don't buy any motherboard that doesn't explicitly say VT-d (or AMD-Vi/IOMMU) in it's specifications or BIOS release notes.

Some motherboards will not yet support VT-d when using the latest Intel Ivybridge or Haswell CPUs. Some manufacturers are planning to release new BIOS versions to support VT-d, but it most likely will not be their top priority.

Many users have reported success with ASrock motherboards. However, you would need to check specifically which CPU / motherboard / BIOS combination does support VT-d (or AMD-Vi). The best is to choose a combination that's been confirmed to work, or, if in doubt, contact the manufacturer.

Do you have any idea what kind of PC you are going to build? What are you going to use it for?

Thanks, I've been reading a lot about the CPU/Motherboard requirements and while problematic, that's easy enough to achieve.

Is there any technical reason the VM can't take exclusive access of the GPU while it's active? Obviously linux would visually inaccessible while the windows VM was running.

The reason I'd like to use the same (good) GPU for both Windows and Linux is that with SteamOS around the corner, Linux Gaming is going to take a step forward and we'll have a better selection of native linux games, which will make having a decent GPU in linux useful.

I'm looking at a i7 4771 (which is a locked 4770k and has VT-d), an ASRock Z87 board, and an ati gpu (currently undecided). Primary use is work (PHP/Mysql Web development, some Java development, general office use) and gaming. While currently overkill, I'd like something that's somewhat futureproof as using my PC for work also, I'd rather not take it apart frequently. I'm currently running an i7 920 which I got in 2009 and was one of the best PC component investments I ever made despite it being overkill for almost every game of the time.
powerhouse
Level 6
Level 6
Posts: 1144
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by powerhouse »

Tz2001 wrote:Thanks, I've been reading a lot about the CPU/Motherboard requirements and while problematic, that's easy enough to achieve.

Is there any technical reason the VM can't take exclusive access of the GPU while it's active? Obviously linux would visually inaccessible while the windows VM was running.

The reason I'd like to use the same (good) GPU for both Windows and Linux is that with SteamOS around the corner, Linux Gaming is going to take a step forward and we'll have a better selection of native linux games, which will make having a decent GPU in linux useful.

I'm looking at a i7 4771 (which is a locked 4770k and has VT-d), an ASRock Z87 board, and an ati gpu (currently undecided). Primary use is work (PHP/Mysql Web development, some Java development, general office use) and gaming. While currently overkill, I'd like something that's somewhat futureproof as using my PC for work also, I'd rather not take it apart frequently. I'm currently running an i7 920 which I got in 2009 and was one of the best PC component investments I ever made despite it being overkill for almost every game of the time.
Seems to be a good choice, just make sure VT-d is supported on the ASrock board in conjunction with the i7 4770 CPU. Overkill is relative - typing this reply on my i7 3930K-based PC while it idles along is definitely total overkill. A guy over at the Asus ROG forum runs different Windows VMs with VGA passthrough on Xen to test/Q&A his software, using a good hardware RAID controller to run SSDs in RAID0 for caching and multiple HDDs in RAID5 for the VMs (see http://rog.asus.com/forum/showthread.ph ... post361172). I'm sure he's saving a hell lot of time with this setup. It all depends on your needs and the money you would be able or willing to spend on it.

If the IGP in the 4770 is good enough for you, you can do with only one discrete graphics card. I'm currently using an AMD HD 7770 for Linux (dom0) and a Nvidia Quadro 2000 for Windows, because I use Windows for photo editing, not gaming. Only $10 more will buy you an AMD HD 7790 with significantly better performance for your dom0, see http://www.videocardbenchmark.net/gpu.p ... 90&id=2502. For the Windows guest (gaming VM) the AMD 7870 or 7950 may be worth considering, or some beefier models if you run a hi-res screen and want maximum quality. CrossFire is also an option if you have a second/third PCI-e 16 slot. Naturally all depends on your screen resolution - see the following to get an idea: http://www.techpowerup.com/reviews/AMD/ ... ire/6.html.

Technical reason for not "sharing" the same GPU for host and guest: Obviously the guest cannot share the GPU while it has exclusive access to it, as you pointed out. The reason why we cannot switch the GPU between host and guest and vice versa is that VGA passthrough works a little different from PCI passthrough. When the Windows guest boots, it initializes the card (this has to do with backward compatibility - in essence this is code that's been written when most of the guys here weren't even born). Once the card is initialized, it cannot be easily reset to work with a different OS. Some cards support FLR (Function Level Reset), but that's a rare thing. Even my Quadro 2000 card doesn't support it. Anyway, I believe FLR would allow the system to reset the graphics card and then it could be re-assigned to the host or guest, whichever. But I wouldn't waste time on looking for FLR.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
Tz2001

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by Tz2001 »

Thanks for the detailed reply.

That makes sense. I'm currently running a Radeon 5870 (and I have an old 8800gtx around here somewhere) in my system along with the i7 920 which I could keep along with the card I'm getting with the upgrade. How does that affect power usage? Not that it's a massive issue, but I need to consider the PSU. Would I need a hefty PSU to handle two power-hungry GPUs or would it not matter if the 5870 was practically idle running only the linux desktop? FYI I currently have a Corsair HX 650w.

Would the card used by the VM use much/any power while the VM was turned off?

One consideration in the original post was a monitor with multiple inputs. Is there no way around that (other than manually switching cables every time, or a hardware switch)? I saw VNC mentioned earlier in the thread, can you VNC into the windows VM from the linux box or does that cause input/output lag (which is obviously not ideal for gaming)? Although thinking about it a simple hardware switch is probably the easiest method.

VT-d is definitely supported on the i7 4771: http://ark.intel.com/products/77656/int ... o-3_90-ghz and I'll do some research on the best motherboard. However, clearly there's no point paying extra for fancy overclocking features when using a locked cpu.

Again, thanks for all your answers!
powerhouse
Level 6
Level 6
Posts: 1144
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by powerhouse »

@Tz2001:

You can use the 8800GTX only in dom0 (Linux), and only with the nouveau driver (not the proprietary Nvidia driver). The 4770 should have an IGP, so better use that for dom0.

I have a 650W Gold rated power supply and don't have issues, though my graphics cards are rather low wattage. Keep in mind that a 650W Gold rated power supply can sustain only around 80% of that, or 520W. Unless you intend to use power-hungry graphic cards, a 650W should do.

For reference, when measuring the power draw of my PC (i7 3930K on Asus Sabertooth X79, 32GB RAM, 5 HDD, 1 SSD, 2 GPUs with each max ~50W) during a stress test, my power meter maxed at 240W. The minimum power draw was 117W. Power management seems to be working.

Regarding monitor, most of them have multiple inputs, at least 2. If you happen to have one with only 1 input, you may be able to find a KVM switch for mouse/keyboard and DVI (or perhaps even HDMI). If you have an HDMI option, go with it if your screen has speakers or a headphone connector.

Forget VNC, it's much too slow!
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
Tz2001

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by Tz2001 »

I'm intending to get a new ATI GPU. The 5870 is really showing its age these days. The 5870, however, I recall is quite a power hungry card. I know it's not an exact science, but if I used the existing 5870 for linux, and say, an R9 270/280 for Windows would that likely be a power issue? I'd rather keep a dedicated GPU in Linux as I'm still interested in keeping up to date with native Linux gaming developments and wouldn't want to mess around reconfiguring my GPUs each time and since I already have the 5870, unless there's a reason not to utilise it, I may as well make use of it and not take CPU power away by using the IGP.

Actually you're right, my monitors both have DVI and HDMI connectors.

However, I do not want to use the inbuilt speakers in my monitor. Can't the sound be output through the normal soundcard? Even if it's not passed through using VT-d, the VM should be able to output sound directly, shouldn't it? I'd rather not have a second set of speakers, and it would be nice to sync up mute/master volume between both OS's by keeping Linux in control of the sound. I've done this with non VT-d based VMs before and sound has seemed fine (albeit only in stereo, not 5.1).

Fair enough! I thought that might be the case with VNC.

Again, thanks for your knowledgeable answers :)


One other question... I'm a little confused about the boot process:
VGA passthrough to GPU as primary adapter: The guest system (domU) boots and uses the passed through graphics card as its primary or only graphics adapter.
VGA passthrough to GPU as secondary adapter: The guest system first boots using the Qemu-dm virtual Cirrus adapter as the primary graphics adapter (this is where you see the BIOS screen and the Windows boot logo). After installation of the graphics driver under Windows and subsequent reboot of the domU, the system will switch to the secondary graphics adapter at some stage of it's boot process. This is the technique described here in this How-To.
What exactly happens here? I'd assume the computer boots as normal into linux, and I'd run the VM software which in turn would boot windows like virtualbox does... but that doesn't seem to be the case implied here? Both operating systems boot at once, so windows and linux are always running, is that right?


Edit: Also, I just wanted to say... this is something I've been looking for for years! After many hours messing around with Wine and always being dreadfully disappointed with performance/compatibility and going back to windows I've either used Windows as my Primary OS or, since I've been back on linux as my primary been dual booting for gaming. For me this is a total godsend and worth the price/set up time. If this works as well as it looks like it does I'll have no reason to have windows installed outside the VM. Hopefully with SteamOS and the software maturing, gaming hardware will have VT-d by default as the technology matures and this kind of set up will be easier to create and commonplace among gamers.
powerhouse
Level 6
Level 6
Posts: 1144
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by powerhouse »

@Tz2001:

1. Sound: In my system, sound is controlled by Linux. Here is my setup:

Linux (dom0): Asus Xonar Essence sound card connected to a deVa stereo amplifier which drives a set of Celestion SL600 Signature speakers (I spare you the details of the cables). For my music collection I use gmusicbrowser with ALSA in a bit perfect setup and disabled pulseaudio, but that is because I care about sound quality.

Windows VM: I passed through a USB port for a USB sound card - see post & picture here: http://forums.anandtech.com/showpost.ph ... tcount=128. The headphone output of that USB dongle is then connected to the line-in port of the Asus Xonar. The ALSA mixer under Linux allows me to switch that port on or off, and I have sliders to control the input signal.

Volume is controlled by my amplifier, with the volume slider under Linux turned to maximum. Unless I use headphones, which are controlled by the volume slider under Linux. In the Windows VM I preset the volume for best output quality (if it's too low, you get hiss, if it's too high the result can be distortion).

How does that work? Under Linux, audio works as expected except that gmusicbrowser takes exclusive control of the sound card with no system sounds or other interferences - exactly the way it should be when listening to music :) . Bit perfect means there is no resampling of the music material. Of course the music is stored in flac files, i.e. lossless.

When running Windows, the sound output of Windows is multiplexed with the sound under Linux, unless gmusicbrowser is playing music (same as with Linux). So in case I would be gaming under Windows, the sound comes out on the same speakers I use for Linux, together with any sound that's on Linux (except music).

In a normal setup with pulseaudio all sound sources would be multiplexed (and resampled to 48kHz if necessary) so you would be able to listen to music, get system sounds, and for instance play a game in Windows with sound output all at the same time.

This USB sound card may look cumbersome (extra USB port, wiring) but believe me, this is the easiest and cheapest solution to the problem.

Since I have a discrete sound card (the Xonar Essence), I could as well passthrough the onboard audio to the Windows guest and thus use the existing audio ports on the back panel and front (actually, the front headphone/mic connectors are wired to the Xonar). But I had this setup running with the onboard sound before I got the Xonar, so why mess with a perfectly working system?

If onboard sound is good enough for you, get a USB sound stick for Windows. If you prefer better sound quality under Windows, get a separate sound card and pass it to Windows, or pass the onboard sound to Windows and use the sound card for Linux. If you buy a sound card, check the ALSA compatibility list: http://www.alsa-project.org/main/index.php/Matrix:Main. Or better even, get a good USB DAC: http://www.stereophile.com/category/com ... o-reviews/. The Dragonfly got good reviews and is still affordable. By the way, USB DACs don't have compatibility issues with Linux, see ALSA page.

2. Power supply: You need to dimension your PSU to meet the power specs of your GPUs and the motherboard/CPU. Haswell CPUs are less power hungry than my Sandybridge-E, which tops the list with 130W TPD. Check the maximum power consumption of all your components, add it up, add some 20-30% spare for future replacements/additions, multiply by 5/4 and get a gold rated or better power supply whose power rating is equal to or bigger than the result. If everything adds up to 500W (example), a 650W PSU would be enough, but if you want some headroom get a 750W PSU. Don't buy a 1200W PSU just in case (unless you plan 4-way crossfire etc.), as an over-sized PSU will just waste electricity and generate heat.

3. Boot process: Using my how-to, you end up booting into Linux. When you want to run Windows, you would enter the command

Code: Select all

sudo xm create /etc/xen/win7.cfg
to start the Windows VM.
I personally use a panel icon linked to a startup script.

Primary / secondary adapter passthrough: My how-to uses "VGA passthrough to GPU as secondary adapter" which means that the Windows VM first boots using a virtual Cirrus VGA adapter and, at a later stage during the boot process (when you get the Windows login prompt), switches to the secondary, real GPU. "VGA passthrough to GPU as primary adapter" means that the Windows VM boots straight away using the real GPU, which is faster and better, but usually a lot more difficult to make it work. Before I start patching and compiling the Xen sources to make primary passthrough work, I'd rather choose secondary passthrough which usually works "out of the box".
In the end you need to switch your screen input to the graphics card that you use for Windows. Hope that explains it.

Dual-booting is a real pain in the neck and like you I've been waiting for a long time to find a better way. I'm very happy with Xen and VGA passthrough. The only downside to it is that the boot time for Windows is rather long (28 seconds until I get the login prompt). But once logged in everything is up to speed. For all practical purposes I can't tell that Windows is running as a VM. I could have tuned my Windows performance even further, for example by passing through some SATA controllers and perhaps an Ethernet card, but I really don't feel the need for it.

A big bonus of this setup is the ease of backing up the Windows VM, and restoring it when needed. It takes around 5 minutes to restore a 70GB Windows partition (actually LVM volume). And it's easy to switch between Windows and Linux.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
Quich

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by Quich »

Hi everyone,

Reporting here a little off-thread, but it might interests some of you.

As I was not able to get sound from the VM using Xen and various configuration (Xen 4.3, 4.4, xend and xl toolstack, different qemu versions), I moved to Archlinux in order to use what is described in this thread (use GPU passthrough with kvm) : https://bbs.archlinux.org/viewtopic.php?id=162768&p=1 (I first tried to compil by myself under/for Mint but I blocked with some dependencies when building qemu).

I'm currently running Kernel 3.13-rc8 with the patches available on the thread (not working + memory leak of qemu without them), as well as the given qemu and seabios.
It took me some time, but I got a Windows 7 x64 VM running via qemu, which supports sound passed directly to the host via emulation (with no lag, at least for youtube videos, did not check for gaming yet) and the GPU card. I still use Synergy to control mouse/keyboard.

I ran 3DMark11 with Xen and Mint 16 dom0, and then with qemu-kvm on Archlinux host. (I ran some other benchmarks when I was using Xen but I unfortunately lost them).

The Xen domU was using 6 vcpus, 3.5GB of ram (machine had 8GB 1333MHz).
The qemu-kvm was using (-smp 6,cores=3,threads=2, I do not know exactly how it is handle by qemu, as CPU-Z is reporting a frequency of 4GHz instead of 3.4GHz), 8GB of ram (I upgraded machine to 32GB 1600MHz).

Test | Xen domU Score | qemu-kvm score
Total | P2121 | P2124
Graphic score | 1868 | 1860 (1899 with Win8 directly booted)
Physics score | 7359 | 8206
Combined score | 2021 | 2034

Xen http://www.3dmark.com/3dm11/7812193
Qemu-kvm http://www.3dmark.com/3dm11/7822210 PassMark http://www.passmark.com/baselines/V8/di ... 8607520383

The results are very similar, and qemu-kvm has the advantage to give all the available memory to the host when the guest is not used.
As I do not need to run any other VM for the moment, it is a perfect working solution for me.
Tz2001

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by Tz2001 »

Quich wrote:Hi everyone,

Reporting here a little off-thread, but it might interests some of you.

As I was not able to get sound from the VM using Xen and various configuration (Xen 4.3, 4.4, xend and xl toolstack, different qemu versions), I moved to Archlinux in order to use what is described in this thread (use GPU passthrough with kvm) : https://bbs.archlinux.org/viewtopic.php?id=162768&p=1 (I first tried to compil by myself under/for Mint but I blocked with some dependencies when building qemu).

I'm currently running Kernel 3.13-rc8 with the patches available on the thread (not working + memory leak of qemu without them), as well as the given qemu and seabios.
It took me some time, but I got a Windows 7 x64 VM running via qemu, which supports sound passed directly to the host via emulation (with no lag, at least for youtube videos, did not check for gaming yet) and the GPU card. I still use Synergy to control mouse/keyboard.

I ran 3DMark11 with Xen and Mint 16 dom0, and then with qemu-kvm on Archlinux host. (I ran some other benchmarks when I was using Xen but I unfortunately lost them).

The Xen domU was using 6 vcpus, 3.5GB of ram (machine had 8GB 1333MHz).
The qemu-kvm was using (-smp 6,cores=3,threads=2, I do not know exactly how it is handle by qemu, as CPU-Z is reporting a frequency of 4GHz instead of 3.4GHz), 8GB of ram (I upgraded machine to 32GB 1600MHz).

Test | Xen domU Score | qemu-kvm score
Total | P2121 | P2124
Graphic score | 1868 | 1860 (1899 with Win8 directly booted)
Physics score | 7359 | 8206
Combined score | 2021 | 2034

Xen http://www.3dmark.com/3dm11/7812193
Qemu-kvm http://www.3dmark.com/3dm11/7822210 PassMark http://www.passmark.com/baselines/V8/di ... 8607520383

The results are very similar, and qemu-kvm has the advantage to give all the available memory to the host when the guest is not used.
As I do not need to run any other VM for the moment, it is a perfect working solution for me.

Very interesting. I don't have my hardware yet, but that looks promising.

I see you're using an i7-4770. Which motherboard are you using with it?


You say "and qemu-kvm has the advantage to give all the available memory to the host when the guest is not used." From what I've read (again, I don't have the hardware to test anything yet) Xen supports a feature called ballooning which does this?

Edit: Looking at it, qemu-kvm looks a little simpler, and seems to handle things in a manner I'd like better: from the KVM faq: "Xen is an external hypervisor; it assumes control of the machine and divides resources among guests. On the other hand, KVM is part of Linux and uses the regular Linux scheduler and memory management. " Considering I'll only use the guest 10-15% of the time, this seems to suit my needs better. I'll have a play around once I get the hardware in the next couple of weeks. Is there any reason KVM cannot run on Mint? I'd move to arch but I prefer the apt package manager as it has more available software.
Quich

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by Quich »

Tz2001 wrote: Very interesting. I don't have my hardware yet, but that looks promising.

I see you're using an i7-4770. Which motherboard are you using with it?


You say "and qemu-kvm has the advantage to give all the available memory to the host when the guest is not used." From what I've read (again, I don't have the hardware to test anything yet) Xen supports a feature called ballooning which does this?

Edit: Looking at it, qemu-kvm looks a little simpler, and seems to handle things in a manner I'd like better: from the KVM faq: "Xen is an external hypervisor; it assumes control of the machine and divides resources among guests. On the other hand, KVM is part of Linux and uses the regular Linux scheduler and memory management. " Considering I'll only use the guest 10-15% of the time, this seems to suit my needs better. I'll have a play around once I get the hardware in the next couple of weeks. Is there any reason KVM cannot run on Mint? I'd move to arch but I prefer the apt package manager as it has more available software.
I do not really know about the ballooning feature and what it is supposed to do.

You can use kvm with Mint, but you will still need to compile your kernel, qemu and seabios because it will not work out-of-the-box. You will need to apply manualy the patch files given on the kvm thread to the (kernel|qemu|seabios) sources and then to compile them.
I failed to compile qemu under Mint because I was trying to build it with the spice feature (and the code from spice git or one of its dependences does not compile) which is not mandatory at all in fact.

I am using an OEM Lenovo CIB85M, which is based on an Intel B85 and is included inside the Lenovo K450 series (check the previous page for more details).
powerhouse
Level 6
Level 6
Posts: 1144
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by powerhouse »

@Quich: I'm glad you found a solution that works for you. nbhs over at the archlinux forum does a great job in making VGA passthrough with kvm more accessible.

About giving all available memory to the host: Tz2001 is correct - it's called "ballooning" and Xen supports it. But because of the way the Linux kernel works, the Xen documentation discourages its use - see http://wiki.xen.org/wiki/Xen_Best_Practices. I don't think that kvm is any different here, so if ballooning is discouraged under Xen, it most probably is also discouraged under kvm.

Xen sound support in Windows guest: In all my posts above I never considered the simplest option - soundhw=... in the win7.cfg file. If I find a little time today I will give it a try. It should support ac97 emulation, though not hda, but that is irrelevant, as long as there are drivers under Windows.
EDIT: I tried and ran with soundhw='ac97' and installed the ac97 driver under Windows. It doesn't work! The Realtek AC97 drivers are ancient (2009). Luckily I made a backup before experimenting with Windows, as it didn't take it kindly when I uninstalled the Realtek driver. In short: forget the soundhw=... option when using qemu-dm.

The following command shows you the supported "emulated" sound devices:

Code: Select all

/usr/lib/xen-4.3/bin/qemu-dm -soundhw ?
The other sound devices are even more antiquated.

EDIT2: Did some more searching. Linux Mint/Ubuntu currently ships with qemu 1.5. qemu 1.7, however, has support for emulated HDA sound device. Then again, somewhere I read that qemu-upstream might break VGA passthrough. I guess I'll wait it out.

Quich, please post the output of:

Code: Select all

inxi -Fn
only the motherboard and CPU part, for the motherboard compatibility list.
Unfortunately Lenovo ... isn't enough.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
Quich

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by Quich »

powerhouse wrote:@Quich: I'm glad you found a solution that works for you. nbhs over at the archlinux forum does a great job in making VGA passthrough with kvm more accessible.

About giving all available memory to the host: Tz2001 is correct - it's called "ballooning" and Xen supports it. But because of the way the Linux kernel works, the Xen documentation discourages its use - see http://wiki.xen.org/wiki/Xen_Best_Practices. I don't think that kvm is any different here, so if ballooning is discouraged under Xen, it most probably is also discouraged under kvm.

Xen sound support in Windows guest: In all my posts above I never considered the simplest option - soundhw=... in the win7.cfg file. If I find a little time today I will give it a try. It should support ac97 emulation, though not hda, but that is irrelevant, as long as there are drivers under Windows.
EDIT: I tried and ran with soundhw='ac97' and installed the ac97 driver under Windows. It doesn't work! The Realtek AC97 drivers are ancient (2009). Luckily I made a backup before experimenting with Windows, as it didn't take it kindly when I uninstalled the Realtek driver. In short: forget the soundhw=... option when using qemu-dm.
That is what I was confronted to. The AC97 emulation works with Windows 7 32, but there is no working driver for 7 x64.

The following command shows you the supported "emulated" sound devices:

Code: Select all

/usr/lib/xen-4.3/bin/qemu-dm -soundhw ?
The other sound devices are even more antiquated.
powerhouse wrote: EDIT2: Did some more searching. Linux Mint/Ubuntu currently ships with qemu 1.5. qemu 1.7, however, has support for emulated HDA sound device. Then again, somewhere I read that qemu-upstream might break VGA passthrough. I guess I'll wait it out.
Yes, that is why I tried with Xen 4.4 (which correct some problem with the xl toolstack) and an up-to-date upstream-qemu. I got the soundcard installed on the 7 x64 domU but I got no sound on the dom0. It was for sure a configuration problem on the dom0, and it should be easy to fix, but I did not try as the GPU passthrough was effectively not working the right way anymore (dom0 freeze, artifacts on the domU...).
powerhouse wrote: Quich, please post the output of:

Code: Select all

inxi -Fn
only the motherboard and CPU part, for the motherboard compatibility list.
Unfortunately Lenovo ... isn't enough.
Launched with root privileges, I do not get anything more than that :

Code: Select all

Machine:   System: LENOVO product: 10120 version: IdeaCentre K450 serial: ES11638761 
           Mobo: LENOVO model: N/A version: 31900003 STD serial: ES11638761 Bios: LENOVO version: I1KT15AUS date: 03/28/2013
CPU:       Quad core Intel Core i7-4770 CPU (-HT-MCP-) cache: 8192 KB flags: (lm nx sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx) 
The motherboard is labeled "Lenovo" and "CIB85M", I will take a picture later.
Locked

Return to “Virtual Machines”