HOW-TO make dual-boot obsolete using XEN VGA passthrough

Questions about virtualization software
Forum rules
Before you post read how to get help. Topics in this forum are automatically closed 6 months after creation.
Lomaxx

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by Lomaxx »

A further note: I had another quick idea. If you could remotely change the monitors input-source when using Synergy and moving the mouse to the other OS's screen, then you wouldn't need to press a button at all. It seems like something like that is doable. See this thread (5th posting): http://sourceforge.net/projects/synergy ... ic/2182433

I am currently a bit exhausted since I have been browsing sources for hardware the whole day. So I only took a quick view at that thread. Please excuse me if it's not what I thought it's about. But I think it's worth a look.

I'm off for today.
earlboy

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by earlboy »

Lomaxx wrote:A further note: I had another quick idea. If you could remotely change the monitors input-source when using Synergy and moving the mouse to the other OS's screen, then you wouldn't need to press a button at all. It seems like something like that is doable. See this thread (5th posting): http://sourceforge.net/projects/synergy ... ic/2182433

I am currently a bit exhausted since I have been browsing sources for hardware the whole day. So I only took a quick view at that thread. Please excuse me if it's not what I thought it's about. But I think it's worth a look.

I'm off for today.
I'm interested in what you're planning to do as well, since I too have dual monitors for my xen x-fire rig. Will check out the links later as soon as a finish fixing my installation. Encountered problems when I use UEFI boot with Xen. Had to use xen 4.2.1, which I had to build from source.

I know I could just go with standard bios, but well, I don't know why I want uefi, maybe just because it looks a lot cooler. :lol:
powerhouse
Level 6
Level 6
Posts: 1144
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by powerhouse »

@Lomaxx: This how-to is all about running Linux as your main OS and Windows as a VM for when you need it - but all this without dual-boot.

Re synergy: Yes, both OS run at the same time and synergy works. However, each time you need administrator privileges in Windows synergy gets thrown off, that is the mouse and keyboard won't work anymore. There is a configuration switch in synergy that's supposed to fix that but it hasn't worked for me. Perhaps I didn't have enough patience to recheck my configuration etc., or because I already had a KVM switch, I removed synergy. When it works (and it does when not trying to do admin stuff) it's really great. I can't say if it's suitable for games on Windows, but it seems very responsive. So I'd say give it try.

Re building a PC: If you are already going to build a PC, I would definitely get VT-d and a VGA passthrough compatible graphics card, even if you decide not to go this way now. You never know how you feel about it tomorrow. You don't need to buy 2 graphics cards, if you get one of these Intel CPUs with onboard GPU. Check the links in the beginning of the tutorial for compatible CPUs. As for motherboards, ASRock seem to be a good choice, but check the compatibility list (there should be some links in the how-to) and the manufacturer's information.
If I were to go shopping now, I'd seriously consider the i7 3770, but a lesser one will do as well as long as it supports VT-d.

If synergy doesn't work out, a USB KVM switch is probably the best option (I use one). If you don't have speakers in your screen (via HDMI), I also recommend investing some $8-$10 in a USB sound adapter and cable. This USB sound adapter goes into a passed-through USB port, with a cable plugged into its earphone port and on the other side into the IN port of the onboard sound. Other than that there is no further investment in hardware for this. Since you have already 2 screens, you won't even need to switch the screen.

Good luck!
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
powerhouse
Level 6
Level 6
Posts: 1144
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by powerhouse »

@earlboy: Did you manage to get UEFI work with Xen? I believe saw UEFI support somewhere on a release note (or was it the roadmap?).

I for myself try to fight the temptations and stick to the principle: Don't touch a running system.

If you made it work, care to share your steps to it? Ah, and also the benefits of UEFI, if there are any.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
Lomaxx

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by Lomaxx »

@powerhouse:

Thanks for the reply. I didn't know about the problem with Synergy in the administrator-mode of Synergy. Good to know that. If I every make it to a xen_vt-d_setup, then I will test if there is a workaround.

As for the hardware: I indeed plan to get a i7 3770 (not the K version). I also thought about the Core i7-3930K c2 (has to be c2 for vt-d), but it would be quite some additional cost also because of the missing internal intel-gfx and thus the need for two GPU-cards. In addition I would need better cooling since it got a max TDP of 130W and a more powerful PSU. So I probably manage to keep myself from buying it.
The mainboard I've got in focus is the Asrock Asrock Z77 Extreme 6. I compared it to the Extreme 4 and noticed that the E6 got a DisplayPort for 2560x1600, which is the resolution that I probably will go for when buying a new monitor and it exactly fits the maximum resolution of the 3770. From what I read in the BIOS-description and comments on the net it SHOULD be VT-d-compatible although I didn't find a full proof for it yet.
I'm not sure about what ram I will buy. On the one hand this is off-topic anyway, while on the other it might help people to get recommendations for a complete VT-d-setup - which isn't that off-topic anymore. All I know is that I will directly get 32GB and probably nothing special (just 1600Mhz).
My hardest decision is which gfx-card I should buy. I already planned to get an Nvidia-card (ASUS GeForce GTX 670 DirectCU II) as it got a very good recommendation from SPCR regarding powerconsumption, performance and noise/silence. Nvidia-support in Xen 4.2 for vga-passthrough seems to be on the way. I still would risk being left without VT-d (at least for a while), as yet it' s a bit early to tell how good it will be and when it will fully implemented and easy to be set up. I will try to find more info on that and on AMD-alternatives.

Other things I wonder:

Is Xen flexible enough to quickly shut down the virtual WindowsOS and launch a VirtualLinux instead without reconfiguration in order to be able to use the separate GPU-card in Linux for something like Blender?

Or is it even possible to shut down the virtual WindowsOS and add the card directly to the Linux-dom0 without a restart/reboot? Same question regarding the RAM.

The following is probably hard to answer: How much watt does such a GPU-card draw when it's not used by any virtual setup? Can it be (nearly) completely turned off (and on again when needed), maybe even without restarting Xen/dom0.

I'll stop asking now and meanwhile try to find answers on my own instead.
earlboy

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by earlboy »

powerhouse wrote:@earlboy: Did you manage to get UEFI work with Xen? I believe saw UEFI support somewhere on a release note (or was it the roadmap?).

I for myself try to fight the temptations and stick to the principle: Don't touch a running system.

If you made it work, care to share your steps to it? Ah, and also the benefits of UEFI, if there are any.
Just got it working now. Still using your guide with the following difference.

- Used linux mint 14 kde edition ( just want to try a different DE) :D
- There is a bug in the current xen kernel (the one ubuntu is using) wherein xen only gets 512MB under UEFI no matter what amount of ram you pass at boot time. This made dom0 DE very sluggish. Had to build 4.1.4 from source, which had the fix for it.
- I also had to ugrade my kernel to 3.8 since I was getting kernel panic when the pciback script gets executed.

I first tried xen 4.2.1 but it's using the xl toolkit instead of xm. Managed to get the vm running but could not get vga passthrough to work.

Windows 7 running full virt with vga passthrough. Will be testing with tera online again as soon as I finish my work.

As for the benefit, well there is a lot of discussion regarding the benefit of uefi floating around in the net.
For me, I did cause I thought it was cool. :lol:
powerhouse
Level 6
Level 6
Posts: 1144
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by powerhouse »

@Lomaxx:

CPU: I have the 3930K and while I like it, it sure comes at a cost in $$$ and power consumption. As I'm writing this, my PC (w/o screen) eats 112W (which is the lowest I can get) without a Windows guest. Unless you have a real need for 6 cores / 12 threads I would definitely go with the 3770 (more about this later).

M/B: Check the links I posted, including this one: http://www.overclock.net/t/1338063/vt-d ... therboards. The list of M/Bs in that thread is far from complete, but I recommend to stick with a board that's been tested and proven. My Asus Sabertooth X79 was not a good choice, though it works (with Marvell SATA controller disabled).

Graphics card: Don't buy the Nvidia - it will be hard to impossible to get it working. I have a "multi-OS" Nvidia graphics card which works fine, but you will need to spend a lot of money to get a "multi-OS" card that works well with games. In short, get an AMD graphics card (I know this is hard coming from Nvidia myself). Not only will your Nvidia card not work with VGA passthrough, it also won't work under Linux dom0 with Xen. Well, at least not with the proprietary Nvidia driver, and what is the use of a good card without a good driver.
Should you discard this advice and go ahead with the Nvidia card, you will most like (99%) have to compile the Xen hypervisor, perhaps with patches applied, and you may also need to compile the Linux kernel. In addition to the patching and compiling you will probably have to read out the graphics card's BIOS. If that BIOS file is too big, you won't be able to use this method. In addition support for Function Level Reset (FLR) may also be required.
If I haven't managed to discourage you, here is some information on getting Nvidia cards to work under Xen with VGA passthrough: http://www.davidgis.fr/documentation/Xe ... h/doc.html and http://www.davidgis.fr/blog/index.php?2 ... ss-through. If you make it with the Nvidia card you have on the list, share a how-to and you'll be honored with "Xen hero" status :D .

Shutdown Windows and launch Linux VM: The shutdown procedure of Windows takes a few seconds when using an SSD. After the Windows shutdown your PCI devices remain detached from dom0:

Code: Select all

sudo xm pci-list-assignable-devices
so you can just use them for a Linux VM (as HVM guest with PCI/VGA passthrough). I haven't tried it, though.

In order to make PCI devices available to dom0 (after you shut down the Windows domU), the PCI devices need to be "reattached":

Code: Select all

virsh nodedev-reattach pci_0000_02_00_0
0000_02_00_0 is the PCI ID (0000 stands for the domain). Do with every PCI device you detached. (Note the underscores instead of the ":" and "." in the xm commands.)

The problem with reattaching the graphics card to dom0 is the driver. I am using the virsh command to reuse my USB controllers for dom0, but never tried using the graphics card. May be worth trying.

RAM: Xen has the option of "memory ballooning", though it is discouraged. If you get 32GB you can give your dom0 4GB or even 8GB of memory, which is far more than enough. So I would stick with fixed memory for dom0 and disable ballooning. My Windows guest gets 24GB and I have a hard time filling even a fraction of that (well, when I start stitching photos I will be able to use it). All my tmp folders in Linux are in RAM, which is good for the SSD (and performance).

RAM hardware: Go with 1600MHz as you plan. Memory can be very tricky, and I had serious issues in my built. It took 2 months, one month at the lab, to finally find a memory kit that worked. Definitely run 48-72 hours of memory test BEFORE even installing anything!!! There should be 0 errors. Different chipsets (X79 versus Z77) and boards may make a huge difference, so perhaps the memory chips (see my signature for what I use).

GPU power draw: My Nvidia Quadro 2000 that I use for Windows draws little power, the maximum specified is 60W. According to my Watt meter the card seems to run on a low power level when not in use. When I run Windows, the power consumptions stays about the same or increases slightly. All in all it behaves as if power management is working. There is a power management option in the guest configuration file: pci_power_mgmt=1

I have it enabled in my config file. Perhaps check this both enabled or disabled to see what difference it makes. There is also a power management option in the PCI passthrough line in the config line. Need to look it up.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
Lomaxx

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by Lomaxx »

Thanks for all the info.

Although the chance of being a Xen-hero is attractive to me ;) and although I am currently still using Gentoo, where I am used to fiddle around a lot, your description of the measures that need to be taken give me little hope of success. Especially the line "GTX 600 series are not supported" in the introduction of patch-site that you linked, doesn't sound promising. I'll try to look for suiting Radeon-alternatives for now.
Lomaxx

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by Lomaxx »

A quick reply related to Synergy: It's seems like Synergy can be made to work in both - administrator-mode and user-mode - simply by running it with administrative rights. Instructions to do so can be found easily on the web.

Also it seems like you can configure it to switch the screen by pressing a shortkey instead of moving the mouse to the edge of a screen. People with a single monitor and gamers that fear to switch the screen while in action might prefer this. But you can also use NumLock to lock the mouse to one screen.

For a few other hints see: http://synergy2.sourceforge.net/tips.html
powerhouse
Level 6
Level 6
Posts: 1144
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by powerhouse »

Re GPU: I don't know how indicative this benchmark is, but check http://www.videocardbenchmark.net/high_end_gpus.html.
Lomaxx wrote:A quick reply related to Synergy: It's seems like Synergy can be made to work in both - administrator-mode and user-mode - simply by running it with administrative rights. Instructions to do so can be found easily on the web.

Also it seems like you can configure it to switch the screen by pressing a shortkey instead of moving the mouse to the edge of a screen. People with a single monitor and gamers that fear to switch the screen while in action might prefer this. But you can also use NumLock to lock the mouse to one screen.

For a few other hints see: http://synergy2.sourceforge.net/tips.html
Thanks for pointing this out - really helpful! I guess I will give it another try.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
powerhouse
Level 6
Level 6
Posts: 1144
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by powerhouse »

Today I installed the Passmark benchmark suite onto my Windows 7 VM and gave it a spin. Below the results summary:

Image

The only weak point is the 3D performance of my graphics card, but this was to be expected. The Nvidia Quadro 2000 is no gaming GPU. Anyway, the graphic card results should be in line with bare metal results.

I've shared the Passmark benchmark result here: http://www.passmark.com/baselines/V8/di ... 6676049655.

I think that's pretty cool for a virtual machine, isn't it?

Now, here some results showing off the latest GPLPV disk driver.

1. LVM partitioned SSD (Sandisk Extreme 120GB):
disk = [ 'phy:/dev/mapper/lm13-win7,hda,w' ]
Image

2. Two LVM partitioned HDDs (WD20EARX-00P), striped:
disk = [ 'phy:/dev/mapper/photos-photo_stripe,hdb,w' ]
Image

3. LVM partitioned HDDs (WD1001FALS-0 and WD5000AAKS-7):
disk = [ 'phy:/dev/mapper/original-photo_raw,hdc,w' ]
Image

Obviously, SSDs perform better. But what is important is that disk I/O performance under Xen compares quite well with bare metal (see comparison drives).
Last edited by powerhouse on Fri Mar 29, 2013 4:18 pm, edited 3 times in total.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
Lomaxx

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by Lomaxx »

powerhouse wrote:I think that's pretty cool for a virtual machine, isn't it?
Yes. It is. :D


There's something else cool: NEC got a technique in their monitors (not sure in which models), that basically acts as implemented KVM-switch. You can connect the keyboard and mouse to the monitor, connect each computer's USB to the monitor (and of course the video-input as well) and then simply switch between computers by the press of a single, physical button on the monitors front (no need to go through an OSD-menu).
The downside is that these monitors are pretty expensive, though seem to have a brilliant picture-quality. Their Nec pa271w is still a recommended monitor although it's something like 2 years old. Maybe nothing for absolute extreme hardcore-gamers, but I read of people using it for gaming and being pleased with the latency. One of the few bad feelings that i have about buying it (beside the price) is the fact that after these years new monitors from NEC might be around the corner.

Also see: http://www.necdisplay.com/documents/Mis ... hPaper.pdf
Last edited by Lomaxx on Mon Mar 25, 2013 3:56 pm, edited 1 time in total.
powerhouse
Level 6
Level 6
Posts: 1144
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by powerhouse »

Lomaxx wrote:There's something else cool: NEC got a technique in their monitors (not sure in which models), that basically acts as implemented KVM-switch. You can connect the keyboard and mouse to the monitor, connect each computer's USB to the monitor (and of course the video-input as well) and then simply switch between computers by the press of a single, physical button on the monitors front (no need to go through an OSD-menu).
The downside is that these monitors are pretty expensive, though seem to have a brilliant picture-quality. Their Nec pa271w is still a recommended monitor although it's something like 2 years old. Maybe nothing for absolute extreme hardcore-gamers, but I read of people using it for gaming and being pleased with the latency. One of the few bad feelings that i have about buying it (beside the price) is the fact that after these years new monitors might be around the corner.

Also see: http://www.necdisplay.com/documents/Mis ... hPaper.pdf
Thanks for the pointer - that is really cool. Funny thing I own the predecessor of the PA271w, the NEC Multisync 2690WUXi with Spectraview display calibrator. This is by far the best screen I ever owned for photo editing (it's a wide gamut screen). Unfortunately it didn't come with the built-in KVM switch. I can't say if this or similar NECs are good for gaming, they are usually bought by graphics artists, designers, and photographers.

EDIT: I had a look at the PA271W specs. Man, they even topped the specs of my screen by a save margin. If $1,200 is not too steep a price, this is one hell of a screen. Just make sure the specs are suitable for gaming, in all other areas it's total overkill.

By the way, my NEC screen was the reason I was so determined about VGA passthrough. Both mine and the PA271W support LUT (lookup table) upload to the screen, so when I calibrate my screen using a calibrator and the Spectraview software it will upload the color profile (or better the corrections) into the screen via DVI cable. The advantage is that the color correction of the screen is done inside the screen using 14-bit per color, unlike the 8-bit values that most graphic cards use.
Needless to say that with my screen (which hasn't got a USB link) the LUT upload can only work when the software has access to the graphics card and can make it upload the data to the screen. So in my case it's not only VGA passthrough for the Windows display driver, but also the NEC Spectraview software needs to communicate with the graphics card which then communicates with the screen.
There are more features (and I'm sure the PA271W supports them too), for example loading different color profiles depending on the screen use (Adobe RGB for photo editing, sRGB for web and general purpose). You can even lock the screen buttons via software, which helps prevent kids or other people screwing up the display settings. The screen also has a really wide viewing angle, and watching HD movies on it is a delight.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
powerhouse
Level 6
Level 6
Posts: 1144
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by powerhouse »

Lomaxx wrote:Thanks for all the info.

Although the chance of being a Xen-hero is attractive to me ;) and although I am currently still using Gentoo, where I am used to fiddle around a lot, your description of the measures that need to be taken give me little hope of success. Especially the line "GTX 600 series are not supported" in the introduction of patch-site that you linked, doesn't sound promising. I'll try to look for suiting Radeon-alternatives for now.
I just ran into the weirdest thread on that: If you are really adventurous, check out this post (and entire threat): http://www.eevblog.com/forum/projects/h ... #msg207550.

It turns out people are hardware modifying (hacking) their Nvidia cards so they identify (and perhaps work like) their professional Quadro etc. counterparts. The above post describes how to get that card running with VGA passthrough under KVM, but that should be the same for Xen.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
Lomaxx

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by Lomaxx »

@powerhouse: What ugly sh...stuff. I'm not referring to the work of the people who hardware-hack, but to the companies who artificially limit the abilities of their cards. It's not new to me, but again and again making me somewhat angry. I wish I could solder that well, but I am lousy at it.

Now I wonder if Xen could fake the cards-ID and tell the driver that it's using a different card. But I doubt that it's that easy otherwise it would have been done. Maybe it's a bios-thing or on a similar, lower level. That's all beyond my scope.
powerhouse
Level 6
Level 6
Posts: 1144
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by powerhouse »

Lomaxx wrote:@powerhouse: What ugly sh...stuff. I'm not referring to the work of the people who hardware-hack, but to the companies who artificially limit the abilities of their cards. It's not new to me, but again and again making me somewhat angry. I wish I could solder that well, but I am lousy at it.

Now I wonder if Xen could fake the cards-ID and tell the driver that it's using a different card. But I doubt that it's that easy otherwise it would have been done. Maybe it's a bios-thing or on a similar, lower level. That's all beyond my scope.
A couple a days ago I saw some post on a KVM mailing list that talked about changing device IDs. I could be misinterpreting this, but in any case the KVM developers are working on improving VGA passthrough. Over at http://www.overclock.net/t/1205216/guid ... t_19602860 there is a description on how to make KVM work.

While it's a pity for us not getting the maximum out of our expensive GPU investments, I can understand that Nvidia needs to reduce manufacturing costs and reduce waste, which is a good thing. The sad story is that one of the features removed from the non-enterprise cards is VGA passthrough (their "multi-OS" feature). This doesn't have to be like that. In fact, were Nvidia to support VGA passthrough and Xen, me and other Xen / KVM users could recommend Nvidia cards. Right now I can only discourage users from buying Nvidia GPUs - they are essentially a pain in the neck for VGA passthrough (except for some outrageously expensive Quadro, Kepler etc. models termed "multi-OS capable").
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
Locuust

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by Locuust »

:D

Thanks for taking the time to do this powerhouse. Back in 2009 I caught wind of the possibility of doing vga pass through and I became obsessed with it for a while (even wrote a post here about it). I realized after trying for days to get it to work that I had missed an important detail - my motherboard didn't support iommu. /Facepalm. I had mistakenly assumed that all AMD 890 chipsets had support. I had a 890GX not a 890FX. I might have swapped my motherboard but I had just replaced a fried MB with the 890GX.

Windows gaming VM here I come. I know there are probably more practical solutions (like dual boxing), but I'm stubborn. This solution is just elegant. In a few years I think this will become the standard to replace dual-booting windows. It even makes sense from a power saving stand point. Keep your discrete GPU powered down when you're not using your Stream processing or gaming VM's. What's more Valve is probably adopting a similar strategy with the Steambox server which will support streaming games to PC's/devices at home. Each game instance could be a separate VM powered by a GPU and then streamed with some souped up remote access scheme. What would be really interesting is if they worked on a standard for virtualizing the actual GPU. That is to say allow allocation of stream processors to different VM's (might already be one). There's a similar standard for IO resource sharing, but I can't remember the name of it off the top of my head. But I digress. The point is I'm excited.

At any rate some one was kind enough to give me a new MB and CPU with the IOMMU/V-td support I need. An ASRock 970 and a Amd FX6300. Now I'm just deciding on the other bits I need.
I have -
ASRock 970 Extreme4
Amd FX-6300 (6 core)
16 Gb Ram
Radeon 7870 for DomU (Found a post stating someone had luck with it under KVM *crosses fingers)

Need -
Second Radeon for Dom0
2nd Sound Card/USB

Want -
Probably upgrade to at least 24Gb
New Monitor

For the second Radeon I'm thinking a 7750 because it's single slot and doesn't need additional power. Should be more than adequate for my Dom0/Mint (day to day, surfing, video/streaming).
On the sound card I'm really torn. On one hand I'm really tempted to get a USB device like you did, but... those things sound like they're hit or miss in terms of sound quality and defect rate. I'm leaning towards getting a PCI-E card but it's hard to justify the extra slot and cash for a decent one.
My current monitor has one vga and one dvi connection. I'm not happy about using vga. Plus it doesn't have a single button input selection in the controls. I do have a spare monitor I could use as a second screen (same problems) but space is limited and I use a ergotron mount on my current display.

Still not sure if I'm going to do the USB KVM switch or not. Seems there are some issues with certain gaming keyboards/mice. Which means I might just have to use multiple Kb/Mice if I want full support under windows. I haven't checked out synergy, but I assume it just emulates a keyboard. As the alternative would be a bit too complicated...afaik. I'll probably make sure the rest of my hardware works before I commit to anything on this.

With any luck I'll have this working next weekend.

P.s. I'm so glad I planned ahead and got a Radeon this time around... yeah it was totally with Xen vga passthru in mind... better lucky than good :P



*Edit*
This morning I was wondering about the cheapest system build one could come up with to support XEN passthrough gaming while maintaining at least 2 cores for both Dom0/DomU. It occurred to me that AMD's Fusion APU's might work. I immediately went about checking for IOMMU support in the A85X chipset and the latest trinity processors. Even if one could get Vga passthrough working on this platform there are some tradeoffs - Number of cores and the potential bottleneck between CPU/Northbridge thanks to the 2 Gb/s UMI interface are the ones that come immediately to mind. However this solution might be a lower cost alternative if performance or fast storage is not a priority. I've just started digging but here is what I've found -

Motherboards (A85X Hudson D4):
Asus F2A85-V PRO - IOMMU option in bios according to Manual*
ASRock FM2A85X Extreme6 - No mention of IOMMU in manual or bios notes.
ASRock FM2A85X Extreme4 - No mention
ASRock FM2A85X Extreme4-M - No mention
Gigabyte GA-F2A85XM-D3H - IOMMU option in bios according to manual*
*Warning I haven't read about anyone successfully using IOMMU on this chipset/cpu combo.

I'm really surprised ASRock doesn't have it. Asus allegedly supports IOMMU but is one of the more expensive boards and brings into question the validity of this choice. You're essentially sacrificing a couple CPU cores for a free graphics card for Dom0 (FX-6300 is roughly the same price as A10-5800k). Still if you're on a budget $60-100 might be worth the trade off? Thankfully gigabyte's board is fairly cheap. I haven't done the math but it might be worth it from a heat/power standpoint too. lol. Sorry, this is mostly a curiosity to me as I already have a motherboard and cpu that is known to work. This might become a more viable alternative in the future if AMD releases chips based on the design in the PS4. It has a 8 Core APU.
powerhouse
Level 6
Level 6
Posts: 1144
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by powerhouse »

I'm glad you found this thread useful. Yeah, I hope this VGA passthrough gets more attention as it is really cool 8) .

As to your hardware:

1. VM graphics card (AMD 7870): If it works in KVM I don't see a reason it shouldn't work for Xen. Have you checked the links I posted, especially then one on overclock.net?

2. dom0 graphics card: Definitely a Radeon and not a Nvidia. I couldn't get the Nvidia proprietary driver work with Xen which is a pity as I usually prefer Nvidia. The 7750 should do. I have a much lesser AMD card and it works, though I hope to replace it soon as it's really border line.

3. Memory (16GB): I don't see why you need more memory than you already got? 8GB for gaming in a Windows VM should be more than enough. Leaving 8GB to Linux dom0 is definite overkill, even if you create a temp file system in RAM as I do. I give Linux 6GB, because I can - 2GB would do, 4GB would be more than enough. My Windows VM got 24GB because I edit large photo files, and with HDR or stitching they can grow really big really fast. Yet, I haven't managed to exceed ~12GB RAM utilization up until now. I planned in some spare RAM for when I add a new camera, perhaps a 36 megapixel one, and for doing more photo stitching, but I don't. In short, unless you have a foreseeable reason, don't add more RAM.

4. Monitor: Check your AMD 7870 specs - does it have onboard sound? If yes, you might find a monitor with HDMI port and builtin speakers, if that suits you. This way you could use either Linux with that monitor, or Windows, as you prefer. The sound module on your m/b can then be used for the other OS.

5. Sound: If you don't choose the solution offered under 4., it much depends and how you use sound. If you need both input and output for your Windows guest, then it's better to get a sound card. I am only using the output, which is looped to my m/b input. I have no need for microphone or other inputs under Windows.
I'm using some cheap external PC speakers, and in my setup with a $5 USB sound stick the Windows sound it mixed with the Linux sound output so I can actually listen to both at the same time on the same speakers. It suits me, but I seldom use it to listen to music (I'm old-fashioned and still have one of the better stereo systems with a CD player). Also, if I listen to music, it is within Linux. The USB sound stick makes some cracking noise when booting Windows, so it's best to turn down volume when starting the domU.

6. KVM switch or synergy: I would try synergy and see if it works for you. You can always get a KVM switch, or just 2 mice and keyboards. I have not enough room on my desk for multiple screens and input devices, so the KVM switch comes in handy.

In general, just get the stuff that you really need for VGA passthrough, that is a second graphics card. You could even try with passing through your one and only graphics card, but that would change the how-to significantly, and is also more challenging to achieve.

Good luck!
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
Locuust

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by Locuust »

powerhouse wrote: As to your hardware:

1. VM graphics card (AMD 7870): If it works in KVM I don't see a reason it shouldn't work for Xen. Have you checked the links I posted, especially then one on overclock.net?
No I haven't I suppose I should be checking up on other hardware setups.
2. dom0 graphics card: Definitely a Radeon and not a Nvidia. I couldn't get the Nvidia proprietary driver work with Xen which is a pity as I usually prefer Nvidia. The 7750 should do. I have a much lesser AMD card and it works, though I hope to replace it soon as it's really border line.
Cool, I found a nice version of the 7750 that is truly single slot. Most of the cards only take one slot on the backplane but their cooling solutions appear to approaching that of dual slot cards in thickness. Nice to have so I don't block more PCI slots.
3. Memory (16GB): I don't see why you need more memory than you already got? 8GB for gaming in a Windows VM should be more than enough. Leaving 8GB to Linux dom0 is definite overkill, even if you create a temp file system in RAM as I do. I give Linux 6GB, because I can - 2GB would do, 4GB would be more than enough. My Windows VM got 24GB because I edit large photo files, and with HDR or stitching they can grow really big really fast. Yet, I haven't managed to exceed ~12GB RAM utilization up until now. I planned in some spare RAM for when I add a new camera, perhaps a 36 megapixel one, and for doing more photo stitching, but I don't. In short, unless you have a foreseeable reason, don't add more RAM.
True enough. I think I got projects crossed in my head. I'm also working on upgrading my home server to do some backend VM's. In which case I'd have more than one DomU. Ahh, the rabbit hole.
4. Monitor: Check your AMD 7870 specs - does it have onboard sound? If yes, you might find a monitor with HDMI port and builtin speakers, if that suits you. This way you could use either Linux with that monitor, or Windows, as you prefer. The sound module on your m/b can then be used for the other OS.
Ahhh.. the HDMI audio.. I hadn't considered that. Unfortunately I do need more than just output for my DomU. Too bad... err wait. Derp. I don't really need inputs in Linux. Even the 7750 has an HDMI port. I think you may have changed my mind. Instead of a second soundcard I'll just get new monitor with speakers (or at least a lineout for the HDMI audio). Two birds one stone. Though I better double check to see if there are any issues with this setup before I go placing my order.
5. Sound: If you don't choose the solution offered under 4., it much depends and how you use sound. If you need both input and output for your Windows guest, then it's better to get a sound card. I am only using the output, which is looped to my m/b input. I have no need for microphone or other inputs under Windows.
I'm using some cheap external PC speakers, and in my setup with a $5 USB sound stick the Windows sound it mixed with the Linux sound output so I can actually listen to both at the same time on the same speakers. It suits me, but I seldom use it to listen to music (I'm old-fashioned and still have one of the better stereo systems with a CD player). Also, if I listen to music, it is within Linux. The USB sound stick makes some cracking noise when booting Windows, so it's best to turn down volume when starting the domU.
The above pretty much solves my sound issue. The sound quality/cracking noise was one of the reasons I was shying away from the USB soundcard.
6. KVM switch or synergy: I would try synergy and see if it works for you. You can always get a KVM switch, or just 2 mice and keyboards. I have not enough room on my desk for multiple screens and input devices, so the KVM switch comes in handy.
Yeah.. I'm not too keen on having two sets. I guess I'll just have to decide if I'll miss the extra features of my gaming kb/mouse.
In general, just get the stuff that you really need for VGA passthrough, that is a second graphics card. You could even try with passing through your one and only graphics card, but that would change the how-to significantly, and is also more challenging to achieve.

Good luck!
lol yeah I dunno if I'm that adventurous. Passing the only graphics card back and forth would probably get messy. I imagine it would mean adding/removing the PCI ID from that PCI Back list and unblacklisting the necessary driver... nah. I want to get it working and not tempt fate.
powerhouse
Level 6
Level 6
Posts: 1144
Joined: Thu May 03, 2012 3:54 am
Location: Israel
Contact:

Re: HOW-TO make dual-boot obsolete using XEN VGA passthrough

Post by powerhouse »

@Locuust:

1. Single card VGA passthrough: IIRC, in order to make do with one GPU, you would need to compile pciback into the kernel and add the pciback option to the grub file. The idea is to bind the graphics card to pciback before the graphics driver (dom0) gets loaded. But don't take my word on that, it's been a while since I played with it. See also http://www.overclock.net/t/1205216/guid ... al-machine for a great tutorial based on Fedora 16. It's a very long thread with tons of useful information.

In order to release a passed through PCI device, use:

Code: Select all

virsh nodedev-reattach pci_0000_0a_00_0
This reattaches the PCI device to dom0. It works well with USB controllers, but I haven't tested my graphics card yet. I have some doubts about reassigning graphics cards to different domUs, or back to dom0.

2. AMD CPUs: You wrote a lot about AMD CPUs. I'm afraid I haven't got a clue about AMD CPUs, the last time I used one was some 7-8 years ago. A few years ago Intel integrated the memory controller into the CPU - I believe AMD uses another chip to host the memory controller, so it would also depend on the motherboard.
For both AMD and Intel, all parts must support IOMMU: CPU, motherboard (chipset), and BIOS, though in some cases the CPU doesn't play part in IOMMU (Intel Core 2 Duo, for example). What I mean is that you need to make sure that the combination of all three supports IOMMU.
Subjects of interest: Linux, vfio passthrough virtualization, photography
See my blog on virtualization, including tutorials: https://www.heiko-sieger.info/category/ ... alization/
Locked

Return to “Virtual Machines”