This weekend I did some hardware upgrade, replacing my AMD Radeon 6450 for a 7770, and adding an Asus Xonar Essence STX sound card (it was hard for me to buy an Asus product, but this sound card is as good as it gets for audiophiles, unless you spend some $$$$$ for a good external DAC). I should have bought a wiring cabinet too as the back of my PC looks like a patch panel in a data center.
After installing the cards, I booted and everything looked fine. But when I checked usb-devices there was an error with my ASMedia SATA/USB3 controller. I thought perhaps putting the ASMedia controller into a different slot might help, but the only way to do so was to exchange the two graphics cards - so I put my dom0 VGA card into the second PCIe 16 slot, and vice versa the domU card into the first, hoping to be able to configure it somehow. Next I rebooted into the BIOS to check the settings. I can't remember what I did exactly, but somehow I must have selected some default / preconfigured setup by accident and saved the settings (anyway I didn't find any VGA related settings). The PC booted showing the normal messages until it got stuck at a blank screen. I thought the graphics cards might be the issue (I use AMD for dom0 and Nvidia for domU) and rebooted into a regular LM14 (without Xen), where I got an Xorg error and a terminal screen. I couldn't find much under Google, whatever little I found I tried, such as some xorg.conf settings, but nothing worked.
Finally I gave up, deleted the xorg.conf file changes and returned the cards to how I installed them the first time. When booting into Xen I would get a black screen, but it booted fine into regular LM14. In Xen, before giving me the black screen, it displayed an HSM error. Sounds familiar, but I didn't pay much attention to it. I thought I had screwed up the Xorg settings so I booted a live LM14 USB stick and restored my Linux partitions from backup. It still didn't boot.
To end this long story, I finally remembered the HSM message and bothered to check the BIOS settings: Lo and behold VT-d was disabled - ouch
! After enabling VT-d Xen/LM14 booted happily into the dom0. The usb-device error with the ASMedia controller magically disappeared too (during the boot process I saw some "ASMedia ... PCIe Gen2" message telling that the BIOS adjusted the PCIe setting to match the card's capabilities).
So, everything works now. The Radeon 7770 is way better than the 6450 - no more tearing when scrolling, and HD video is smooth as it should be at 1900x1200 or so. Didn't even have to touch the driver, as the installed fglrx-updates driver package recognized the new hardware.
The only thing that keeps bothering me is that I can't install my primary (dom0/Linux) display card into the second PCIe-16 slot and the guest VGA into the first PCIe-16 slot. Any suggestions?
Note: The reasons I want to exchange the two is that the AMD 7770 is a two slot card and totally blocks one PCIe-1 slot, and the ASMedia controller in the next slot is half blocking its cooling fan. Next to my second PCIe-16 slot is an old PCI slot that I have no use for. My Nvidia Quadro 2000 takes only 1 slot and rearranging the cards would free 1 PCIe-1 slot and leave more space to the AMD 7770 graphics card for better cooling. I sometimes wonder who designs these PC motherboards