Reporting back on the Matter of running two identical Graphics card on my Ubuntu 16.04.
I have two Radeon R9 290 graphics cards on my machine an finally found
a way to lock them from the radeon driver individually using vfio-pci instead of pci-stub.
There are two ways to get this working.
Version A is described below and uses the external dracut package to specify the nessesary scripts.
And Version B using the Mint/Ubuntu default tools including the info of powerhouse about two
post below this one, for the same cause.
So for anybody struggling with passing through devices with identical pci-id's try the following
modification to powerhouse's manual replacing the pci-stub part:
Please Note !!! :
This actions require at least kernel 3.19+ and a exchange of the initramfs-tools stack against dracut
First install the dracut package for modifying the initramfs image:
Code: Select all
sudo apt-get install dracut dracut-core
Next modify
/etc/default/grub and add
rd.driver.pre=vfio-pci the line
GRUB_CMDLINE_LINUX=
So you get a grub file similar to this:
Code: Select all
GRUB_DEFAULT=0
GRUB_HIDDEN_TIMEOUT=0
GRUB_HIDDEN_TIMEOUT_QUIET=true
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
GRUB_CMDLINE_LINUX="intel_iommu=on "
Rebuild grub:
For more information on the dracut package and its Grub commandline options see:
dracut-core
dracut.cmdline
Create the file
/etc/modprobe.d/local.conf and add:
Code: Select all
install vfio-pci /sbin/vfio-pci-override-vga.sh
Now look up the devices you want to be locked by vfio-pci to be later passed through using:
Wich yields in my case:
Code: Select all
04:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii PRO [Radeon R9 290] [1002:67b1]
04:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii HDMI Audio [1002:aac8]
05:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller [1b21:1242]
06:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii PRO [Radeon R9 290] [1002:67b1]
06:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii HDMI Audio [1002:aac8]
Remember to pass through both the graphics and the audio in case of a graphics card.
Create the following script:
/sbin/vfio-pci-override-vga.sh
Code: Select all
#!/bin/sh
DEVS="0000:06:00.0 0000:06:00.1""
for DEV in $DEVS; do
echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override
done
modprobe -i vfio-pci
And replace the DEV with the ones of the devices you want to pass through. This works for both graphics cards and other devices like sound cards.
And make it executable:
Code: Select all
sudo chmod 755 /sbin/vfio-pci-override-vga.sh
Create the file
/etc/dracut.conf.d/local.conf and add:
Code: Select all
add_drivers+="vfio vfio_iommu_type1 vfio_pci vfio_virqfd"
install_items+="/sbin/vfio-pci-override-vga.sh /usr/bin/find /usr/bin/dirname"
Regenerate initramfs via dracut using:
Check if the script was sucessfully added:
Code: Select all
sudo lsinitramfs -l /boot/initramfs-4.4.0-22-generic.img | grep vfio
Adapt your kernel version accordingly.
It should show all scripts and modules you just added:
Code: Select all
drwxr-xr-x 3 root root 0 Jun 2 00:04 lib/modules/4.4.0-22-generic/kernel/drivers/vfio
drwxr-xr-x 2 root root 0 Jun 2 00:04 lib/modules/4.4.0-22-generic/kernel/drivers/vfio/pci
-rw-r--r-- 1 root root 72390 May 13 01:57 lib/modules/4.4.0-22-generic/kernel/drivers/vfio/pci/vfio-pci.ko
-rw-r--r-- 1 root root 46078 May 13 01:57 lib/modules/4.4.0-22-generic/kernel/drivers/vfio/vfio.ko
-rw-r--r-- 1 root root 23126 May 13 01:57 lib/modules/4.4.0-22-generic/kernel/drivers/vfio/vfio_iommu_type1.ko
-rw-r--r-- 1 root root 11486 May 13 01:57 lib/modules/4.4.0-22-generic/kernel/drivers/vfio/vfio_virqfd.ko
-rwxr-xr-x 1 root root 157 Jun 1 01:43 sbin/vfio-pci-override-vga.sh
Now Reboot
And check if the vfio-pci was loaded as intendend:
Showing:
Code: Select all
04:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii PRO [Radeon R9 290] [1002:67b1]
Subsystem: Hightech Information System Ltd. Hawaii PRO [Radeon R9 290] [1787:2343]
Kernel driver in use: radeon
Kernel modules: radeon
04:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii HDMI Audio [1002:aac8]
Subsystem: Hightech Information System Ltd. Hawaii HDMI Audio [1787:aac8]
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
05:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller [1b21:1242]
Subsystem: ASUSTeK Computer Inc. ASM1142 USB 3.1 Host Controller [1043:8696]
Kernel driver in use: xhci_hcd
06:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii PRO [Radeon R9 290] [1002:67b1]
Subsystem: Micro-Star International Co., Ltd. [MSI] Hawaii PRO [Radeon R9 290] [1462:3081]
Kernel driver in use: vfio-pci
Kernel modules: radeon
06:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii HDMI Audio [1002:aac8]
Subsystem: Micro-Star International Co., Ltd. [MSI] Hawaii HDMI Audio [1462:aac8]
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
It should have attached to the devices you had set in the vfio-pci-override-vga.sh file.
I hope this may solve the problem for those of you with dual graphics cards.
References taken:
VFIO tips and tricks
vfio-pci users maillinglist