KVM GPU passthrough latency issues - Intel X299

Questions about virtualization software
Forum rules
Before you post please read how to get help
Post Reply
th2000125
Level 1
Level 1
Posts: 1
Joined: Sat Jan 20, 2018 7:23 am
Location: United Kingdom
Contact:

KVM GPU passthrough latency issues - Intel X299

Post by th2000125 » Sat Jan 20, 2018 9:52 am

Hello everyone,

I wanted to start by saying, hello! and sorry for being a first post help pleading newbie. I could have sworn I had an account but apparently not.

Brief explanation: I'm currently building a project with some friends, I have and always will use Linux as my primary OS but to game I have been forced to go to the darkside that is Windows. Anyway, I have never succeeded to successfully passthrough my GPU via KVM until now and this system is supposed to be our solution to LANs. After a first success I had to fix (code 43) for the Nvidia card. Then I had to work out why I was having abysmal performance in, to put it bluntly everything. And now, I seem to be at the final hurdle, I have got games to work, I can passthrough system audio to the host or use the HDMI stream (which is what we will be using at LANs or I hoped so) at the moment I am getting latency spikes that don't ruin single player games but multiplayer, lets just say there is much to be desired. The spikes appear to destroy my real-time audio abilities, I get pop's every few seconds 3 - 5 but the host audio is unaffected so in every other scenario so I don't know what to do.

*edit*
I forgot to add, when my PC starts it "lags" every few seconds. CLI and system monitor both freeze for periods of time and then come back and go. This happens for around 3 minutes and then it appears to be fine afterwards. I really have no idea how I would go around debugging that. If I need to make a second post I would appreciate some direction :)

Hardware:
CPU: 7820X @ 4.5GHz
RAM: 32GB 4x8GB
NIC: Intel Gigabit, AQC107 (tried passing through as I thought the bridge might be adding latency but no avail)
SSD: Host@500GB Samsung 850 Evo and one of the VM's @500GB Samsung 850 Evo M.2 with a few hard drives
GPU: Host@GTX1060 (pcie x8mode), VM@1080ti (pcie x8mode - I can't run at it at x16 due to whacky BIOS config)
PSU:HX1000i

Environment:
OS: Linux Mint 18.3 (Had to install 18.2 first as 18.3 live USB just kernel panic'd)
Kernel: 4.14.13 (I had so many issues going past Linux Kernels 4.8+, I get an ACPI error on boot with all of them)
Software: Compiled against library: libvirt 2.2.0, Using library: libvirt 2.2.0, Using API: QEMU 2.2.0, Running hypervisor: QEMU 2.6.2, OVMF 05/01/2018.

/etc/default/grub

Code: Select all

GRUB_DEFAULT=0
#GRUB_HIDDEN_TIMEOUT=0
#GRUB_HIDDEN_TIMEOUT_QUIET=true
GRUB_TIMEOUT=2
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 elevator=deadline transparent_hugepage=always intel_idle.max_cstate=1 isolcpus=0-3,8-11 nohz_full=0-3,8-11 rcu_nocbs=0-3,8-11"
GRUB_CMDLINE_LINUX=""
/etc/initramfs-tools/modules

Code: Select all

#pci_stub ids=10de:1b06,10de:10ef,1d6a:d107
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
vhost-net
/etc/modprobe.d/local.conf

Code: Select all

options vfio-pci ids=10de:1b06,10de:10ef
/etc/sysctl.conf

Code: Select all

# Set hugetables / hugepages for KVM single guest
vm.nr_hugepages = 9000
kernel.sched_min_granularity_ns=10000000
kernel.sched_wakeup_granularity_ns=15000000
vm.dirty_ratio=10
vm.dirty_background_ratio=5
vm.swappiness=10
kernel.shmmax = 9437184000
VM config

Code: Select all

<domain type='kvm'>
  <name>Silica</name>
  <uuid>e343d9b5-8f83-45f1-b6df-ed3a0a95c81e</uuid>
  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <memoryBacking>
    <hugepages/>
  </memoryBacking>
  <vcpu placement='static' current='8'>16</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='3'/>
    <vcpupin vcpu='4' cpuset='8'/>
    <vcpupin vcpu='5' cpuset='9'/>
    <vcpupin vcpu='6' cpuset='10'/>
    <vcpupin vcpu='7' cpuset='11'/>
    <emulatorpin cpuset='0,8'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-xenial'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/Silica_VARS.fd</nvram>
    <bootmenu enable='yes'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='whatever'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='4' threads='2'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/kvm-spice</emulator>
        <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/disk/by-id/ata-Samsung_SSD_850_EVO_500GB'/>
      <target dev='sdb' bus='sata'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/disk/by-id/ata-HGST_HDN726040'/>
      <target dev='sdc' bus='sata'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/disk/by-id/ata-WDC_WD40EZRZ'/>
      <target dev='sdd' bus='sata'/>
      <address type='drive' controller='0' bus='0' target='0' unit='3'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/var/lib/libvirt/images/virtio-win-0.1.141.iso'/>
      <target dev='sde' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='5'/>
    </disk>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='nec-xhci'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </controller>
    <interface type='direct'>
      <mac address='52:54:00:ad:66:b3'/>
      <source dev='enp0s31f6' mode='private'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x17' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x17' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x14' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
</domain>
[/b]Not inuse audio passthrough[/b]

Code: Select all

<qemu:commandline>
    <qemu:env name='QEMU_AUDIO_DRV' value='pa'/>
    <qemu:env name='QEMU_PA_SAMPLES' value='1024'/>
    <qemu:env name='QEMU_AUDIO_TIMER_PERIOD' value='150'/>
    <qemu:env name='QEMU_PA_SERVER' value='/run/user/1000/pulse/native'/>
  </qemu:commandline>
Image

I have gone over so much data trying to work out where I am going wrong. Arch Linux documents and all kinds of forums, I have tried pci_stub vs vfio-pci and can't notice any difference. CPU pinning, Different Kernels, Latency monitoring after changing settings. I don't mind buying a DAC but for each of us to have to do that when we have already invested a fair bit in this build doesn't seem right and either way I need to fix this latency.

Any help would be seriously appreciated! Thanks so much for reading. :D

Post Reply

Return to “Virtualization”