kernel: NVRM: Xid 79, GPU has fallen off the bus.

[bsas]

Recently my desktop has started crashing to a black screen with the audio still going while gaming in a few different games launching from Steam requiring a hard reboot but i think this is a graphics related problem. This has not happened without a game running for multiple days uptime but happens pretty quickly when gaming. This is new as of a month or so ago. Some relevant logs:

Code: Select all

11:29:14 kernel: nvidia-modeset: ERROR: GPU:0: Failed to query display engine channel state: 0x0000c67e:2:0:0x0000000f
11:29:14 kernel: nvidia-modeset: ERROR: GPU:0: Failed to query display engine channel state: 0x0000c67e:2:0:0x0000000f
11:29:14 kernel: nvidia-modeset: ERROR: GPU:0: Failed to query display engine channel state: 0x0000c67e:4:0:0x0000000f
11:29:14 kernel: nvidia-modeset: ERROR: GPU:0: Failed to query display engine channel state: 0x0000c67e:6:0:0x0000000f
11:29:11 systemd: headsetcontrol-notificationd.service: Failed with result 'exit-code'.
11:29:11 systemd: Started Headsetcontrol-Notificationd.
11:29:01 kernel: NVRM: A GPU crash dump has been created. If possible, please run
NVRM: nvidia-bug-report.sh as root to collect this data before
NVRM: the NVIDIA kernel module is unloaded.
11:29:01 kernel: NVRM: A GPU crash dump has been created. If possible, please run
NVRM: nvidia-bug-report.sh as root to collect this data before
NVRM: the NVIDIA kernel module is unloaded.
11:29:01 kernel: NVRM: GPU 0000:09:00.0: GPU has fallen off the bus.
11:29:01 kernel: NVRM: Xid (PCI:0000:09:00): 79, pid='<unknown>', name=<unknown>, GPU has fallen off the bus.
11:29:01 kernel: NVRM: GPU at PCI:0000:09:00: GPU-9db9eb2c-b306-6338-81ee-ff74fa5838c4

Code: Select all

System:
  Kernel: 5.15.0-101-generic x86_64 bits: 64 compiler: gcc v: 11.4.0 Desktop: Cinnamon 6.0.4
    tk: GTK 3.24.33 wm: muffin vt: 7 dm: LightDM 1.30.0 Distro: Linux Mint 21.3 Virginia
    base: Ubuntu 22.04 jammy
Machine:
  Type: Desktop Mobo: ASUSTeK model: TUF GAMING X570-PLUS (WI-FI) v: Rev X.0x
    serial: <superuser required> UEFI: American Megatrends v: 2802 date: 10/21/2020
CPU:
  Info: 12-core model: AMD Ryzen 9 5900X bits: 64 type: MT MCP smt: enabled arch: Zen 3 rev: 2
    cache: L1: 768 KiB L2: 6 MiB L3: 64 MiB
  Speed (MHz): avg: 2406 high: 3950 min/max: 2200/3700 boost: enabled cores: 1: 2145 2: 2047
    3: 3586 4: 2052 5: 3590 6: 2052 7: 2196 8: 2195 9: 2195 10: 2195 11: 2195 12: 2194 13: 2192
    14: 2130 15: 3950 16: 2048 17: 3580 18: 2048 19: 2195 20: 2193 21: 2195 22: 2196 23: 2193
    24: 2195 bogomips: 177266
  Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 svm
Graphics:
  Device-1: NVIDIA GA102 [GeForce RTX 3080 Lite Hash Rate] vendor: ASUSTeK driver: nvidia
    v: 535.161.07 pcie: speed: 2.5 GT/s lanes: 16 ports: active: none off: DP-2,HDMI-A-1,HDMI-A-2
    empty: DP-1,DP-3 bus-ID: 09:00.0 chip-ID: 10de:2216 class-ID: 0300
  Display: x11 server: X.Org v: 1.21.1.4 driver: X: loaded: nvidia
    unloaded: fbdev,modesetting,nouveau,vesa gpu: nvidia display-ID: :0 screens: 1
  Screen-1: 0 s-res: 3840x2520 s-dpi: 96 s-size: 1016x667mm (40.0x26.3") s-diag: 1215mm (47.8")
  Monitor-1: DP-2 pos: primary,bottom-r res: 2560x1440 dpi: 92 size: 708x399mm (27.9x15.7")
    diag: 813mm (32")
  Monitor-2: HDMI-0 pos: top-center res: 1920x1080 hz: 60 dpi: 82 size: 598x336mm (23.5x13.2")
    diag: 686mm (27")
  Monitor-3: HDMI-1 pos: primary,top-left res: 1920x1080 hz: 60 dpi: 82
    size: 598x336mm (23.5x13.2") diag: 686mm (27")
  OpenGL: renderer: NVIDIA GeForce RTX 3080/PCIe/SSE2 v: 4.6.0 NVIDIA 535.161.07
    direct render: Yes
Audio:
  Device-1: NVIDIA GA102 High Definition Audio vendor: ASUSTeK driver: snd_hda_intel v: kernel
    pcie: speed: 16 GT/s lanes: 16 bus-ID: 09:00.1 chip-ID: 10de:1aef class-ID: 0403
  Device-2: AMD Starship/Matisse HD Audio vendor: ASUSTeK driver: snd_hda_intel v: kernel pcie:
    speed: 16 GT/s lanes: 16 bus-ID: 0b:00.4 chip-ID: 1022:1487 class-ID: 0403
  Device-3: Logitech G733 Gaming Headset type: USB driver: hid-generic,snd-usb-audio,usbhid
    bus-ID: 1-1:2 chip-ID: 046d:0afe class-ID: 0300 serial: <filter>
  Sound Server-1: ALSA v: k5.15.0-101-generic running: yes
  Sound Server-2: PulseAudio v: 15.99.1 running: yes
  Sound Server-3: PipeWire v: 0.3.48 running: yes
Network:
  Device-1: Intel Wireless-AC 9260 driver: iwlwifi v: kernel pcie: speed: 5 GT/s lanes: 1
    bus-ID: 04:00.0 chip-ID: 8086:2526 class-ID: 0280
  IF: wlp4s0 state: down mac: <filter>
  Device-2: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet vendor: ASUSTeK driver: r8169
    v: kernel pcie: speed: 2.5 GT/s lanes: 1 port: f000 bus-ID: 05:00.0 chip-ID: 10ec:8168
    class-ID: 0200
  IF: enp5s0 state: up speed: 1000 Mbps duplex: full mac: <filter>
  IF-ID-1: docker0 state: down mac: <filter>
  IF-ID-2: lxcbr0 state: down mac: <filter>
Bluetooth:
  Device-1: Intel Wireless-AC 9260 Bluetooth Adapter type: USB driver: btusb v: 0.8 bus-ID: 3-5:2
    chip-ID: 8087:0025 class-ID: e001
  Report: hciconfig ID: hci0 rfk-id: 0 state: up address: <filter> bt-v: 3.0 lmp-v: 5.1
    sub-v: 100 hci-v: 5.1 rev: 100
Drives:
  Local Storage: total: 3.64 TiB used: 649.86 GiB (17.4%)
  ID-1: /dev/nvme0n1 vendor: Samsung model: SSD 990 PRO 2TB size: 1.82 TiB speed: 63.2 Gb/s
    lanes: 4 type: SSD serial: <filter> rev: 3B2QJXD7 temp: 46.9 C scheme: GPT
  ID-2: /dev/sda vendor: Seagate model: ST2000DM008-2FR102 size: 1.82 TiB speed: 6.0 Gb/s
    type: HDD rpm: 7200 serial: <filter> rev: 0001 scheme: MBR
Partition:
  ID-1: / size: 1.79 TiB used: 649.85 GiB (35.5%) fs: ext4 dev: /dev/nvme0n1p2
  ID-2: /boot/efi size: 607.8 MiB used: 6.1 MiB (1.0%) fs: vfat dev: /dev/nvme0n1p1
Swap:
  ID-1: swap-1 type: partition size: 724 MiB used: 0 KiB (0.0%) priority: -2 dev: /dev/nvme0n1p3
USB:
  Hub-1: 1-0:1 info: Hi-speed hub with single TT ports: 6 rev: 2.0 speed: 480 Mb/s
    chip-ID: 1d6b:0002 class-ID: 0900
  Device-1: 1-1:2 info: Logitech G733 Gaming Headset type: Audio,HID
    driver: hid-generic,snd-usb-audio,usbhid interfaces: 4 rev: 2.0 speed: 12 Mb/s power: 20mA
    chip-ID: 046d:0afe class-ID: 0300 serial: <filter>
  Hub-2: 2-0:1 info: Super-speed hub ports: 4 rev: 3.1 speed: 10 Gb/s chip-ID: 1d6b:0003
    class-ID: 0900
  Hub-3: 3-0:1 info: Hi-speed hub with single TT ports: 6 rev: 2.0 speed: 480 Mb/s
    chip-ID: 1d6b:0002 class-ID: 0900
  Device-1: 3-5:2 info: Intel Wireless-AC 9260 Bluetooth Adapter type: Bluetooth driver: btusb
    interfaces: 2 rev: 2.0 speed: 12 Mb/s power: 100mA chip-ID: 8087:0025 class-ID: e001
  Device-2: 3-6:3 info: ASUSTek AURA LED Controller type: HID driver: hid-generic,usbhid
    interfaces: 2 rev: 2.0 speed: 12 Mb/s power: 16mA chip-ID: 0b05:18f3 class-ID: 0300
    serial: <filter>
  Hub-4: 4-0:1 info: Super-speed hub ports: 4 rev: 3.1 speed: 10 Gb/s chip-ID: 1d6b:0003
    class-ID: 0900
  Hub-5: 5-0:1 info: Hi-speed hub with single TT ports: 4 rev: 2.0 speed: 480 Mb/s
    chip-ID: 1d6b:0002 class-ID: 0900
  Device-1: 5-3:2 info: Unicomp Unicomp R7_2_122_5250_Kbrd_v7_47 type: Keyboard
    driver: hid-generic,usbhid interfaces: 1 rev: 2.0 speed: 12 Mb/s power: 100mA chip-ID: 17f6:0865
    class-ID: 0301
  Device-2: 5-4:3 info: Razer USA DeathAdder Elite type: Mouse,Keyboard
    driver: hid-generic,usbhid interfaces: 3 rev: 2.0 speed: 12 Mb/s power: 500mA chip-ID: 1532:005c
    class-ID: 0300
  Hub-6: 6-0:1 info: Super-speed hub ports: 4 rev: 3.1 speed: 10 Gb/s chip-ID: 1d6b:0003
    class-ID: 0900
Sensors:
  System Temperatures: cpu: N/A mobo: N/A gpu: nvidia temp: 56 C
  Fan Speeds (RPM): N/A gpu: nvidia fan: 0%
Repos:
  Packages: 2928 apt: 2885 flatpak: 43
  No active apt repos in: /etc/apt/sources.list
  Active apt repos in: /etc/apt/sources.list.d/docker.list
    1: deb [arch=amd64 signed-by=/usr/share/keyrings/docker.gpg] https: //download.docker.com/linux/ubuntu jammy stable
  Active apt repos in: /etc/apt/sources.list.d/official-package-repositories.list
    1: deb http: //packages.linuxmint.com virginia main upstream import backport
    2: deb http: //archive.ubuntu.com/ubuntu jammy main restricted universe multiverse
    3: deb http: //archive.ubuntu.com/ubuntu jammy-updates main restricted universe multiverse
    4: deb http: //archive.ubuntu.com/ubuntu jammy-backports main restricted universe multiverse
    5: deb http: //security.ubuntu.com/ubuntu/ jammy-security main restricted universe multiverse
  Active apt repos in: /etc/apt/sources.list.d/rael-gc-ubuntu-xboxdrv-jammy.list
    1: deb [signed-by=/etc/apt/keyrings/rael-gc-ubuntu-xboxdrv-jammy.gpg] https: //ppa.launchpadcontent.net/rael-gc/ubuntu-xboxdrv/ubuntu jammy main
  Active apt repos in: /etc/apt/sources.list.d/spotify.list
    1: deb http: //repository.spotify.com stable non-free
  Active apt repos in: /etc/apt/sources.list.d/waydroid.list
    1: deb [signed-by=/usr/share/keyrings/waydroid.gpg] https: //repo.waydro.id/ jammy main
Info:
  Processes: 472 Uptime: 3h 6m wakeups: 0 Memory: 62.7 GiB used: 4.29 GiB (6.8%) Init: systemd
  v: 249 runlevel: 5 Compilers: gcc: 11.4.0 alt: 11/12 Client: Cinnamon v: 6.0.4 inxi: 3.3.13

This is my first post here let me know what else I can supply or try. I have been googling this with minor success but mostly old posts. I have tried changing the driver for the RTX 3080 but currently using the recommended one 535. Changing this didnt seem to make a difference.

edit: also i want to say this started after I upgraded to Virginia but that is a guess as to timing when this started.

[bsas]

No game running:

Code: Select all

Thu Mar 28 09:54:00 2024       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.161.07             Driver Version: 535.161.07   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 3080        Off | 00000000:09:00.0  On |                  N/A |
|  0%   50C    P8              40W / 340W |   1091MiB / 10240MiB |      6%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      1440      G   /usr/lib/xorg/Xorg                          786MiB |
|    0   N/A  N/A      2591      G   cinnamon                                     68MiB |
|    0   N/A  N/A      3537      G   ...apHandler --variations-seed-version      128MiB |
|    0   N/A  N/A     40390      G   ...ictureAPI --variations-seed-version       55MiB |
|    0   N/A  N/A     62441      G   ...local/share/Steam/ubuntu12_32/steam        4MiB |
|    0   N/A  N/A     62592      G   ./steamwebhelper                              5MiB |
|    0   N/A  N/A    184701      G   vmware-remotemks                             22MiB |
+---------------------------------------------------------------------------------------+

Running glmark2:

Code: Select all

Thu Mar 28 10:25:21 2024       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.161.07             Driver Version: 535.161.07   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 3080        Off | 00000000:09:00.0  On |                  N/A |
| 53%   46C    P8              40W / 340W |   1174MiB / 10240MiB |      5%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      1440      G   /usr/lib/xorg/Xorg                          877MiB |
|    0   N/A  N/A      2591      G   cinnamon                                    127MiB |
|    0   N/A  N/A      3537      G   ...apHandler --variations-seed-version       74MiB |
|    0   N/A  N/A     40390      G   ...ictureAPI --variations-seed-version       41MiB |
|    0   N/A  N/A     62441      G   ...local/share/Steam/ubuntu12_32/steam        4MiB |
|    0   N/A  N/A     62592      G   ./steamwebhelper                              5MiB |
|    0   N/A  N/A    184701      G   vmware-remotemks                             22MiB |
+---------------------------------------------------------------------------------------+

glmark2 results:

Code: Select all

=======================================================
    glmark2 2021.02
=======================================================
    OpenGL Information
    GL_VENDOR:     NVIDIA Corporation
    GL_RENDERER:   NVIDIA GeForce RTX 3080/PCIe/SSE2
    GL_VERSION:    4.6.0 NVIDIA 535.161.07
=======================================================
[build] use-vbo=false: FPS: 7663 FrameTime: 0.130 ms
[build] use-vbo=true: FPS: 13943 FrameTime: 0.072 ms
[texture] texture-filter=nearest: FPS: 13806 FrameTime: 0.072 ms
[texture] texture-filter=linear: FPS: 13756 FrameTime: 0.073 ms
[texture] texture-filter=mipmap: FPS: 14126 FrameTime: 0.071 ms
[shading] shading=gouraud: FPS: 13851 FrameTime: 0.072 ms
[shading] shading=blinn-phong-inf: FPS: 13671 FrameTime: 0.073 ms
[shading] shading=phong: FPS: 14053 FrameTime: 0.071 ms
[shading] shading=cel: FPS: 14095 FrameTime: 0.071 ms
[bump] bump-render=high-poly: FPS: 13071 FrameTime: 0.077 ms
[bump] bump-render=normals: FPS: 14290 FrameTime: 0.070 ms
[bump] bump-render=height: FPS: 14229 FrameTime: 0.070 ms
[effect2d] kernel=0,1,0;1,-4,1;0,1,0;: FPS: 14020 FrameTime: 0.071 ms
[effect2d] kernel=1,1,1,1,1;1,1,1,1,1;1,1,1,1,1;: FPS: 13195 FrameTime: 0.076 ms
[pulsar] light=false:quads=5:texture=false: FPS: 13801 FrameTime: 0.072 ms
[desktop] blur-radius=5:effect=blur:passes=1:separable=true:windows=4: FPS: 6923 FrameTime: 0.144 ms
[desktop] effect=shadow:windows=4: FPS: 10702 FrameTime: 0.093 ms
[buffer] columns=200:interleave=false:update-dispersion=0.9:update-fraction=0.5:update-method=map: FPS: 2137 FrameTime: 0.468 ms
[buffer] columns=200:interleave=false:update-dispersion=0.9:update-fraction=0.5:update-method=subdata: FPS: 3450 FrameTime: 0.290 ms
[buffer] columns=200:interleave=true:update-dispersion=0.9:update-fraction=0.5:update-method=map: FPS: 3004 FrameTime: 0.333 ms
[ideas] speed=duration: FPS: 12300 FrameTime: 0.081 ms
[jellyfish] <default>: FPS: 12499 FrameTime: 0.080 ms
[terrain] <default>: FPS: 2508 FrameTime: 0.399 ms
[shadow] <default>: FPS: 11229 FrameTime: 0.089 ms
[refract] <default>: FPS: 7797 FrameTime: 0.128 ms
[conditionals] fragment-steps=0:vertex-steps=0: FPS: 13994 FrameTime: 0.071 ms
[conditionals] fragment-steps=5:vertex-steps=0: FPS: 14030 FrameTime: 0.071 ms
[conditionals] fragment-steps=0:vertex-steps=5: FPS: 14069 FrameTime: 0.071 ms
[function] fragment-complexity=low:fragment-steps=5: FPS: 13984 FrameTime: 0.072 ms
[function] fragment-complexity=medium:fragment-steps=5: FPS: 14042 FrameTime: 0.071 ms
[loop] fragment-loop=false:fragment-steps=5:vertex-steps=5: FPS: 13777 FrameTime: 0.073 ms
[loop] fragment-steps=5:fragment-uniform=false:vertex-steps=5: FPS: 13623 FrameTime: 0.073 ms
[loop] fragment-steps=5:fragment-uniform=true:vertex-steps=5: FPS: 12859 FrameTime: 0.078 ms
=======================================================
                                  glmark2 Score: 11651 
=======================================================

[SMG]

Recently my desktop has started crashing to a black screen with the audio still going while gaming in a few different games launching from Steam requiring a hard reboot but i think this is a graphics related problem. This has not happened without a game running for multiple days uptime but happens pretty quickly when gaming. This is new as of a month or so ago.

Welcome to the forum, bsas.

If this is the correct link, the latest BIOS/UEFI is version 5003. There are quite a few improve system performance and improve system stability in the ones you do not have.

Machine:
Type: Desktop Mobo: ASUSTeK model: TUF GAMING X570-PLUS (WI-FI) v: Rev X.0x
serial: <superuser required> UEFI: American Megatrends v: 2802 date: 10/21/2020

I also recommend upgrading to a newer kernel to see if that helps.

Instructions for upgrading to the 6.5 kernel:

• Open Update Manager.
• Select View > Linux Kernels and click Continue.
• Make sure 6.5 is selected on the left panel and then click the top-most option on the right panel. An "Install" button will appear.
• Install the kernel and then reboot for it to become active.

[bsas]

Thanks for the reply, SMG.

I have upgraded the kernel to 6.5 via a recommendation in IRC along with some other items which have not resolved the problem. I will look into the BIOS update and report back.