Kernel Panic with PCIe riser and Radeon RX570 and amdgpu drivers

[agent37]

I have a Radeon RX 570 that doesn't play nice in my multi-gpu rig that I use for mining, so I wanted to put it on my linux mint system with a PCI riser card to have it just do its thing.

When I plug it in to the main pci slot, I can boot and it works fine, but I don't want to run the display and mine on it because eventually the display part crashes. So I have a $50 pci vid (1G radeon 6350) card I use to run the display and I am trying to use a PCI riser for the 570.

If I do it without the amdgpu drivers installed, it hangs at a blinking cursor on boot. When I install the amdgpu driver (normal, pro, whatever) it gives me these errors and usually kernel panics.

https://imgur.com/a/8j3JbRD

I have 2 systems with the same setup. Msi B550m PRO-VDH WIFI, ryzen 3950x, 16G 2666mHz corsair memory. Same result no matter which motherboard I try it on.

If I boot into windows, it works fine.

Has anyone else had to do anything in particular to get the riser working on Linux Mint / Ubuntu 20.x?

I have tried iommu=soft and iommu=pt and amd_iommu=off. Same result every time.

I have searched high and low and haven't ran across anything specifically mentioning pci risers being an issue. Any help or a nudge in the right direction is appreciated.

[SMG]

Welcome to the forum, agent37.

One of the reasons Section 5 of READ: How To Get Help! explains the data we need to be able to help you is so we can see how Mint views your system. Please give us information about your install by entering this command in a terminal: inxi -Fxxxrz
Click </> from the mini toolbar above the textbox where you type your reply and then place your cursor between the code markers and paste the results of the command between the code markers [code]Results[/code]. This will let us know how Mint sees your hardware.

With a Ryzen 3950x I would assume you would want to be running the 5.8 kernel, but we do not know if you are doing that. It's possible that kernel might help run the display and mine without there being crashes.

[agent37]

Thanks, I was sort of looking for just things to look up and try rather than the ultimate solution, but thanks for the help! Here's the readout. I updated to 5.8 in part to fix this bad wifi connection speed issue I was having.

Anyway, here are the results:

Code: Select all

System:    Kernel: 5.8.0-53-generic x86_64 bits: 64 compiler: N/A Desktop: Cinnamon 4.8.6 wm: muffin 4.8.1 dm: LightDM 1.30.0 
           Distro: Linux Mint 20.1 Ulyssa base: Ubuntu 20.04 focal 
Machine:   Type: Desktop Mobo: Micro-Star model: B550M PRO-VDH WIFI (MS-7C95) v: 1.0 serial: <filter> 
           UEFI: American Megatrends LLC. v: 2.72 date: 04/09/2021 
CPU:       Topology: 16-Core (2-Die) model: AMD Ryzen 9 3950X bits: 64 type: MT MCP MCM arch: Zen L2 cache: 8192 KiB 
           flags: avx avx2 lm nx pae sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 svm bogomips: 223988 
           Speed: 1866 MHz min/max: 2200/3500 MHz boost: enabled Core speeds (MHz): 1: 1865 2: 2802 3: 2195 4: 2198 5: 2198 
           6: 2197 7: 2199 8: 2199 9: 1866 10: 2790 11: 1862 12: 1860 13: 1864 14: 2198 15: 2201 16: 2192 17: 2200 18: 2197 
           19: 2201 20: 2197 21: 2198 22: 2200 23: 2200 24: 1864 25: 2199 26: 2200 27: 2198 28: 2199 29: 2197 30: 2196 
           31: 2199 32: 2197 
Graphics:  Device-1: Advanced Micro Devices [AMD/ATI] Cedar [Radeon HD 5000/6000/7350/8350 Series] 
           vendor: Hightech Information System driver: radeon v: kernel bus ID: 2b:00.0 chip ID: 1002:68f9 
           Display: x11 server: X.Org 1.20.9 driver: ati,radeon unloaded: fbdev,modesetting,vesa resolution: 1920x1080~60Hz 
           OpenGL: renderer: AMD CEDAR (DRM 2.50.0 / 5.8.0-53-generic LLVM 11.0.0) v: 3.3 Mesa 20.2.6 compat-v: 3.1 
           direct render: Yes 
Audio:     Device-1: Advanced Micro Devices [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300/7300 Series] 
           vendor: Hightech Information System driver: snd_hda_intel v: kernel bus ID: 2b:00.1 chip ID: 1002:aa68 
           Device-2: Advanced Micro Devices [AMD] Starship/Matisse HD Audio vendor: Micro-Star MSI driver: snd_hda_intel 
           v: kernel bus ID: 2d:00.4 chip ID: 1022:1487 
           Sound Server: ALSA v: k5.8.0-53-generic 
Network:   Device-1: Intel Dual Band Wireless-AC 3168NGW [Stone Peak] driver: iwlwifi v: kernel bus ID: 29:00.0 
           chip ID: 8086:24fb 
           IF: wlo1 state: up mac: <filter> 
           Device-2: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet vendor: Micro-Star MSI driver: r8169 v: kernel 
           port: f000 bus ID: 2a:00.0 chip ID: 10ec:8168 
           IF: enp42s0 state: up speed: 1000 Mbps duplex: full mac: <filter> 
Drives:    Local Storage: total: 465.76 GiB used: 29.87 GiB (6.4%) 
           ID-1: /dev/nvme0n1 vendor: Crucial model: CT500P2SSD8 size: 465.76 GiB speed: 31.6 Gb/s lanes: 4 serial: <filter> 
           rev: P2CR010 scheme: GPT 
Partition: ID-1: / size: 456.96 GiB used: 29.87 GiB (6.5%) fs: ext4 dev: /dev/nvme0n1p2 
Sensors:   System Temperatures: cpu: 61.5 C mobo: N/A gpu: radeon temp: 48 C 
           Fan Speeds (RPM): N/A 
Repos:     No active apt repos in: /etc/apt/sources.list 
           Active apt repos in: /etc/apt/sources.list.d/google-chrome.list 
           1: deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main
           Active apt repos in: /etc/apt/sources.list.d/official-package-repositories.list 
           1: deb http://mirrors.usinternet.com/mint/packages ulyssa main upstream import backport
           2: deb http://archive.ubuntu.com/ubuntu focal main restricted universe multiverse
           3: deb http://archive.ubuntu.com/ubuntu focal-updates main restricted universe multiverse
           4: deb http://archive.ubuntu.com/ubuntu focal-backports main restricted universe multiverse
           5: deb http://security.ubuntu.com/ubuntu/ focal-security main restricted universe multiverse
           6: deb http://archive.canonical.com/ubuntu/ focal partner
Info:      Processes: 519 Uptime: N/A Memory: 15.56 GiB used: 1.67 GiB (10.7%) Init: systemd v: 245 runlevel: 5 Compilers: 
           gcc: 9.3.0 alt: 9 Shell: bash v: 5.0.17 running in: gnome-terminal inxi: 3.0.38 

This is without the Radeon RX570 plugged in, because the system won't boot fully into Mint when it is.

[SMG]

This is without the Radeon RX570 plugged in, because the system won't boot fully into Mint when it is.

You mean it won't boot when you have both cards in with the RX 570 on the PCI riser? That is what I understood to be the case from your initial post.

When I plug it in to the main pci slot, I can boot and it works fine, but I don't want to run the display and mine on it because eventually the display part crashes. So I have a $50 pci vid (1G radeon 6350) card I use to run the display and I am trying to use a PCI riser for the 570.

If I do it without the amdgpu drivers installed, it hangs at a blinking cursor on boot. When I install the amdgpu driver (normal, pro, whatever) it gives me these errors and usually kernel panics.

What do you mean "do it without the amdgpu drivers installed"? The drivers come from the kernel and install automatically when one boots. What are you doing to prevent the amdgpu driver from installing?

[agent37]

You mean it won't boot when you have both cards in with the RX 570 on the PCI riser? That is what I understood to be the case from your initial post.

correct

What do you mean "do it without the amdgpu drivers installed"? The drivers come from the kernel and install automatically when one boots. What are you doing to prevent the amdgpu driver from installing?

I mean downloading the drivers from AMD and using amdgpu-install / amdgpu-pro-install -opencl=legacy / -opencl=rocm,legacy

Then I run amgpu-uninstall to remove and it freezes at a blinking cursor. Remove the 570 and the riser, boots right up.

I am getting a new GPU in a few days. I will give that a shot and see what happens. Any suggestions for things to try/search/etc... I'm happy to do the homework.

Thanks again.

[SMG]

I mean downloading the drivers from AMD and using amdgpu-install / amdgpu-pro-install -opencl=legacy / -opencl=rocm,legacy

We only recommend downloading drivers from the AMD website if someone is having specific graphic issues that the pro drivers are known to fix. One does NOT need to get drivers from the AMD website to get amdgpu drivers. The amdgpu drivers in the kernel actually come from AMD.

Then I run amgpu-uninstall to remove and it freezes at a blinking cursor.

That does not always remove all the files. It's possible the amdgpu driver is being black-listed because not all the files were removed and that's why you are getting the blinking cursor.

Run ls /etc/modprobe.d/ to see if there is an amdgpu blacklist file. That file has been known to not always remove when the AMD pro driver is removed.

Remove the 570 and the riser, boots right up.

The 6350 card uses the radeon driver and not the amdgpu driver. That might be why it works since it's very possible the amdgpu driver is being blocked because the uninstall did not remove everything. The AMD pro driver does not have a radeon component.

[agent37]

The only reason I got the amdgpu drivers from AMD is because I couldn't figure out how to install/enable opencl, which I need.

This is the result of listing /etc/modprobe.d/:

Code: Select all

total 44
-rw-r--r-- 1 root root 2507 Jul 30  2015 alsa-base.conf
-rw-r--r-- 1 root root  154 Feb 16  2020 amd64-microcode-blacklist.conf
-rw-r--r-- 1 root root  325 Mar 12  2020 blacklist-ath_pci.conf
-rw-r--r-- 1 root root 1518 Mar 12  2020 blacklist.conf
-rw-r--r-- 1 root root  210 Mar 12  2020 blacklist-firewire.conf
-rw-r--r-- 1 root root  677 Mar 12  2020 blacklist-framebuffer.conf
-rw-r--r-- 1 root root  156 Jul 30  2015 blacklist-modem.conf
lrwxrwxrwx 1 root root   41 May 14 15:28 blacklist-oss.conf -> /lib/linux-sound-base/noOSS.modprobe.conf
-rw-r--r-- 1 root root  583 Mar 12  2020 blacklist-rare-network.conf
-rw-r--r-- 1 root root  127 Jan 22  2020 dkms.conf
-rw-r--r-- 1 root root  154 Nov 11  2020 intel-microcode-blacklist.conf
-rw-r--r-- 1 root root  347 Mar 12  2020 iwlwifi.conf

I don't 100% remember, but I think I did a wipe/install of mint and it still didn't work. I will try again just to make sure.

When I ran amdgpu-uninstall, I did notice there were two files or directories that couldn't be removed, so I went in and manually deleted them. I don't remember what they were. I'll post back with the results of the wipe/install in a few. Thanks again.

[agent37]

Trying to boot off installation usb stick with both cards connected, I get this then it freezes on the LM green dot boot screen:

[SMG]

The only reason I got the amdgpu drivers from AMD is because I couldn't figure out how to install/enable opencl, which I need.

I do not have a lot of experience with that, but I understand that to be a reason for getting the pro drivers. I usually recommend running a Timeshift snapshot before installing graphics drivers. That makes it much easier to get a clean "undo" of the changes.

This is the result of listing /etc/modprobe.d/:

I do not see the blacklist file which I have seen in some other installs.

Trying to boot off installation usb stick with both cards connected, I get this then it freezes on the LM green dot boot screen:

Are you using the Linux Mint 20.1 "Ulyssa" - Cinnamon (Edge, 64-bit) with the 5.8 kernel?

Check your BIOS/UEFI for a setting for PCI-E link. I seem to recall helping someone who just changed MoBo's and the system kept crashing and it was because of that setting? I will have to see if I can find the thread. I think you do not want it to be "auto", but I'm going to have to do some searching to find the thread where that issue came up to see what the setting should be.

[agent37]

Thanks, I'll try an install with that image and let you know the results.

Edit: I am pretty sure it should be gen1 for riser boards, if I am thinking of the same PCI setting in the bios. I'll double check when I reinstall with the new image.

[agent37]

OK! PROGRESS! I am booted up with both cards installed. I tweaked some more bios settings and that seems to have done the trick. Now the issue is that the cheapo card is using the old radeon drivers (I think) and the RX570 is using the amdgpu opencl driver. (at least that is what some googling is pointing me toward when I search for the error I'm getting when I try to start the mining software):

Code: Select all

amdgpu_device_initialize: DRM version is 2.50.0 but this driver is only compatible with 3.x.x.
Segmentation fault (core dumped)

Is there a way to somehow set a variable to ignore the cheap card?

[SMG]

OK! PROGRESS! I am booted up with both cards installed. I tweaked some more bios settings and that seems to have done the trick.

Glad you got it working because that means I can stop looking for that thread I can't seem to find.

Now the issue is that the cheapo card is using the old radeon drivers (I think) and the RX570 is using the amdgpu opencl driver. (at least that is what some googling is pointing me toward when I search for the error I'm getting when I try to start the mining software):

Code: Select all

amdgpu_device_initialize: DRM version is 2.50.0 but this driver is only compatible with 3.x.x.
Segmentation fault (core dumped)

Is there a way to somehow set a variable to ignore the cheap card?

Is there a way to set a variable so the mining software ignores the cheap card? The cheap card is using is using radeon. The cheap card is not using amdgpu, so I would not think that is a factor in the error since the error indicates amdgpu_device_initialize.

[agent37]

It would be cool to figure out if there's a way to do it, but I just ordered the cheapest vid card supported by the amdgpu driver, so that should solve the issue.

I'm willing to do some homework on a fix if you point me in the right direction.

[agent37]

Is there a way to set a variable so the mining software ignores the cheap card? The cheap card is using is using radeon. The cheap card is not using amdgpu, so I would not think that is a factor in the error since the error indicates amdgpu_device_initialize.

Yeah, I will monkey around with that and see what I can do. Thanks for your help! It is much appreciated.

[SMG]

Is there a way to set a variable so the mining software ignores the cheap card? The cheap card is using is using radeon. The cheap card is not using amdgpu, so I would not think that is a factor in the error since the error indicates amdgpu_device_initialize.

Yeah, I will monkey around with that and see what I can do. Thanks for your help! It is much appreciated.

I found this older thread which indicates, "DRM version 2.x.y is the radeon kernel driver, the amdgpu kernel driver is version 3.x.y. If you want to use the latter, you may need to blacklist the former.". However, that sounds to me like you would not be using the cheap card for graphics if you did that.

What is the output of inxi -Gx? Are both the radeon and amdgpu drivers listed?

I'm wondering if you now made changes in BIOS whether you could successfully use just the RX570? I do not know enough about mining software to know if that would work or whether you definitely need two cards.

[agent37]

Hoo boy, I went down a rabbit hole last night. I ended up trying to install Mint 18.3 and using the older AMD drivers.. That didn't work either, even though my googling turned up a bunch of success doing that.

I definitely had radeon blacklisted, made no difference. I might try the next release of the AMD drivers, but for now I'm just going to wait for my amdgpu compatible card to arrive and pretend this never happened.

Thanks again for your help.

[SMG]

Hoo boy, I went down a rabbit hole last night. I ended up trying to install Mint 18.3 and using the older AMD drivers.. That didn't work either, even though my googling turned up a bunch of success doing that.

LM18.3 is no longer supported. Only LM19 and LM20 have support.