[SOLVED] random Freeze/reboots on asus H470, mate 20.1 64bit

[SMG]

Sorry, output as a picture this time.

Pictures work.

It appears device 1d.0 is the NVMe drive controller and device 1d.3 is the Ethernet card. I had read yesterday the message for the Ethernet card is commonly seen, so it may be that neither of these are the source of the issue.

DPC = Deferred Procedure Call
IRQ = Interrupt ReQuest
Linux generic IRQ handling indicates "The generic interrupt handling layer is designed to provide a complete abstraction of interrupt handling for device drivers."

Best as I can tell, these messages indicate the system has given a special designation for handling any issues from these two devices. I would not think these are the issue.

Maybe something else of interest. The crashreport mentioned

1983 /usr/bin/gdk-pixbuf-thumbnailer

I seem to recall that coming up when helping someone troubleshoot Cinnamon crashes. I seem to recall it being related to other software the person had installed. It should not crash the entire system.

btw, mcelog could not be installed

I see in Checking Out Machine Check Exception (MCE) Errors in Linux that is an older package and "we’re being told the mcelog package functionality has been replaced by rasdaemon."

However, the person was able to get mcelog installed and had problems using rasdaemon. Maybe something in the article or the comments for it might give you some ideas.

The article indicates, "Machine check exceptions (MCEs) can occur for a variety of reasons ranging from undesired or out-of-spec voltages from the power supply, from cosmic radiation flipping bits in memory DIMMs or the CPU, or from other miscellaneous faults, including faulty software triggering hardware errors."

If you are able to get a copy of the log of the prior boot (one that crashes), maybe we can get a better idea of what is happening.

[Ivo-G]

I have reset the bios to optimised defaults and again disabled the fast boot. No difference.

Also sent a mail to asus tech support, knowing full well they would likely put their middle finger in front of my face while politely telling me they don't support linux (and so they did in a speedy reply).

Tried booting several different linux live distros. Who knows one might somehow work or at least barf out an error that actually identifies the problem.

Linux lite
holds during boot at
rc.local[2990]: A connection to the bus can't be made
Could also not be turned off and also had
stdin: Invalid argument

RebornOS 2021-03-13
Shows the processor zero errors during boot, but able to boot up the system
Kernel 5.11.6-arch1-1
Froze and rebooted randomly.

Solus 4.2 budgie
Could not even get past the first 3 lines in boot, Stuck in less then a sec.

Sparky linux 5.14 lxqt
kernel 4.19
Able to boot briefly
Froze and rebooted randomly

Ubuntu 20.10
Able to boot briefly
Froze and rebooted randomly

Zorin 15.3 core
Able to boot and stay up a bit longer
Froze and rebooted randomly

Also tried newest versions of clonezilla and redo resque again so I could at least save the currently still perfectly functional win10pro64
Clonezilla still gives processor zero errors, not able to functionally boot
Redo resque is able to boot, but also froze randomly.

pics of the boot results: https://drive.google.com/drive/folders/ ... sp=sharing

Called the computerstore for help, and if they had a similar MB in a functional desktop handy to test a live distro, but no luck so far.
I did hear the asus x470 works well with linux, but it has an amd chipset, so comparing the differences with the intel h470 is probably a bit useless.

And the last error report I could eventually get to (getting to this always takes a few reboots):
https://termbin.com/55t0

Did not get to the mcelog yet. There are only so many crashes one can stomach each day without knowing what causes it...

[SMG]

Linux lite
holds during boot at
rc.local[2990]: A connection to the bus can't be made
Could also not be turned off and also had
stdin: Invalid argument

That can often relate to the boot order in the BIOS.

And the last error report I could eventually get to (getting to this always takes a few reboots):
https://termbin.com/55t0

Signal: 4 (ILL) is an illegal instruction. The file it is trying to thumbnail seems to be file:///home/HDD/THETA/Moon.ico. (Is that a thumbnail for sdc?)

The fact you are seeing essentially the same thing with all these distros points to hardware. I do not know how much the hardware is new, but I came across two ideas as I was researching some of the items which came up.

1) Do all your drives have the latest firmware installed?

2) Are you positive your PSU and the individual rails on your mobo can handle the load you have on them?

You have a lot of drives installed. While reading through this thread Correct BIOS NVME M.2 Settings and other oddities while trying to find possible similarities to your situation, the post at the top of that page I linked brought back a memory of someone on this forum who was having an issue. It turned out to be a power-sharing issue related to the rails being used when drives were placed in certain slots. One slot it worked and the other it did not.

Food for thought from the post at the top of that page I linked, "how many sata drives can you connect while the nvme slot is occupied? I know that with some motherboards 2 of the ports will drop down in speed when nvme is used." I realize that is a different board with a different cpu, but the beginning of this thread (yours) started with a discussion of how the system was seeing a raid that you didn't have. Maybe the issue is the set-up of the drives. Which ones are pulling how much power versus how much power is available. Maybe that is the reason for the instability.

[Ivo-G]

The file it is trying to thumbnail seems to be file:///home/HDD/THETA/Moon.ico. (Is that a thumbnail for sdc?)

I killed the moon right after I first posted that msg... (or the .ico at least)
I used to have that icon file as a custom trash-bin icon in windows, but it's not being used anywhere currently.

Are you positive your PSU and the individual rails on your mobo can handle the load you have on them?

I should even be able to place one more sata in the current config. I checked it before I even installed any of the drives.
Just to be absolutely sure, I again physically removed all the satas. As usual, it did not change a thing.
And the power supply is a 500W, should be more than enough, I had the same type on my previous system with 7 HDD's
Windows isn't making a fuzz about any of it either.

In the meantime there is an update to the kernels. Took only like 15 crashes to get those installed.
Obviously didn't fix anything. Decided to install kernel 5.10 again, and to my enormous surprise, that worked like a charm Even able to reboot the system myself this time (have not been able to do that much, it's usually rather "involuntary")
I created these termbin logs right after the successful boot into kernel 5.10: ippv and vgxx
...and about 30sec later it crashed again...
After reboot I created the termbin logs again: onb3 and 81y8

processor zero is in there again, but also several raid6 lines. Again that raid config!

The crashing itself can happen early in the boot process to minutes after bootup and everywhere in between. It's extremely random, but opening apps or installing anything seems to increase the instability greatly.

[Ivo-G]

mcelog installed, output in attachment

[SMG]

mcelog installed, output in attachment

MCE error: MCA: Internal parity error is a case where the original poster ended up swapping the cpu's in two different machines and then they both worked without errors (very end of answer 1).

I also thought the fourth answer related to UEFI and clocking was interesting. I had read your mobo has the capability to change (turbo-like mode) without it specifically being an overclocking, but I think that is something you have to proactively set.

I should even be able to place one more sata in the current config. I checked it before I even installed any of the drives.
Just to be absolutely sure, I again physically removed all the satas. As usual, it did not change a thing.
And the power supply is a 500W, should be more than enough, I had the same type on my previous system with 7 HDD's
Windows isn't making a fuzz about any of it either.

I participated on a thread which spanned multiple months as we tried to determine why someone's Mint was randomly rebooting. That did not happen on the person's Windows install. Someone early in the thread suggested it might be the PSU, but it was a brand new PSU that had enough wattage to handle the load. Turns out that PSUs can vary in how much power they supply to each of the rails. The specs for that PSU was running the low end on the rail which powered the drive on which Mint was located. Therefore, it didn't take much change to drop power to the drive and then resume power to it. As I recall, that system only had two drives, so it was not a matter of how many drives were installed.

Your issue may not be related to the PSU, but the example I just stated was not even the one of which I was originally thinking when I posted my prior comment.

In the meantime there is an update to the kernels. Took only like 15 crashes to get those installed.

Apparently, when updating the kernel os-prober runs. I saw that in ippv. I'm still going through the rest.

From ippv:
The below can only be corrected with a firmware update for the drive (from the drive manufacturer). I've seen the line on logs from other computers so I do not expect this is what is causing the problem.

Code: Select all

Mar 24 08:24:36 OMEGA-DESKTOP kernel: nvme nvme0: missing or invalid SUBNQN field.

This looks unusual. There were actually more lines than these three. I found this systemd pstore: Tool to archive contents of pstore which describes it, but I'm not quite sure what the data is or what we can do with it.

Code: Select all

Mar 24 08:24:36 OMEGA-DESKTOP systemd-pstore[721]: PStore dmesg-efi-161657062208001 moved to /var/lib/systemd/pstore/161657062/dmesg-efi-161657062208001
Mar 24 08:24:36 OMEGA-DESKTOP systemd-pstore[721]: PStore dmesg-efi-161657062209001 moved to /var/lib/systemd/pstore/161657062/dmesg-efi-161657062209001
Mar 24 08:24:36 OMEGA-DESKTOP systemd-pstore[721]: PStore dmesg-efi-161657062210001 moved to /var/lib/systemd/pstore/161657062/dmesg-efi-161657062210001

You have multiple ntfs drives, so I'm not sure to which one this line refers.

Code: Select all

Mar 24 08:24:38 OMEGA-DESKTOP ntfs-3g[779]: Warning : using problematic uid==0 and gid!=0

The below happened twice. sde is the problem-child drive I mentioned earlier in the thread.

Code: Select all

Mar 24 08:24:38 OMEGA-DESKTOP udisksd[887]: Error probing device: Error sending ATA command IDENTIFY DEVICE to '/dev/sde': Unexpected sense data returned:
                                            0000: 70 00 01 00  00 00 00 0a  00 00 00 00  00 1d 00 00    p...............
                                            0010: 00 00 00 00  00 00 00 00  00 00 00 00  00 00 00 00    ................
                                             (g-io-error-quark, 0)

The crashing itself can happen early in the boot process to minutes after bootup and everywhere in between. It's extremely random, but opening apps or installing anything seems to increase the instability greatly.

Sounds like anything which requires a high cpu load.

[SMG]

A lot of the same messages were repeated in all the logs, so I'm just going to remark on what might be new information.

vgxx
You moved to the 5.10.0-051000-generic on this log. I am not familiar with that specific version.

In another thread, some people with Comet Lake were not able to get the i915 (Intel) graphics driver to load. The fix was to use the latest Ubuntu 5.10-oem kernel. I do not know if it has any other fixes which might benefit Comet Lake, but that i915 fix is not in newer mainline kernels. If you want to try it, install the latest 5.10-oem kernel with sudo apt install linux-oem-20.04b. It's currently supported and receiving updates. I did not mention it earlier because you are using your Nvidia card, but since you've installed a 5.10 kernel, might want to try the Ubuntu-supported one.

onb3
This log produced some lines which seem to indicate this issue may be related to the cpu and how it is being handled.

I found the following thread CMCI Storm Detected and MCE Hardware Error which mentioned possible hardware problems.

Code: Select all

Mar 24 08:38:48 OMEGA-DESKTOP kernel: mce: CMCI storm detected: switching to poll mode

But I also found this thread CMCI storm detected: switching to poll mode in which the last post mentioned using two kernel parameters, processor.max_cstate=1 and intel_idle.max_cstate=0 to mitigate issues.

From what I could find, those parameters modify the cpu's ability to enter power-saving states. There are always pluses and minuses to using kernel parameters and at least one of the minuses I found was these can cause more power consumption (which as a result means more heat generation).

Then I later found this line in the log:

Code: Select all

Mar 24 08:29:06 OMEGA-DESKTOP kernel: mce: CPU0: Thermal monitoring enabled (TM1)

Researching that I found this thread Linux 20.04 MCE Kernel Errors which indicated kernel parameter intel_idle.max_cstate=1 solved the issue which included freezes. That is a different value for intel_idle.max_cstate that what was mentioned in the other thread.

These parameters were used on processors older than what you have. I do not know if they might be appropriate for Comet Lake given newer processors are usually designed at least a little different from older ones.

You have multiple ntfs drives, so I'm not sure to which one this line refers.

Code: Select all

Mar 24 08:24:38 OMEGA-DESKTOP ntfs-3g[779]: Warning : using problematic uid==0 and gid!=0

I found this message repeatedly in onb3 and it relates to sda1(Theta_Tau), sdc2 (THETA), sdd2(OMNICRON), and sde2(LAMBDA) in that log. I guess that means either none of them are problem or all of them are.

81y8
The below is a common message I've seen in most (maybe all) of the logs. It indicates there are messages which are not being printed.

Code: Select all

[Wed Mar 24 08:42:53 2021] mce_notify_irq: 9 callbacks suppressed

Hopefully something in this post or the prior one will give you some ideas.

[Ivo-G]

since you've installed a 5.10 kernel, might want to try the Ubuntu-supported one.

I'm now running the oem 5.10
Like everything else so far: no difference at all.
Did not get to new logs yet, but doubtful they will reveal anything new anyway.

using two kernel parameters, processor.max_cstate=1 and intel_idle.max_cstate=0 to mitigate issues.

If I understand correctly, I should enter these in grub?
I tried both and one by one. Grub doesn't seem to accept these lines.

You have multiple ntfs drives, so I'm not sure to which one this line refers.

I suspect sdb. I have a couple of 2TB WD green HDD laying around, will replace the WD red 2TB soon.
It's not the main problem though, the system is still unstable even with all the drives disconnected.
And I have the WD data diagnostics running under windows, and just to be sure did a manual scan on all the drives and there are nowhere any problems to be found.

It indicates there are messages which are not being printed.

I'll try to update mcelog, maybe that will give something new.

Other than that, I'm not sure what else I can still try.
I do have a functional win10pro64 (that does not indicate ANY problem anywhere), isn't there anything I can do there? At least I would not get the constant crashing while looking up anything (It's starting to get REALLY frustrating). If any kind of error can be found in the hardware under windows I could return that hardware for replacement.

[Ivo-G]

I found how to enter the lines in grub (grub cmdline), but crashed during the grub update and now no longer able to boot into linux :/
EDIT: can't even get mint live booted after several tries.
I'm giving up for today. It's not even noon here and allready had to endure like 50 crashes. I've reached the limit

[Ivo-G]

I ran the intel processor diagnostics tool, not a single thing wrong with the cpu. Everything completely in the green.
Installed samsung magician for the ssd, all in the green and firmware was allready up to date
I ran de WD tools again for my sata hdd's, nothing wrong to be found anywhere.
Tested the external mybook 12tb on my other pc with both win10pro64 and mint mate64 20.1, not a problem to be found anywhere in either system.

[SMG]

You have verified that Win10 did not turn fast boot back on? Some people have found that has happened.

I just helped someone on another thread who has an i7-10700 on a ASRock B560 Steel Legend. The problem turned out just to be the Intel i915 graphics driver and the 5.10-oem kernel fixed the issue, but I had that person run journalclt to try and diagnose the issue. In that log there was just one set of mce messages (shown below) and no messages regarding the efi (which has been consistently showing in your logs). That was output while running a 5.12 kernel (before switching to the 5.10-oem).

Code: Select all

kernel: mce: [Hardware Error]: Machine check events logged
kernel: mce: [Hardware Error]: CPU 0: Machine Check: 0 Bank 6: ee0000000040110a
kernel: mce: [Hardware Error]: TSC 0 ADDR fef1e680 MISC 43880000086 
kernel: mce: [Hardware Error]: PROCESSOR 0:a0655 TIME 1616552620 SOCKET 0 APIC 0 microcode e2

Do you just have the one efi partition for both Windows and Mint? I do not have a lot of experience troubleshooting efi issues. I would have thought since the system was booting that all was well, but maybe there is more to it than that.

[SMG]

I had read your mobo has the capability to change (turbo-like mode) without it specifically being an overclocking, but I think that is something you have to proactively set.

This is the article which mentioned what I referenced in the post.
Intel Comet Lake Non-K CPUs ‘Power Limit’ Overclocking on ASUS, ASRock, MSI H470, B460 & H410 Motherboards Detailed & Compared

I do not know what the default or optimized states in the BIOS might have for individual settings, but maybe somehow that is a factor in this issue?

[Ivo-G]

Haven't checked the bios yet what setting it has regarding the sort of overclocking. But next weekend probably i'm going to dismantle the system and try to reïnstall only linux on a single blank ssd, without any other sata or nvme. If even that still doesn't work...
Currently I can't even get the live mint distro booted anymore.

[SMG]

Hopefully, a new month will bring new results. I have not come across any new ideas, but will post if I think of anything or come across something someone else has seen.

When one boots to a live session of Mint, Mint is looking at all the attached drives and partitions. In the case of UEFI, it is also looking for the EFI information on the computer. You may already know that, but wanted to pass it along in case it might somehow be a factor.

[Ivo-G]

We can now also rule out that anything ON the nvme is causing the instability.
It has to be something in the MB that Linux can't handle for some very dark reason.
I could try putting the MB on a leash by reducing it's builtin capabilities like the turbo mode. But from what I found online so far, Linux should not have a problem with it at all. Also not fond of reducing the capability of my system so significantly that even my previous desktop would beat it.

Is there a way to test the MB itself and the psu? I found some apps for the hardware, but non free. And I don't have a compatible psu laying around to replace it for testing purposes.

Some of the uefi settings:
https://drive.google.com/drive/folders/ ... sp=sharing

[SMG]

According to this Intel® Core™ i5-10400 Processor Specs the Processor Base Frequency is 2.90 GHz and the Max Turbo Frequency 4.30 GHz. Your current setting of Target CPU Turbo-Mode Frequency of 4300MHz fits Intel's specs and the current frequency of 2900 MHz is the design spec.

Is there a way to test the MB itself and the psu? I found some apps for the hardware, but non free. And I don't have a compatible psu laying around to replace it for testing purposes.

Hardware testing is not something with which I have much experience.

Some of the uefi settings:
https://drive.google.com/drive/folders/ ... sp=sharing

The voltages showing have the 3.3V at 3% over and the others are tighter tolerances. I've read 3-5% tolerance is normal specs for a PSU, so unless those values are varying while the computer is running, I would think they are fine. However, I defer to others who most likely have more hardware experience than I do.

The only thing which caught my eye was the C states option (shown in 18.44.30.jpg), but I do not know enough about changing BIOS/UEFI options for it to make suggestions.

Maybe someone with more motherboard experience will have some suggestions.

[Ivo-G]

Finally managed to get mint reïnstalled.

using two kernel parameters, processor.max_cstate=1 and intel_idle.max_cstate=0 to mitigate issues.

kernel parameter intel_idle.max_cstate=1 solved the issue which included freezes.

I tried both the commandsets, they were both unsuccesfull.
I entered them in grub like this:
GRUB_CMDLINE_LINUX_DEFAULT="intel_idle.max_cstate=1"
And
GRUB_CMDLINE_LINUX_DEFAULT="processor.max_cstate=1 intel_idle.max_cstate=0"
Is this the correct procedure to enter those commands?

The system is still able to boot (most of the time) with either of the command, no change in systemstability at all.
Upgraded to kernel 5.10 again, still no change at all.

[SMG]

Is this the correct procedure to enter those commands?

Yes, that is the correct placement. If you edited grub to add then, then you would run sudo update-grub after saving the file for them to take effect.

If you just edited grub while booting then they would only be used on that specific boot.

What is odd about those is the second is basically a reversal of the first (intel_idle.max_cstate value) which is why I wasn't too sure how they might affect the system.

The system is still able to boot (most of the time) with either of the command, no change in system stability at all.
Upgraded to kernel 5.10 again, still no change at all.

I did more searching, but did not find anything specific. It seems to me it is somehow related to hardware, but I could not find any reason why it would be fine on Windows yet so very unstable in Linux.

The only posts I've seen on the forum with that family of processors is with a problem getting the onboard Intel graphics to work. That requires uses the 5.10-oem kernel because that is the only kernel with the Intel patch in it. Otherwise one gets a black screen on boot.

[Ivo-G]

I guess I'll have to wait for a hardware and uefi specialist to show up that can tell if any of this is a problem

Some of the uefi settings:
https://drive.google.com/drive/folders/ ... sp=sharing

Anyway, thanks for all the help so far SMG

[Ivo-G]

You will NOT believe this! My system is playing a joke on me I think!
I'm typing this from within linux mint, and it has been up and running smooth for over 10 minutes without a problem...
BUT NOTHING CHANGED!

Even mounted all the drives, installed some apps, rebooted without a problem... I don't get it...

Don't get me wrong, i'm VERY happy! But... why now? Nothing changed...

EDIT: Some settings would not open, like power management, and some took very long to open, but most seemed to be fine. Eventually it did crash seemingly random again, but after almost a full hour this time. Now it's again so unstable I can't even get it booted up anymore.
This is so weird...