Random crashes - how to find what's causing it?

[peter269]

I'm getting occasional random crashes, I've been trying to pinpoint a cause and I've run out of ideas. Can anyone suggest how to pin it down further please?

I installed Linux Mint in December 2020 on an old laptop, and it's crashed 5 or 6 times since then. Screen just goes blank, laptop leds go off, there is a click which I think is the disc parking, then nothing. Boots up OK after crash.

inxi report:

Code: Select all

System:    Kernel: 5.4.0-65-generic x86_64 bits: 64 compiler: gcc v: 9.3.0 Desktop: Cinnamon 4.8.6 
           wm: muffin dm: LightDM Distro: Linux Mint 20.1 Ulyssa base: Ubuntu 20.04 focal 
Machine:   Type: Laptop System: ASUSTeK product: K54C v: 1.0 serial: <filter> 
           Mobo: ASUSTeK model: K54C v: 1.0 serial: <filter> BIOS: American Megatrends v: K54C.202 
           date: 11/03/2011 
Battery:   ID-1: BAT0 charge: 23.6 Wh condition: 26.0/37.8 Wh (69%) volts: 16.5/14.4 
           model: ASUSTek K53--27 serial: <filter> status: Charging 
CPU:       Topology: Dual Core model: Intel Celeron B815 bits: 64 type: MCP arch: Sandy Bridge 
           rev: 7 L2 cache: 2048 KiB 
           flags: lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx bogomips: 6385 
           Speed: 1441 MHz min/max: 800/1600 MHz Core speeds (MHz): 1: 1578 2: 1252 
Graphics:  Device-1: Intel 2nd Generation Core Processor Family Integrated Graphics 
           vendor: ASUSTeK driver: i915 v: kernel bus ID: 00:02.0 chip ID: 8086:0106 
           Display: x11 server: X.Org 1.20.9 driver: modesetting unloaded: fbdev,vesa 
           resolution: 1366x768~60Hz 
           OpenGL: renderer: Mesa DRI Intel HD Graphics 2000 (SNB GT1) v: 3.3 Mesa 20.2.6 
           compat-v: 3.0 direct render: Yes 
Audio:     Device-1: Intel 6 Series/C200 Series Family High Definition Audio vendor: ASUSTeK 
           driver: snd_hda_intel v: kernel bus ID: 00:1b.0 chip ID: 8086:1c20 
           Sound Server: ALSA v: k5.4.0-65-generic 
Network:   Device-1: Qualcomm Atheros AR9285 Wireless Network Adapter vendor: Foxconn 
           driver: ath9k v: kernel port: e040 bus ID: 02:00.0 chip ID: 168c:002b 
           IF: wlp2s0 state: up mac: <filter> 
           Device-2: Qualcomm Atheros AR8151 v2.0 Gigabit Ethernet vendor: ASUSTeK driver: atl1c 
           v: 1.0.1.1-NAPI port: a000 bus ID: 04:00.0 chip ID: 1969:1083 
           IF: enp4s0 state: down mac: <filter> 
Drives:    Local Storage: total: 298.09 GiB used: 32.02 GiB (10.7%) 
           ID-1: /dev/sda vendor: Seagate model: ST320LT020-9YG142 size: 298.09 GiB 
           speed: 3.0 Gb/s serial: <filter> 
Partition: ID-1: / size: 291.92 GiB used: 32.02 GiB (11.0%) fs: ext4 dev: /dev/sda5 
USB:       Hub: 1-0:1 info: Full speed (or root) Hub ports: 2 rev: 2.0 chip ID: 1d6b:0002 
           Hub: 1-1:2 info: Intel Integrated Rate Matching Hub ports: 6 rev: 2.0 
           chip ID: 8087:0024 
           Device-1: 1-1.2:3 info: Alcor Micro Asus Integrated Webcam type: Video driver: uvcvideo 
           rev: 2.0 chip ID: 058f:a014 
           Hub: 2-0:1 info: Full speed (or root) Hub ports: 2 rev: 2.0 chip ID: 1d6b:0002 
           Hub: 2-1:2 info: Intel Integrated Rate Matching Hub ports: 6 rev: 2.0 
           chip ID: 8087:0024 
           Hub: 3-0:1 info: Full speed (or root) Hub ports: 2 rev: 2.0 chip ID: 1d6b:0002 
           Hub: 4-0:1 info: Full speed (or root) Hub ports: 2 rev: 3.0 chip ID: 1d6b:0003 
Sensors:   System Temperatures: cpu: 57.0 C mobo: N/A 
           Fan Speeds (RPM): cpu: 2000 
Repos:     No active apt repos in: /etc/apt/sources.list 
           Active apt repos in: /etc/apt/sources.list.d/google-earth-pro.list 
           1: deb [arch=amd64] http: //dl.google.com/linux/earth/deb/ stable main
           Active apt repos in: /etc/apt/sources.list.d/official-package-repositories.list 
           1: deb http: //www.mirrorservice.org/sites/packages.linuxmint.com/packages ulyssa main upstream import backport
           2: deb http: //archive.ubuntu.com/ubuntu focal main restricted universe multiverse
           3: deb http: //archive.ubuntu.com/ubuntu focal-updates main restricted universe multiverse
           4: deb http: //archive.ubuntu.com/ubuntu focal-backports main restricted universe multiverse
           5: deb http: //security.ubuntu.com/ubuntu/ focal-security main restricted universe multiverse
           6: deb http: //archive.canonical.com/ubuntu/ focal partner
Info:      Processes: 189 Uptime: 15h 06m Memory: 3.75 GiB used: 2.08 GiB (55.5%) Init: systemd 
           v: 245 runlevel: 5 Compilers: gcc: 9.3.0 alt: 9 Client: Unknown python3.8 client 
           inxi: 3.0.38 

I know it's an old laptop, it was very slow under Windows but at least it never crashed. It now runs Linux Mint fine except for the occasional crashes.

What I've done so far-
Eliminated low battery - last 2 crashes battery level was over 80%
Eliminated overheating - last time it had only been running a few minutes before crashing, and temperatures were around 50'C when I looked at sensors

Here's an extract from syslog at the time of the last crash-

Feb 12 20:28:04 peter-K54C systemd[1]: NetworkManager-dispatcher.service: Succeeded.
Feb 12 20:29:46 peter-K54C kernel: [63847.957051] [UFW BLOCK] IN=wlp2s0 OUT= MAC=01:00:5e:00:00:01:10:d7:b0:05:ec:82:08:00 SRC=192.168.1.254 DST=224.0.0.1 LEN=36 TOS=0x00 PREC=0xC0 TTL=1 ID=0 DF PROTO=2
Feb 12 20:30:02 peter-K54C CRON[48710]: (root) CMD ([ -x /etc/init.d/anacron ] && if [ ! -d /run/systemd/system ]; then /usr/sbin/invoke-rc.d anacron start >/dev/null; fi)
\00\00\00\00\00\00\00\00\00\00\... (loads more like this) .... 0\00\00\00\00\00\00\00Feb 12 20:38:45 peter-K54C kernel: [ 0.000000] microcode: microcode updated early to revision 0x2f, date = 2019-02-17
Feb 12 20:38:45 peter-K54C kernel: [ 0.000000] Linux version 5.4.0-65-generic (buildd@lcy01-amd64-018) (gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)) #73-Ubuntu SMP Mon Jan 18 17:25:17 UTC 2021 (Ubuntu 5.4.0-65.73-generic 5.4.78)

20:28 - this is the last of the commands from previous resume
20:29 - UFW. This is normal, I get them every 2 minutes
20:30 - CRON. I don't know what this is doing, but I get one or two lines like this shortly after each resume, so I assume it's normal.
At about 20:32 system crashed, and wrote a whole load of corrupted characters to syslog.
20:38 - start of the reboot

I found the same corruption in ufwlog, which makes me think that maybe UFW was trying to log a firewall block at the instant of the crash. Could it be UFW that's crashing?
Here's an extract from ufwlog-

Feb 12 20:29:46 peter-K54C kernel: [63847.957051] [UFW BLOCK] IN=wlp2s0 OUT= MAC=01:00:5e:00:00:01:10:d7:b0:05:ec:82:08:00 SRC=192.168.1.254 DST=224.0.0.1 LEN=36 TOS=0x00 PREC=0xC0 TTL=1 ID=0 DF PROTO=2
\00\00\00\00\00\00\00\00\00\00\00\00\00\00\ ... etc

Where else could I look please? Are there other logs that might show what caused the crash?
Thanks

[Termy]

When did you first notice this happening?

You might also want to check dmesg --human for the kernel ring buffer. Another thing you can try is journalctl for the SystemD Journal. Lastly, you can check APT's log with less /var/log/apt/history.log for around the data/time you noticed this happening, to see if it coincides with any package changes, such as for kernel upgrades.

Do you see any errors at boot-time? If unsure, check: cat /var/log/boot.log

Now let's talk hardware:

* Do you notice any weird smells?
* DO you hear any other abnormal sounds?
* Is there any screen corruption?
* Has the laptop had anything spilled on it, lately?
* Has the laptop been knocked or dropped, lately?
* Has processing performance suffered, lately?
* Have you been getting regular errors, lately?

Where lately is before you noticed this problem.

[peter269]

Thanks for the suggestions Termy. Here's some more information.

When did it start? - I don't remember exactly but the first time was a few days after I installed Linux mint 20.0 back in December. I didn't do anything about it at the time because I was on a steep learning curve, and I thought perhaps it would go away with updates (or I would just abandon Linux and buy a new laptop). It has crashed several times since and I've now started to keep a log. The last two times were 3rd and 12th Feb. Both times fairly soon after I had 'resumed", and both times doing something innocuous like checking emails and watching a Youtube video.

I have updated numerous times since December, including kernel updates and migrating to 20.1, all without problems. There were no updates happening at the time of the crashes, I don't allow automatic updates.

I looked at the logs you suggested - dmesg only records since the last reboot so wasn't very helpful. I did find /var/log/kernlog and that shows exactly the same corruption but no more information.
I also found the systemd journal. This contained one extra line of CRON and no corruption. The last entry was at 20:30 but I think it was about 20:32 when it crashed. Does the "session closed" CRON record mean anything? Here's an extract

Code: Select all

Feb 12 20:28:04 peter-K54C systemd[1]: NetworkManager-dispatcher.service: Succeeded.
Feb 12 20:29:46 peter-K54C kernel: [UFW BLOCK] IN=wlp2s0 OUT= MAC=01:00:5e:00:00:01:10:d7:b0:05:ec:82:08:00 SRC=192.168.1.>
Feb 12 20:30:02 peter-K54C CRON[48709]: pam_unix(cron:session): session opened for user root by (uid=0)
Feb 12 20:30:02 peter-K54C CRON[48710]: (root) CMD ([ -x /etc/init.d/anacron ] && if [ ! -d /run/systemd/system ]; then /u>
Feb 12 20:30:02 peter-K54C CRON[48709]: pam_unix(cron:session): session closed for user root
-- Reboot --
Feb 12 20:38:30 peter-K54C kernel: microcode: microcode updated early to revision 0x2f, date = 2019-02-17

I tried looking at bootlog, but got "permission denied"

Finally, I too have wondered about a hardware problem, though it's strange that it started almost straight after installing Linux. I'm not aware of any damage that happened in December, and as I said I've had no crashes on Windows prior. No smells, loss of performance, or any other symptoms.

I've got memtest86 - I'll try running it overnight tonight. I don't know if there's any form of non-destructive hard disc test I could do?

[SMG]

I have an HP laptop with the same Celeron B815 cpu, 4GB of memory, and Cinnamon. However, I have all the options in the Effects app turned off.

I turned them off because I do not like windows animations, but I've been told numerous times since then that is probably why I do not have issues with my install given the specs of the computer. Turning the Effects off lightens the load on the hardware. Maybe doing that will help your situation.

[Termy]

Fancy a project? Wanna learn some cool stuff? Know how to load from an earlier kernel version? Have a backup in-case things don't work out?

If you answered yes to those questions, then you might want to give glkfu(1) a go. The reason I suggest it is because you can use it to install all sorts of different kernel versions, both old and up to and including the very latest stable kernel straight from kernel.org. You can use glkfu -l to list all of the available kernel versions, and you might find this web page interesting.

Everything you said leads me to believe it's probably not a hardware problem but could be something related to hardware support. A lot of people running 5.4 and up who're running on old hardware are having issues.

As an added bonus, using glkfu(1) to get your kernels will allow you the chance to drastically lighten them and offer chances to improve system performance. The whole thing is interactive. You'll want plenty of space for compilation, and note that you will be the maintainer of your kernel versions, so it falls to you to ensure you're updated. I do this myself to get the latest kernels for my Ubuntu-based system.

Beyond that, I'm afraid I'm not sure. Hopefully someone else can chime in with experience with your problems in Linux using an older machine.

[ThaCrip]

Since no one mentioned it, I would try 5.8 kernel and see if that fairs any better and it's a easy enough thing to try...

Update Manager > View > Linux Kernels.

[Moonstone Man]

... I would try 5.8 kernel ...

Perhaps you would, but the OP shouldn't because the machine is at least 10 years old.

[Moonstone Man]

When did it start? - I don't remember exactly but the first time was a few days after I installed Linux mint 20.0 back in December.

That apparent relationship might be sheer synchronicity.

Your machine is at least 10 years old so you need to be prepared for an unfixable hardware issue. Also, the crashes might seem random but if you've got dying components then they won't be random at all. They will occur if the machine is under load or getting hot, or just moving the machine slightly, but only your usage patterns will reveal the true culprit.

I think you should check for these:

[ThaCrip]

... I would try 5.8 kernel ...

Perhaps you would, but the OP shouldn't because the machine is at least 10 years old.

I don't think there is any specific reason why he can't try it, right?

I had my basic motherboard since May 2012 and it's probably released in about 2011. so it's tech is about 10 years old and my system actually works better on 5.8 than 5.4 (I had a few bugs that 5.8 solved).

I figure worst case, the OP can simply revert back to 5.4 if 5.8 does not work. it would seem the OP does not have much to lose by trying the 5.8 kernel but maybe a bit of their time. I suggested it since it's a simple enough thing to try.

[Moonstone Man]

I don't think there is any specific reason why he can't try it, right?

There is absolutely no evidence that it's a kernel issue. That is one technical reason. Technical support isn't done by making random suggestions; it's done by problem solving based on evidence. If it turns out to be an actual kernel issue then it was sheer pot luck that you were right.

[ThaCrip]

I don't think there is any specific reason why he can't try it, right?

There is absolutely no evidence that it's a kernel issue. That is one technical reason. Technical support isn't done by making random suggestions; it's done by problem solving based on evidence. If it turns out to be an actual kernel issue then it was sheer pot luck that you were right.

But can't their be stability issues on one kernel and not on another? ; if so, it don't seem like a stretch for my suggestion to be worth a shot. it might not help at all, but it's a easy enough thing to try. plus, he was saying it was fine on Windows but not on Linux.

p.s. the issues I had were not random system crashes but stuff like HDMI audio disappearing/Trash can temporarily working and then not etc.

[Moonstone Man]

But can't their be stability issues on one kernel and not on another? ...

Either you have not read the thread or you have not understood it. In particular see @Termy's last response specifically about kernels.

I'm out. This is just derailing the thread.

[ThaCrip]

But can't their be stability issues on one kernel and not on another? ...

Either you have not read the thread or you have not understood it. In particular see @Termy's last response specifically about kernels.

I'm out. This is just derailing the thread.

Yes, but unless I am mistaken that's using kernels that the Mint team did not directly approve of like how it is with 5.4 and 5.8 currently. hence, more risky, correct?

so while one could try it, 5.8 is probably the better initial choice in terms of kernels since the Mint team officially allows it through the Update Manager.

[Termy]

@OP, alternatively, and I apologise for not mentioning this sooner, you could try installing an older version of Linux Mint which is still being supported. By version I don't mean a point-release (such as the difference between 18.1 and 18.2). An older version of Linux Mint, it being based on Ubuntu, will offer repositories which contain older kernels, thus bypassing the whole glkfu(1) thing.

IIRC, Ubuntu 20.04's repositories (on which the latest Mint is based) frustratingly don't include kernels older than 5.4.

There is absolutely no evidence that it's a kernel issue.

I disagree.

It being a hardware issue (that is, a fault in the hardware), while not ruled out, doesn't appear as likely. One of the most common problems people seem to be having with symptoms like these, at least from what I've observed on this message board, seems to be related to the kernel. These same people who're running on either too old or too new a hardware on a distribution of Linux with either too new or too old a kernel, respectively.

Remember, the kernel is the layer between the hardware and the software; it translates hardware components into something the software can understand. It's not a stab in the dark, I'd argue, but an educated guess based on the limited amount of evidence laid out to us. At least in my case, I suggest that it could be the kernel, and to gain evidence to the contrary, would be as easy as installing alternative kernels aligning the age of the hardware.

That being said, I wouldn't myself bother installing 5.8, because it's newer than the kernel with which the OP is potentially having issues, so in that regard, and considering the age of the machine, I do agree with you.

[peter269]

Thanks for the interest and all the suggestions, I think the Linux community is wonderful!
Here's a progress update:

Overheating - no. The laptop had only been resumed for a few minutes when it crashed, and I checked the temps with 'sensors' straight after I rebooted.

Vibration/moving - no. I've tried some fairly vigorous tap testing, including the battery, without getting it to fail.

Memory - ran disctest86 for 10 hours last night, zero errors.

Effects - thanks for that suggestion, I had no idea that they existed! I've now turned them all off (can hardly tell the difference), though I don't think it is the problem since I was not opening or closing any windows or dialogues last time it crashed, just watching Youtube. The laptop runs well between crashes with no obvious speed problems or glitches.

Disc test. Since my last post, I've now found the system "disk" app, and spent a bit of time learning how to interpret the SMART disc stats. I've also run an extended test. For the five "Pre-fail" (ie.condition-based) tests, the normalised results are as follows (100 is "normal", >100 is better, <100 is poorer) -
*Read Error rate: 119, worst 99, threshold 6 (119 = better than normal)
*Spin-up time: 99, worst 99, threshold 85
*Reallocated sector count: 0 sectors
*Seek error rate: 83, worst 60, threshold 30 (= a bit low)
*Spin-up retry count: 0 retries
There is one "old age" parameter that could be a bit concerning :
*Hardware ECC recovered: 47, worst 41, threshold 0

Given that the disc is about 10 years old, I think these results are pretty healthy. All attributes are shown as "OK", with the exception of air-flow temperature which shows one failure in the past (could be any time in the last 10 years I think).

I've now installed kernel 5.8. Having looked at the difference between 5.4 and 5.8, I doubt if it will make a difference, but I've got nothing to lose. If that fails then I might start looking at earlier kernels but I don't want to lose some of the latest functionality like Warpinator. I might just live with the risk (or I could even treat myself to a newer laptop).

[ThaCrip]

That being said, I wouldn't myself bother installing 5.8, because it's newer than the kernel with which the OP is potentially having issues, so in that regard, and considering the age of the machine, I do agree with you.

I am no expert but I heard other people (one who's a 'senior administrator' on another tech forum) claim 5.4 is generally a more problematic kernel and that something like 5.8 etc might be a better option for many. hell, even Linus Torvalds appeared to like 5.8 as he mentioned, "So I didn't really expect this, but 5.8 looks to be one of our biggest releases of all time."

so given that info, I won't be surprised if 5.8 works better where 5.4 fails. plus, like I mentioned before 5.8 definitely works better on my older system than 5.4 does as just changing from 5.4 to the 5.8 kernel it solved a few bugs I was having.

but thanks for your comment.

p.s. but it appears the OP is currently testing 5.8. so we shall find out soon enough if it's any better or not. either way, I think the 5.8 kernel was worth a shot as it's not much effort to try it.

[ricardogroetaers]

I'm getting occasional random crashes, I've been trying to pinpoint a cause and I've run out of ideas. Can anyone suggest how to pin it down further please?

Please, the command output was missing:
sudo inxi -mxxx

I suppose a "clean" installation of Linux Mint Xfce 64 bits would be better.

[Termy]

...

Good points. Whether it solves the problems or not, I agree that it was worth a try. I've heard some negative things about 5.4 as well.

[peter269]

I think it's time I gave an update on this problem.
Since I last posted (Feb), I've had about 10 further crashes, and moving to kernel 5.8 from 5.4 made no difference.
The only pattern I can find in the crashes is that they all happen between 5 and 10 minutes after booting.

When I boot from cold, I've noticed that the fan doesn't start straight away, but starts up when the temperature gets to about 50deg, which normally takes about 5 to 10 minutes from cold. I don't think that's coincidence!

So my conclusion is that it's the fan trying to start up that occasionally causes a crash. Maybe the surge in current causes a voltage dip, and/or a glitch in the processor or memory, or maybe there's some sort of firmware problem. I don't think it will be software, though I still can't explain why it never happened in the 10 years I was running Windows, and started almost immediately after I installed Linux.

One last question before I mark this as solved - should I stick with 5.8 kernel, or should I go back to 5.4?
5.4 is supported until Dec 2025, but according to Wikipedia 5.8 was only supported to Nov 2020 (though I'm getting updates for both). What happens when 5.8 reaches end of support - does it somehow supersede 5.4 and become LTS?

Thanks in advance

[Larry78723]

Are the crashes happening only when running on battery, not plugged in? If so, you might need to replace your battery.
From your inxi: Battery: ID-1: BAT0 charge: 23.6 Wh condition: 26.0/37.8 Wh (69%) volts: 16.5/14.4