[RESOLVED] Kernel panic with 5.4.0-84.

[Landon]

HI, if this belongs over in the Boot' forum please move it there.

Linux Mint Mate 19.3.

Last night my PC had been running fine with absolutely no problems at all, Update Manager had wanted to update to Kernel 5.4.0-84 along with some other updates, I let it do its thing and then it wanted to reboot so I let it reboot and it did nothing but end up at a black screen.

I reset the PC and then it ended up at a Kernel Panic text crash error referring to 5.4.0-84, another reboot didn't help, powering off didn't help, and I finally ended up at the option to Normal Boot, or Recovery Boot using '84, or Recovery Boot using '81.

'84 just resulted in ending up back at the Kernel Panic error, I tried again using Recovery Boot '81 and in the end it fixed everything and eventually I was back at my Desktop again.

That would have been the end of it and I would have just considered it a fluke on my end of things but a Google search on my phone while I was trying to fix this brought up someone else that also had a Kernel Panic error with Kernel 5.4.0-84, which I don't know if it was ever resolved or not which just leads me to believe that there is a problem with '84 that needs to get looked at before it causes anyone else problems.

So... with all of that being said I'm not doing any more Kernel updates until I move on to version 20.xx and I will be using a different hard drive so that I can put this one back in if things go bad.

[SMG]

HI, if this belongs over in the Boot' forum please move it there.

Done.

Last night my PC had been running fine with absolutely no problems at all, Update Manager had wanted to update to Kernel 5.4.0-84 along with some other updates, I let it do its thing and then it wanted to reboot so I let it reboot and it did nothing but end up at a black screen.

The most recent 5.4 kernel is 5.4.0-89, so it is a bit perplexing as to why yours might have been asking to only update to the -84 version.

Kernel panics can be caused by many different things, so we would really need to know what the panic was in order to help you. It is possible that information is currently sitting in the journalctl and we might be able to access it.

If you would like to pursue investigating this, the following command lists the last 10 boots made by your computer.
journalctl --list-boots | tail -n 10
If you can tell which one is the one which had the kernel panic, we can get information about that specific boot to see what might have happened. Let us know if you think you know which one it is and we can provide a command to get that information.

[Landon]

Here is the info that it gave...

Code: Select all

landon@landon-desktop ~ $ journalctl --list-boots | tail -n 10
  -9 67e17c46bdf6457e894a4cf869c4f637 Sun 2021-08-08 08:20:03 PDT—Tue 2021-08-10 18:30:23 PDT
  -8 30a0096ab1804c4cb33802ba76d7f101 Tue 2021-08-10 20:16:27 PDT—Thu 2021-08-12 08:41:51 PDT
  -7 705db09cb7c743d5bec58adf89e8114f Thu 2021-08-12 10:59:20 PDT—Fri 2021-08-13 15:28:54 PDT
  -6 66bb0baf7f8043fcbd08b35ccaffe9b9 Sun 2021-08-15 19:27:38 PDT—Sun 2021-08-29 08:38:19 PDT
  -5 c04cd88a9e544136ad0df780bb8b51bb Sun 2021-08-29 08:39:34 PDT—Sun 2021-08-29 09:05:53 PDT
  -4 745a75721e7b487bb152d65a76a0d91a Sun 2021-08-29 09:11:41 PDT—Sun 2021-08-29 09:19:42 PDT
  -3 7ba0d2ece50a4f1e9c94ac5b178c579a Sun 2021-08-29 09:20:50 PDT—Sun 2021-08-29 15:31:22 PDT
  -2 359567f58965455f9b46d4c69c2a712e Sun 2021-08-29 15:52:41 PDT—Fri 2021-09-17 17:54:05 PDT
  -1 ff1b8934dd164e0c963d57cc81b68713 Sat 2021-09-18 17:31:46 PDT—Sun 2021-10-24 23:18:20 PDT
   0 a903f61565f4470c9df3fe1938a9e55b Sun 2021-10-24 23:36:46 PDT—Mon 2021-10-25 17:00:42 PDT
landon@landon-desktop ~ $ 

The " -1 ff1b8934dd164e0c963d57cc81b68713 Sat 2021-09-18 17:31:46 PDT—Sun 2021-10-24 23:18:20 PDT " is the reboot that occured last night.

I'm not sure what all of that hexadecimal information means, I'm sure that it means something to someone, I could probably convert it to ASCII but it probably still wouldn't mean anything.

One ongoing problem that I have had that may have contributed to this is that the "Boot" folder is limited to something like 200-ish megs and is always running out of room, there is really no way to fix that other that starting over with a fresh install, so as soon as I get it ordered tonight a new hard drive is on the way.

[SMG]

I'm not sure what all of that hexadecimal information means, I'm sure that it means something to someone, I could probably convert it to ASCII but it probably still wouldn't mean anything.

We just need the number at the very beginning which is -1

One ongoing problem that I have had that may have contributed to this is that the "Boot" folder is limited to something like 200-ish megs and is always running out of room, there is really no way to fix that other that starting over with a fresh install, so as soon as I get it ordered tonight a new hard drive is on the way.

How many kernels do you currently have installed? With that tiny of a boot folder, you would not want to keep any more than two or three.

Instructions to remove kernels:

• Open Update Manager.
• Select View > Linux Kernels and click Continue.
• Select a kernel branch on the left panel and then click on a kernel which is listed as Installed in the middle column on the right pane.
• A Remove button will appear. Click it to remove the kernel.
• Repeat that until you have no more than two or three total kernels installed (one will be Active and the other two Installed).

The terminal command to get the information about the boot which had a kernel panic is:
journalctl -b -1 | nc termbin.com 9999
It will return with a url address that you should post in your next reply.

That command prints the journal log information of the prior boot cycle (the one you said had a kernel panic) and sends it to termbin.

[Landon]

I'm not sure what all of that hexadecimal information means, I'm sure that it means something to someone, I could probably convert it to ASCII but it probably still wouldn't mean anything.

We just need the number at the very beginning which is -1

One ongoing problem that I have had that may have contributed to this is that the "Boot" folder is limited to something like 200-ish megs and is always running out of room, there is really no way to fix that other that starting over with a fresh install, so as soon as I get it ordered tonight a new hard drive is on the way.

How many kernels do you currently have installed? With that tiny of a boot folder, you would not want to keep any more than two or three.

Instructions to remove kernels:

• Open Update Manager.
• Select View > Linux Kernels and click Continue.
• Select a kernel branch on the left panel and then click on a kernel which is listed as Installed in the middle column on the right pane.
• A Remove button will appear. Click it to remove the kernel.
• Repeat that until you have no more than two or three total kernels installed (one will be Active and the other two Installed).

The terminal command to get the information about the boot which had a kernel panic is:
journalctl -b -1 | nc termbin.com 9999
It will return with a url address that you should post in your next reply.

That command prints the journal log information of the prior boot cycle (the one you said had a kernel panic) and sends it to termbin.

I currently have three Kernels installed...

--------------------------
5.4.0-87 Installed
5.4.0-84 Installed
5.4.0-81 Active
--------------------------

Here is that link that the journalctl -b -1 | nc termbin.com 9999 command generated:

https://termbin.com/er4x

There is a lot of data in there, is there something specific in that to look for?

EDIT

It may have been the -2 boot, I checked out that data dump and it has a ton of errors in it.

https://termbin.com/wz9d

[SMG]

There is a lot of data in there, is there something specific in that to look for?

The idea was to look for the kernel panic that I thought happened after you rebooted the computer. However, the data in this listing is for September 18th. There's no kernel panic there.

Looking back at your listing, it appears you booted the computer on August 29th, September 19, and then October 24th.

Code: Select all

  -2 359567f58965455f9b46d4c69c2a712e Sun 2021-08-29 15:52:41 PDT—Fri 2021-09-17 17:54:05 PDT
  -1 ff1b8934dd164e0c963d57cc81b68713 Sat 2021-09-18 17:31:46 PDT—Sun 2021-10-24 23:18:20 PDT
   0 a903f61565f4470c9df3fe1938a9e55b Sun 2021-10-24 23:36:46 PDT—Mon 2021-10-25 17:00:42 PDT

It may have been the -2 boot, I checked out that data dump and it has a ton of errors in it.

No kernel panic there either.

Last night my PC had been running fine with absolutely no problems at all, Update Manager had wanted to update to Kernel 5.4.0-84 along with some other updates, I let it do its thing and then it wanted to reboot so I let it reboot and it did nothing but end up at a black screen.

I reset the PC and then it ended up at a Kernel Panic text crash error referring to 5.4.0-84, another reboot didn't help, powering off didn't help, and I finally ended up at the option to Normal Boot, or Recovery Boot using '84, or Recovery Boot using '81.

What do you mean when you said "reset the PC"? What steps did you take?

[Landon]

There is a lot of data in there, is there something specific in that to look for?

The idea was to look for the kernel panic that I thought happened after you rebooted the computer. However, the data in this listing is for September 18th. There's no kernel panic there.

Looking back at your listing, it appears you booted the computer on August 29th, September 19, and then October 24th.

Code: Select all

  -2 359567f58965455f9b46d4c69c2a712e Sun 2021-08-29 15:52:41 PDT—Fri 2021-09-17 17:54:05 PDT
  -1 ff1b8934dd164e0c963d57cc81b68713 Sat 2021-09-18 17:31:46 PDT—Sun 2021-10-24 23:18:20 PDT
   0 a903f61565f4470c9df3fe1938a9e55b Sun 2021-10-24 23:36:46 PDT—Mon 2021-10-25 17:00:42 PDT

It may have been the -2 boot, I checked out that data dump and it has a ton of errors in it.

No kernel panic there either.

Last night my PC had been running fine with absolutely no problems at all, Update Manager had wanted to update to Kernel 5.4.0-84 along with some other updates, I let it do its thing and then it wanted to reboot so I let it reboot and it did nothing but end up at a black screen.

I reset the PC and then it ended up at a Kernel Panic text crash error referring to 5.4.0-84, another reboot didn't help, powering off didn't help, and I finally ended up at the option to Normal Boot, or Recovery Boot using '84, or Recovery Boot using '81.

What do you mean when you said "reset the PC"? What steps did you take?

Well... I literally reached over and pressed the [RESET] button on the PC, when that just ended up in the same place I reached over and held down the power button until the PC powered off.

I just went through 0, -1, -2, -3. and -4. and all I can find is logs that reference 5.4.0-81, so I don't know where the log for '84 ended up, maybe hitting the reset button killed it before it could log anything, idk at this point.

But here is a picture that I took of my screen last night, sorry that it is bit blurry but you can clearly see the '84 error.

[SMG]

I currently have three Kernels installed...
--------------------------
5.4.0-87 Installed
5.4.0-84 Installed
5.4.0-81 Active
--------------------------

Linux Mint is set up to automatically boot to the highest number kernel which means it should have been booting to 5.4.0-87.

Instead, it attempted to boot to 5.4.0-84. It appears to me it failed before the boot was able to happen which is probably why it does not show up in any of the boot logs. It failed before it reached the point of being able to be logged. In those types of cases, photographs are the only way to capture the information.

In the photograph you provided is the message Kernel Panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0).

One of the many responses to that thread is one which states, "In my situation the problem was that /boot was at 100% capacity, so the last 2 kernel updates had not completed successfully, hence on reboot when GRUB2 selected the latest Kernel, it failed."

You have a very, very small boot folder. Given that the most recent 5.4 kernel is not even on the list you provided, I would recommend going into Update Manger and removing both the 5.4.0-87 and the 5.4.0-84 kernels. Also make sure you do not have any 5.3, 5.0, or 4.15 kernels installed on your system. Clear out as much space as possible in the boot folder.

Then install the 5.4.0-89 kernel (the most recent one available for the 5.4 series) which should show in your Update Manager. Watch to make sure there are no errors listed during the install process. If all looks good, then restart your computer to update the kernel.

[Landon]

I currently have three Kernels installed...
--------------------------
5.4.0-87 Installed
5.4.0-84 Installed
5.4.0-81 Active
--------------------------

Linux Mint is set up to automatically boot to the highest number kernel which means it should have been booting to 5.4.0-87.

Instead, it attempted to boot to 5.4.0-84. It appears to me it failed before the boot was able to happen which is probably why it does not show up in any of the boot logs. It failed before it reached the point of being able to be logged. In those types of cases, photographs are the only way to capture the information.

In the photograph you provided is the message Kernel Panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0).

One of the many responses to that thread is one which states, "In my situation the problem was that /boot was at 100% capacity, so the last 2 kernel updates had not completed successfully, hence on reboot when GRUB2 selected the latest Kernel, it failed."

You have a very, very small boot folder. Given that the most recent 5.4 kernel is not even on the list you provided, I would recommend going into Update Manger and removing both the 5.4.0-87 and the 5.4.0-84 kernels. Also make sure you do not have any 5.3, 5.0, or 4.15 kernels installed on your system. Clear out as much space as possible in the boot folder.

Then install the 5.4.0-89 kernel (the most recent one available for the 5.4 series) which should show in your Update Manager. Watch to make sure there are no errors listed during the install process. If all looks good, then restart your computer to update the kernel.

5.4.0-89 was waiting in the Update Manager and I let it install but didn't let anything else and then rebooted, the reboot went fine there were no errors or crashes.

"Linux Kernels" is now showing:
==============
5.4.0-89 Active
5.4.0-81 Installed
==============

There are no other installed Kernels in sections 4.15; 4.18; 5.0; 5.3; or 5.4.

Oddly though there is a "linux-4.15.0-161.169 Linux Kernel Headers for development" waiting to be installed in "Update Manager", I haven't had a 4.15 Kernel on here in a while.

Regarding the BOOT folder... GParted is reporting it as a 243 meg partition with about 62 megs of free space, the current 5.4.0-89 download that just came through was about 86 megs and I am assuming that the next Kernel update will be even larger so I will be out of space yet again.

But... my new hard drive is coming on Friday which will be a fresh install of 20.x which will hopefully resolve this BOOT folder thing once and for all and considering that everything goes well the current hard drive will be retired from service after what I believe is probably 12 years.

[SMG]

5.4.0-89 was waiting in the Update Manager and I let it install but didn't let anything else and then rebooted, the reboot went fine there were no errors or crashes.

That is good to hear.

Oddly though there is a "linux-4.15.0-161.169 Linux Kernel Headers for development" waiting to be installed in "Update Manager", I haven't had a 4.15 Kernel on here in a while.

I found this What is 'linux-libc-dev'? "Linux kernel headers for development" for another kernel series in software updater which seems to indicate it is related to something else in the system? I'm not completely sure what it does.

Regarding the BOOT folder... GParted is reporting it as a 243 meg partition with about 62 megs of free space, the current 5.4.0-89 download that just came through was about 86 megs and I am assuming that the next Kernel update will be even larger so I will be out of space yet again.

You will have to remove the older kernel before downloading a new one to make sure you do not run into the same issue you did this time.

We do not recommend making a separate boot folder when installing Mint. Most people just let it be part of their root partition. Both the kernel and the operating system itself are larger than what they were in the past, so using past recommendations can cause issues.

For LM20, if you decide to have separate root and home partitions (instead of one large root partition), we recommend root be 30-50GB in size.

It sound like this is issue is resolved. Please go to your first post in this topic, click the pencil icon, and add [SOLVED] to the title so others know you are no longer seeking help on this issue.

[Landon]

The BOOT folder I think goes clear back to like version 13 or something which was probably adequate for its time but not now, which is why I'm going for the fresh install this time.

Thank you for the help and advice I am going to consider this resolved.

[SMG]

The BOOT folder I think goes clear back to like version 13 or something which was probably adequate for its time but not now, which is why I'm going for the fresh install this time.

Yes, sizes have definitely changed since then if you have been upgrading in place. We have had people post here who did a fresh install of LM20, but used old sizing criteria which is why I wanted you to be aware of the newer recommendations before you did your fresh install.

Just for future reference (since you have the new drive already on the way and will soon be switching to it): If one has space on a drive, it is possible to increase partition size depending upon where the free space is located in relation to the partition which needs increased. The drive has to be unmounted (so one usually makes the change using a live usb session of Mint) and one needs to be careful, but it is possible.