[Solved] Random Kernel Panic on boot - not syncing

[ugly]

One of my PCs fails to boot occasionally and I get a kernel panic. I have not really found a pattern. Most of the time it works. It seems like it happens more often when I boot when the monitor is off, but this seems like an unlikely cause.

I have noticed some messages showing up on shutdown, but I have not been able to catch them.

This is what I get for the kernel panic:

Code: Select all

[     0.356846]  Initramfs unpacking failed: junk in compressed archive
[     0.986721]  Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[     0.986804]  CPU: 3 PID: 1 Comm: swapper/0 Not tainted 4.13.0-26-generic #29~16.04.2-Ubuntu
[     0.986884]  Hardware name: ZOTAC ZBOX-CI527/CI547/ZBOX-CI527/CI547, BIOS B331P205 03/15/2017
[     0.986964]  Call Trace:
[     0.987032]   dump_stack+0x63/0x8b
[     0.987099]   panic+0xe4/0x23d
[     0.987167]   mount_block_root+0x1fd/0x2ac
[     0.987235]   mount_root+0x38/0x3a
[     0.987302]   prepare_namespace+0x13f/0x194
[     0.987370]   kernel_init_freeable+0x20e/0x237
[     0.987440]   ? rest_init+0xc0/0xc0
[     0.987507]   kernel_init+0xe/0xfc
[     0.987573]   ret_from_fork+0x1f/0x30
[     0.987769]  Kernel Offset: 0x2ac00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

And this is the result of inxi -FxGz:

Code: Select all

System:    Host: media-ZBOX Kernel: 4.13.0-26-generic x86_64 (64 bit gcc: 5.4.0)
           Desktop: Cinnamon 3.6.7 (Gtk 2.24.30)
           Distro: Linux Mint 18.3 Sylvia
Machine:   Mobo: ZOTAC model: ZBOX-CI527/CI547 v: XX
           Bios: American Megatrends v: B331P205 date: 03/15/2017
CPU:       Dual core Intel Core i3-7100U (-HT-MCP-) cache: 3072 KB
           flags: (lm nx sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx) bmips: 9600
           clock speeds: max: 2400 MHz 1: 2400 MHz 2: 2400 MHz 3: 2400 MHz
           4: 2400 MHz
Graphics:  Card: Intel Device 5916 bus-ID: 00:02.0
           Display Server: X.Org 1.18.4 driver: intel
           Resolution: 1920x1080@60.00hz
           GLX Renderer: Mesa DRI Intel HD Graphics 620 (Kaby Lake GT2)
           GLX Version: 3.0 Mesa 17.4.0-devel - padoka PPA Direct Rendering: Yes
Audio:     Card Intel Device 9d71 driver: snd_hda_intel bus-ID: 00:1f.3
           Sound: Advanced Linux Sound Architecture v: k4.13.0-26-generic
Network:   Card-1: Intel Ethernet Connection I219-LM
           driver: e1000e v: 3.2.6-k bus-ID: 00:1f.6
           IF: enp0s31f6 state: down mac: <filter>
           Card-2: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
           driver: r8169 v: 2.3LK-NAPI port: e000 bus-ID: 01:00.0
           IF: enp1s0 state: up speed: 1000 Mbps duplex: full mac: <filter>
           Card-3: Intel Wireless 3165 driver: iwlwifi bus-ID: 02:00.0
           IF: wlp2s0 state: down mac: <filter>
Drives:    HDD Total Size: 480.1GB (8.2% used)
           ID-1: /dev/sda model: SanDisk_SDSSDA48 size: 480.1GB
Partition: ID-1: / size: 46G used: 13G (29%) fs: ext4 dev: /dev/sda1
           ID-2: /home size: 387G used: 18G (5%) fs: ext4 dev: /dev/sda6
           ID-3: swap-1 size: 8.00GB used: 0.00GB (0%) fs: swap dev: /dev/sda5
RAID:      No RAID devices: /proc/mdstat, md_mod kernel module present
Sensors:   System Temperatures: cpu: 51.5C mobo: 29.8C
           Fan Speeds (in rpm): cpu: N/A
Info:      Processes: 205 Uptime: 2 min Memory: 809.9/7900.8MB
           Init: systemd runlevel: 5 Gcc sys: 5.4.0
           Client: Shell (bash 4.3.481) inxi: 2.2.35 

I've seen some suggestions about /boot being at 100%, but I'm not seeing anything on my PC about a /boot partition.

This has happened with previous kernels in the 4.10 and 4.11 line. I just recently updated to 4.13, but it still happens.

Any help would be appreciated.

[jglen490]

A kernel usually happens when the bootloader can't find a kernel at the expected location. You don't have a dedicated /boot partition (that's a good thing) and your / partition is not even close to being dangerously full. You might check to see how many kernel files you have in /boot, but I'm thinking it may be that Grub2 needs some fixing.

I can't honestly say that I've ever had to fix my grub installation, but I do know there are some good grub rescue packages available. As to the number of kernels you might have, typically you shouldn't need more that your current kernel, plus one or two older kernels. You can check what kernels you have either via Update Manager or actually looking in the /boot directory.

Sometimes a kernel panic happens if a drive is failing, but that's probably not associated with random panics.

[ugly]

Thanks for the reply.

I only have two kernels. I'll look into some grub rescue tools.

The drive is a new SSD. It's only been used lightly for a few months. It has crossed my mind that I got a faulty SSD. Although, I can't say I've noticed any other weirdness.

[Mazda81]

I have the same issue with one of my machines, it seems to happen after updates are issued for initramfs-tools.
I use the following workaround [found on this forum] to get the machine to boot.

Boot using a live Linux DVD.
When the countdown starts press the down arrow to highlight the first entry "Start Linux Mint"
Press the Tab key & edit the kernel command line "boot=casper" with "root =/dev/sda1/
Press enter to continue to boot. [Note that sda1 is correct for my system but may be different for yours]

After the machine has booted up open the terminal and enter "sudo dpkg-reconfigure initramfs-tools" without the quotation marks.

This works for me so hopefully it should work for you.

[ugly]

I have the same issue with one of my machines, it seems to happen after updates are issued for initramfs-tools.
I use the following workaround [found on this forum] to get the machine to boot.

Boot using a live Linux DVD.
When the countdown starts press the down arrow to highlight the first entry "Start Linux Mint"
Press the Tab key & edit the kernel command line "boot=casper" with "root =/dev/sda1/
Press enter to continue to boot. [Note that sda1 is correct for my system but may be different for yours]

After the machine has booted up open the terminal and enter "sudo dpkg-reconfigure initramfs-tools" without the quotation marks.

This works for me so hopefully it should work for you.

Unfortunately, this did not work for me.

A kernel usually happens when the bootloader can't find a kernel at the expected location. You don't have a dedicated /boot partition (that's a good thing) and your / partition is not even close to being dangerously full. You might check to see how many kernel files you have in /boot, but I'm thinking it may be that Grub2 needs some fixing.

I can't honestly say that I've ever had to fix my grub installation, but I do know there are some good grub rescue packages available. As to the number of kernels you might have, typically you shouldn't need more that your current kernel, plus one or two older kernels. You can check what kernels you have either via Update Manager or actually looking in the /boot directory.

I also tried Rescatux to do a grub rescue. This also did not work.

I can't really see any pattern. Sometimes it boots fine. And once it boots, I seem to be able to reboot as often as I want. But when I actually shut down, especially if I leave it unpowered for a while, when I try to turn it on again, I'll get the kernel panic. And once I get the kernel panic, I'll often get it several times in a row. Then it will eventually start booting fine again.

I haven't really paid a lot of attention if particular kernel versions are worse. It seems like the 4.13 kernels are worse. But I assumed that the latest kernels would have better support for Kaby Lake.

[catweazel]

Code: Select all

[     0.356846]  Initramfs unpacking failed: junk in compressed archive
[/quote]
Are you dual booting with a UEFI install of Windwoes, and have you played around with Grub in any way?

[ugly]

Are you dual booting with a UEFI install of Windwoes, and have you played around with Grub in any way?

No dual boot. This is a new SSD that never had anything on it but Linux Mint (it's a bare-bones mini-PC and this is the only drive I have used).

In the BIOS, the Boot Mode is set to 'Legacy Only'.

I have made changes to Grub. This is what I currently have:

Code: Select all

# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_HIDDEN_TIMEOUT=0
GRUB_HIDDEN_TIMEOUT_QUIET=true
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet pci=noaer"
GRUB_CMDLINE_LINUX=""

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
GRUB_GFXMODE=1280x1024x24

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

I made the change GRUB_GFXMODE=1280x1024x24 after I had the kernel panics

I also changed to GRUB_CMDLINE_LINUX_DEFAULT="quiet pci=noaer" after getting the kernel panics in response to https://askubuntu.com/questions/771899/ ... -corrected. I noticed the PCIe Bus error severity=Corrected on shutdown, and thought it might be part of the kernel panic issue. But it has made no difference that I can tell.

[thx-1138]

...No guarantee the following will fix it, but it certainly wouldn't hurt...run:

Code: Select all

sudo tune2fs -c 1 /dev/sda1

This will run fsck on next boot. Then, after logging in:

Code: Select all

sudo update-initramfs -uk all

[ugly]

...No guarantee the following will fix it, but it certainly wouldn't hurt...run:

Code: Select all

sudo tune2fs -c 1 /dev/sda1

This will run fsck on next boot. Then, after logging in:

Code: Select all

sudo update-initramfs -uk all

Things seemed improved for a while after doing this, but eventually I started getting the kernel panics again.

[thx-1138]

...the fact that you don't have a /boot partition & at the same time you are using 'Legacy' instead of UEFI is what puzzles me - if it's of any help, check here:
http://blog.runtux.com/2011/01/10/186/
https://alien.slackbook.org/blog/initra ... d-archive/
So, maybe you've hit a limitation of BIOS? Also note that the articles refer to lilo, not grub2...
Maybe you should better make a /boot partition and install grub there?

[ugly]

Since I have a backup (and nothing particularly important on the PC anyway), I decided to re-install and keep things simple.

My BIOS has 3 options for 'Boot Mode' : Legacy, Pure UEFI and Win7 UEFI.

I decided to try 'Pure UEFI' and when installing, I selected the option to Erase disk and install Linux Mint 18.3. So nothing fancy.

So now I have:
sda2 at 438.7G for /
sda3 at 8G for [SWAP]
sda1 at 512M for /boot/efi

It installed and booted fine. I tried a couple reboots and everything seemed okay. I did all the initial updates (without installing any new software other than the mint codecs), including upgrading to the latest 4.13 kernel. It booted a few more times after the updates and seemed okay. Turned it off for a while, then tried to boot again and got another kernel panic.

This one is a little different:

Code: Select all

Failed to execute /init (error -2)
Kernel panic - not syncing: No working init found.  Try passing init=option to kernel.  See Linux Documentation/admin-guide/init.rst for guidance.
CPU: 2 PID: 1 Comm: swapper/0 Not tanited 4.13.0-32-generic #35~16.04.1-Ubuntu
Hardware name: ZOTAC ZBOX-CI527/CI547/ZBOX-CI527/CI547, BIOS B331P205 03/15/2017

After spewing a stream of obscenities, I re-installed again. Same thing Pure UEFI with Erase disk and install Linux Mint 18.3.

This time I installed only Level 1 and Level 2 updates. Made sure all level 3 and 4 updates are unchecked. So I am still on the 4.10.0-38-generic Kernel.

I've rebooted multiple times last night and today. So far no kernel panics.

Maybe something not playing nice with my hardware in 4.13? Although I have had kernel panics in 4.11 before I re-installed (not sure about 4.10).

Edit - Scratch that, failed to boot again on 4.10.

[ugly]

I've done a bit more testing.

Tried Ubuntu 17.10. Same errors. Tried a 4.15 kernel, same errors.

Tried Manjaro. I didn't get the typical Unable to mount root fs on unknown-block(0,0) error, but there were common freezes on startup. I did find someone on Manjaro with the similar ZBOX as mine and they had an issue with the display driver: https://forum.manjaro.org/t/after-insta ... ss/36066/2. But uninstalling xserver-xorg-video-intel didn't help.

I've tried using a different SSD, and I got the same error.

I've run memtest for 6 hours and no errors were found.

I've even gotten the error when trying to boot from a Mint 18.3 ISO off a USB stick.

I occasionally get different errors when it fails to boot:

Code: Select all

Initramfs unpacking failed: junk in compressed archive
Couldn't get size. 0x8000000000000000e
MODSIGN: Couldn't get UEFI db list
Couldn't get size. 0x8000000000000000e
/scripts/init-top/udev: line 24: /lib/systemd/sytstemd-udevd: not found
usb 1-8: device descriptor read/64, error -71
usb 1-8: device descriptor read/64, error -71
usb 1-8: device descriptor read/64, error -71
usb 1-8: device descriptor read/64, error -71
usb 1-8: device not accepting address 7, error -71
usb 1-8: device not accepting address 8, error -71
usb usb1-port8: unable to enumerate USB device
Gave up waiting for root device.  Common problems:
- Boot args (cat /proc/cmdline)
	- Check rootdelay= (did the system wait long enough?)
	- Check toot= (did the system wait for the right device?)
- Missing modules (cat /proc/modules; ls /dev)
ALERT!	UUID=9b09b895-2c7f-4959-8c29-ea9911182c6 does not exist.  Dripping to a shell!

I've also seen Initramfs unpacking failed: junk in compressed archive on boot but was still able to boot successfully.

I used the Media Creation Tool to try a Windows 10 install. Seems to work, but one time I did receive a BSOD with the error: "page fault in nonpaged area". But I've only seen that once.

I think it might be time to consider it a hardware error and contact the OEM. But I'd like to see if I could reproduce the BSOD one more time.

If anyone else had any insight, it would be appreciated.

[ugly]

I marked this as solved.

Looks like it was a hardware issue.

I found a cheap stick of DDR4 RAM on ebay to use as a test. No issues since replacing the RAM.

The RAM that was causing the problem didn't show any errors with memtest, or with the Windows Memory Diagnostic. It also ran fine with various stress tests.

Thanks for the help.

[NicoAarts]

Hi all,
Updating my kernel to 4.15.0-43 on my Mint 18.3 Xfce also resulted in exact the same error code as provided above.
After a quest of hours the solution for me was rather simple...
My problem was as follows:
Size boot partition: 780 MB
Boot partition used memory: 768 MB

So at the update of the kernel to 4.15.0-43 my boot partition was full (which I noticed in a much later phase of my quest...) resulting in an error and an unbootable system.
I tested my RAM memory and even after 5 passes of the test I had zero errors, so my 4 GB of RAM seemed to be OK.
Nevertheless I considered purchasing new RAM memory.
I also considered a fresh install of Mint 18 Xfce.

I booted my system with a live Mint 18 DVD and looked at my SSD drive with Gparted in order to perform a new fresh install. However at a certain moment I noticed that my boot partition of ca 780 MB was nearly full.
Then with help of this forum I used Update Manager-->View-->Linux Kernels to look at my kernels. It appeared that I had about 16 !!! installed Linux kernels which apparently filled up my boot partition!

Having noticed this I did thrink back my data partition and expanded my boot partition to 3 GB with Gparted.

Also with help of this forum I learned how to get into GRUB by pushing the shift key while booting my PC.
From the GRUB menu I did chose to boot with an older kernel which went flawless. And then I had my Linux Mint PC live and kicking again. What a relief...
Immediately Update manager provided me again with the update to the 4.15.0-43 kernel. I did chose to update to this kernel and now everything went flawless due to the fact that there was enough room on my expanded 3 GB boot partition.

And today I removed about 10 older kernels using Update Manager-->View-->Linux Kernels. Just select an older installed kernel and choose "Remove" (one by one). Now I still have 5 older kernels installed and 4.15.0-43 is my active kernel.

I wondered why I did not have any problem on my 12 years old Sony Vaio laptop whith the update to kernel 4.15.0-43. It appears I do have one big 800 GB boot partition on my laptop including the boot files. So no storage limitations there...

So my advice is to first look at the occupation of the boot partition. If this is nearly full you can be back on track with resizing the partition with Gparted in maybe 10 minutes or so as mentioned above.

And notice that I did not use the terminal to solve the problem !

Many thanks and happy New Year to the Linux Community!

[gnxfce]

Hey man. How are you doing?
You have done so much writing . It was necessary for me. Thanks for that