[Solved] Kernel installation does not stop

[feodor]

I am on Linux Mint 20.2 cinnamon and have tried to install a new kernel (5.4.0-88, updating from 5.4.0-86). Installation seems to run smoothly until it updates grub when is gets stuck with the oldest kernel I have (5.4.0-74):

Code: Select all

feodor:~/> sudo dpkg --configure -a
[sudo] password for feodor:             
Setting up linux-image-5.4.0-88-generic (5.4.0-88.99) ...
Processing triggers for linux-image-5.4.0-88-generic (5.4.0-88.99) ...
/etc/kernel/postinst.d/dkms:
 * dkms: running auto installation service for kernel 5.4.0-88-generic
   ...done.
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-5.4.0-88-generic
/etc/kernel/postinst.d/zz-update-grub:
Sourcing file `/etc/default/grub'
Sourcing file `/etc/default/grub.d/50_linuxmint.cfg'
Sourcing file `/etc/default/grub.d/init-select.cfg'
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.4.0-88-generic
Found initrd image: /boot/initrd.img-5.4.0-88-generic
Found linux image: /boot/vmlinuz-5.4.0-86-generic
Found initrd image: /boot/initrd.img-5.4.0-86-generic
Found linux image: /boot/vmlinuz-5.4.0-84-generic
Found initrd image: /boot/initrd.img-5.4.0-84-generic
Found linux image: /boot/vmlinuz-5.4.0-74-generic
Found initrd image: /boot/initrd.img-5.4.0-74-generic

and then nothing happens.

Stupidly, I did a hard reboot (turn off power) yesterday after waiting for a few hours and getting the computer started again this morning took forever (30 minutes?), so I am a bit worried by now. What should I do?

[JoeFootball]

Stupidly, I did a hard reboot (turn off power) ...

Correct, that's not a good idea, as the file system can be corrupted that way.

Should you find yourself in a scenario where you cannot shutdown normally, try this first. Holding down the Alt key, press SysRq (often shared with the PrtSc key), then r e i s u, and then o to Shutdown (or b to Reboot). This sequence should close your system cleanly.

Onward ...

What should I do?

Here's what I would do ...

-- Verify my off-disk backups, in case the file system is corrupted, or something else is amiss, or for all the other reason I can't think of.

-- Boot to a live session of Linux Mint, run GParted from the application menu, right-click on the offending partition(s), and select "Check", then "Apply" (the check mark), where it then runs a suite of commands to fix and tune the file system.

-- Ensure that my root partition isn't too full, where too many old kernels are usual suspects, so I would manually delete any old ones (keep at least one prior for safety) via Update Manager (View > Linux Kernels).

-- Make sure my future kernel clean-up was automated in Update Manager (Edit > Preferences > Automation, and in the Automatic Maintenance section, select: "Remove obsolete kernels and dependencies").

-- Execute sudo update-grub. I think this may be already done via the kernel removal, but I'd want to be sure.

[feodor]

Thanks for the suggestions (and for agreeing that turning the power off is a really stupid solution - knowing it beforehand just makes it worse). It really is most appreciated.

Removing old kernels is something I will try; I only have 4 right now, so I imagine that it is not those filling up my root partition. But the grub updater seems to get stuck on the oldest one, so getting rid of that may help. "Automation" is definitely something that I will turn on.

I may need to work up a bit of courage before using GParted. But your instructions are quite clear, so I am not as worried as I otherwise would have been.

Right now "Generating grub configuration file ... " is still stuck (7 hours now), so I suppose that I have to kill it, remove the oldest kernel, update-grub and/or dpkg --configure -a ... I think I'll leave that for tomorrow (working up some courage, give you or somebody else the chance to scream "don't to that, you ...").

Feodor

[JoeFootball]

... so I imagine that it is not those filling up my root partition.

Regardless of what may or may not be filling your system, you can assess how full/free things are via: df -h

I may need to work up a bit of courage before using GParted.

A good backup regimen makes for a good courage booster. I like to work this mind exercise to measure how well off I am: Without warning, my hard drive has physically failed, making any recovery of files via a live session (or otherwise) not possible, and the hard drive needs to be replaced. Am I prepared?

Another idea for partition maintenance is to setup automated file system checks, for example, sudo tune2fs -c 10 /dev/sda2 (lower-case c) sets the sda2 partition to be checked after 10 boots. And if I wanted to force a check on the next boot, sudo tune2fs -C 11 /dev/sda2 (upper-case C) pushes the counter past the 10 boot threshold. Note that your partition(s) may be different than sda2, and that the boot quantity threshold is up to you.

[feodor]

One of many fun aspects of linux is always learning new things

Code: Select all

feodor:~/>df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            7.8G     0  7.8G   0% /dev
tmpfs           1.6G  2.5M  1.6G   1% /run
/dev/sda2       453G  119G  312G  28% /
tmpfs           7.8G  464M  7.4G   6% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/sda1       511M  5.4M  506M   2% /boot/efi
tmpfs           1.6G   56K  1.6G   1% /run/user/1000

Not too sure about what this tells me specifically (I could google, so I'm not really asking); it seems to me that nothing is filling up. If there is anything looking really worrying, please tell me. I'll look into tune2fs when I have fixed this (learn something new).

I completely agree on the backup principles. I have backups on a separate harddrive (which for some reason is not mounted today, but it worked yesterday) and dropbox (plus the most important stuff on Mega), so what worries me is that I thought I couldn't even boot from a usb - it looked like that for a bit. So if it breaks I can do a fresh install and only lose time.

But I prefer to fix the problem if I can (and learn along the way).

A bit of courage tomorrow and I'll see what happens...

Thanks again

[JoeFootball]

One of many fun aspects of linux is always learning new things

I couldn't agree more. I've been using Linux for over 20 years, and I learn new things about it all the time.

/dev/sda2 453G 119G 312G 28% /

It's not drive space that's the issue, as you have only used 28% of your / (root) partition, so an easy cleanup isn't the fix. I can't say why the grub update is hanging.

I would try removing those older kernels anyway, just leaving 1-2 prior ones for safe keeping. I don't know if it will fix your problem, but it's worth a try.

[feodor]

I left my machine on all night and this morning dpkg had stopped running:

Code: Select all

feodor:~/>sudo dpkg --configure -a
[sudo] password for feodor:             
Setting up linux-image-5.4.0-88-generic (5.4.0-88.99) ...
Processing triggers for linux-image-5.4.0-88-generic (5.4.0-88.99) ...
/etc/kernel/postinst.d/dkms:
 * dkms: running auto installation service for kernel 5.4.0-88-generic
   ...done.
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-5.4.0-88-generic
/etc/kernel/postinst.d/zz-update-grub:
Sourcing file `/etc/default/grub'
Sourcing file `/etc/default/grub.d/50_linuxmint.cfg'
Sourcing file `/etc/default/grub.d/init-select.cfg'
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.4.0-88-generic
Found initrd image: /boot/initrd.img-5.4.0-88-generic
Found linux image: /boot/vmlinuz-5.4.0-86-generic
Found initrd image: /boot/initrd.img-5.4.0-86-generic
Found linux image: /boot/vmlinuz-5.4.0-84-generic
Found initrd image: /boot/initrd.img-5.4.0-84-generic
Found linux image: /boot/vmlinuz-5.4.0-74-generic
Found initrd image: /boot/initrd.img-5.4.0-74-generic
  WARNING: Device /dev/sdb not initialized in udev database even after waiting 10000000 microseconds.
Adding boot menu entry for UEFI Firmware Settings
done

So the problem seems to be the external hard drive (/dev/sdb/ - I imagine that is the external disk). A new problem to think about.

I have removed the oldest kernel and the update manager hangs at exactly the same spot again, after the oldest remaining kernel when generating the grub config file, probably trying to initialize /dev/sdb/ ... at least I know that I just have to wait a zillion milliseconds

Feodor

[karlchen]

Hello, Feodor.

Seems as bit as if a disk device /dev/sdb is referenced somewhere, although the device is no longer connected to the machine.
Would you please, share the output of the command cat /etc/fstab with us?
Also, please, share the output of the command lsblk -o +UUID with us.

Hopefully this will help identify /dev/sdb and maybe, too, why it negatively affects the "dpkg" activities.

Regards,
Karl

[feodor]

Hi Karl,

I get

Code: Select all

feodor:~/>cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/sda2 during installation
UUID=f9c8a5d6-a02a-4574-9cea-53a5cd5a4dcd /               ext4    errors=remount-ro 0       1
# /boot/efi was on /dev/sda1 during installation
UUID=5D1D-C07C  /boot/efi       vfat    umask=0077      0       1
/swapfile                                 none            swap    sw              0       0

Code: Select all

feodor:~/>lsblk -o +UUID
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT UUID
sda      8:0    0   477G  0 disk            
├─sda1   8:1    0   512M  0 part /boot/efi  5D1D-C07C
├─sda2   8:2    0 460.6G  0 part /          f9c8a5d6-a02a-4574-9cea-53a5cd5a4dcd
└─sda3   8:3    0  15.9G  0 part            7938979d-ecab-4ed0-bd13-8b6891b2f887
sdb      8:16   0   1.8T  0 disk            
sr0     11:0    1  1024M  0 rom       

I have a harddisk on the computer and a external harddisk with two partitions; one is encrypted ext4 (I think), which I use for backups and time shifts, the other is fat4 and not really used. As far as I can tell, the external harddisk is on and connected to the desktop. It worked last week, at least, when I had to restore I file I had managed to overwrite.

Feodor

[karlchen]

Thanks, Feodor, for sharing the requested details with us.
As you mention an ext4 and a FAT32 formatted disk, let me ask you to repeat the second command in a slightly modified way, please:
Please, share the output of the commandline lsblk -o +FSTYPE,UUID with us.

The fstab output, which you shared, tells that the disk device /dev/sdb is not in automatically mounted by the system. Else it would be mentioned in /etc/fstab, either using the device name /dev/sdbX (where x would be a number) or by using the UUID.

The lsblk output, which you had shared does not give any UUID for the device /dev/sdb. All it can determine is this sdb 8:16 0 1.8T 0 disk . This looks a bit as if the device does not answer.

Could you please, check whether the syslog file logs some complaints about device /dev/sdb?
Please, share the output of the commandline grep "sdb" /var/log/syslog* with us.

Thanks,
Karl

[feodor]

Hi Karl,

Code: Select all

feodor:~/>lsblk -o +FSTYPE,UUID 
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT FSTYPE UUID
sda      8:0    0   477G  0 disk                   
├─sda1   8:1    0   512M  0 part /boot/efi  vfat   5D1D-C07C
├─sda2   8:2    0 460.6G  0 part /          ext4   f9c8a5d6-a02a-4574-9cea-53a5cd5a4dcd
└─sda3   8:3    0  15.9G  0 part            swap   7938979d-ecab-4ed0-bd13-8b6891b2f887
sdb      8:16   0   1.8T  0 disk                   
sr0     11:0    1  1024M  0 rom        

Output from grep "sdb" /var/log/syslog* is 7637 lines long, to much to post, so I am attaching it instead. No, it is too large to attach, so I am attaching a sample. I imagine that it repeats the same errors/messages repeatedly. If there is something specific I need to look for, let me know. If you need the entire 1.1M, maybe I can try to put it somewhere for you to download.

Feodor

[karlchen]

Hello, feodor.

I am not quite sure whether your collection of messages, which keep on re-occuring, in the file sdb-syslogs.txt, should alert you of a disk hardware problem.

This may also depend on when precisely the problem, which you reported in your first post, has started.
I mean has installing kernel updates, with the /dev/sdb 1.9 TB harddisk attached, initially worked?
Has the problem started only recently?

In any case I would check all cable connections to this 1.8 TB harddisk, power cable as well as data cable. In case this harddisk is attached using a USB cable, I would change the USB cable and maybe, too, the USB port, in order to find out whether syslog stops logging all those messages.

If possible check the disk health by inspecting the smartmon values. Those should be available in the application "Disks", after selecting the device /dev/sdb. Hamburger menu => Smart values and self-check.

Searching for the error message near the top of sdb-syslog.txt, blk_update_request: I/O error, dev sdb, sector 3907029160 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0, also returned an older Launchpad bug report. https://bugs.launchpad.net/ubuntu/+sour ... ug/1856434
The reported error occurred on Ubuntu 19.10. No Linux Mint, however, is based on Ubuntu 19.10. Either on Ubuntu 18.04: Mint 19.x. Or on Ubuntu 20.04: Mint 20.x. By the way the issue was resolved by a firmware update of the affected machine in the end. - I doubt a bit that this Launchpad thread apllies to your case, Feodor.

--
Close to the top of the messages in sdb-syslog.txt, too: [sdb] tag#21 Add. Sense: Logical unit is in process of becoming ready
Wondering whether the device is configured to go to sleep, while not actively used, and needs too much time to wake up again.
--

Wondering a bit, whether as a workaround it may be possible to instruct update-grub to ignore /dev/sdb and leave it alone. - Disconnecting the disk before an upate-grub might be a workaround as well.
(The last operation of any kernel installation / uninstallation is alway an update-grub, invoked by dpkg, which executes the post installation or post removal script as appropriate.)

Regards,
Karl

[feodor]

Hi Karl,

Thank you for the effort. And the suggestions; I will definitely try most if not all of them.

Having located the problem to most likely being related to the external hard drive, I am not as worried as I was yesterday. I think first thing to try is to unplug it, see what happens, plug it in again. Maybe it's just a glitch. Unplug update grub, plug in again. After that check cables (it is connected via usb) etc. If I can get it back online, I'll check smartmon values (and google what they are); "Disks" can see the external drive but only offers to format it (and that must be a bit premature).

I have not consciously configured the external drive to go to sleep. I've only plugged it in and been happy that it just works

I have updated kernels before without any problems, so why it happens now is a bit of a mystery (to me).

Anyway, plenty of things to try out now, and I am feeling ok again. Thanks for your help

Feodor

[feodor]

Embarrassingly, plugging the external hard drive out and in again (in a new usb port) seems to have solved the problem.

Thank you, Karl and Joe(?), for your help.

I'll mark this as solved and open a new thread if the problem reappears.

Feodor

[karlchen]

Great, Feodor.
Then let's keep our fingers crossed that the culprit was just the previously used USB port and that the problem will not return.
Cheers,
Karl

[JoeFootball]

Thank you, Karl and Joe(?), for your help.

You're welcome, and I'm glad it was a easy fix once you discovered it. I didn't suspect another device when it wasn't present in your df output, so good investigative thinking on Karl's part.