systemd-udev-settle.service causing long boot time.

[KFGeronimo]

I've had long boot times since I started using Mint. I used systemd-analyze blame and this is what I got:

Code: Select all

2min 18ms systemd-udev-settle.service
   5.774s NetworkManager-wait-online.service
    500ms mullvad-early-boot-blocking.service
    335ms networkd-dispatcher.service
    240ms blueman-mechanism.service
    220ms lightdm.service
    219ms plymouth-quit-wait.service
    174ms dev-sda3.device
    125ms udisks2.service
    120ms systemd-resolved.service
    112ms systemd-fsck@dev-disk-by\x2duuid-5CF0\x2d5DA3.service
    103ms user@1000.service
    102ms accounts-daemon.service
     89ms systemd-timesyncd.service
     88ms ModemManager.service
     85ms lvm2-monitor.service
     84ms apparmor.service
     80ms ubuntu-system-adjustments.service
     66ms avahi-daemon.service
     66ms bluetooth.service
     65ms systemd-modules-load.service
     64ms NetworkManager.service
     63ms systemd-udev-trigger.service

What is going on to cause that service to take so long?

I'm also dealing with udevd also preventing me from shutting down. This is all I got on that.

[rene]

I do not know what in the end the direct issue will be. The udevadm settle delay seems to most often be connected with "fake block device" systems such as LVM, but LVM is a part of every encrypted Ubuntu/Mint install whereas I never in fact see this service being asked about, so it's seeimgly not generally an issue with LVM. I expect it's in your case related to that "mullvad" thing that we see...

But whatever the specific issue is here, systemd-udev-settle is in any case deprecated -- see man systemd-udev-settle.service -- and if there's no actual reason your specific system needs it, then the solution might very well still just be

Code: Select all

systemctl mask systemd-udev-settle.service

Something pulls in that service as a dependency, and I as said expect it's that mullvad thing, but as long as nothing in fact stops working all should be fine.

[gcristof]

Posting this additional info here to hopefully help save folks some time in trying to fix this very aggravating slow boot issue...
I just spent a day and a half trying to figure out why my Linux Mint 21.1 system which used to boot in only 6 seconds, is now taking 2 minutes to boot every time. Looking at logs I finally tracked the issue down to this systemd-udev-settle service.

So I masked the service as suggested by @rene, and now my system is back to booting up in 6 seconds flat again!

I then found this man page for systemd-udev-settle.service here: https://www.man7.org/linux/man-pages/ma ... ce.8.html
It says that the service is Deprecated, and specifically states:

"Using this service is not recommended. There can be no guarantee
that hardware is fully discovered at any specific time, because
the kernel does hardware detection asynchronously, and certain
buses and devices take a very long time to become ready, and also
additional hardware may be plugged in at any time. Instead,
services should subscribe to udev events and react to any new
hardware as it is discovered. Services that, based on
configuration, expect certain devices to appear, may warn or
report failure after a timeout. This timeout should be tailored
to the hardware type. Waiting for systemd-udev-settle.service
usually slows boot significantly, because it means waiting for
all unrelated events too."

So I was wondering why this service is still being used.. looking into it a bit further I suspect this post has something to do with it still being enabled in Mint/Ubuntu:
https://github.com/opensvc/multipath-to ... -775037968

[rene]

Good that it helped, but as to this bit...

I then found this man page for systemd-udev-settle.service here: https://www.man7.org/linux/man-pages/ma ... ce.8.html

If you would've, as advised, just typed man systemd-udev-settle.service into a terminal you would've found that same manpage on your own system

[KFGeronimo]

I do not know what in the end the direct issue will be. The udevadm settle delay seems to most often be connected with "fake block device" systems such as LVM, but LVM is a part of every encrypted Ubuntu/Mint install whereas I never in fact see this service being asked about, so it's seeimgly not generally an issue with LVM. I expect it's in your case related to that "mullvad" thing that we see...

But whatever the specific issue is here, systemd-udev-settle is in any case deprecated -- see man systemd-udev-settle.service -- and if there's no actual reason your specific system needs it, then the solution might very well still just be

Code: Select all

systemctl mask systemd-udev-settle.service

Something pulls in that service as a dependency, and I as said expect it's that mullvad thing, but as long as nothing in fact stops working all should be fine.

Masking udev did solve my boot issue. Thank you for that

I am still dealing with the /oldroot thing preventing my shutdown sadly.

[gcristof]

ok, so while my slow boot issue was indeed resolved by masking the systemd-udev-settle.service, that ended up caused another intermittent issue to occur on my system, so I had to dig a bit deeper to find a better solution in my case..

The problem was that I use a usb docking station to connect my laptop to an external monitor, keyboard/mouse, and usb drives. After having masked the service, I began noticing that occasionally the monitor would no longer "wake up" from power save mode when I connected my laptop to the dock.

I then realized that the slow boot issue only occurs when the laptop is connected to the dock. I re-enabled the service and the slow boot issue reappeared, but if I disconnect from the dock and boot it standalone, it boots up fast as usual. So I then also verified that the slow boot issue went away completely (even while connected to the dock) after uninstalling the DisplayLink software (although my external monitor would then no longer get detected).

I finally found this post on the DisplayLink site which suggests a fix to the DisplayLink udev.sh script. Apparently DisplayLink drivers rely somewhat on the udev services. https://support.displaylink.com/forums/ ... udev-rules

So I located that displaylink script on my system at /opt/displaylink/udev.sh, and applied the fix simply adding the "--no-block" option to the line: systemctl start --no-block displaylink-driver

That seems to fix the issue entirely. The systemd-udev-settle.service is still starting up, but it no longer gets stuck waiting for the DisplayLink drivers to finish loading. I wondered if perhaps that might still cause some race condition issues with my external monitor not waking up occasionally, but so far after 5 days of testing, it has been working perfectly every time.

[rene]

Very good digging; thank you for sharing!

[dunkirk]

ok, so while my slow boot issue was indeed resolved by masking the systemd-udev-settle.service, that ended up caused another intermittent issue to occur on my system, so I had to dig a bit deeper to find a better solution in my case..

The problem was that I use a usb docking station to connect my laptop to an external monitor, keyboard/mouse, and usb drives. After having masked the service, I began noticing that occasionally the monitor would no longer "wake up" from power save mode when I connected my laptop to the dock.

I then realized that the slow boot issue only occurs when the laptop is connected to the dock. I re-enabled the service and the slow boot issue reappeared, but if I disconnect from the dock and boot it standalone, it boots up fast as usual. So I then also verified that the slow boot issue went away completely (even while connected to the dock) after uninstalling the DisplayLink software (although my external monitor would then no longer get detected).

I finally found this post on the DisplayLink site which suggests a fix to the DisplayLink udev.sh script. Apparently DisplayLink drivers rely somewhat on the udev services. https://support.displaylink.com/forums/ ... udev-rules

So I located that displaylink script on my system at /opt/displaylink/udev.sh, and applied the fix simply adding the "--no-block" option to the line: systemctl start --no-block displaylink-driver

That seems to fix the issue entirely. The systemd-udev-settle.service is still starting up, but it no longer gets stuck waiting for the DisplayLink drivers to finish loading. I wondered if perhaps that might still cause some race condition issues with my external monitor not waking up occasionally, but so far after 5 days of testing, it has been working perfectly every time.

Sadly this doesn't work if your docking station is connected to 4 external monitors. More specifically, one of them is connected through a "DisplayPort over USB C". I'm using the Dell UD22 docking station, which supports 4 external monitors only if the DisplayLink driver is installed. I have installed the driver. Everything is fine except the long boot time.

After I added "--no-block" to the udev.sh script, the boot time goes back to 6 seconds. However, the monitor that connects through that "DisplayPort over USB C" port went blank after the quick reboot

If I want all 4 monitors to light up after booting up, I have to endure the painfully slow boot (measured exactly 2 mins 1 sec after the Linux Mint logo showed up).

[Reddog1]

Here's a hint, if you have superfluous service hanging your shutdown (with the 90 second timeout). Edit /etc/systemd/system.conf and remove the '#' from the #DefaultTimeoutStopSec=90s, and change the 90s to something reasonable, 15 or 20, and save. The change will take affect after a reboot.
systemd has a problem with some services, and I have to do this with all of my arch-based VM's hanging on a virtualbox service that doesn't want to close on shutdown. Out of frustration I shortened the timeout and have had zero problems doing it, but of course, YMMV. You can try it and see if there are any adverse effects, because the change is easy to reverse.

This may work without any problems when applied to the startup timeout, IF the boot continues 'normally' after the timeout completes. One way to easily see if the timeout is in play is to edit /etc/default/grub and remove quiet splash.