Deliver Me From "SYSTEMD" HELL!

[Spearmint2]

I was playing around testing something with Disks and not thinking, put two / mount points, one on each another drive's partition, one existing for current system, the other / for old 17.3 system. Forgetting NEVER do that. I changed it back, then rebooted. All should have been fine, but NO! The boot took me to the tty screen with login> and so I did, and tried using "start x" and also "mdm" and "sudo service mdm start" and "systemctl start mdm" but nothing worked. Error msg was always about some missing

com.ubuntu.upstart, or maybe com/ubuntu/upstart, but think it was the first. I haven't run timeshift yet, but did try that in the tty screen, but kept getting "read only" errors and it failed.

Finally at the boot menu I'd entered my 17.3 old system and changed fstab to what it needed to be, also manually changed the one problem bootline in the grub.cfg, but on reboot to 18.3 still got the upstart error msg.

I'm really missing System V init, not happy with this newer systemd/systemctl stuff.

I finally noticed an entry in the boot menu with the word "upstart" by it, and thankful to whomever put that in the Advanced section, since it works to boot back into the GUI of my 18.2 MATE system.

What's my options to fix this without doing a timeshift recovery. Thankfully I have one from Oct 6, just yesterday. I'd sort of like to know how to get out of such a situation in future, since although I can use the Advanced boot option for "upstart", it looks like it maybe bypassed the Upstart stuff, since my normal menu boot line continues to fail, always going to tty screen.

If I can't fix it manually, I'll run the timeshift and hope that solves it, but would like to know any other method if it happens again and I didn't have a near current timeshift backup to solve it. I don't yet even know if that will fix it.

[Spearmint2]

I did a comparison of my first bootline and the second one that has "upstart" named in it. After removing all lines that were exactly the same, this is what remains of each. They are divided by =======================

Code: Select all

 --class ubuntu   'gnulinux-simple-fafbd832-cde5-4060-9464-1effc5caad07'
	
	==================================================================================================
with Linux 4.4.0-146-generic (upstart)' --class ubuntu   'gnulinux-4.4.0-146-generic-init-upstart-fafbd832-cde5-4060-9464-1effc5caad07'
echo	'Loading Linux 4.4.0-146-generic ...'
init=/sbin/upstart
echo	'Loading initial ramdisk ...'

So, I think I'm going to copy that entire "upstart" menuentry in the "Advanced Menu" area, paste into grub.cfg above it, and use it as my first default choice. It's possible I ran the older update-grub while in Mint 17.3 system, since that's the only one was able to access quickly, and it may have butchered the first menuentry for Mint 18.3 ? I remember when I ran Mint 14 along with Kubuntu 12.04, if I ran one system update-grub, it created a different grub.cfg than if I did it from the other system. I did however, run the current update-grub once I got back into my 18.3 system using that "upstart" menuentry, and it didn't correct that first "menuentry", but went back to failing to boot into the GUI.

[Spearmint2]

seems I'm back on the System V init system only now, systemd is dead?

Code: Select all

$ systemd-analyze blame
Failed to parse reply: No such property 'FirmwareTimestampMonotonic'
$ systemd
Trying to run as user instance, but the system has not been booted with systemd.
$ systemctl
Failed to list units: No such method 'ListUnitsFiltered'

maybe tomorrow run timeshift restore on it, see if that gets the systemd back. Do that after I see if anything seems broken otherwise. I reinstalled some of the systemd that Package Manager showed as installed. No luck, still same problem.

This systemd/systemctl stuff sits on top of the system like humpty dumpty sat on the wall, and then took a great fall, and couldn't be put back together again.

[Spearmint2]

Still booting with "Upstart" method instead of "Systemd". I'm kinda liking it, reminds me of the way 17.3 worked. Unfortunately, the newer systemctl and system-* commands can't be used with it. When I boot now I am only using 300MB, and before when I booted it was between 500MB-600MB. I did find two pages that give some interesting feedback on the relationship between upstart and systemd. I might give that a tryout, just to see what happens. Otherwise things seem to run OK. A chance to learn something new I guess. I can hopefully always use the timeshift later.

https://www.howtoinstall.co/en/ubuntu/x ... ion=remove

https://wiki.ubuntu.com/SystemdForUpstartUsers

If you get odd boot message when a boot fails,remember the "upstart" method in the Advanced area of Boot Menu may get you back into your system.

[Spearmint2]

What really upsets me at this point, since Upstart is booting my system OK, is how easily the systemd method can fail so miserably by a seemingly small mistake, one that was corrected even before reboot. I liked the 17.3 without systemd, thought maybe the systemd starting in v18 by 18.3 would be hardened by then. Now I'm considering removing it altogether from 18.3, clearing all it's orphaned parts out.

[Spearmint2]

More reading, from 2014.

https://www.tecmint.com/systemd-replaces-init-in-linux/

It's odd that we've gone from parallel drives to serial drives (PATA vs SATA), and that's an improvement, but for booting the latest rage seems to move the opposite direction from serial boot process (asynchronous) to a parallel process. As the saying goes, "something here just doesn't add up".

Linus Torvalds, Chief architect of Linux kernel, feels attitude of key developer of systemd towards users and bug reports do not seems ok. It was also reported that systemd philosophy is weird and a foreign way to control system processes. The same has been recorded from Patric Volkerding and other notable Linux Users and Developers as well as over online forum, time-to-time.

https://www.2daygeek.com/enable-disable ... l-command/

[Spearmint2]

Reading more, it seems this systemd is sort of a "wrapper" around the actual Mint OS, otherwise it wouldn't boot just fine also in "Upstart" init method. It adds about 150-200 MB more to the boot process, compared to when I checked a boot immediately after systemd boot was used in past to what I find when using the Upstart method to boot up.

Anyway, that's something worth remembering, that if someone isn't able to boot with the normal boot under systemd if something went wrong with it, then the Advanced Mint boot line section provides this alternate "Upstart" method to get booted. Once that systemd is broken, even running update-grub won't restore booting under it.

Maybe they should add that "tip" into the release notes.

I'm sort of enjoying running it this way, most things systemd provided I want to check are found in the logs anyway, easy enough with Log Viewer. Feels like being back on 17.3 again, which I liked a lot.

So far as I can see, the main "advantage" of systemd is parallel booting process instead of serial boot, or asynchronous method, so a bit faster boot time. I know many are obsessed with faster boot time, but that's not such a big thing with me, since I may go months with just using Suspend anyway.

[rene]

It's odd that we've gone from parallel drives to serial drives (PATA vs SATA), and that's an improvement, but for booting the latest rage seems to move the opposite direction from serial boot process (asynchronous) to a parallel process. As the saying goes, "something here just doesn't add up".

Thank you. Priceless...

[Spearmint2]

I have found 5 popular distros which do NOT contain the systemd "wrapper" program. I've decided however, I'm going to follow some information in above links, and some other pages, to rip systemd out of my Mint 18.3 and see if it still runs well using "Upstart" init system instead. Supposedly that's possible without too much difficulty. When I do, I'll post results back here.


Knoppix

PC LinusOS

MX Linux

Devuan (very devoted to NOT systemd)

Puppy Linux

[LanceM]

I'd like to know if Timeshift fixes it. Wowzers I'd of thought you'd make a Clonzilla image before such an undertaking. Nice to know even you can mess up

[Spearmint2]

I do have my user profile backed up.

Here's how it went. It didn't. It turns out the Upstart boot is also subject under systemd. I thought it was a "stand alone" boot process. It can be installed as such, which is what I should have done. I ran this command to remove systemd.

Code: Select all

sudo apt-get remove systemd

Don't do that!

The option in boot menu for "upstart" was gone after I ran update-grub.

Don't do that #2!

I should have first installed Upstart, so it would be stand alone, but stupidly I'd assumed it already was setup that way.
Never assume.

However, I was able to reach the splash/GUI animated Login screen I have from metacity. I think MDM was running that, so don't know why it didn't also enter my profile with it running too.

That systemd seems to have a finger in every pie of the system.

I did the login, and hard drive light flashed showing accesses were going on. I ended up at a black screen, totally black, except for a mouse cursor which did work with mouse, even though it's a wireless mouse. That's it.

I then tried changes in the boot menu line, like adding init=/sbin/init and some other things I'd hoped would work, but didn't. Always the same thing hapened after GUI login page. I even tried taking out some insmod stuff that wasn't specifically for msdos, which I'd not seen in older boot menus.

I got tired of messing with it, backed up my profile from the home folder using caja in superuser mode, which was easy since I still have Mint 17.3 on this computer and that booted OK.

I've downloaded the 18.3 ISO, checked it's hash (I'd done the mint upgrade method from a 17.3 clone of my system I'm now on, to get to 18.3 before. I've used mkusb to make a bootable USB flash.

So, my plan now is to do a "dirty install" as we called it in windows, and reinstall 18.3 on top of what I already have there, using something else choice, and no formatting done, etc. I've used that method in past with no problems in Mint, since version 14.

I'll see how it goes in a bit.

[Spearmint2]

I renamed my /home to /__home, then did the "dirty install" but unlike all the other times since version 14, that didn't work out. So I deleted everything from the partition, except the /__home folder and did a fresh install. I had to chown the /__home folder to the new user, and moved a lot from it into the new home, especially firefox and thunderbird files. Still have some of the others to do later on.

I discovered something new at firefox. They have a new type of firefox that you unzip into a folder in your user's area, and it all works there without putting all the *.so files in the system. Seems like a possible security risk to me. You can have two different firefox programs open running at the same time. They want you to sign up for an account to "sync" stuff from older firefox into it. Mint 18.3 32bit came with version 57, but all the problems people have had with version 69, I'll wait to upgrade it from the update manager.

Mostly all is running OK now. Maybe some future time I'll install a second copy of 18.3 and take another shot at seeing if systemd can be removed from it, just for the experience. Had enough experience with that for now.

[pbear]

Guess this means we won't be seeing an Upstart respin any time soon.

[Spearmint2]

Guess this means we won't be seeing an Upstart respin any time soon.

LOL, don't give up yet. When I'm not tired of this, rested up, l may challenge myself again on it. I bet you and rene even might try it, but follow those ubuntu suggestions in link how to do. In order. In other words "asynchronously" or serially, LOL.

If systemd is hardened, so some odd, somewhat innocuous thing like I did, a simple mistake I corrected first, didn't collapse the house of cards, then I think they added some great analytical means of problem solving. I also think some problems have been added. The big selling point seems to be parallel loading of an OS, instead of one by one process, in serial order, but some things often require a serial process, lined up properly. I suspect many of these get combined into one, but may not always avoid a clash, head-on crash. Supposedly parallel booting makes things faster, but it's what happens AFTER a fast boot process that concerns me most, like all the other time I'm actually using the OS. If the boot can be speeded up in such manner, OK, but does that justify the OS "wrapper" additions that systemd has become?

ONE RING TO RULE THEM ALL? (one program to rule them all?)

[deepakdeshp]

I read with interest the progress spearmint2 is making.Good luck to him in this. I am sure he has appropriate backup like Clonezilla or time shift , preferably both.
I personally wouldn't venture into it, with the beleif that far more knowledgeable persons have taken the decision and indented it.
I haven't found any increase in system start up time from Mint 18 to 19. It is more or less same at 3 minutes. I had opened a thread to reduce it, but that didn't help.
Here is a good link
https://unix.stackexchange.com/question ... nd-systemd

[Spearmint2]

Great link!

The mention, talking about the shorter commands compared to longer commands needed with systemd rings strong with me, and the conversion of clear text to binary. That's not always true, but often.

Logs

In upstart, the logs are normal text files in the /var/log/upstart directory, so you can process them as usual:

cat /var/log/upstart/foo.log
tail -f /var/log/upstart/foo.log

In systemd logs are stored in an internal binary format (not as text files) so you need to use journalctl command to access them:

sudo journalctl -u foo
sudo journalctl -u foo -f

It's a better presentation using Log Viewer than the journalctl provides too, easier to quickly read/scan it seems. Maybe it's just my eyesight?

As I've said before, I'm not a "fast boot" junkie, I can wait a bit if necessary, since I often just use Suspend anyway.

4 months with no reboot.

121days-up.jpg (41.68 KiB) Viewed 3402 times

[rene]

I bet you and rene even might try it [ ... ]

Shall not speak for him but can assure you that most certainly I will not. I am here, on what idiots like me would call the weenie-side of Linux, in the first place due to having given up in that respect; transferred from Arch after its adoption of systemd forced it user into a uniform low-level setup anyway lest said user needed to basically do it all itself --- and again on every "rolling" update --- turning Arch into a distribution for children who though/think that typing pacman -S fifty times in a row rather than letting an installer do so makes one one of the Cool Kids.

That was at the time of Mint 17 which used Upstart and booted faster than any of my carefully tweaked Arch systems in the first place -- which made me go "Ah. Uh." when I idly tried it. Upstart was a good init system, and certainly for those that in fact cared about the end-user being able to easily interfere. At the time systemd bootups were a collection of race-conditions --- on Arch as well as everywhere else.

Things with systemd have improved a bit, even if not in a fundamental sense as compared to Upstart's explicit "bottom-up" approach to dependencies, and unification góóód, and... in any case, that ship has sailed when Debian switched to systemd. It means there's no relevant enough party anymore to maintain init-scripts for a few thousands packages and a few decades to come. Had it not been for the damned Ubuntu Contributor License Agreement things might have turned out different -- but they did not. Fighting a battle for Upstart now is much like campaigning for the dodo-bird's animal rights.

I.e., have fun there, but, no, I won't be joining you....

[Spearmint2]

I loved Mint 17.3 and wish 18.3 was just polished up and added more, but leave the way it booted, the control methods, the commands already learned, etc. I'll use Mint 18.3 till I discover if I can appreciate it more, but may also be looking around again as I did when Kubuntu was upgrading from 12.04 which I'd liked more than the later versions. I'm worried that systemd may be one of those additions that when one thing in it breaks, it affects more other things that otherwise wouldn't have been. Maybe Mint maybe one day will decide to create a Devuan based version, as the current ones are Ubuntu based, which is based on Debian.

[rene]

The only thing that could still happen is a fork of systemd, and the most major thing that could be expected from that is said fork dropping parts of systemd that are not in fact "the init system systemd" (but the DNS resolver, or the NTP client, or the login manager, or udev, or...). The init system would seem incredibly unlikely to not be here to stay at this point; even if only since providing uniformity is actually a good idea there, in the context of third party developers.

[tweeks]

Any updated on the state of a decent systemd-free fork? 17/3/rosa expires soon, right? I don't mind a compatibility layer (e.g. using systemctl, journa,ctl, blah blah blah).. I just don't trust the full beast.. SystemD even reads your LUKS boot block-device passphrase!?! Wedgeware would be fine with me.

Last I checked, Devuan was not very nice.. any other decent options?

Tweeks