Power Glitch > Logo > No Boot

[WHVW]

Well, it happened again....another power glitch and now an install won't boot.
The usual tricks, such as "Repair Filesystem" in Disks (which always works) says that there are no problems. I tried "repair" anyway...nix. Yet the logo is about as far as I can get. For a while I was able to get it to boot on another (identical, except 8 Gigs vs. 16 Gigs of memory) machine after a bit of fiddling, but not now.
I tried two Timeshift restores...nope. (maybe should be to another drive?)
I have seen "no drive detected" error messages. (yet I can read files on it)
I have seen "you must load the kernel first" messages, but of course no hint on how to effect this.
The "waddayawannaboot" screen (I don't know its proper name) has shown Ubuntu, 19.1 & 19.3 choices...none work. As I recall, the machine was running 19.3, but it displays the leaf logo, not the circular one.
Any ideas?

Thanks.

[Cosmo.]

If you are able to boot into recovery mode do a file system check there. Note, that there is the possibility, that the file system might get damaged beyond repair, especially if the power problem should reappear on your site repeatedly.

[WHVW]

Cosmo:

I was able to get into recovery mode a couple of times..it said that everything was fine, and that's when I was able to boot into the system once but if I turned it off and back on....no go.
It is obviously still able to function, (occasionally) so everything must still be there.
Is there a way to watch what it is doing and manually correct the bad instruction/file? Or re-write the boot sector from the install disk? Or lay the system and home files over a new install? Or Timeshift to an other HD?
Are any of those ideas viable?
Thanks

[Cosmo.]

If the file system is broken, you have only one choice: Backup everything, reformat the disk and install new.

[mikeflan]

Is there a way to watch what it is doing and manually correct the bad instruction/file?

Hit escape during boot. If you want to make this permanent, turn off quiet splash.
https://deeptalk.lambdalabs.com/t/ubunt ... splash/443

[WHVW]

Thanks, mikeflan for the "quiet splash" edit tip. One question, though: since the system won't boot, do I need some other command such as "sudo update grub2 /etc/default/grub /dev/sdb1" (I know that syntax is wrong, but you get the idea) to get the grub on the bad disk to update rather than the currently operating system disk?

Thanks

[mikeflan]

I know of no magic commands. I think Cosmo is right - you should reinstall if you don't have a whole disk image to reload.

[Reddog1]

If this was my system, I'd be questioning the viability of hardware. First guess is hard disk, but also power supply. If your 'outage' was preceded by a voltage spike (lightening strike, 2 opposite phases touching momentarily) a problem with line monitoring/safety equipment on the incoming line could very possibly fry something in your system. In my professional life I've seen this. Hopefully not your problem, but worth considering, especially if your system wasn't plugged into a working surge arrestor. And yes, surge arrestors can fail after one time operation, resulting in no protection for the next time.

Good luck

[WHVW]

Reddog1, mikeflan:

Yes, the HD would be one of the first suspects if it were not for the fact that Disks kept saying that there was nothing wrong with the file system. I get a lot of .5 to 1 second power interruptions here, and that's what always seems to do it. There was one that night.
I pulled the important files from the disk, and did some re-installs.
But I'm curious: Most of these cases stop at the logo. Is there some sort of analysis programme that allows one to watch the boot process and show where it went off the rails?
Thanks

[rene]

But I'm curious: Most of these cases stop at the logo. Is there some sort of analysis programme that allows one to watch the boot process and show where it went off the rails?

mikeflan;'s advised deletion of "quiet splash" from the kernel command-line -- although you can't then do that now by editing /etc/default/grub since you're not booting.

When you boot I take it you (can) get into the Grub menu -- the "waddayawannaboot" screen -- so from said menu and with the selection on the entry you in fact want to boot use e to edit the entry, and delete quiet splash from the linux line, then F10 to boot.

[WHVW]

rene

O.K. I did that: that is, hit the "e" key and delete "quiet boot" from the Linux line. Here's what I got:
"OK stopped light display manager
starting to detect available GPUs and deal with any system changes"
If I do nothing, it will just stay there.
If I hit cntrl+alt+del, I will get a brief glimpse of the activities, then a black screen with a blinking cursor in the upper left. Nothing more.
(on a new boot) If I hit cntrl+alt+del twice I will get the brief glimpse of activities, then its back to the black screen with blinking cursor. Nothing more.
I also looked at the etc/defaults/grub file, the quiet splash line would have to be added, since it is not there.

Thanks.

[rene]

It's not too useful. Might be the case that LightDM got damaged on-disk -- but might essentially also be the case that more or less anything might, and certainly from here it's impossible to say. I'd reinstall.

[WHVW]

rene

Oh, I have reinstalled. At this point, I'm looking at this as a chance to learn something about the deep machinations of a HD, and how to become good enough to be able get in there, patch around and make it work in spite of itself.
Its a Sherlock Holmes sort of thing, actually. There is a sequence of events: one of them has gone off the rails and stopped the train. The first thing is to find out just which one. That done, one next needs to find out why the next event is not happening. Are the bits that point to the next file bad? or, are the bits that identify the next sector bad? Or something else? Knowing what they should be, why couldn't a disk editor be used to correct them? Or build a "bridge" to another part of the disk and back?
It's not much different than fixing a transmitter, actually. You go through it, stage by stage, confirming that each stage is doing what it should, until you find the one that isn't. Then you investigate that stage, correct the problem and you're done.

[rene]

It's a bit of a lost cause I'm afraid. The power-glitch cause says it's almost certainly at heart a filesystem damage issue, and/but fsck as per your report not finding anything (anymore, maybe/presumably) and the eventual issue seeming to be a dying/hanging graphical environment means that trying to remedy things by e.g. reinstalling components is going to be a potentially near endless game of whack-a-mole -- certainly across the internet, i.e., without first hand experience of what the previously good situation was and/or what in every last detail happened and is happening now.

[AndyMH]

I get a lot of .5 to 1 second power interruptions here

Save yourself future grief and get a UPS. We get frequent power outages and I wouldn't be without one. I have my router, switches, NAS and desktop on it. If I have a prolonged outage it gives me plenty of time to shutdown the desktop and fire up a laptop.

[WHVW]

AndyMH, Cosmo, mikeflan, Reddog1, rene & All:

Yes, that UPS is the only way to go.

As for the (original) drive/install itself, I have already pulled off all the important files and transferred them to another drive with a fresh install that works just fine. For me, this is a perfect opportunity to learn how to fix things at the "component" level...there are (after all) people who know how to do such things, and, with a little help from you folks, maybe I can become one of them.

Since all critical files have been transferred to a fresh install on a different drive, the original drive/install is now superfluous. Therefore, I can experiment without fear of ruining anything. (So) this is a pure quest for knowledge, I would really like to know how to work at that level, hence all suggestions and ideas are welcome and worthwhile.

Thanks.

[rene]

Trying to patch up a filesystem manually ("with a disk editor") is as said a lost cause; first because you are generally not going to be able to know what on the filesystem-level the correct situation was/is after parts of its internal data structures have gone missing due to an e.g. bad sector, and secondly, since if fsck doesn't tell you, you will not even be able to know that there is a problem left anywhere until you per chance run into an e.g. missing or incomplete or corrupted file.

On a block-device level, i.e., the canvas underlying said filesystem, SMART values can be telling as to e.g. bad sectors, and you might be even able to "repair" things by doing a firmware-level format on the drive. In the case of actually bad sectors this "repairs" only in the sense of retiring those sectors and mapping internal spares in their place but perhaps good enough.

If the drive upon connect is say /dev/sdz, you can after sudo apt-get install smartmontools run first of all

Code: Select all

sudo smartctl -A /dev/sdz

to look at the SMART attributes. While theoretically speaking vendor specific, attribute 0 tends to be a read error count; you'd like it at 0, and high values can other than trouble with the drive as such also indicate cable/controller issues.

Attribute 5 tends to be a "reallocated sector count" and on at least HDD something you want at 0 lest the drive has already run out of its internal pool of spare sectors: in that case more are likely imminent, with every one guaranteed to damage something that happens to live there on the filesystem level. Often there's also a "currently pending sectors" count in e.g. attribute 197 that shows sectors that are found to have an issue and are waiting to be retired by the firmware.

SMART also allows for running tests. For example

Code: Select all

sudo smartctl -t short /dev/sdz

runs a short self-test, on HDD on the order of a few minutes. sudo smartctl -l selftest /dev/sdz after those few minutes shows result; sudo smartctl -a /dev/sdz shows all, including both foregoing.

If you find e.g. bad sectors and/or self-test failures it can help to run the drive through a "secure erase", i.e., a firmware-level format. This of course fully empties the drive so only potentially useful after all data-recovery done. For this you first from

Code: Select all

sudo hdparm -I /dev/sdz

note the last part about "Security" features, e.g.

Code: Select all

Security: 
	Master password revision code = 65534
		supported
	not	enabled
	not	locked
	not	frozen
	not	expired: security count
		supported: enhanced erase
	XXXmin for SECURITY ERASE UNIT. YYYmin for ENHANCED SECURITY ERASE UNIT.

If it in your case says "locked" or "frozen" instead of "not" such, suspend your machine with echo -n mem | sudo tee /sys/power/state and wake it up again; another sudo hdparm -I /dev/sdz should then show it not locked/frozen any more.

Note the XXX minutes that the operation is going to take, and set a drive password with

Code: Select all

sudo hdparm --security-set-pass foobar /dev/sdz

Then run a secure erase with

Code: Select all

sudo hdparm --security-erase foobar /dev/sdz

This will hang for those mentioned XXX minutes while the drive is doing its thing; after it verify that the set password has been disabled again with another sudo hdparm -I /dev/sdz ("not enabled" in the above shown section) and repeat looking at SMART attributes and running self-tests.

For any of this see man smartctl and/or man hdparm. Former also shows that you can use

Code: Select all

sudo smartctl -t long /dev/sdz

for a longer test.

In any case all of this concerns the block-device level; at the file-system level you are again basically SoL once fsck feels all hunky-dory. Starting over with a new filesystem, i.e, repartitioning/reformatting the device after potentially the above to make sure the device itself is again semi-valid, tends to be the best you can in a practical sense do.

[WHVW]

rene :

Thank you very much for your great little tutorial; I have printed it out for easy reference. After I am through here, I will download "smartmontools" and follow the Yellow Brick Road.

Thanks once again, (and) I'll post any interesting developments here.

p.s. If they let me in, I'm going to pay A LOT of attention to the man behind the curtain.....

[rene]

Note; I on reread noticed I said hdparm --security-set-password whereas it needs to be hdparm --security-set-pass. Just now edited.