big hairy dual-boot mess [SOLVED]

[linx255]

Hi,

I'm using Mint 18 Mate 64 and need a dual boot setup with special requirements and I'm having trouble getting it to work. But before I get into the details of that let me explain where I'm coming from and the whole reason I got into this mess.

Formerly, I had one /boot partition that was shared between the primary OS on /dev/sda5 and a secondary OS on /dev/sda6. Before ever booting into the one on sda6 I modify its fstab to point to the correct UUID. Grub recognizes both OS' on sda5 and sda6 and allows me to boot into either with no problems.

This setup worked fine until I needed each OS to be able to boot with different kernels. The reason for this is I need to test a new kernel over a period of months while having the old kernel available for booting into on the other-- in the event the newer one freezes / inoperability. I couldn't simply snapshot /boot before installing a kernel, test a new kernel, and restore the old /boot snapshot in the event of problems because I'd be left without a bootable machine ( besides LiveUSB, which isn't productive enough since it doesn't save my customizations ). So it seems each OS needs its own /boot ( in my case, their own /boot partitions ), and recently some of you told me sharing /boot was a bad idea.


My special requirements are:

A) I still need the two file systems synced ( via scripts that I execute when I make a change to one and want to boot into the other with those changes carried over ) so they're identical and either can be used with the same setup and relied on for work / productivity if the other is down or in need of maintenance.

B) I didn't want to have to manually select kernels from the advanced menu in Grub every time I boot up so my idea was to have grub.cfg manually rigged to boot each with a different kernel of my choosing. Yes, I know everyone says not to modify grub.cfg but how else do I get it configured the way I want? It doesn't generate the file with my setup in mind. ( However, I need to re-assess my whole strategy before I start researching how to customize grub.cfg the right way. )


My initial idea was to just re-partition and have two /boots ( sda1, sda2 ), with the MBR pointing to sda1 ( with boot flag ), the OS on sda5 pointing to /boot on sda1 in fstab. The other /boot on sda2 ( with no boot flag ) is only referenced by grub menu but its config file isn't referenced or accessed by MBR, but the OS on sda6 points to sda2 in fstab.

So I backed up all my original file systems and data, including original /boot and proceeded to re-partition the whole drive as follows:

/dev/sda1 primary /boot
/dev/sda2 secondary /boot
/dev/sda3 extended
/dev/sda5 primary OS
/dev/sda6 secondary OS
/dev/sda7 data

Then I synced my backed up file systems back to sda5 and sda6, set the UUIDs of each, modified fstab in each OS to point to the correct UUIDs, and finally modified grub.cfg in /boot on sda1 to point to the correct UUIDs. ( By the way, I always use this file system syncing, fstab modification, UUID setting method for system maintenance and have never had problems, strange as my method may. I don't recall ever modifying grub.cfg before, however. )

Popped knuckles and... Upon selecting the primary OS on sda5 to boot, it said in a text-only resolution "Missing Operating System" for a second , then loaded the Mint splash screen with the animated dots, and never loaded the login screen . F1 revealed no error messages; it just said it was searching for Btrfs ( or something like that ) and that /dev/sda5 was clean.

I found no way to debug the "Missing Operating System" error nor the failure to load the login screen. I thought everything was in place, and now I'm questioning my whole approach again.

All I can do now is restore my original parition scheme and grub.cfg and cross my fingers that I'll have my old setup back. But I must attempt to get my new setup to work so the primary OS boots with the old kernel and the secondary boots with the new, whether that's having one /boot that works with my original setup and a customized grub.cfg or two /boots and whatever else makes it work.

I need help finding the path forward for my needs, or if I'm in the right direction but made mistakes then I need to troubleshoot what went wrong with the "Missing Operating System" message and hanging at Mint logo splash screen, and/or how do configure grub.cfg the kosher way while meeting my requirements. Better ideas for implementation are welcome, however, the two requirements above remain.

My machine is currently only running via LiveUSB so my customizations are absent and productivity ( business and personal ) is hence severely lacking. I hope this can be resolved within a few days because I'm not able to do a lot of essential things.

Thank you very much!

[linx255]

UPDATE: I just noticed I forgot to assign the correct UUID in fstab to my newly created data partition. Betting this caused problem, but now that the whole day was burned and I already restored my original setup I'll probably have to wait til next weekend to try again. However, I'd still like to hear what you have to say.

[linx255]

Alright, success!

Now I have:
/boot on /dev/sda1 ( has boot flag )
/boot on /dev/sda2 ( no boot flag )
Mint on /dev/sda5, linked to /boot on /dev/sda1
copy of Mint on /dev/sda6, linked to /boot on /dev/sda2

MBR points to /dev/sda1.
Grub allows me to boot in /dev/sda5 or /dev/sda6 after manually modifying the UUIDs in grub.cfg to point to the correct /boot for /dev/sda6.
Grub on /dev/sda2 is unused and irrelevant.
Fstab on each copy of Mint is modified to point to the correct UUID for /boot and / as well.
And...

Everything works fine.
The first boot I got a brief message: "Missing Operating System" but it boots into either copy fine and I find no other problems yet.
Why would it say that when clearly the operating system is present and runs perfectly?
Does it not like that I have two /boot with only one having a boot flag?
Initial research yielded nothing but will try again tonight.
Anyway, I haven't seen the message again.

Now that I have independent /boot directories for each OS I can test a new kernel on one /boot and not the other while still syncing the rest of the OS.
I'd appreciate any feedback, particularly pertaining to the safety and efficacy of this configuration.
I think all my bases are covered except for manually modifying grub.cfg.
If any problems arise as a result of modifying grub.cfg my plan is to simply allow the system to re-generate it when it needs and then modify it again.
To prevent crashing related to grub.cfg I've made it immutable.
If any update needs to change grub.cfg, it should give me an error message, in which case I'll un-immute it, retry the update, and modify grub.cfg afterward.
No big deal right?
I'm maintaining multiple version of backups of OS; should be able to restore if anything catastrophic happens, and I can always go back to the way I had it before, including my previous configuration with one /boot on /dev/sda1.

Thanks

[fabien85]

My opinion is that you made a very complicated setup (that I didnt really follow to be honest) and you may want to consider the following option to simplify your life :
timeshift
This is the number 1 option for backup and system snapshots in Mint. It's integrated by default since Mint 18.3, and can be installed from the repositories for Mint 17-17.3 and 18-18.2.
It can snapshot the whole system and allows you to restore it at any moment. A life saver.

[linx255]

My setup is hard to explain, but it's simple to me.

Timeshift doesn't work when the OS is unbootable.
I take regular snapshots of my OS without needing any application.
My maintenance scripts can take care of everything even if the OS becomes unbootable. ( Hence dual boot setup )
If timeshift can be configured to run on any OS or LiveUSB drive and select the target drive (as opposed to the current drive) to restore and it takes into account that I store /boot on /dev/sda1, then I'll consider using it.
However, my scripts are so efficient and customized for my setup I probably don't need anything else.

My setup simply requires two identical OS copies, each with its own / partition and /boot partition.
( The master boot record points to /boot on /dev/sda1 which is linked to the OS on /dev/sda5 and the other /boot on /dev/sda2 never boots but is linked to /dev/sda6.
Grub is custom configured to allow me to boot into either OS, however, by referencing the UUIDs, and the OS on /dev/sda6 never knows that /boot on /dev/sda2 is unused except to determine which kernel to run ( AFAIK ). I hope you followed all that.

/boot for the secondary OS is just there so that I don't have to share /boot across multiple OS' which would create two problems:
1) If corrupted neither OS' would be bootable, where as if just /boot on the second OS is corrupted the first is still in tact and bootable.
2) I wouldn't be able to run my current working kernel on the primary while testing a new kernel on the secondary without having to specify which kernel to boot with in Grub, which is a hassle.

This setup allows me to perform maintenance on one copy from the other all the while still having a environment conducive to my work, and avoiding using LiveUSB unless even Grub doesn't boot, in which case LiveUSB is the only way to boot.
If I had only one OS and it became unbootable I'd have to pull out the LiveUSB drive to restore it, which is a huge pain and inconvenience because of security procedures, opening the chassis, change some jumpers and BIOS settings, and worst of all the default Mint settings are not conducive to productivity for me.
With two OS' I have a lifeboat OS built-in my internal hard drive and down-time is minimized, while my work environment and productivity are maintained.

Figuring out how to do this was complicated but now that it's done and seems to work it should be smooth sailing, and now I don't have the headaches I once did.

[lsemmens]

Rather than stuff around with Grub, could you not set up a system whereby each OS resides on a separate disk and, when needed, just change the Boot order in BIOS. All your data could reside on a third disk which may, or may not necessarily be physically installed in your system, or connected via USB.

[linx255]

No because my machine is a compact fanless PC with only space for one physical hard drive and USB drives aren't fast enough. Also, changing boot order on the fly on a regular basis isn't practical especially in this setup/environment. My goal is not to have to mess with anything hardly ever and everything needs to be mostly automated ( not do anything more than click a button to activate certain tasks to automate maintenance ). Although, ideally, yes, I'll explore having a larger machine that could house multiple physical drives when/if funds permit! ( I'm regretting having a compact PC for many reasons. )

Anything less than that severely interferes with productivity. Luckily, I've had time to develop this setup since it's gotten to where I won't have time for much more hassles like this, and besides occasional natural OS crashes and recovery via scripts from the other working OS I think I've mostly achieved my goal as much as I can. So far what I got's working great.

If grub.cfg becomes a bottleneck for my goal, such as changing in ways I don't want after an update, I'll have to make a script modify the menu entries ( leaving everything else alone ), which can be run prior to every shutdown, so that Grub doesn't break access to either OS ( try to assign the wrong UUIDs to the wrong entries ). It would be nice if grub, fstab, and gparted could all work together and be edited in one GUI with tabs and some graphical switches and fields that re-code grub and fstab for you so that all UUIDs would all be synchronized across these config files and partitions, and grub menu entries would be solely under the control of the user and not some update that re-writes them for apparently no good reason. ( If it's going to scan for bootable OS' and make menu entries it should ask me if the configurations it picks are correct and if I want to modify them. I shouldn't have to become a grub guru to maintain menu entries, IMHO. ) However, I've long come to accept that using Linux means workarounds in order to get what I want.

[fabien85]

If you are happy with your setup, then I dont have any advice.

I will just share some of what I'm doing, which may or may not give you ideas for future
- as bootloader I'm using refind
the feature that is an advantage in this context is that refind auto-detects all bootable OSes. So there is no need to maintain a config file like grub.cfg which should change everytime you change your OS setup. Also the thing doesnt break if you wipe any OS, since it lives independently of them.
The catch is that to use refind you have to boot in EFI mode, not the legacy mode (with an MBR) that you are using.
- I do have a backup OS on another partition of the drive. To distinguish the two OSes, I have labeled their partition, and refind shows the partition labels. (also the OSes dont run the same kernel series so I can distinguish them like that)
I dont mirror files on my main OS to the backup one though, because I choosed timeshift as my backup solution
- my home directory is on a separate partition and shared between both OSes
- my main OS is backed up automatically by timeshift to another partition
if the main OS would be unbootable (never happened yet) I could boot the backup OS and use timeshift to restore the main OS to one of the last save points.
- I have yet another backup to a ssh server via "Back in Time", in order to have something in a different physical location

in total it's a bit complicated because it grew with time (at first I had neither timeshift nor a backup OS, just Back in Time), but I used only solutions available in the Mint repositories.
If I had to redo it from scratch, maybe I would do it differently. I miss the possibility to do backup via ssh in timeshift though (maybe it's possible, I haven't looked in details) ; it's important to have a backup on a different drive (hard disk fail) and a different location (physical security, e.g. in case of theft).

[linx255]

Thanks.

I'm refusing to use EFI because I think it's dumb, overcontrolling, and not really that much more secure than BIOS. I'm waiting for Coreboot or some open-source alternative to BIOS to start being compatible and hopefully my next machine will boot off that.

I used to use labels then switched to UUIDs because of some problems.

The primary purpose of my backup OS is to restore the primary. The secondary purpose is to be a perfect mirror, otherwise it's not nearly as useful.

Hm... Sharing home directory between OS'... Well, if the home directory gets corrupted then both OS' are affected. Granted, I have backups. I'm just trying to prevent failure, downtime, and work of course.

My OS' are backed up to another partition as well, however, shared with other data. Hm... I wonder if I should make a 2nd data partition just for OS backups though.

It seems like I read about many ways ssh security is broken; I don't think having that extra attack surface would work for me, though it would eliminate the attack surface of only having one physical location. If the data could be end-to-end encrypted in real-time with some pgp/axolotl engine, then we'd really be talking.

My scripts also backup / and /boot partitions independently, though always synchronously. That way I can restore independently, which I've done when /boot got corrupted on numerous occasions.