Encrypted Raid1 Recovery

[brcre]

Problem:
The OS is loaded on its own drive.
My home drive is an Encrypted Raid1.
The OS drive failed, while running. Everything froze, couldn't exit, nothing. I restarted the system and powered off.

Code: Select all

Alt PrtScn reisub

I have a new drive now and a fresh install of Mint 21.1
I cannot access my Raid1 drives. Everything I check indicates they are fine and stuff is untouched, but I cannot access the data.

• Original fstab
  /dev/mapper/mint--vg-root / ext4 noatime,errors=remount-ro 0 1
  #/dev/sda2
  UUID=041bfcca-f468-4eec-89fe-84ba0de05fc3 /boot ext2 defaults 0 2
  #/dev/sda1
  UUID=28BB-C74E /boot/efi vfat umask=0077 0 1
  #Swap
  /dev/mapper/cryptswap1 none swap sw 0 0
  #dev/sdd
  UUID=61431159-cb6a-4a69-830e-7e27a979ab3e /tmp ext4 noatime,errors=remount-ro 0 1
  #/dev/md0
  /dev/mapper/0 /home ext4 rw,errors=remount-ro 0 1

• Original crypttab
  sda3_crypt UUID=b91ab150-2dc6-46f6-ac6a-92a70dbde4d4 none luks,discard
  #Swap
  cryptswap1 UUID=60322181-8c0c-43e8-8f13-24367fd4abbe /dev/urandom swap,offset=1024,cipher=aes-xts-plain64
  #home
  0 UUID=d92357f7-1935-49cc-a716-de9f25e9c0b5 /etc/fsdrive.conf luks

• New fstab
  #/dev/sda2
  UUID=7aa945a5-0476-402e-8b9a-dca083c3f0f5 / ext4 errors=remount-ro 0 1
  #/dev/sda1
  UUID=DF0A-0AD6 /boot/efi vfat umask=0077 0 1
  #Swap
  /swapfile none swap sw 0 0

• New crypttab
  #Unknown
  #sda3_crypt UUID=b91ab150-2dc6-46f6-ac6a-92a70dbde4d4 none luks,discard
  #Swap
  cryptswap1 UUID=60322181-8c0c-43e8-8f13-24367fd4abbe /dev/urandom swap,offset=1024,cipher=aes-xts-plain64
  #Raid1
  0 UUID=004d8960-1adb15cd-149755aa-819c3f2a /etc/fsdrive.conf luks

The steps I took to try and mount the Raid1

Code: Select all

sudo apt-get install mdadm

Code: Select all

sudo mdadm --create /dev/md0 --level=mirror --metadata=1.2 --raid-devices=2 /dev/sd[b-c]1

Code: Select all

cat /proc/mdstat

Personalities : [raid1]
md0 : active raid1 sdc1[1] sdb1[0]
7813893120 blocks super 1.2 [2/2] [UU]
bitmap: 0/59 pages [0KB], 65536KB chunk
unused devices: <none>

Code: Select all

sudo mdadm --detail /dev/md0

/dev/md0:
Version : 1.2
Creation Time : Fri Feb 3 18:57:12 2023
Raid Level : raid1
Array Size : 7813893120 (7.28 TiB 8.00 TB)
Used Dev Size : 7813893120 (7.28 TiB 8.00 TB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent

Intent Bitmap : Internal

Update Time : Sat Feb 4 10:17:36 2023
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0

Consistency Policy : bitmap

Name : chimera:0 (local to host chimera)
UUID : 004d8960:1adb15cd:149755aa:819c3f2a
Events : 11205

Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 33 1 active sync /dev/sdc1

Code: Select all

sudo cryptsetup -v luksAddKey /dev/md/0 /etc/fsdrive.conf

Device /dev/md/0 does not exist or access denied.
Command failed with code -4 (wrong device or file specified).

Code: Select all

sudo modprobe dm-crypt

Code: Select all

lsblk

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 232.9G 0 disk
├─sda1 8:1 0 512M 0 part /boot/efi
└─sda2 8:2 0 232.4G 0 part /
sdb 8:16 0 7.3T 0 disk
└─sdb1 8:17 0 7.3T 0 part
└─md0 9:0 0 7.3T 0 raid1
sdc 8:32 0 7.3T 0 disk
└─sdc1 8:33 0 7.3T 0 part
└─md0 9:0 0 7.3T 0 raid1
sr0 11:0 1 1024M 0 rom
sr1 11:1 1 1024M 0 rom

Code: Select all

sudo mkdir /mnt/data

Code: Select all

sudo mount /dev/md0 /mnt/data

mount: /mnt/data: /dev/md0 already mounted or mount point busy.

Code: Select all

sudo mount /dev/mapper/0 /mnt/data

mount: /mnt/data: special device /dev/mapper/0 does not exist.

Code: Select all

cryptsetup luksOpen /dev/md0 crypt1

Device /dev/md0 does not exist or access denied.

Code: Select all

cryptsetup luksOpen /dev/sdb1 crypt1

Device /dev/sdb1 does not exist or access denied.

Code: Select all

cryptsetup luksOpen /dev/sdc1 crypt1

Device /dev/sdc1 does not exist or access denied.

Your thoughts? Anything helps. Thanks in advance

[DisturbedDragon]

The disk is encrypted, you should not be able to access it. If you could, there would be no point in encrypting them. You need the backup key, but since you are asking here, I am guessing you do not have that. Without the key you are pretty well hosed.

[brcre]

What specific data would I need to acquire?

I do know the keyphase I typed in to unlock the drive.
I have another encrypted drive that is not a raid config, but stand alone.
When I plug it in and use "disks" to unlock it, it asks for the pass phrase, I type it and the drive unlocks.
I was expecting the RAID to act the same way.

[rene]

Code: Select all

#/dev/md0
/dev/mapper/0 /home   ext4 rw,errors=remount-ro 0 1

Little weird that that would say 0 1 rather than 0 2 if you didn't edit your setup yourself, and in both that and the overall context my question would be how you originally created this setup in the first place. Don't remember anymore if that was once different but these days RAID isn't offered in the installer. Did you do anything manual/special?

Although yes, your LUKS-header isn't expected to be gone or anything and you would as such need only your passphrase, I am worried that your mdadm --create will have destroyed your existing array: you shouldn't have created a new array buy have assembled the existing one. Now, maybe if it was created in the exact same way as before and no data as such was overwritten that it'll still work but certainly I'm unsure of that.

Let us take things one step at a time: if you boot cleanly, does without doing anything yourself cat /proc/mdstat show that e.g. /dev/md0 has been auto-assembled? If not, try

Code: Select all

sudo mdadm --assemble --scan

and see what you end up with: hopefully an e.g. /dev/md0 comprising your /dev/sd{b,c}1.

If so I suppose you now try to luksOpen that MD device:

Code: Select all

cryptsetup open /dev/md0 0

Success? Do you now have a /dev/mapper/0 again? If so, can you mount it with sudo mount /dev/mapper/0 /mnt/data?

This however is the point where I expect that mdadm --create may have messed you. /dev/mapper/0 is in both your old (and new) crypttab and /etc/fstab and does seem to say you weren't additionally using LVM even if you did for your root partition /dev/mapper/mint--vg-root, i.e., another indication that you once set this up custom. But without LVM the above is expected to be what you once setup. You'd of course preferably have a recollection of what you once did...

[brcre]

Rene,
Attached are the instructions I used to put together the RAID and mount the disks originally.
I also added the top of my install instructions.

Side bar:
I copy the notes forward from the last install and then follow them as closely as possible and update them as necessary so I get a similar install every time.
Anyway...

This is different because its not a planned install, but a recovery and things were not behaving as I'd expect them to.

boot cleanly, does without doing anything yourself cat /proc/mdstat show that e.g. /dev/md0 has been auto-assembled? If not, try

Code: Select all

chimera:~$ cat /proc/mdstat

Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdb1[1] sda1[0]
7813893120 blocks super 1.2 [2/2] [UU]
bitmap: 0/59 pages [0KB], 65536KB chunk

unused devices: <none>

Code: Select all

chimera:~$ sudo mdadm --assemble --scan

[sudo] password:
chimera:~$

I'm considering looking into having the HDD rebuilt/cloned/recovered.
Your opinion...with what you know so far...worth it?

[rene]

Why did you stop where you did? Does sudo cryptsetup open /dev/md0 0 provide you /dev/mapper/0 and can you mount that as instructed?

[brcre]

Code: Select all

chimera:~$ sudo cryptsetup open /dev/md0 0

[sudo] password:
Device /dev/md0 is not a valid LUKS device.

[rene]

There is too much delay between responses: this is an involved subject and a responder then each time has to go back and see what was said/done already and re-think it all again.

I will as such leave it at this but, yes, I do believe that says that your mdadm create will have disturbed the device enough that now recovery is an involved process. I believe TestDisk may work in theory if it can still find the encrypted volume so that might be an option.

[brcre]

I apologize for my slow response.
I've got "life" getting in the way of this drama.
I really appreciate the time you've given to this so far.
I also really appreciate the straight forward action items. Things I can actually do and test.

I guess I'm going to try for a hail mary, but that option will take me months to rebuild the original drive one memory chip at a time. I'm assuming its the device controller that went out. If it is one of the memory chips I'll end up tearing down an unusable drive just to build an unusable device!