[SOLVED] I am in a muddle with gunzip and cannot decompress a backup to an external ssd.

Quick to answer questions about finding your way around LMDE as a new user.
Forum rules
There are no such things as "stupid" questions. However if you think your question is a bit stupid, then this is the right place for you to post it. Stick to easy to-the-point questions that you feel people can answer fast. For long and complicated questions use the other forums within the support section.
Before you post read how to get help. Topics in this forum are automatically closed 6 months after creation.
Locked
User avatar
medea
Level 1
Level 1
Posts: 41
Joined: Tue May 30, 2023 7:48 am

[SOLVED] I am in a muddle with gunzip and cannot decompress a backup to an external ssd.

Post by medea »

Hello All Minty Persons, This is my first post.

I've been using Linux Mint (LMDE debian) as my only driver for a year now and have honestly never been happier with an OS in my life, so thank you everyone here who has contributed to that (developers, admins, forum q&a particpants). Being windows-free is such a liberating experience, and without you (all and collectively) I could never have known that feeling. You are doing truly good work.

My problem in a sentence.

I followed a tutorial on how to compress the contents of an ssd into a gunzip backup file (which semed to work), but on follwing the instructions to restore it by uncompressing the gunzip backup back onto the ssd, it never seems to work (I am now wondering if it is me or the tutorial is outdated/wrong).

My problem in detail.

1. I formatted a brand new 500gb ssd to exfat (using Linux disks app) to be able to use it as an external harddrive connected via usb (using one of those cases with cables made for the purpose) .
This seemed to work fine. I chose exfat in case I ever needed to access it via windows.

2. I put about 150gb of films on it and all seemed fine. Once transferred from my Linux ext4 computer to the exfat 500gb ssd (connected via usb) I double-checked and all files had transfered ok.

3. This is where problem happened.
I had an old windows 7 64bit system which had a few films on I wanted to grab, so I connected the usb external ssd.
As soon as I did this, I saw a notification that windows was installing a driver on it. It said it had done so successfully, but when I tried to open the external ssd it showed empty.
(I have found out since that windows 7 doesn't recognise exfat, so maybe it wrote over or corrupted the exfat driver.)
Guessing that windows installing drivers had messed something up, I shutdown and went back to my linux system to see if I could still access my 150gb of files.

4. As soon as I connected it to linux via usb and tried to access it I got a error pop up saying that the drive was corrupted due to a bad block (sorry I can't post that pop up error in full here, but I didn't save it as I thought I could find fix this problem on my own).
I couldn't access it at all, so I went to the webs to find a solution.

5. This is what I decided would be a good tutorial* on how to fix it (a recent tutorial on an established forum with replies saying it worked).
https://www.maketecheasier.com/repair-c ... ive-linux/
*On revisiting the page today the tutorial seems to be missing step 4, but I'm not sure of that was the case when I followed the tutorial.

6. I followed 'Take a Compressed Full Backup Image' instructions and it created a compressed gunzip backup file of around 120gb in my home folder (supposedly the compressed 150gb of files on the 500gb external exfat ssd drive). So far so good.
I then tried the 'fix corrupt block' FSCK instructions, but that didn't work.
So it seemed the only option was to wipe and reformat the external ssd drive to exfat (which I did using linux disks) then restore my 150gb of compressed files to my reformatted exfat ssd from the gunzip backup.

7. This is where I am stuck. No matter what I do I can't get that compressed gunzip backup to uncompress back onto the drive.
This was my first time using gunzip and command line instructions, and though I think I followed everything to the letter, something is not right.
I have searched the webs and tutorial videos and tried various ways around this (as far as my abilities allow me) but it is just beyond me.

The command line instructions I used here are exactly as shown in the tutorial linked, I just substituted my computer and external drive names and locations to suit my system.

What the problem may be (my limited understanding).
a) User error. I try my best but I am not a trained tech anything, so I may just be doing it wrong.
b) When I compressed the files using gunzip, I would have been compressing files on an exfat drive (external ssd) to a backup on an ext 4 linux OS. Maybe I missed a step to make this work properly.
c) The tutorial is outdated and/or wrong.

Final notes.
A GUI option to make this compressing/decompressing process clearer/simpler would have really helped. I am autistic and often get muddled by written instructions, as a consequence I always do better if I can *see* what I am trying to do laid out in front of me in a simple visual format (a flow chart for example). This helps me mentally follow the order in which the system works (a >b>c etc)
Command line instructions are very text heavy, often producing long strings of text when enacted that makes no sense to me, so this may be a big reason I am getting lost-stuck trying to do this.

This is already long post and I am wary of putting people off helping by writing too much.
If you need any more information please ask (eg. I kept notes of the exact instructions I used to compress and (try to) decompress).

A huge thank you to anyone who can offer ideas, suggestions, instructions, links, as to how I can fix this.

Medea.
Last edited by LockBot on Thu Nov 30, 2023 11:00 pm, edited 3 times in total.
Reason: Topic automatically closed 6 months after creation. New replies are no longer allowed.
be a good animal.
User avatar
Coggy
Level 5
Level 5
Posts: 639
Joined: Thu Mar 31, 2022 10:34 am

Re: I am in a muddle with gunzip and cannot decompress a backup to an external ssd.

Post by Coggy »

So I guess you have a file called something like flashback.img.gz on your Linux computer. I'm not convinced that an image of a corrupt disk is all that much use to be honest. If you do restore the image, you will just get your corrupt disk back.

It seems that you are trying to restore this corrupt image. Step 5 should restore the backup to the disk. But the id will have changed if you formatted it in the meantime - you will need to review /dev/disk/by-id and figure out the new id to write the image back to. Then you still have to find some way of fixing the corrupt disk. Recovering images from a corrupt filesystem can be tricky.
Why do you think you have not managed to restore the image to the disk? Any error message?

If you do manage to restore the image, the disk will change its id back to what it was before. unplugging and re-plugging the disk will refresh the /dev/disk/by-id entry and show if the id has changed back again.
rene
Level 20
Level 20
Posts: 12212
Joined: Sun Mar 27, 2016 6:58 pm

Re: I am in a muddle with gunzip and cannot decompress a backup to an external ssd.

Post by rene »

As Coggy said, plus: when I follow that tutorial link now, both step 4 (backup) and 5 (restore) use gzip -c which is wrong: step 5 should use gunzip -c instead.
User avatar
medea
Level 1
Level 1
Posts: 41
Joined: Tue May 30, 2023 7:48 am

Re: I am in a muddle with gunzip and cannot decompress a backup to an external ssd.

Post by medea »

@Coggy »
So I guess you have a file called something like flashback.img.gz on your Linux computer.

Hey Coggy, many thanks for the reply.
Yes, I've added more details below that show that, plus the command lines I tried.

If you do restore the image, you will just get your corrupt disk back.

I haven't got that far, so impossible to say until I at least get something back.

But the id will have changed if you formatted it in the meantime - you will need to review /dev/disk/by-id and figure out the new id to write the image back to.

I have tried to address that (see below) by choosing the same external ssd name when I reformatted, and then using ls command to get partition information.
Whether I adapted this information properly, I can't say. I tried to, but it didn't work.

Why do you think you have not managed to restore the image to the disk? Any error message?

Because at the end of the command process (copied out in another post below) my destination ssd external drive was totally empty.
As the gunzip image is about 120gb, and (in theory anyway) should contain a compressed image of about 150gb of film files (which I backed up with gunzip from the external drive with the bad block in the driver), extracting it onto the external ssd should put those 150gb files on it. Unless I completely misunderstand the process.

If you do manage to restore the image, the disk will change its id back to what it was before.

If I could get that far and recover my compressed files, I don't mind what the ssd id becomes. I would just like to get that far.
be a good animal.
User avatar
medea
Level 1
Level 1
Posts: 41
Joined: Tue May 30, 2023 7:48 am

Re: I am in a muddle with gunzip and cannot decompress a backup to an external ssd.

Post by medea »

@rene »
As Coggy said, plus: when I follow that tutorial link now, both step 4 (backup) and 5 (restore) use gzip -c which is wrong: step 5 should use gunzip -c instead.

Thank you for the reply, Rene.

That is an interesting twist. I will give that a go as soon as I can clear space on my OS to put the gunzip image (120gb) back.
As I only have a 240gb ssd for my OS I just copied it (still compressed) to the external 500gb ssd (until I can find a way to uncompress it properly) to free up space for other things.

Do you think that is an error in the original tutorial (as people had replied that it it worked for them, I felt confident trying it)?
Or is it because the tutorial is for Ubuntu and Debian systems use it differently?
be a good animal.
User avatar
medea
Level 1
Level 1
Posts: 41
Joined: Tue May 30, 2023 7:48 am

Re: I am in a muddle with gunzip and cannot decompress a backup to an external ssd.

Post by medea »

Some Important system info that (5) of the 'How To Get Help' guide says to include:
I am using Linux Mint Debain LMDE OS (all updated) on a 64bit system with Cinnamon DE.

As for exact command line instructions I used, this is what I first used:
(*I have anonymised names and serial numbers (using capital letters) to avoid accidentally compromising personal data.)

ANYONE@computer:~$ sudo gzip -c /home/ANYONE/backups/flashback.img.gz | sudo dd of=/dev/sdb status=progress

And this is what the command line said it did:

[sudo] password for ANYONE:
123080835584 bytes (123 GB, 115 GiB) copied, 10973 s, 11.2 MB/s
240392352+1 records in
240392352+1 records out
123080884235 bytes (123 GB, 115 GiB) copied, 10973.7 s, 11.2 MB/s
ANYONE@computer:~$

Which looks like it did something (it took 3 hours to run), but when I opened the 480 external drive I thought I had uncompressed the image zip to, it was empty.

I wondered if the mistake was happening because I was using the drive name, so I doublechecked using the ls command to find out if my external ssd name was something else. This is what it said:

s /dev/disk/by-id
result
ANYONE@computer:~$ ls /dev/disk/by-id
ata-HL-DT-ST_DVD+_-RW_LONGNUMBER1
ata-Samsung_SSD_850_EVO_250GB_LONGNUMBER2
ata-Samsung_SSD_850_EVO_250GB_LONGNUMBER2-part1
ata-Samsung_SSD_850_EVO_250GB_LONGNUMBER2-part2
ata-Samsung_SSD_850_EVO_250GB_LONGNUMBER2-part3
ata-SanDisk_SSD_PLUS_480GB_LONGNUMBER3
wwn-0xLONGNUMBER4
wwn-0xLONGNUMBER5
wwn-0xLONGNUMBER6
wwn-0xLONGNUMBER6-part1
wwn-0xLONGNUMBER6-part2
wwn-0xLONGNUMBER6-part3
ANYONE@computer:~$

I think that means I was originally using a name instead of the ssd number (and the ssd number is what I should use).
So then I tried using LONGNUMBER3 instead of the ssd name for the destination external ssd, like this:

ANYONE@computer:~$ sudo gzip -c /home/ANYONE/backups/flashback.img.gz | sudo dd of=/dev/disk/by-id/ata-SanDisk_SSD_PLUS_480GB_LONGNUMBER3 status=progress

And this is what happened in the command line:

ANYONE@computer:~$ sudo gzip -c /home/ANYONE/backups/flashback.img.gz | sudo dd of=/dev/disk/by-id/ata-SanDisk_SSD_PLUS_480GB_LONGNUMBER3 status=progress
[sudo] password for ANYONE:
123080540672 bytes (123 GB, 115 GiB) copied, 3368 s, 36.5 MB/s
240392352+1 records in
240392352+1 records out
123080884235 bytes (123 GB, 115 GiB) copied, 3368.76 s, 36.5 MB/s
ANYONE@computer:~$

So exactly the same result as before (process runs but nothing appears on destination ssd when process completes).
In sum, both variants of the commands I tried gave the impression (like a 'dummy run') that they are decompressing the gunzip image to the external ssd external drive, but when the process finishes and I open the destination drive there is nothing there (which is frustrating after the hours long wait).

As I said in the first post, I have tried many different permutations of the uncompress instructions given in the tutorial, but they all end up with the same results as above (I don't want to bore anyone trying to help me with this by overloading my replies with all my other attempts).

Some other thoughts.
I have peazip, which I know how to use to extract rar files, but I am not sure if it could help in this situation?

Gunzip seems to be a command line only option, which I really wish I had never gotten into now, as it seem very picky about every grammatical and syntactical element, which makes it a minefield for newcomers trying to adapt other's command line instructions to their own systems.
As mentioned before, I do better with a GUI with clear binary options 'do it/undo it' and a big button to press to make it all happen.
99% of the time Linux Mint offers this (or super simple command line instructions), which I'm sure is a big reason why members here like it so much as a system.

But gunzip commands seems to be one area Mint hasn't addressed so far with a GUI solution (it may exist, but I haven't found it despite much searching).
As such, I often feel I am 'poking around in a bucket of sand with a stick and hoping for the best' when using the command line, which probably goes a long way to explaining any glaring errors in my self-adapted instructions I posted above.
I am just trying my best to find a solution for myself with the info/tools I have found.

Thank you if anyone can tell from these failed commands what I am doing wrong.
be a good animal.
rene
Level 20
Level 20
Posts: 12212
Joined: Sun Mar 27, 2016 6:58 pm

Re: I am in a muddle with gunzip and cannot decompress a backup to an external ssd.

Post by rene »

medea wrote: Thu Jun 01, 2023 10:13 am

Code: Select all

ANYONE@computer:~$ sudo gzip -c /home/ANYONE/backups/flashback.img.gz | sudo dd of=/dev/sdb status=progress
This compressed again your existing and already compressed disk image /home/ANYONE/backups/flashback.img.gz and then wrote that doubly compressed file to /dev/sdb. Yes, it's a simple matter of an error in the tutorial. What you want instead is

Code: Select all

ANYONE@computer:~$ sudo gunzip -c /home/ANYONE/backups/flashback.img.gz | sudo dd of=/dev/sdb status=progress
For good measure, use sync after it completes, and certainly before you remove/replug the drive.

As Coggy said this will restore any and all to /dev/sdb including partitioning and including the filesystem errors that will still keep you from mounting the drive and copying off the mentioned video files (although fortunately you are using a direct /dev/sdb-type specification so that you'll at least not have to deal with changing UUIDs -- but be sure that /dev/sdb is still the correct specifier when you do this). I.e., you'd be back at the start.

exFAT is not a good idea in Linux. Please first of all try your best to locate a Windows 10/11 system to then CHKDSK the drive from: it has much better chance of fixing things. Only when absolutely not possible should you after restoring the original situation as per above try and muck about with a (partly) closed and patented, i.e., practically and/or legally unimplementable by Linux, filesystem such as exFAT from Linux.
User avatar
medea
Level 1
Level 1
Posts: 41
Joined: Tue May 30, 2023 7:48 am

Re: I am in a muddle with gunzip and cannot decompress a backup to an external ssd.

Post by medea »

A big thank you for the reply Rene (and sorry for delay in my reply but mintforums seemed to be offline this week whenever I tried to reply).

To get my confidence up in using gunzip to compress from usb drive to backup, then uncompress from backup to usb drive (confidence couldn't be lower following my repeated failures with this so far), I am going to experiment with gunzip compressing some simple files from a usb key to backup img, and then uncompressing back.
And, only once I can do that right, try to restore the big 480gb one (because that takes hours to run).

Just to be super-sure of my approach before i start, are these two commands definitely the compress/uncompress commands to do that (substituting drive details where appropriate)?

1) to compress files from usb key to gzip compressed backup file in my homefolder:
ANYONE@computer:~$ sudo dd if=/dev/disk/by-id/dev/sdb status=progress | gzip -c > /home/ANYONE/backups/flashback.img.gz

2) to uncompress files from compressed backup file in my homefolder onto my usb key:
ANYONE@computer:~$ sudo gzip -c /home/ANYONE/backups/flashback.img.gz | sudo dd of=/dev/sdb status=progress

That seems to me to be the ying/yang of it.

Re sync.

For good measure, use sync after it completes, and certainly before you remove/replug the drive.

This may make me seem naive, but I have never used sync before. So I will have to look into what sync is and does and do as you say once I have a grasp of the idea (any reliable beginner-friendly links welcome).

Re restoring bugs along with files and exfat.

As Coggy said this will restore any and all to /dev/sdb including partitioning and including the filesystem errors that will still keep you from mounting the drive and copying off the mentioned video files (although fortunately you are using a direct /dev/sdb-type specification so that you'll at least not have to deal with changing UUIDs -- but be sure that /dev/sdb is still the correct specifier when you do this). I.e., you'd be back at the start.

Once restored, I will (as you advised) ask someone with Win10/11 if they will let me do a checkdisk on that system, and go from there. If it works, great. If not, well, at least I tried.

exFAT is not a good idea in Linux. Please first of all try your best to locate a Windows 10/11 system to then CHKDSK the drive from: it has much better chance of fixing things. Only when absolutely not possible should you after restoring the original situation as per above try and muck about with a (partly) closed and patented, i.e., practically and/or legally unimplementable by Linux, filesystem such as exFAT from Linux.

From my reading prior to formatting my external usb drive, i was of the impression that exfat was considered the best option for compatibility for usb external drives that may move between linux and windows worlds (most compatible across systems fat32 has 4gb limit per file, ntfs windows only, ext 2,3,4 linux only). What do you recommend as an alternative?

On that theme, as i understand it, in this case i will have to uncompress my gunzip backup image onto an external usb drive of the same format as it was originally compressed from (i may be misunderstanding this).
As the backup image was compressed from an exfat drive, it will therefore need to be uncompressed back onto an exfat drive again (just to get my files back, at least).

As a more general thought, after the drama (and potential data loss) of this episode, I am now considering keeping my usb drives strictly linux OS compatible only (so one of ext 2, 3, 4) going on.
Though not sure if same corruption would still arise (windows auto-forcing its own drivers onto the usb drive on insertion, as i understand it, and corrupting existing drivers) if using ext 2,3,4 instead of fat.
Windows certainly is a selfish and thoughtless brute!

Thank you again for any help you (or anyone who wants to contribute thoughts on this) can suggest.
be a good animal.
rene
Level 20
Level 20
Posts: 12212
Joined: Sun Mar 27, 2016 6:58 pm

Re: I am in a muddle with gunzip and cannot decompress a backup to an external ssd.

Post by rene »

Don't have much time currently, so only briefly...
medea wrote: Sat Jun 10, 2023 1:59 pm Just to be super-sure of my approach before i start, are these two commands definitely the compress/uncompress commands to do that (substituting drive details where appropriate)?

1) to compress files from usb key to gzip compressed backup file in my homefolder:
ANYONE@computer:~$ sudo dd if=/dev/disk/by-id/dev/sdb status=progress | gzip -c > /home/ANYONE/backups/flashback.img.gz
No, not if=/dev/disk/by-id/dev/sdb, but -- if that's the device specifier it got upon inseruion; check in e.g. Disks -- but just if=/dev/sdb (the -c parameter to gzip is fine but you can also leave it out).
medea wrote: Sat Jun 10, 2023 1:59 pm 2) to uncompress files from compressed backup file in my homefolder onto my usb key:
ANYONE@computer:~$ sudo gzip -c /home/ANYONE/backups/flashback.img.gz | sudo dd of=/dev/sdb status=progress
No, you are wanting to unzip the previously zipped data so (once again) want gunzip -c there. You more over don't need to prefix that part with sudo.

Will have more time later.
User avatar
medea
Level 1
Level 1
Posts: 41
Joined: Tue May 30, 2023 7:48 am

Re: I am in a muddle with gunzip and cannot decompress a backup to an external ssd.

Post by medea »

Thank you Rene, you genius! That's it, I can do it with the help of those final tweaks to the commands.

I have twice compressed files on my usb key (usb key containing just a single mp3 file) to a compressed backup, and back successfully.

After creating the backup I used disks to delete the partition, create a new partition, then format the partition, remembering to mount the usb key after formatting (the first time I tried I forgot, so I learned from that).

Btw the usd key identifier is /dev/sdb1, hence the difference between that and the /dev/sdb in the original instructions we began with (which referred to my 480gb usb external drive, which has the corruption problems).

I wanted to experiment with getting the process of compressing and uncompressing the usb key absolutely watertight before going back to try it on the 480gb usb drive, as compressing/uncompressing data on that takes hours to run through.

Anyway, I will present it below in a form that anyone else who gets stuck with this process as I have (resulting in several nervous breakdowns and tantrums) can adapt to their own system, just by changing the computer name from ANYONE to their computer name, and changing the /dev/sdb part to whatever their usb drive is called (in disks, name under device location).

So here goes...

ME TRYING THE ABOVE ADVICE (THANKS TO RENE) - IT WORKS!
(make sure usb key is mounted after deleting contents and reformatting!)

COMPRESSING USB KEY TO BACKUP.

command I used to compress usb key to backup:
sudo dd if=/dev/sdb1 status=progress | gzip -c > /home/ANYONE/backups/flashback.img.gz

got this:
ANYONE@computer:~$ sudo dd if=/dev/sdb1 status=progress | gzip -c > /home/ANYONE/backups/flashback.img.gz
[sudo] password for ANYONE:
7983141376 bytes (8.0 GB, 7.4 GiB) copied, 375 s, 21.3 MB/s
15630336+0 records in
15630336+0 records out
8002732032 bytes (8.0 GB, 7.5 GiB) copied, 375.904 s, 21.3 MB/s
ANYONE@computer:~$

result:
8gb usb key compressed to 8gb folder (called flashback.img.gz) in backups containing 3.7gb compressed file.

UNCOMPRESSING BACKUP TO USB KEY

command I used to uncompress backup to usb key:

gunzip -c /home/ANYONE/backups/flashback.img.gz | sudo dd of=/dev/sdb1 status=progress

got this:
ANYONE@computer:~$ gunzip -c /home/ANYONE/backups/flashback.img.gz | sudo dd of=/dev/sdb1 status=progress
[sudo] password for ANYONE:
7996621312 bytes (8.0 GB, 7.4 GiB) copied, 738 s, 10.8 MB/s
15630336+0 records in
15630336+0 records out
8002732032 bytes (8.0 GB, 7.5 GiB) copied, 738.761 s, 10.8 MB/s
ANYONE@computer:~$

result:
files successfully uncompressed from backup to the usb key, and compressed file still in backups (so can keep or delete) thanks to -c command.


I hope the above generic template for gzip and gunzip commands helps someone who is as stuck as I was with this process, as Rene's (and Coggy's) help got me to a stage where I could even do this at all.

And thank you again to Rene and Coggy for taking the time to consider my problem and support and advise me through this thorny hell-command of a process.
be a good animal.
User avatar
medea
Level 1
Level 1
Posts: 41
Joined: Tue May 30, 2023 7:48 am

Re: I am in a muddle with gunzip and cannot decompress a backup to an external ssd.

Post by medea »

- this post is a kind of post-script and loose-end aside to the main issue of compressing and uncompressing with gzip gunzip -

@Rene

exFAT is not a good idea in Linux. Please first of all try your best to locate a Windows 10/11 system to then CHKDSK the drive from: it has much better chance of fixing things. Only when absolutely not possible should you after restoring the original situation as per above try and muck about with a (partly) closed and patented, i.e., practically and/or legally unimplementable by Linux, filesystem such as exFAT from Linux.

I will do this, though it may be a week or so until I can access a win10/11 system as I only have an old win7 (64bit) here at home.
As you are recommending win10/11 I assume this is because win7 does not recognise the exfat format (which the restored corrupted disk image will be).

But if you have a recommendation for a better cross linux/windows format (which I thought exfat was) that you think I should use instead of exfat going on, I would be very grateful to hear your opinion.

Alternatively, if I decide to try to keep linux usb drives and windows usb drives separate from now on (though muddles happen, which is why choosing a cross system format seems tempting) which of ext2, ext3, ext4 is best for external usb drives?
I use my usb drives mostly to back up media, lots of files sizes from 5mb single songs to 5gb HD films, for example.

Thank you for any thoughts and suggestions.
be a good animal.
User avatar
axisofevil
Level 4
Level 4
Posts: 388
Joined: Mon Nov 14, 2011 12:22 pm

Re: [SOLVED] I am in a muddle with gunzip and cannot decompress a backup to an external ssd.

Post by axisofevil »

+1 on the exfat thing.
I'd recommend that you go for ext4 - its pretty standard.

Worth bearing in mind that video files are normally highly compressed, so attempting to use compression won't reduce the file size significantly.
rene
Level 20
Level 20
Posts: 12212
Joined: Sun Mar 27, 2016 6:58 pm

Re: [SOLVED] I am in a muddle with gunzip and cannot decompress a backup to an external ssd.

Post by rene »

Warning. If you used (as you said) dd if=/dev/sdb [etc.] to backup then you need to also use dd of=/dev/sdb [etc.] to restore, i.e., not /dev/sdb1 (and again only after making sure that /dev/sdb is the correct device specifier).

Explanation is:

1. A block-device is "a byte-canvas" only; something that can store and retrieve blocks of bytes at a given block-address (historically and/or as far as software is concerned "a block" tends to be 512 bytes). In this case this "device" is what would be referred to as e.g. /dev/sdb.

2. A partition of any such device is a range of block-addresses between some given starting and ending block-number, as stored in a table -- the device's partition table -- on the device itself, normally at the start of the device. This is what would be referred to as e.g. /dev/sdb1, /dev/sdb2, etc.

3. A filesystem tends to live on a partition (but does so sometimes directly on the device itself) and is no other than a bunch of blocks used by the filesystem for housekeeping plus the blocks containing your actual on said filesystem hosted data.

2 and 3 can for many purposes be identified with one another -- but 1 on the one hand and 2 or 3 on the other can not. What was damaged in your case was 3, what you as per previous reporting backed up was 1, and what you are now suggesting to backup/restore in the USB-stick case is 2. That's clearly not particularly consistent.

When you originally backed up your device using if=/dev/sdb you backed up all of the device: its partition table and the actual partition(s)/filesystem(s) that lived on them. Certainly then you should now also restore that whole kit and caboodle to the device itself, and not to just one of its partitions.

Also then, you needn't pre-partition the device before restoring if you backed up the device itself, since said partitioning will immediately be overwritten again by the backed up partitioning. Most certainly you needn't have formatted anything, i.e., needn't have gone to to the trouble of creating an empty filesystem on the partition, since certainly that entire filesystem will be overwritten immediately again. Extramost certainly you should by the way not mount anything while doing this: you "mount" a filesystem but said filesystem is about to be overwritten out from under said mount.

Yes, after the restore you're still stuck with the same damage to the filesystem, 3 from above, as before since said damaged filesystem is what you as part of the entire device backed up and would now restore. Yes, you'd in the case of an exFAT filesystem preferably use Windows to check/repair said damaged filesystem. Yes, 10/11 probably best but if your Windows 7 install can deal with exFAT that'd be fine as well; I earlier gathered from you that it couldn't and do as a not Windows user not know myself.

exFAT has been open-sourced but its patents have historically been an issue for Linux implementations and I do not know details of current state. For all I know it's currently in fact no worse than using NTFS for interoperability -- and/but I do not know since I don't care about device-level interoperability with Microsoft's spyware. You'd be surprised how many "Linux problems" all of a sudden cease to exist when you just give Windows a swift kick in the butt and out the door.

ext2 and ext3 are mere previous versions of ext4 and latter should be it for most Linux-only use. An added benefit is that unlike NTFS and (ex)FAT a UNIX filesystem type such as ext4 in fact supports the to UNIX-systems central concept of UNIX file/directory ownership and permissions -- although some consider it a drawback again to then also having to understand them.
Last edited by rene on Mon Jun 12, 2023 2:28 pm, edited 2 times in total.
User avatar
axisofevil
Level 4
Level 4
Posts: 388
Joined: Mon Nov 14, 2011 12:22 pm

Re: [SOLVED] I am in a muddle with gunzip and cannot decompress a backup to an external ssd.

Post by axisofevil »

Linux will support a NTFS file system / partition - but Linux holds a lot of meta data which is absent from NTFS - such as user, group ownerships and permission attributes which are absent from NTFS.

Only use NTFS for Windows data files - absolutely no programs or system files!

So no reason why you shouldn't create an extra partition where you can copy files from/to as an intermediate staging post for Windows stuff.
User avatar
medea
Level 1
Level 1
Posts: 41
Joined: Tue May 30, 2023 7:48 am

Re: [SOLVED] I am in a muddle with gunzip and cannot decompress a backup to an external ssd.

Post by medea »

Many thanks to for the replies since my last post.

It's taken me a while to find time to get back to this, as I have had to clear everything off the ssd (the one I made the original corrupted image of before I wiped and reformatted it) before attempting that.

Anyway, the proof of the pudding...

ANYONE@computer:~$ gunzip -c /home/ANYONE/backups/flashback.img.gz | sudo dd of=/dev/sdb status=progress
[sudo] password for ANYONE:
480098259456 bytes (480 GB, 447 GiB) copied, 14916 s, 32.2 MB/s
937703088+0 records in
937703088+0 records out
480103981056 bytes (480 GB, 447 GiB) copied, 14916.2 s, 32.2 MB/s
ANYONE@computer:~$

So, thanks to the help of all those who contributed code and advice, the uncompressing of the gzip flashback disk image worked perfectly (around 4 hours) and I got back to where I started.

Happy with that at least working, this is where things got even more interesting.

I originally had to go the gzip backup disk image route as I was completely unable to mount the usb external ssd due to an error.
Report dialog of that from that time:

'Unable to mount DISKNAME.
Error mounting /dev/sdb at /media/ANYONE/DISKNAME:
Wrong fs type, bad superblock on /dev/sdb, missing codepage or helper program, or other error.'

But this time, after the restoring the disk image to the ssd, it did mount and I was able to see inside.
What I found was around 20 folders all with padlocks icons on, which on opening revealed no data.
Not the 150GB of films all in separate folders and all neatly indexed in directories under each director's name, that had originally been there before it broke.
Disappointing but also tantalizing, as something had clearly worked... to an extent.

I had originally tried to use fsck on the ssd before getting into this whole gzip backup drama, but it hadn't worked, perhaps as the drive was unmountable.
But now I had a drive that clearly was mounted, I thought I had nothing to lose and everything to gain by giving it another go (I noticed the command process relied on the exfatprogs utility to process, a utility pre-installed with LMDE5 Linux Mint).

And I got this:
ANYONE@computer:~$ sudo fsck /dev/sdb
[sudo] password for ANYONE:
fsck from util-linux 2.36.1
exfatprogs version : 1.1.0
checksum of boot region is not correct. xxxxxxxxx, but expected xxxxxxxxxx
boot region is corrupted. try to restore the region from backup. Fix (y/N)? y
/dev/sdb: clean. directories 103, files 459
/dev/sdb: files corrupted 0, files fixed 0
ANYONE@computer:~$

Clean we like!

And lo! When I returned to my once stricken-down and seemingly unsaveable ssd, there was EVERYTHING... all in their directories, all present and correct, as if time itself had been rewound.
So 'happy' became 'thrilled' and everyone lived happily ever after.

Again, I can't thank all those who helped me get to this point enough. Hugest respect and gratitude to you all, because I had completely run out of ideas (and patience) as to how to resolve this on my own, before I decided to sign up here and ask a question.

Other things.

@Rene
"Warning. If you used (as you said) dd if=/dev/sdb [etc.] to backup then you need to also use dd of=/dev/sdb [etc.] to restore, i.e., not /dev/sdb1 (and again only after making sure that /dev/sdb is the correct device specifier)."

Thank you Rene, this I took on board and triple-checked all my commands for being consistent with this (taking my device name from the device name as given in the 'disks' LMDE utility).

"Also then, you needn't pre-partition the device before restoring if you backed up the device itself, since said partitioning will immediately be overwritten again by the backed up partitioning."

As you recommended, I just restored it to an empty unformatted ssd.

"Extramost certainly you should by the way not mount anything while doing this: you "mount" a filesystem but said filesystem is about to be overwritten out from under said mount."

I was extremely glad you mentioned this, as I might have inadvertently popped a flashdrive into a socket to grab something without really appreciating that it may reset drive identifiers.
So I adopted an 'Apollo 13' tactic, by sticking a piece of paper over the sockets I usually use to remind me 'NO!' while the restore ran its course.

Re your comments about how I might set about fixing the disk image once restored, see my explanation of how process unfolded above. Luckily I was able to avoid that step.

Re Exfat v ext4.

@Rene
"exFAT has been open-sourced but its patents have historically been an issue for Linux implementations and I do not know details of current state. For all I know it's currently in fact no worse than using NTFS for interoperability -- and/but I do not know since I don't care about device-level interoperability with Microsoft's spyware. You'd be surprised how many "Linux problems" all of a sudden cease to exist when you just give Windows a swift kick in the butt and out the door."

I have been Windows free 100% Linux for last year or two (for many of the same reasons you state), I just thought that keeping the drive open to being connected with as many operating systems as possible was a useful policy, in case I ever needed to share anything from that drive with a windows user.
But as a consequence of my problems with it, my view on exfat has changed entirely.

@axisofevil
"+1 on the exfat thing.
I'd recommend that you go for ext4 - its pretty standard."


This now strikes me as a wise recommendation. Recommendation accepted.

Going on.

Having gone through the tech nightmare I have been through with this, what I have done between my last post and this has been to buy another 500GB ssd, format it to ext4, and transfer all those retrieved files from the recovered exfat ssd to the new ext4 ssd, so that absolutely everything is now stored on ext4.
I then deleted that exfat ssd once empty, and reformatted it to ext4.
This is a practice and policy I intend to continue with (going on) as I no longer use any windows systems on computers I use on a daily basis.

I do keep a handful of smaller than 65GB usb flashdrives that are formatted to fat32 (which have never given me any problems on linux or windows) which I shall keep as they are, in order to have something to use for moving files smaller than 4GB between operating systems (should the circumstance arise).
But those aside, its definitely ext4 for me from now.

All the best to all who contributed to this thread, and I look forward to seeing you again here soon.

Medea.
be a good animal.
rene
Level 20
Level 20
Posts: 12212
Joined: Sun Mar 27, 2016 6:58 pm

Re: [SOLVED] I am in a muddle with gunzip and cannot decompress a backup to an external ssd.

Post by rene »

medea wrote: Mon Jun 19, 2023 5:10 pm And lo! When I returned to my once stricken-down and seemingly unsaveable ssd, there was EVERYTHING... all in their directories, all present and correct, as if time itself had been rewound.
Congrats :)
Locked

Return to “Beginner Questions”