Backups and imaging - need for a tutorial?

[br1anstorm]

It's a well-worn cliche - everyone ought to backup their system and data. It's also true that many people don't realise the need for backups until the inevitable crash or failure occurs.... and then it's too late!

I have myself sought forum advice on backup tools and strategies, and this has revealed that almost everyone has a different approach and/or uses different tools. See for example:

Getting backups organised viewtopic.php?f=47&t=284746&p=1576134#p1576134;

Best way to backup and restore viewtopic.php?f=90&t=245350&hilit

Backups - best options viewtopic.php?f=60&t=276859

There are so many permutations, and most comments describe individual personalised schemes. My feeling is that a simple "idiot's guide" outlining all the options and essential facts, is needed - ie a tutorial. Does such a comparative guide and overview exist somewhere already?

The fact that Mint 19 comes with two specific programs (Timeshift and mintbackup) preinstalled suggests that a guide to these, and other, options would be useful. I don't have the knowledge or skill to write it. Is there someone who can? Among the aspects which need clarification, I'd put the following:

1) the difference between a disk image, a system image (or snapshot), and the preservation (backup) of data files. This is fundamental, yet a lot of discussion fudges the difference, and the word "backup" is loosely used for each, or all three;

2) following on from that, an explanation of the tools for disk imaging - like Clonezilla or ReDo, or indeed Macrium Reflect, all of which are disk-imaging apps;

3) similarly, a description of system-imaging tools such as Timeshift. This would need to include specific warnings about their limitations - like Timeshift only working with ext4-formatted partitions and not NTFS, and the importance of choosing the right destination and allocating enough space in order to avoid the trap which some (including me) many have fallen into of Timeshift rapidly filling up its designated partitions if the default settings are used, and causing the entire OS to seize up. I think some proprietary "backup" or imaging apps (eg Aomei and Acronis if you are a Windows user) write their backup files, or compress them, in formats which can only be read by the program that created them. How true is that in the world of Linux "backups" too?

4) the program-choices for saving or backing up data and personal files/photos etc. I think of DejaDup and BackInTime, for example. And maybe LuckyBackup too? I don't know whether the mint backup tool in Mint 19 is equivalent to these or not.....as I've never used it. What category does Aptik fit into - system, or data, or both?

5) for non-newbies who want to use the terminal rather than a GUI, a tutorial ought also to cover rsync (or is it grsync?), btrfs, and cp-ax (whatever that does....). Are there other command-line options? And why would one use them rather than a program with a GUI?

Some of these questions might be misconceived, and reveal my own lack of understanding. I wonder if anyone might volunteer to produce a complete guide?

[bob466]

People will tell you all sorts of wonderful ways to Backup...but what works for them might not work for others and confuses the hell out of Newbies.

I wrote this Tutorial...viewtopic.php?f=42&t=264431 because there wasn't one and works just great for Linux. In combination with Timeshift I'm ok.

It's not hard to understand...you need something that backs up everything and Macrium does just that...it creates a System Image of the whole HDD or SSD and stores it on an External HDD...if your HDD/SSD fails you just put the Image on a new HDD/SSD...done.

I agree with you there should be Tutorials for other Linux Backup Tools especially Clonezilla which many people on this Forum push so hard...maybe one day.

[ricardogroetaers]

In my modest opinion the tutorial will depend on the program used.
For example.
I use a good old DOS program to make complete copies of partitions or the entire disk.
It would be unwise to do a tutorial about a DOS program here.

[br1anstorm]

Interesting comments in both replies so far....

I agree that a tutorial about DOS would probably not be wise or useful for most Mint users.

I'd forgotten, but have just revisited, the good tutorial by bob466 on how to use Macrium Reflect (a program which has other supporters on this forum). But - for some folk a big BUT - you need a Windows setup in order to use it.

And that sort-of illustrates my point. What is perhaps needed is not a tutorial on using this, or that, specific program in this or that way, but rather a comparative overview of all the most commonly used programs which simply describes what each program does, and the pros and cons or particular requirements of each. At the moment such information is scattered in posts all across the forum.

Just as an illustrative example of some of the fragments of info I happen to have picked up:

Clonezilla - provides a disk image, seems to be reliable, but has an old-fashioned GUI;
ReDo - also provides a disk image, very simple and user-friendly interface, still available but no longer supported, some reliability questions (may not boot up for restoring);
Macrium Reflect - needs Windows but can work with Linux as noted above;
Acronis/Aomei - Windows only?

Timeshift - a system image tool. Linux-only? Requires ext4 formatting. Can rapidly fill up the space allocated to it unless default frequency of snapshots is adjusted. Not useful for backing up personal data, it's a system-restore tool;
SystemBack - similar, Linux only, discontinued....
Cronopete - dunno. Anybody tried it? Said to be a Linux equivalent to the Mac Time Machine (whatever that does.... I'm not a Mac user);

DejaDup - takes backups of your (/home?) folder(s). Can't specify how long to keep backups - it keeps going till the space is full!
BackInTime - ditto. But more flexible (eg can specify how many backups to keep, and for how long). Doesn't offer encrypted backups whereas DejaDup does. Allows selective browsing and restoration of individual files - DejaDup's ability to do so is limited.
LuckyBackup - er, don't know much about it. But it is I think included with some other distros - like MX?

Then there seem to be various other options which have no GUI but need expertise in using the command line. rsync? Duplicity?

I have no idea what the mintbackup utility built into 19.x does and how it compares with any of these. And there are other angles where I am simply ignorant - like how much do each or all of these compress the backups they take (important when thinking about allocating space)? Do they all work with ext4 and NTFS and/or FAT formatting? Do they each use their own special file-formats or are they inter-operable..... and so on....

[Finston Pickle]

I think that the original suggestion for a complete tutorial guide is very valid.

I have found that the live DVD version of Acronis 9 (which may well be written Linux and is many years old) is fine for creating a .ext4 Linux Mint O/S Disc image onto a NFTS external HDD drive (which can be larger than the orginal PC HDD). Unfortunately the stored files are in a propriatary format - not a .iso. Great and flexible if you have the Acronis live DVD version to clone/restore to other HDDs - not much help if you do not.

[itsme4401]

About REDO:
Quote from:https://redobackup.blogspot.com/ Easy Backup, Recovery & Quick Restore
Redo Backup and Recovery is so simple that anyone can use it. It is the easiest, most complete disaster recovery solution available. It allows bare-metal restore. Bare metal restore is not only the best solution for hardware failure, it is also the ultimate antivirus: Even if your hard drive melts or gets completely erased by a virus, you can have a completely-functional system back up and running in as little as 10 minutes.


redo backup download data recovery solution back
Download Version 1.0.4Released 2013-12-20

I like it, however!

[AndyMH]

I like it, however!

So do I, but it has limitations. For some it won't boot from a usb stick (I had to isohybrid the redo iso to get mine to work) and "gets completely erased by a virus" doesn't work. It uses dd to copy the partition table (first 34 512B sectors), but misses the fact that grub on an MBR drive hides in the gap between the end of the MBR and start of the first partition (normally 1MB). Which explains why when I used it to clone an HDD to SSD a couple of years ago I had to use boot-repair to fix it. It was only recently that I 'looked under the hood' to find out how it works (it's a PERL script).

[bob466]

My mechanical HDD is 23% full...Macrium took just over 1 hr to create a System Image ...Clonezilla on the same HDD was running some 3.5 hrs later at which point I cancelled it...could have gone on for several hours more for all I know...it's not for me.

[jglen490]

Tools are tools. One can Google all about all the Linux backup tools, and probably Windows tools, too.

The first and foremost - don't have a plan, MAKE a plan. Write down what is a prioritized list of goals. That list will lead you to the correct tools, after you do the research. And DO research, don't spend a lot of time asking for opinions. Ask 100 people, you'll get 101 opinions. The decision is on you, so invest the time.

Some people think saving the OS environment is paramount, others thinks data is the most important. The same tool can do both, whether it's an imaging tool or a blockwise/bitwise/bytewise copy tool.

Maybe it's important to consider what to do with a dual boot environment. And maybe that complicates tool choice, or simplifies it with an outlay of money. Write that down.

Why do you want to image/backup something? If it's about recovery from failure, or if it's about access to lost files, or if it may be both. Make a plan.

Where will the backup/image be stored. Maybe you want to just save changes right where the actual data is. Maybe you believe that disk failure is somewhat more important. Maybe a more complex plan is good for you.

Put all that into a plan, along with whatever else turns you on. Make it in written form, and once you have written down. Keep it. So if something isn't quite right later, you can change the plan, and write down those changes. Keep it!

Here's my use case. If you're not interested, that's fine, skip to the next topic in the forum.
My plan is written down, in a text file on my PC - and it's backed up
I have a / partition, a /home partition, and a SWAP partition, all on two drives. The / partition is on a SSD, along with SWAP. /home is on a WD spinner
I don't backup my / partition. All in all, it takes less time to reinstall, than it does to make several images/backups over time.
Of course SWAP is not an issue or a backup source.
So that leaves /home. I do not, and never will make a backup that involves saving file changes to the same drive as the files. Therefore, any backup goes to a location other than the drive with /home.
I try to make a backup about once every 7 - 10 days. Chances are pretty good that nothing of consequence will be lost in that time period.
My external target to receive the backups, is a set of three/four spinners that are at least the same size as the drive that has /home.
I rotate those drives and keep a record of each as it is used in the text file mentioned above. Each drive is uses a GPT partition table and is formatted with one ext4 partition.
Each external drive is named: Backup1, Backup2, Backup3, Backup4.
I use the command line executable

Code: Select all

rsync

, in a pre-written set of four commands (one for each named drive) that are saved in that same text file described above. Each command follows this pattern:

Code: Select all

sudo rsync -auv /home /media/john/Backup#

where the # is 1,2,3,or 4.
Each time one of the drives is used, I record the date at the end of the line that has the command for that named drive.

The result is the ability to preserve each file in /home, including the ".files" that contain configurations and other data for the programs I use. Do I backup .cache? Sure, if I forget to delete those files. It doesn't hurt, because I can pick and choose exactly which file to get in a recovery situation. And I have had to recover files in a PEBCAK situation

[Larry78723]

jglen, I fully agree with your comments regarding the need for analysis and planning for backup. In my case, I have limited download bandwidth so having a snapshot of / is important to me. Anytime I install new software I do a Timeshift snapshot. I do a lot of file editing, so I do a daily rsync backup of /home via a cron job. All snapshots/backups are made to a dedicated drive on my laptop and once a week I make a Timeshift snapshot and rsync backup to an external SSD. Like you, I maintain a file where I log all changes to the system and all manual snapshots/backups made.

[br1anstorm]

Very systematic and logical advice from jglen490 - outlining a plan-making process which in an ideal world every user would follow!

I fear however that quite a few people simply want to "point and click", and if backing-up/cloning/imaging requires an elaborate process they will shy away from it.

And with all due respect, Larry78723, with your comment that you ".... do a daily rsync backup of /home via a cron job..." you've lost me (and many others) already! "rsync"? What's that and how? "cron job"? Translation please. Shows how easy it is, when you are proficient in using the terminal, to overlook the fact that many people still dare not go there.

My idea in the OP was rather to suggest a series of guidelines for non-experts which clarifies exactly what the loosely-used word "backup" actually means - essentially based on the kind of thinking jglen describes. Something on these lines:

- if you want to make and preserve an image of your entire disk (system, data and all) then clone it. You can use Clonezilla, ReDo or xxxxxx (and then the guide could flag up the idiosyncrasies and advantages of each program, and where/how to save the image);

- if you want to save and then restore your operating system (and profile settings?) in case it crashes, make a system image. Use Timeshift, or xxxxxx (and then add any specific advice about points to watch or how to adjust settings);

- if you want to ensure you don't lose any important personal files, documents or data, then backup your /home.... or data partition. You can use BackInTime, DejaDup or xxxxxxxx. (then add advice on where to keep the backup and how often to do it depending on usage).

- if you want to do it all using the terminal, then these are the ways to do it..... (whole big chapter with detailed commands!).

[Moem]

I think it's a fine idea... Now can you find someone to make it happen? That's the big question here.

[AndyMH]

My thinking as well, happy to help, but busy on other things backup related

[jollyjack]

As a fairly ignorant use, can I add my 5 cents worth?

I try and have a two line defence. I back up the system (/) overnight with TimeShift. I back up data (/home) overnight with BackInTime. All gui-based - I need that. I also use Clonezilla every so often, maybe once a month. This is my fall-back, if it all goes wrong. (OK, the gui isn't user friendly, but I just take the defaults - it works).

Does that sound sensible?

Thanks

[Larry78723]

br1anstorm, you made a great point in your comment about my post, thank you. Basically, I'm doing exactly what jollyjack is doing using BackInTime to backup his /home folder.

BackInTime is basically a GUI that uses rsync to backup your /home folder.

[jglen490]

rsync, grsync, Luckybackup, and so many others are based on rsync. It's a simple yet flexible command line tool.

A cron job is a method of setting up a task that will run on a schedule that the user establishes, doing what the user wants done, while specifying where that task will be done. It, too, is a command line tool.

There are dozens of books about Linux and the wonders of the command line. Everything that happens in a Linux GUI is based on something that also occurs, or can easily occur, on the command line. Code writing is a skill, not everyone has the desire to learn that skill and that's fine. Not doing the command line has nothing to do with intellect or "realness", nor does it put you in a lower category of human being.

I say this only to make this point, understand what Linux is regardless of how you choose to actually use Linux.

[slipstick]

Timeshift - a system image tool. Linux-only? Requires ext4 formatting. Can rapidly fill up the space allocated to it unless default frequency of snapshots is adjusted. Not useful for backing up personal data, it's a system-restore tool

TimeShift will also work with btrfs format. I've never tried it, as I use ext4.

BackInTime - ditto. But more flexible (eg can specify how many backups to keep, and for how long). Doesn't offer encrypted backups whereas DejaDup does. Allows selective browsing and restoration of individual files - DejaDup's ability to do so is limited.

BackInTIme does offer encrypted backups. In BIT Settings->General, click on the "Mode" button and there are 4 choices: "Local", "Local encrypted" (uses EncFs), "SSH", and "SSH encrypted".

I like the flexibility of BIT - I have two Mint versions (triple boot with Windows 7). The two Mint versions (currently 18.3 and 19.1) share a common Data partition which is mounted in the /home of whichever of those two Mint systems is running. I have three different backup profiles: in LM19.1 the Main profile backs up to profile 1 on the backup drive and the Data profile backs up to profile 3 on the backup drive. In LM18.3, the Main profile backs up to profile 2 on the backup drive and the Data profile backs up to the same profile 3 on the backup drive that is used for LM19.1. This keeps my personal data the same for both systems, including e-mail and FF bookmarks because I have moved the Thunderbird and FF profiles to the Data partition (symlinked to /home in either system). The backup data is not compressed, but the backups are incremental, so the space used is reasonable if you don't keep too many backups. Uncompressed data also makes it easy to restore individual files.

I agree that a tutorial is probably needed as how to backup is a constantly occurring question, but I don't have the knowledge to write one. TS and BIT are the only ones I am familiar with (and not with all the features of either one), along with Clonezilla which I seldom use, so have to relearn every time I use it.

[SilentBlue]

I use Timeshift for system backups and I use a Linux Mint Live USB drive when I need to restore to a previous snapshot:

• Timeshift saved me several times and I am really happy it came installed with Linux Mint 19.1 and was advertised for in the Welcome screen.
• My Timeshift backups are stores on a separate partition. I am not sure is very useful but it has the following advantages:

1. It prevents Timeshift from filling up the main partition. Mind that one then needs to regularly check that Timeshift has enough space to perform its backup as in the contrary it will just fail silently. In case space is missing, I just delete former backups until enough space is available again, and maybe I also reajust the schedule and the amount of snapshots to be kept. Mind that Timeshift uses hardlinks, so deleting a snapshot might not release as much space as one might anticipate. To try and have an idea about which backups take the more space, I open Timeshift, go to Browse, go up one level in the directories (to the directory `timeshift`) and then right-click on the directory `snapshots` and open it with Baobab to analyse its content.
2. When I search for files from the root folder `/`, having the Timeshift backups in a separate partition prevents the multiple hardlink copies to appear in the search result. But maybe this issue only occurs if one opens Nemo as root, which should be avoided anyway.

I use BackInTime 1.2 for my `/home/user` backups, and store them on an external drive.

• I first tried the MintBackup tool provided by default, but I found it very limited. For instance it cannot do incremental backups. And I think there were other limitations but I cannot remember them exactly just now.
• BackInTime is very fast and extremely flexible. But this means that one needs to carefully read the documentation to take advantage of its flexibility.
• An interesting feature of BIT is that it can backup you files on an NTFS partition while still leaving the files accessible to a Windows system and while still being able to backup and restore the Linux filesystem-specific attributes of the files.
• One can set up a backup profile to automatically perform a backup at regular time intervals.... or automatically when the backup drive is mounted (very cool!).
• One of the limitations of BackInTime I stepped into is that all the backup profiles have to be stored to the same destination folder (one cannot store the backups of one profile to one drive and store the backups of another profile to another drive).
• Mind also that BackInTime 1.2 doesn't play fully smoothly with backups from earlier BackInTime versions (1.1.x). So it is best to start fresh from BackInTime 1.2.

I actually think that the Linux Mint team should invite their users to use BackInTime instead of the current MintBackup tool. I can see that the latter is very simple to use and introduce, but it is also very limited in its features. Maybe with a short introductory step-by-step guide in the Welcome screen this learning curve for using BackInTime could be easily overcome.

[Spiderspoon]

[*]I first tried the MintBackup tool provided by default, but I found it very limited. For instance it cannot do incremental backups. And I think there were other limitations but I cannot remember them exactly just now.

I have just done my 2nd backup with the provided Backup Tool. Having not used this computer much between the 2 backups, they both appear to be about the same. Hence they are not incremental backups. I guess that would mean deleting the previous backup and only keeping the latest.
I think I might try the suggested BackupInTime.

Thanks

[AndyMH]

I think all the mint backup tool does is make a .tar.gz copy of whatever you are backing up. From another post, I also think that it splits the archives at 4GB. The documentation is non-existent (or I haven't found it). I think it is a poor tool and should be removed from the default installation and replaced with something like backintime.