Incremental backup - what app are you guys using?

All Gurus once were Newbies
Forum rules
There are no such things as "stupid" questions. However if you think your question is a bit stupid, then this is the right place for you to post it. Please stick to easy to-the-point questions that you feel people can answer fast. For long and complicated questions prefer the other forums within the support section.
Before you post please read how to get help
Post Reply
afora
Level 2
Level 2
Posts: 79
Joined: Mon Aug 26, 2019 7:35 pm

Incremental backup - what app are you guys using?

Post by afora » Wed Dec 04, 2019 6:55 pm

What's everybody using for implementing incremental archiving / backup?
Thanks!

ajgringo619
Level 6
Level 6
Posts: 1484
Joined: Thu Mar 01, 2018 8:36 pm
Location: San Diego, California, USA

Re: Incremental backup - what app are you guys using?

Post by ajgringo619 » Wed Dec 04, 2019 7:00 pm

I use a custom-built rsync script.
Mint 19.2 XFCE
kernel 4.15.0-72-generic
AMD FX-8350 Eight-Core w/16 GB RAM
Nvidia GeForce 960 w/2GB RAM (440.44)
Image

User avatar
pbear
Level 9
Level 9
Posts: 2744
Joined: Wed Jun 21, 2017 12:25 pm
Location: San Francisco

Re: Incremental backup - what app are you guys using?

Post by pbear » Wed Dec 04, 2019 11:35 pm

Bear in mind there are two kinds of incremental backup. One kind takes a base line image then calculates changes, so it takes all the backup files together to reconstruct the file system. Usually not feasible to pick through the backup for individual files, but requires the least time and bandwidth to maintain. Another kind works with whole files, but only updates the backup for files which have changed. This is as distinguished from a backup which copies everything from scratch every time. So, which kind do you mean? Also, do you mind whether it's command line or do you want GUI?

Anyhoo, to answer the question, I use rsync in the second sense of incremental (which is the default in a local backup). Don't have a script in the technical sense, just a list of commands in a file which I copy and paste when I want to backup. Which I'm pretty good about doing frequently.
Time flies like an arrow. Fruit flies like a banana.
If your problem has been solved, please edit the thread title.

ozbear
Level 1
Level 1
Posts: 12
Joined: Sun Dec 01, 2019 5:35 pm

Re: Incremental backup - what app are you guys using?

Post by ozbear » Thu Dec 05, 2019 1:49 am

On a (potentially) side note, now that my Windows/Linux dual boot machinery is working, it appears that my Windows Acronis True Image Home backup is picking up the Linux partitions now too, since there was no exclusion rule for them. Weird. Whether or not they are restorable/usable I haven't had time to test yet, but if they are it is a cheaper solution than Acronis for Linux. Hope that wasn't too off-topic.
Within Linux I use rsync.

Regards, Oz

afora
Level 2
Level 2
Posts: 79
Joined: Mon Aug 26, 2019 7:35 pm

Re: Incremental backup - what app are you guys using?

Post by afora » Thu Dec 05, 2019 5:04 pm

pbear wrote:
Wed Dec 04, 2019 11:35 pm
So, which kind do you mean? Also, do you mind whether it's command line or do you want GUI?

Anyhoo, to answer the question, I use rsync in the second sense of incremental (which is the default in a local backup). Don't have a script in the technical sense, just a list of commands in a file which I copy and paste when I want to backup. Which I'm pretty good about doing frequently.
Ok, thank you for that - just an incremental backup with an option to restore a point in time from it (which I realise is not quite trivial to do manually from my experience in Windows, unless you take the full backup once in a while).

Script based solutions would be fine, however, I was thinking if there's some GUI front end to help generating command line parameters at least at the start, that would help too. I hope I'm not asking for too much...

User avatar
Larry78723
Level 7
Level 7
Posts: 1702
Joined: Wed Jan 09, 2019 7:01 pm
Location: Jasper County, SC, USA

Re: Incremental backup - what app are you guys using?

Post by Larry78723 » Thu Dec 05, 2019 5:14 pm

Are you looking for a system backup solution or a data backup solution? There are both command line and GUI tools available.
Image

afora
Level 2
Level 2
Posts: 79
Joined: Mon Aug 26, 2019 7:35 pm

Re: Incremental backup - what app are you guys using?

Post by afora » Thu Dec 05, 2019 6:17 pm

Data backup only, I sorted system imaging with Clonezilla which works impeccably.

User avatar
Larry78723
Level 7
Level 7
Posts: 1702
Joined: Wed Jan 09, 2019 7:01 pm
Location: Jasper County, SC, USA

Re: Incremental backup - what app are you guys using?

Post by Larry78723 » Thu Dec 05, 2019 6:33 pm

afora, I use BackInTime, which is actually a front end for rsync with scheduling capabilities.
Image

User avatar
pbear
Level 9
Level 9
Posts: 2744
Joined: Wed Jun 21, 2017 12:25 pm
Location: San Francisco

Re: Incremental backup - what app are you guys using?

Post by pbear » Thu Dec 05, 2019 11:58 pm

afora wrote:
Thu Dec 05, 2019 5:04 pm
... just an incremental backup with an option to restore a point in time ...
To me, that implies snapshots, for which Back in Time is probably the best choice. I've not used, so I'll leave it to Larry (and others) to answer any questions. By contrast, what I do with rsync is a conventional state of the file system as of the last backup sort of backup.

If it turns out BIT doesn't tickle your fancy, take a look at Grsync, which is a simple GUI front end for rsync. One of things it can do is report the Terminal commands it's planning to run. One thing Grsync doesn't have, though, is scheduling capabilities.
Time flies like an arrow. Fruit flies like a banana.
If your problem has been solved, please edit the thread title.

Petermint
Level 5
Level 5
Posts: 734
Joined: Tue Feb 16, 2016 3:12 am

Re: Incremental backup - what app are you guys using?

Post by Petermint » Fri Dec 06, 2019 1:14 am

Backintime (uses rsync) and rsync copy everything. With Backintime backing up to the Ext4 file system, Backintime can create existing files as links to the existing copy of the file from a previous backup. The first backup takes forever because you copy all the files. The next backup to the same partition physically copies only new and changed files. Everything else speeds through because the file becomes just one link which is just an inode write, not a file copy.

If you backup to NTFS or most other common file systems, you loose the space saving through links.

Naturally you have several external disks in rotation for backup. The first copy to each disk is a full copy. The next is 5 minutes instead of 5 hours.

Timeshift can do the same simple backup but has only one backup profile. Backintime can have multiple profiles for different backups. My main desktop computer has one profile to backup the SSDs to dinosaur disks in the same machine. A separate profile backs up the wobbly old rotating disks to external wobbly new USB 3 disks.

afora
Level 2
Level 2
Posts: 79
Joined: Mon Aug 26, 2019 7:35 pm

Re: Incremental backup - what app are you guys using?

Post by afora » Fri Dec 06, 2019 1:27 am

This is fantastic advice!

Thank you everybody!

afora
Level 2
Level 2
Posts: 79
Joined: Mon Aug 26, 2019 7:35 pm

Re: Incremental backup - what app are you guys using?

Post by afora » Fri Dec 06, 2019 1:28 am

pbear wrote:
Thu Dec 05, 2019 11:58 pm
One thing Grsync doesn't have, though, is scheduling capabilities.
Is there any advantage of not using cron for scheduling?

ZakGordon
Level 4
Level 4
Posts: 356
Joined: Thu Feb 12, 2015 11:07 am

Re: Incremental backup - what app are you guys using?

Post by ZakGordon » Fri Dec 06, 2019 6:34 am

ozbear wrote:
Thu Dec 05, 2019 1:49 am
On a (potentially) side note, now that my Windows/Linux dual boot machinery is working, it appears that my Windows Acronis True Image Home backup is picking up the Linux partitions now too, since there was no exclusion rule for them. Weird. Whether or not they are restorable/usable I haven't had time to test yet, but if they are it is a cheaper solution than Acronis for Linux. Hope that wasn't too off-topic.
Within Linux I use rsync.

Regards, Oz
Thanks for that info. I have not looked into this yet for my own dual boot setup, so your input has given me an option to look into :)

Usually the way i do backups is to create a complete new total system image every month or so. Important data (say documents or accounts etc) is just copied to USB key as required and in general for my uses i rarely loose that much info with the monthly schedule. I had been wondering about the dual-boot setup though.
Laptop overheating issues? Check links below:
TLP
itsfoss guide
If none of the above fixes the issue, moving from Cinnamon to XFCE will give around -5 to -10 degrees C improvements.

User avatar
pbear
Level 9
Level 9
Posts: 2744
Joined: Wed Jun 21, 2017 12:25 pm
Location: San Francisco

Re: Incremental backup - what app are you guys using?

Post by pbear » Fri Dec 06, 2019 2:03 pm

afora wrote:
Fri Dec 06, 2019 1:28 am
pbear wrote:
Thu Dec 05, 2019 11:58 pm
One thing Grsync doesn't have, though, is scheduling capabilities.
Is there any advantage of not using cron for scheduling?
Not sure I understand the question. I mentioned Grsync doesn't have scheduling because some users regard that as a deal killer. Regular rsync can be scheduled with cron, though I don't do it. Haven't ever heard of anyone using cron with Grsync, but that doesn't mean it can't be done. If it turns out the answer is "yes," please report back as that would be useful to know. This topic comes up often.
Last edited by pbear on Fri Dec 06, 2019 9:29 pm, edited 1 time in total.
Time flies like an arrow. Fruit flies like a banana.
If your problem has been solved, please edit the thread title.

Petermint
Level 5
Level 5
Posts: 734
Joined: Tue Feb 16, 2016 3:12 am

Re: Incremental backup - what app are you guys using?

Post by Petermint » Fri Dec 06, 2019 5:08 pm

Some of the backups are designed for purely manual use or for servers that are always on. When you use rsync in Backintime, you get scheduling, you get messages when the backup is scheduled overnight but the computer is off, and you get messages when the external USB disk is unplugged.

bt101
Level 1
Level 1
Posts: 38
Joined: Thu Feb 14, 2013 12:30 am

Re: Incremental backup - what app are you guys using?

Post by bt101 » Mon Dec 09, 2019 9:09 pm

I'll throw-in.

I use rdiff-backup. It has no GUI and I would say it is not the easiest to learn (however like anything, once you know it, it's not bad).
It is based on rsync. There's a million ways you can run it, but the simplest/quickest way is to run it just like rsync (rdiff-backup source_dir dest_dir).

Just like rsync, it makes a mirror of your source. However it also makes a special folder at the root of your backup that contains all of incremental data for previous backups. You don't have to worry about that folder or what's in it. Whenever you repeatedly run rdiff-backup, the target backup ends up as a mirror of the current source, and the older incremental data in the special folder is updated. So 99% of the time, when you want to restore a file, you typically want the last version right? So you can get that file from the backup with no special tools as the backup is just a bunch of files/folders that mirror the source. If you do want an older version of a file, you just use a special command to get it (that program will re-assemble the file from the latest copy and the incremental info in that special folder).

So you might say...my backup is going to be at least as big as the source (because it is a mirror) plus it has that incremental info... that's not efficient because my traditional backup does compression!

Well, if you look at your traditional backup (let's say you do fulls every week with daily incrementals and you compress them)... you really should have more than one full (in case one gets messed up). So you have at least two fulls and likely more. Those can add up to more than a mirror.

The fellow who wrote the program did a good job because it is super fast and efficient. For the incremental info, it only stores the changes to the files and it efficiently only sends those changes to the backup.

Now I know that the first thought is...I'll just run rdiff-backup as a cron job and send the backup to a USB key or a CIFS share. You can certainly do this, however there is a better way. You can install rdiff-backup on one server and all of your client machines. You then do all of you configuration on the server and schedule the backups there. It will reach out to the client machines via ssh to run in a client/server mode. The advantages are:
  • all config and scheduling is one on the server (you just need to install an ssh key on the clients)
  • the backup info is sent across the wire encrypted (so you could backup across the internet)
  • the backup are not exposed to the client (so if the client clicks on a bad email, they can't destroy their backups)
So what's the downside? Aside from not being the easiest to setup and no GUI, the only downside I find is that the backup is not encrypted. You could take steps to store the backup on an encrypted drive so that the info is encrypted "at rest" but when it's online it is not encrypted.

If you want an encrypted backup, another one that I am using is duplicity with the duply frontend. Again, these are both setup with config files (no GUI). The duply frontend just makes config and running a bit easier. This is a more traditional backup where you would do (say) weekly fulls and daily incrementals. They are compressed and encrypted, They can/are also sent to the target via ssh (so not exposing some share to the client where ransomware could strike it). Since these backups are encrypted, you could just send them to any repository on the net.

Here the thing... once rdiff-backup does its initial seed, it runs super quick for every subsequent backup. That's especially good if you have a huge store of (let's say) videos. If there is a huge store of data but little changes, rdiff-backup will fly through it in seconds and be done. Even if you have large files that have parts of them that are changed, rdiff-backup only send the changes (very efficient). The more traditional backup will cripple your cpu and network when doing those weekly fulls. When I notice my laptop scorching my leg, I know that duply/duplicity is running. So I try and only use duply/duplicity for smaller stuff that I want encrypted. rdiff-backup is good for everything else.


I had my linux server drive die (and that server was running and storing tons of stuff). I successfully brought back the whole thing from my rdiff-backup backup.

User avatar
Termy
Level 5
Level 5
Posts: 769
Joined: Mon Sep 04, 2017 8:49 pm
Location: UK
Contact:

Re: Incremental backup - what app are you guys using?

Post by Termy » Wed Dec 11, 2019 6:02 pm

The only backup solutions I need and use are BackMeUp (spoiler alert, I wrote it) and the amazing Clonezilla. I hands down have to recommend Clonezilla, especially if you're exploring Linux distributions, or going more in-depth into Linux.
Here to help.

I'm LearnLinux (LL) on YouTube: https://www.youtube.com/channel/UCfp-lN ... naEE6NtDSg
I'm also terminalforlife (TFL) on GitHub: https://github.com/terminalforlife

User avatar
AZgl1500
Level 11
Level 11
Posts: 3743
Joined: Thu Dec 31, 2015 3:20 am
Location: Oklahoma where the wind comes sweeping down the plains
Contact:

Re: Incremental backup - what app are you guys using?

Post by AZgl1500 » Thu Dec 12, 2019 12:06 am

I prefer LuckyBackup because I can use NEMO and directly look at any single file in the backups.
the directories are carbon copies of the originals.

It only adds additional files not previously copied, or if a file is modified, it catches that too.

I assume everyone is backing up to an external drive, as I am.

Petermint
Level 5
Level 5
Posts: 734
Joined: Tue Feb 16, 2016 3:12 am

Re: Incremental backup - what app are you guys using?

Post by Petermint » Thu Dec 12, 2019 2:59 am

Backintime also produces "carbon copies" you can browse. Each snapshot is a separate directory by date/time of backup. The first snapshot is everything. The next snapshots show all files but contain only changed files. The magic works only on Ext4 because Ext4 is the only common file system to allow multiple file entries to point to the same file.

Post Reply

Return to “Newbie Questions”