[SOLVED] Incremental backup - what app are you guys using?

All Gurus once were Newbies
Forum rules
There are no such things as "stupid" questions. However if you think your question is a bit stupid, then this is the right place for you to post it. Please stick to easy to-the-point questions that you feel people can answer fast. For long and complicated questions prefer the other forums within the support section.
Before you post please read how to get help
afora
Level 4
Level 4
Posts: 203
Joined: Mon Aug 26, 2019 7:35 pm

[SOLVED] Incremental backup - what app are you guys using?

Post by afora »

What's everybody using for implementing incremental archiving / backup?
Thanks!

EDIT: 27 July 2020
------------------------------

In the end I settled on writing my own rsync script. Here's my thoughts I hope will help somebody:

USE CASE:
A small office 3-5 workstations with files on local as well as on a file server, a couple of servers. Less than 4TB of user data. We do not need encryption as our volumes are encrypted. We do not backup our data over the internet. Three profiles of incremental backups:
- local-to-local and server-to-server - on a 20 minute rotational schedule (addresses accidental deletes/changes by users)
- local-to-server - every 12 hours (addresses local workstation failures/theft)
- server-to-offsite - every 7 days

1. NO -> Borg and other deduplication solutions
Although deduplication is an attractive feature, Borg simply does not work for us. Cons:

- even with a GUI front end (Vorta) you cannot easily traverse the FS tree structure without awkward mounting/unmounting. If you are running into a couple of dozen of snapshots, Borg's interface becomes very annoying. I want to see the file structure instantaneously and effortlessly. And I cannot.

- the GUI front end I tried (Vorta) comes in a flatpak of 0.5GB on Mint 19.3, it's just silly.

- Users occassionally need to diff two snapshots to see what changed as effortlessly as possible - that's very cumbersome. Productivity of users is my concern, not storage restrictions.

- I have not bought into bitwise deduplication. There is no free cheese, you either need storage or CPU, given our use case, it just does not make sense. YMMV.

2. NO -> Backintime and similar.
I only considered Backintime as other comparable apps (e.g. Timeshift, Grsync, rsnapshot) are either too lightweight or not very well maintained or/and both. Backintime was promising but:

- It has really poor shell functionality, basically being a GUI focussed tool. On a headless server, it's buggy, unreliable, and configurations are very backward (plus see the next point). E.g. see viewtopic.php?f=90&t=326229

- cron scheduling is a killer which is only managed from GUI (in fact cron entries get reset if you start the GUI).

- ssh support is half cooked, as is handling file permissions (e.g. see viewtopic.php?f=90&t=325385). I'm sure there may be workarounds, I am not that interested in the app to spend my time on it.

- it could be fine for a Windows-refugee user whose only workflow is GUI, for us it's just a fail. Shame, and annoyingly, the application was very close to being what we need.
Last edited by afora on Mon Jul 27, 2020 4:12 am, edited 6 times in total.
ajgringo619
Level 8
Level 8
Posts: 2095
Joined: Thu Mar 01, 2018 8:36 pm
Location: San Diego, California, USA

Re: Incremental backup - what app are you guys using?

Post by ajgringo619 »

I use a custom-built rsync script.
Mint 19.3 XFCE [kernel 5.4.0-51-generic]
Ryzen 7 2700X Eight-Core w/16 GB RAM
AMD Radeon R9 290 w/4GB RAM
Image
pbear
Level 15
Level 15
Posts: 5665
Joined: Wed Jun 21, 2017 12:25 pm
Location: San Francisco

Re: Incremental backup - what app are you guys using?

Post by pbear »

Bear in mind there are two kinds of incremental backup. One kind takes a base line image then calculates changes, so it takes all the backup files together to reconstruct the file system. Usually not feasible to pick through the backup for individual files, but requires the least time and bandwidth to maintain. Another kind works with whole files, but only updates the backup for files which have changed. This is as distinguished from a backup which copies everything from scratch every time. So, which kind do you mean? Also, do you mind whether it's command line or do you want GUI?

Anyhoo, to answer the question, I use rsync in the second sense of incremental (which is the default in a local backup). Don't have a script in the technical sense, just a list of commands in a file which I copy and paste when I want to backup. Which I'm pretty good about doing frequently.
ozbear
Level 1
Level 1
Posts: 13
Joined: Sun Dec 01, 2019 5:35 pm

Re: Incremental backup - what app are you guys using?

Post by ozbear »

On a (potentially) side note, now that my Windows/Linux dual boot machinery is working, it appears that my Windows Acronis True Image Home backup is picking up the Linux partitions now too, since there was no exclusion rule for them. Weird. Whether or not they are restorable/usable I haven't had time to test yet, but if they are it is a cheaper solution than Acronis for Linux. Hope that wasn't too off-topic.
Within Linux I use rsync.

Regards, Oz
afora
Level 4
Level 4
Posts: 203
Joined: Mon Aug 26, 2019 7:35 pm

Re: Incremental backup - what app are you guys using?

Post by afora »

pbear wrote:
Wed Dec 04, 2019 11:35 pm
So, which kind do you mean? Also, do you mind whether it's command line or do you want GUI?

Anyhoo, to answer the question, I use rsync in the second sense of incremental (which is the default in a local backup). Don't have a script in the technical sense, just a list of commands in a file which I copy and paste when I want to backup. Which I'm pretty good about doing frequently.
Ok, thank you for that - just an incremental backup with an option to restore a point in time from it (which I realise is not quite trivial to do manually from my experience in Windows, unless you take the full backup once in a while).

Script based solutions would be fine, however, I was thinking if there's some GUI front end to help generating command line parameters at least at the start, that would help too. I hope I'm not asking for too much...
User avatar
Larry78723
Level 8
Level 8
Posts: 2460
Joined: Wed Jan 09, 2019 7:01 pm
Location: Jasper County, SC, USA

Re: Incremental backup - what app are you guys using?

Post by Larry78723 »

Are you looking for a system backup solution or a data backup solution? There are both command line and GUI tools available.
Image
afora
Level 4
Level 4
Posts: 203
Joined: Mon Aug 26, 2019 7:35 pm

Re: Incremental backup - what app are you guys using?

Post by afora »

Data backup only, I sorted system imaging with Clonezilla which works impeccably.
User avatar
Larry78723
Level 8
Level 8
Posts: 2460
Joined: Wed Jan 09, 2019 7:01 pm
Location: Jasper County, SC, USA

Re: Incremental backup - what app are you guys using?

Post by Larry78723 »

afora, I use BackInTime, which is actually a front end for rsync with scheduling capabilities.
Image
pbear
Level 15
Level 15
Posts: 5665
Joined: Wed Jun 21, 2017 12:25 pm
Location: San Francisco

Re: Incremental backup - what app are you guys using?

Post by pbear »

afora wrote:
Thu Dec 05, 2019 5:04 pm
... just an incremental backup with an option to restore a point in time ...
To me, that implies snapshots, for which Back in Time is probably the best choice. I've not used, so I'll leave it to Larry (and others) to answer any questions. By contrast, what I do with rsync is a conventional state of the file system as of the last backup sort of backup.

If it turns out BIT doesn't tickle your fancy, take a look at Grsync, which is a simple GUI front end for rsync. One of things it can do is report the Terminal commands it's planning to run. One thing Grsync doesn't have, though, is scheduling capabilities.
Petermint
Level 6
Level 6
Posts: 1166
Joined: Tue Feb 16, 2016 3:12 am

Re: Incremental backup - what app are you guys using?

Post by Petermint »

Backintime (uses rsync) and rsync copy everything. With Backintime backing up to the Ext4 file system, Backintime can create existing files as links to the existing copy of the file from a previous backup. The first backup takes forever because you copy all the files. The next backup to the same partition physically copies only new and changed files. Everything else speeds through because the file becomes just one link which is just an inode write, not a file copy.

If you backup to NTFS or most other common file systems, you loose the space saving through links.

Naturally you have several external disks in rotation for backup. The first copy to each disk is a full copy. The next is 5 minutes instead of 5 hours.

Timeshift can do the same simple backup but has only one backup profile. Backintime can have multiple profiles for different backups. My main desktop computer has one profile to backup the SSDs to dinosaur disks in the same machine. A separate profile backs up the wobbly old rotating disks to external wobbly new USB 3 disks.
afora
Level 4
Level 4
Posts: 203
Joined: Mon Aug 26, 2019 7:35 pm

Re: Incremental backup - what app are you guys using?

Post by afora »

This is fantastic advice!

Thank you everybody!
afora
Level 4
Level 4
Posts: 203
Joined: Mon Aug 26, 2019 7:35 pm

Re: Incremental backup - what app are you guys using?

Post by afora »

pbear wrote:
Thu Dec 05, 2019 11:58 pm
One thing Grsync doesn't have, though, is scheduling capabilities.
Is there any advantage of not using cron for scheduling?
ZakGordon
Level 5
Level 5
Posts: 862
Joined: Thu Feb 12, 2015 11:07 am

Re: Incremental backup - what app are you guys using?

Post by ZakGordon »

ozbear wrote:
Thu Dec 05, 2019 1:49 am
On a (potentially) side note, now that my Windows/Linux dual boot machinery is working, it appears that my Windows Acronis True Image Home backup is picking up the Linux partitions now too, since there was no exclusion rule for them. Weird. Whether or not they are restorable/usable I haven't had time to test yet, but if they are it is a cheaper solution than Acronis for Linux. Hope that wasn't too off-topic.
Within Linux I use rsync.

Regards, Oz
Thanks for that info. I have not looked into this yet for my own dual boot setup, so your input has given me an option to look into :)

Usually the way i do backups is to create a complete new total system image every month or so. Important data (say documents or accounts etc) is just copied to USB key as required and in general for my uses i rarely loose that much info with the monthly schedule. I had been wondering about the dual-boot setup though.
Laptop overheating? Check link here:itsfoss guide . A move from Cinnamon to XFCE can give a -5 to -10 degrees C change on overheating hardware.

Build a modern dual-boot Ryzen Win7/Linux Mint PC:Tutorial
pbear
Level 15
Level 15
Posts: 5665
Joined: Wed Jun 21, 2017 12:25 pm
Location: San Francisco

Re: Incremental backup - what app are you guys using?

Post by pbear »

afora wrote:
Fri Dec 06, 2019 1:28 am
pbear wrote:
Thu Dec 05, 2019 11:58 pm
One thing Grsync doesn't have, though, is scheduling capabilities.
Is there any advantage of not using cron for scheduling?
Not sure I understand the question. I mentioned Grsync doesn't have scheduling because some users regard that as a deal killer. Regular rsync can be scheduled with cron, though I don't do it. Haven't ever heard of anyone using cron with Grsync, but that doesn't mean it can't be done. If it turns out the answer is "yes," please report back as that would be useful to know. This topic comes up often.
Last edited by pbear on Fri Dec 06, 2019 9:29 pm, edited 1 time in total.
Petermint
Level 6
Level 6
Posts: 1166
Joined: Tue Feb 16, 2016 3:12 am

Re: Incremental backup - what app are you guys using?

Post by Petermint »

Some of the backups are designed for purely manual use or for servers that are always on. When you use rsync in Backintime, you get scheduling, you get messages when the backup is scheduled overnight but the computer is off, and you get messages when the external USB disk is unplugged.
bt101
Level 2
Level 2
Posts: 61
Joined: Thu Feb 14, 2013 12:30 am

Re: Incremental backup - what app are you guys using?

Post by bt101 »

I'll throw-in.

I use rdiff-backup. It has no GUI and I would say it is not the easiest to learn (however like anything, once you know it, it's not bad).
It is based on rsync. There's a million ways you can run it, but the simplest/quickest way is to run it just like rsync (rdiff-backup source_dir dest_dir).

Just like rsync, it makes a mirror of your source. However it also makes a special folder at the root of your backup that contains all of incremental data for previous backups. You don't have to worry about that folder or what's in it. Whenever you repeatedly run rdiff-backup, the target backup ends up as a mirror of the current source, and the older incremental data in the special folder is updated. So 99% of the time, when you want to restore a file, you typically want the last version right? So you can get that file from the backup with no special tools as the backup is just a bunch of files/folders that mirror the source. If you do want an older version of a file, you just use a special command to get it (that program will re-assemble the file from the latest copy and the incremental info in that special folder).

So you might say...my backup is going to be at least as big as the source (because it is a mirror) plus it has that incremental info... that's not efficient because my traditional backup does compression!

Well, if you look at your traditional backup (let's say you do fulls every week with daily incrementals and you compress them)... you really should have more than one full (in case one gets messed up). So you have at least two fulls and likely more. Those can add up to more than a mirror.

The fellow who wrote the program did a good job because it is super fast and efficient. For the incremental info, it only stores the changes to the files and it efficiently only sends those changes to the backup.

Now I know that the first thought is...I'll just run rdiff-backup as a cron job and send the backup to a USB key or a CIFS share. You can certainly do this, however there is a better way. You can install rdiff-backup on one server and all of your client machines. You then do all of you configuration on the server and schedule the backups there. It will reach out to the client machines via ssh to run in a client/server mode. The advantages are:
  • all config and scheduling is one on the server (you just need to install an ssh key on the clients)
  • the backup info is sent across the wire encrypted (so you could backup across the internet)
  • the backup are not exposed to the client (so if the client clicks on a bad email, they can't destroy their backups)
So what's the downside? Aside from not being the easiest to setup and no GUI, the only downside I find is that the backup is not encrypted. You could take steps to store the backup on an encrypted drive so that the info is encrypted "at rest" but when it's online it is not encrypted.

If you want an encrypted backup, another one that I am using is duplicity with the duply frontend. Again, these are both setup with config files (no GUI). The duply frontend just makes config and running a bit easier. This is a more traditional backup where you would do (say) weekly fulls and daily incrementals. They are compressed and encrypted, They can/are also sent to the target via ssh (so not exposing some share to the client where ransomware could strike it). Since these backups are encrypted, you could just send them to any repository on the net.

Here the thing... once rdiff-backup does its initial seed, it runs super quick for every subsequent backup. That's especially good if you have a huge store of (let's say) videos. If there is a huge store of data but little changes, rdiff-backup will fly through it in seconds and be done. Even if you have large files that have parts of them that are changed, rdiff-backup only send the changes (very efficient). The more traditional backup will cripple your cpu and network when doing those weekly fulls. When I notice my laptop scorching my leg, I know that duply/duplicity is running. So I try and only use duply/duplicity for smaller stuff that I want encrypted. rdiff-backup is good for everything else.


I had my linux server drive die (and that server was running and storing tons of stuff). I successfully brought back the whole thing from my rdiff-backup backup.
User avatar
Termy
Level 5
Level 5
Posts: 776
Joined: Mon Sep 04, 2017 8:49 pm
Location: UK
Contact:

Re: Incremental backup - what app are you guys using?

Post by Termy »

The only backup solutions I need and use are BackMeUp (spoiler alert, I wrote it) and the amazing Clonezilla. I hands down have to recommend Clonezilla, especially if you're exploring Linux distributions, or going more in-depth into Linux.
Here to help.

I'm LearnLinux (LL) on YouTube: https://www.youtube.com/channel/UCfp-lN ... naEE6NtDSg
I'm also terminalforlife (TFL) on GitHub: https://github.com/terminalforlife
User avatar
AZgl1500
Level 14
Level 14
Posts: 5179
Joined: Thu Dec 31, 2015 3:20 am
Location: Oklahoma where the wind comes sweeping down the plains
Contact:

Re: Incremental backup - what app are you guys using?

Post by AZgl1500 »

I prefer LuckyBackup because I can use NEMO and directly look at any single file in the backups.
the directories are carbon copies of the originals.

It only adds additional files not previously copied, or if a file is modified, it catches that too.

I assume everyone is backing up to an external drive, as I am.
Linux Mint 19.3 Cinnamon
Petermint
Level 6
Level 6
Posts: 1166
Joined: Tue Feb 16, 2016 3:12 am

Re: Incremental backup - what app are you guys using?

Post by Petermint »

Backintime also produces "carbon copies" you can browse. Each snapshot is a separate directory by date/time of backup. The first snapshot is everything. The next snapshots show all files but contain only changed files. The magic works only on Ext4 because Ext4 is the only common file system to allow multiple file entries to point to the same file.
afora
Level 4
Level 4
Posts: 203
Joined: Mon Aug 26, 2019 7:35 pm

Re: Incremental backup - what app are you guys using?

Post by afora »

Are there any principal differences to be aware between rdiff-backup and backintime? Something like a deal breaker like encrypted mirror or ssh'ing over the internet?

Even though what you are saying about backintime is very appealing, my worry is if it creates an ocean of daily hardlinks the backup file system will eventually drown itself. Or may be I misunderstand how it's done. Also I do not worry about not being able to backup hard links with backintime, as I do not use any on the data partition (the one I need an incremental routine on). I use hard links only on the O/S partition which gets reimaged with Clonezilla and I dont need incremental granular backup there.

That's absolutely fantastic feedback guys, it's not only me but I'm sure many people will appreciate this invaluable commentary!
Post Reply

Return to “Newbie Questions”