[SOLVED] Incremental backup - what app are you guys using?

All Gurus once were Newbies
Forum rules
There are no such things as "stupid" questions. However if you think your question is a bit stupid, then this is the right place for you to post it. Please stick to easy to-the-point questions that you feel people can answer fast. For long and complicated questions prefer the other forums within the support section.
Before you post please read how to get help
User avatar
AZgl1500
Level 14
Level 14
Posts: 5217
Joined: Thu Dec 31, 2015 3:20 am
Location: Oklahoma where the wind comes sweeping down the plains
Contact:

Re: Incremental backup - what app are you guys using?

Post by AZgl1500 »

I use LuckyBackup in manual mode only, so the storage requirements are very miniscule.
Linux Mint 19.3 Cinnamon
pbear
Level 15
Level 15
Posts: 5664
Joined: Wed Jun 21, 2017 12:25 pm
Location: San Francisco

Re: Incremental backup - what app are you guys using?

Post by pbear »

afora wrote:
Mon Dec 16, 2019 12:14 am
... my worry is if it creates an ocean of daily hardlinks the backup file system will eventually drown itself.
I don't bother with snapshots for data files, but I think you're unduly concerned. It takes a long time for snapshots to get significantly larger than source, as not much changes day-to-day. If you're paying attention at all, you'll spot a problem long before you run out of space, which is unlikely to happen anyway. And if an issue arises, it's easy to tailor the backup plan to the situation, e.g., only keep a one-month look back.

No experience with rdiff-backup, so can't comment.
afora
Level 4
Level 4
Posts: 203
Joined: Mon Aug 26, 2019 7:35 pm

Re: Incremental backup - what app are you guys using?

Post by afora »

pbear wrote:
Mon Dec 16, 2019 1:56 pm
you'll spot a problem long before you run out of space, which is unlikely to happen anyway.
Yeah, I agree, my concern was more about a sheer number of hardlinks - I do not know how they are represented in *inux but if they are like ordinary (but empty) files, the FS has to keep track of them, and hence having tens or hundreds of millions of them eventually may be a problem. Not because the FS will be out of space, but because it's just too large of a file tree to keep and traverse. But I may be wrong...
pbear
Level 15
Level 15
Posts: 5664
Joined: Wed Jun 21, 2017 12:25 pm
Location: San Francisco

Re: Incremental backup - what app are you guys using?

Post by pbear »

No reason to think that's a problem and, as mentioned, easily fixed. If you don't want to use snapshots, don't. No need to make stuff up, though.
afora
Level 4
Level 4
Posts: 203
Joined: Mon Aug 26, 2019 7:35 pm

Re: Incremental backup - what app are you guys using?

Post by afora »

I do want to use snapshots, I think it's an extremely useful feature I really love the idea of. I was interested in any tradeoffs.
User avatar
AZgl1500
Level 14
Level 14
Posts: 5217
Joined: Thu Dec 31, 2015 3:20 am
Location: Oklahoma where the wind comes sweeping down the plains
Contact:

Re: Incremental backup - what app are you guys using?

Post by AZgl1500 »

afora wrote:
Mon Dec 16, 2019 6:40 pm
I do want to use snapshots, I think it's an extremely useful feature I really love the idea of. I was interested in any tradeoffs.
I have daily snapshots set up. keep 5.

got me out of trouble last week, I was messing around and did something that screwed things up royally.
bad enough I could not fix it.

so, ran the last daily and wallah!
all good, but lost the last 2 or 3 little tiny tweeks that I had done.
Linux Mint 19.3 Cinnamon
pbear
Level 15
Level 15
Posts: 5664
Joined: Wed Jun 21, 2017 12:25 pm
Location: San Francisco

Re: Incremental backup - what app are you guys using?

Post by pbear »

afora wrote:
Mon Dec 16, 2019 6:40 pm
I was interested in any tradeoffs.
If I were using snapshots for data files, I'd use a look back of a month. Plenty of time to catch oopsies, but limited footprint. First time I did it, though, I'd see what happens with no limit, just cuz I'm curious that way. Heck, I set up a separate Timeshift partition just so I could watch what it does. YMMV.
afora
Level 4
Level 4
Posts: 203
Joined: Mon Aug 26, 2019 7:35 pm

Re: [SOLVED] Incremental backup - what app are you guys using?

Post by afora »

In the end I settled on writing my own rsync script. Here's my thoughts I hope will help somebody:

USE CASE:
A small office 3-5 workstations with files on local as well as on a file server, a couple of servers. Less than 4TB of user data. We do not need encryption as our volumes are encrypted. We do not backup our data over the internet. Three profiles of incremental backups:
- local-to-local and server-to-server - on a 20 minute rotational schedule (addresses accidental deletes/changes by users)
- local-to-server - every 12 hours (addresses local workstation failures/theft)
- server-to-offsite - every 7 days

1. NO -> Borg and other deduplication solutions
Although deduplication is an attractive feature, Borg simply does not work for us. Cons:

- even with a GUI front end (Vorta) you cannot easily traverse the FS tree structure without awkward mounting/unmounting. If you are running into a couple of dozen of snapshots, Borg's interface becomes very annoying. I want to see the file structure instantaneously and effortlessly. And I cannot.

- the GUI front end I tried (Vorta) comes in a flatpak of 0.5GB on Mint 19.3, it's just silly.

- Users occassionally need to diff two snapshots to see what changed as effortlessly as possible - that's very cumbersome. Productivity of users is my concern, not storage restrictions.

- I have not bought into bitwise deduplication. There is no free cheese, you either need storage or CPU, given our use case, it just does not make sense. YMMV.

2. NO -> Backintime and similar.
I only considered Backintime as other comparable apps (e.g. Timeshift) are either too lightweight or not very well maintained or/and both. Backintime was promising but:

- It has really poor shell functionality, basically being a GUI focussed tool. On a headless server, it's buggy, unreliable, and configurations are very backward (plus see the next point). E.g. see viewtopic.php?f=90&t=326229

- cron scheduling is a killer which is only managed from GUI (in fact cron entries get reset if you start the GUI).

- ssh support is half cooked, as is handling file permissions. I'm sure there may be workarounds, I am not that interested in the app to spend my time on it.

- it could be fine for a Windows-refugee user whose only workflow is GUI, for us it's just a fail. Shame, and annoyingly, the application was very close to being what we need.
User avatar
AZgl1500
Level 14
Level 14
Posts: 5217
Joined: Thu Dec 31, 2015 3:20 am
Location: Oklahoma where the wind comes sweeping down the plains
Contact:

Re: [SOLVED] Incremental backup - what app are you guys using?

Post by AZgl1500 »

afora wrote:
Mon Jul 27, 2020 3:57 am

- cron scheduling is a killer which is only managed from GUI (in fact cron entries get reset if you start the GUI).
I use cron for automated system updates, and I wrote the cron file manually, not thru a gui, with a editor.

took me a bit to learn how to, but I enjoyed "being in control" of the when, and the how.

my system backups are done manually via Timeshift to extUSB, and LuckyBackup to extUSB ...
granted, you have a much more demanding case, than I, as a single user.

but, my point is, the 'cron' file does NOT get wiped out when I edit it.
using the crontab job file to setup Automatic Updates
sudo crontab -e - creates and opens a root crontab in the default CLI text editor.
Linux Mint 19.3 Cinnamon
Post Reply

Return to “Newbie Questions”