[SOLVED] Incremental backup - what app are you guys using?

[AZgl1800]

I use LuckyBackup in manual mode only, so the storage requirements are very miniscule.

[pbear]

... my worry is if it creates an ocean of daily hardlinks the backup file system will eventually drown itself.

I don't bother with snapshots for data files, but I think you're unduly concerned. It takes a long time for snapshots to get significantly larger than source, as not much changes day-to-day. If you're paying attention at all, you'll spot a problem long before you run out of space, which is unlikely to happen anyway. And if an issue arises, it's easy to tailor the backup plan to the situation, e.g., only keep a one-month look back.

No experience with rdiff-backup, so can't comment.

[afora]

you'll spot a problem long before you run out of space, which is unlikely to happen anyway.

Yeah, I agree, my concern was more about a sheer number of hardlinks - I do not know how they are represented in *inux but if they are like ordinary (but empty) files, the FS has to keep track of them, and hence having tens or hundreds of millions of them eventually may be a problem. Not because the FS will be out of space, but because it's just too large of a file tree to keep and traverse. But I may be wrong...

[pbear]

No reason to think that's a problem and, as mentioned, easily fixed. If you don't want to use snapshots, don't. No need to make stuff up, though.

[afora]

I do want to use snapshots, I think it's an extremely useful feature I really love the idea of. I was interested in any tradeoffs.

[AZgl1800]

I do want to use snapshots, I think it's an extremely useful feature I really love the idea of. I was interested in any tradeoffs.

I have daily snapshots set up. keep 5.

got me out of trouble last week, I was messing around and did something that screwed things up royally.
bad enough I could not fix it.

so, ran the last daily and wallah!
all good, but lost the last 2 or 3 little tiny tweeks that I had done.

[pbear]

I was interested in any tradeoffs.

If I were using snapshots for data files, I'd use a look back of a month. Plenty of time to catch oopsies, but limited footprint. First time I did it, though, I'd see what happens with no limit, just cuz I'm curious that way. Heck, I set up a separate Timeshift partition just so I could watch what it does. YMMV.

[afora]

In the end I settled on writing my own rsync script. Here's my thoughts I hope will help somebody:

USE CASE:
A small office 3-5 workstations with files on local as well as on a file server, a couple of servers. Less than 4TB of user data. We do not need encryption as our volumes are encrypted. We do not backup our data over the internet. Three profiles of incremental backups:
- local-to-local and server-to-server - on a 20 minute rotational schedule (addresses accidental deletes/changes by users)
- local-to-server - every 12 hours (addresses local workstation failures/theft)
- server-to-offsite - every 7 days

1. NO -> Borg and other deduplication solutions
Although deduplication is an attractive feature, Borg simply does not work for us. Cons:

- even with a GUI front end (Vorta) you cannot easily traverse the FS tree structure without awkward mounting/unmounting. If you are running into a couple of dozen of snapshots, Borg's interface becomes very annoying. I want to see the file structure instantaneously and effortlessly. And I cannot.

- the GUI front end I tried (Vorta) comes in a flatpak of 0.5GB on Mint 19.3, it's just silly.

- Users occassionally need to diff two snapshots to see what changed as effortlessly as possible - that's very cumbersome. Productivity of users is my concern, not storage restrictions.

- I have not bought into bitwise deduplication. There is no free cheese, you either need storage or CPU, given our use case, it just does not make sense. YMMV.

2. NO -> Backintime and similar.
I only considered Backintime as other comparable apps (e.g. Timeshift) are either too lightweight or not very well maintained or/and both. Backintime was promising but:

- It has really poor shell functionality, basically being a GUI focussed tool. On a headless server, it's buggy, unreliable, and configurations are very backward (plus see the next point). E.g. see viewtopic.php?f=90&t=326229

- cron scheduling is a killer which is only managed from GUI (in fact cron entries get reset if you start the GUI).

- ssh support is half cooked, as is handling file permissions. I'm sure there may be workarounds, I am not that interested in the app to spend my time on it.

- it could be fine for a Windows-refugee user whose only workflow is GUI, for us it's just a fail. Shame, and annoyingly, the application was very close to being what we need.

[AZgl1800]

- cron scheduling is a killer which is only managed from GUI (in fact cron entries get reset if you start the GUI).

I use cron for automated system updates, and I wrote the cron file manually, not thru a gui, with a editor.

took me a bit to learn how to, but I enjoyed "being in control" of the when, and the how.

my system backups are done manually via Timeshift to extUSB, and LuckyBackup to extUSB ...
granted, you have a much more demanding case, than I, as a single user.

but, my point is, the 'cron' file does NOT get wiped out when I edit it.

using the crontab job file to setup Automatic Updates
sudo crontab -e - creates and opens a root crontab in the default CLI text editor.