Can somebody recommend a backup/file sync solution?

[gene0915]

Linux newbie here so please be gentle and speak as if you're talking to a 5 year old.

Was using Linux Mint 18.x for over a year then recently switched to Mint 19. Cinnamon desktop.

My /mnt/md0 array is RAID 6 and has a total capacity of 40TB with 22TB's in use. XFS file system.

My /mnt/backup array is RAID 5 and has a total capacity of 24TB with 22TB's in use. EXT4 file system.

Every morning at 8am, Luckybackup kicks off and looks for new files on md0 and backs them up to the backup array. Been using this process for a while and have no complaints. Until recently.

Last night, I was accessing a Samba share on md0 copying over new data and easing old stuff. After I was done, I decided to go ahead and kick off LuckyBackup early. During the backup, I noticed that a few files generated a red, I/O error then towards the end, everything was in red. I didn't look through the logs but I noticed that on my backup array, stuff that I deleted from the Samba share (on /md0) wasn't deleted from the /backup array.

The following are questions I'm just "thinking out loud about" since I'm not home to test my theory but would like to solicit feedback.

Q) Should LuckyBackup have deleted files off the /backup array that don't exist on the md0 array?

Q) Maybe LuckyBackup isn't a file syncing utility.... it just backs up everything onto /backup and doesn't delete stuff that was deleted from the source?

I will check more tonight but I think I only see this behavior when files are deleted from Samba shares. I'm -almost- positive that files I delete on /md0 non-Samba shares are nuked on the /backup array when LuckyBackup is doing it's daily backup. Or heck, maybe it hasn't been all this time and I just never noticed?!?

Something else that was weird. In Nemo, it said that /backup had about 830 gigs free. But when I went into the 'Menu' button in Cinnamon and typed 'disks' and went into that applet, it shows me all my drives and arrays. When I scroll down to the 24TB backup array, it shows 1.1TB free. Why the discrepancy? I know that with EXT4, by default, it reserves X amount of drive space to help with fragmentation. A LONG, LONG time ago, I issues some commands to shrink that down to 1% or 2%. Would that explain it? (Nemo shows the free space (after the shrink command I did) and the Disk app shows the raw value?)


Beyond all that, is there another (better) backup program I should be using? I was thinking about looking into something that does real-time monitoring and delete files instantly from the target if they are deleted from the source. Although, I sort of like the daily backup idea. This way, if I accidentally delete something, I can recover it from the /backup array.

[gm10]

I can't help you with Luckybackup because I never used it. The description says it's a front-end for rsync though, so it should be fine for synchronization. Might be something with your settings or the permission level you run it under, I suggest to check its logs.

Regarding your free space question, yes, the difference is the reserved blocks. Depending on your nemo settings there may be additional discrepancy due to decimal and binary units for free space display. gnome-disks uses decimal, nemo can use either.

[gene0915]

I can't help you with Luckybackup because I never used it. The description says it's a front-end for rsync though, so it should be fine for synchronization. Might be something with your settings or the permission level you run it under, I suggest to check its logs.

Regarding your free space question, yes, the difference is the reserved blocks. Depending on your nemo settings there may be additional discrepancy due to decimal and binary units for free space display. gnome-disks uses decimal, nemo can use either.

That was it. I issued a command to remove all the space that EXT4 was reserving on the backup array and now, free space lines up on the Disk app and Nemo.

On another note, I think I'll need to replace LuckyBackup. I verified last night that files deleted from a Samba share by Windows machines are not being deleted when LuckyBackup backs up data. They just keep accumulating on the LuckyBackup destination array.

[gm10]

On another note, I think I'll need to replace LuckyBackup. I verified last night that files deleted from a Samba share by Windows machines are not being deleted when LuckyBackup backs up data. They just keep accumulating on the LuckyBackup destination array.

Consider simply running rsync directly then. The --delete parameter will delete files from the destination that don't exist on the source, for example. It's strange the LuckyBackup would not be able to use this.

[gene0915]

On another note, I think I'll need to replace LuckyBackup. I verified last night that files deleted from a Samba share by Windows machines are not being deleted when LuckyBackup backs up data. They just keep accumulating on the LuckyBackup destination array.

Consider simply running rsync directly then. The --delete parameter will delete files from the destination that don't exist on the source, for example. It's strange the LuckyBackup would not be able to use this.

If I remember correctly, in the LB configuration screen, the box about 'delete files that don't exist on source' is indeed checked but that's a good idea. I'll see if I can find out the exact rsync command that LB runs and just open up a terminal and call it that way, verifying that the --delete switch is there.