[solved] A curious backup issue

[Cammo]

Hi forum,

I've made a backup bash script using rsync to sync files between my laptop and USB. Basically, I use rsync -uai x y to backup x to y. Now, the files transfer fine, but interestingly the script doesn't update old files on y with the newer versions from x. It just skips them. Isn't the -u switch supposed to over-write the old versions with the new one?

The script runs without any errors and the log files confirm that new files are being written, but old ones are being skipped rather than overwritten.

Any help is much appreciated.

Thanks,

Cammo

. The full source code is attached below, but

Code: Select all

#!/bin/bash
# Purpose:      For syncing files between two drives, where changes might occur
#           	on either. (I use it to backup documents between a PC and USB)
#
#		Rsyncs $OLD-PATH to $NEW-PATH, ignoring newer files on $NEW-PATH. Then, rsyncs $NEW-PATH onto $OLD-PATH, ignoring newer files.
#		The output is redirected to $logLocation, so you can grep through what's been copied later at your leisure.

# Usage:      ./ backup-script.sh
# Author:      Cammo
# Date:        Mon 12 May 2014
# Version:      0.5
# Disclaimer:   Feel free to use it, but don't blame me if your computer conks out.

#=====================================================================
#              Enter your required params here...
OldPath='/mnt/Storage/Uni/'
mountPoint='/media/cameron/UUI'
NewPath="$mountPoint/Uni/"
myName='cameron'

now=$(date +"%F")
logLocation="/home/$myName/Documents/BackupLogs/backup-output-$now.txt"
#=====================================================================

chown $myName: $mountPoint; chown $myName: $logLocation
mount '/dev/sdb1' $mountPoint>>$logLocation
mkdir "$mountPoint/Uni">>$logLocation
echo "">>$logLocation
echo "">>$logLocation
echo $(date +"%c")>>$logLocation
echo ''>>$logLocation
echo 'Hard drive path = ' $OldPath>>$logLocation
echo 'USB path = ' $NewPath>>$logLocation
echo ''>>$logLocation
echo "Starting  backup from $OldPath to $NewPath" >>$logLocation
echo ''>>$logLocation


rsync -uai $OldPath $NewPath>>$logLocation
echo 'Syncing Old->New done'>>$logLocation
rsync -uai $NewPath $OldPath>>$logLocation
echo 'Syncing New->Old done'>>$logLocation
chown -R $myName: $NewPath>>$logLocation
echo 'taking ownership of new files -> done'>>$logLocation

[WharfRat]

Try this

Create a file in an empty new folder

Code: Select all

echo "I was created on $(date)" > afile

rsync the file to the flash drive then update afile

Code: Select all

echo "I was updated on $(date)" > afile

Add -n to the rsync options to see if it is detected to get synced e.g., rsync -naui

-n is a trial run, nothing will actually get updated. If it is listed then remove the -n, let it run then diff the files.

Good luck

[Cammo]

Sorry for the late reply WharfRat, I've been bogged down in assignments over the last few days. Getting ready for exam season, and all the joys that entails!

Ok, I ran WharfRat's suggestion, here's the code I used:

Code: Select all

cameron@cameron-ThinkPad-X220 /mnt/Storage/Test $ echo  "I was created on $(date)" > test.txt
cameron@cameron-ThinkPad-X220 /mnt/Storage/Test $ rsync -a /mnt/Storage/Test/ /media/cameron/sdcard/Test/ 
rsync: failed to set times on "/media/cameron/sdcard/Test/.": Operation not permitted (1)
rsync: mkstemp "/media/cameron/sdcard/Test/.test.txt.OORhUA" failed: Permission denied (13)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1070) [sender=3.0.9]
cameron@cameron-ThinkPad-X220 /mnt/Storage/Test $ sudo !!
sudo rsync -a /mnt/Storage/Test/ /media/cameron/sdcard/Test/ 
[sudo] password for cameron: 
cameron@cameron-ThinkPad-X220 /mnt/Storage/Test $ echo  "I was modified on $(date)" > test.txt
cameron@cameron-ThinkPad-X220 /mnt/Storage/Test $ sudo rsync -uain /mnt/Storage/Test/ /media/cameron/sdcard/Test/
.d...p..... ./
>f.stp..... test.txt
cameron@cameron-ThinkPad-X220 /mnt/Storage/Test $ sudo rsync -uai /mnt/Storage/Test/ /media/cameron/sdcard/Test/
.d...p..... ./
>f.stp..... test.txt

When I opened up the backed up copy of test.txt, it was "I was modified..." etc as expected. But when I ran the same command from my script, it did the same thing it did before - copied new files over, but didn't overwrite modifications.

I note, however that my tests only ran when I sudo'd rsync. I'm guessing this is because /media/cameron/sdcard is owned by root, so my normal user can't touch it. I thought that root privileges was the problem, so I tried running my backup script as root - both by sudo'ing it and even (out of desperation) su'ing and entering in the code line by line. Close, but still no cigar.

So, I tried chown'ing the mount point to cameron (my normal user), but got the following error:

Code: Select all

cameron@cameron-ThinkPad-X220 /mnt/Storage/Test $ sudo chown -R cameron: /media/cameron/sdcard
chown: changing ownership of ‘/media/cameron/sdcard/Test’: Operation not permitted

And the same error when I logged in as root actual (sudo -i) (which is strange, because that account let me fix grub up when it died, and I can't imagine my backup point needing higher privileges than that!)

Thanks as always,

Cammo

[WharfRat]

Cammo,

Yes, the problems you're experiencing is all permissions, i.e., not being able to update the file's timestamp.

I use rsync -aviuh and never have a problem as long as the permissions are set correctly.

What's the file system of the sdcard If it's not a linux file system then that's most likely the problem.

[Cammo]

Hi WharfRat,

My SD card is a 32G SD-HC, which registers as vfat. I've checked that it isn't write-protected and I can write to it by sudo'ing - except, oddly enough, with rsync --update.

Do you have any ideas on how to fix the problem? I'd prefer to avoid reformatting my sdcard to ext-n (because I also use it on a Win system) if at all possible.

I've tried using the chown command to take over the mount point, but as mentioned in my last post, I get permission denied errors, even when running it as root.

[WharfRat]

Cammo,

The FAT filesystems do not understand Linux permissions at all. If you backed-up anything that's permission critical like su or umount in /bin that has the setuid bit set it's not going to restore with those permissions.

You can try mounting the partition with the options uid=1000,gid=1000 for access while backing-up - that's about it.

Good luck

[Cammo]

Thanks, that fixed it for me.

Much appreciated