Two sets of files...

[Arch_Enemy]

I have a master and a backup for my photos.

Some of the photos have been resized for posting, sharing and on-line exhibits. Unfortunately, the full size originals are on one drive and the downsized copies are on another, with the same file names.

I tried using rsync with the --ignore-existing and --size-only parameters, but of course this did not do what I want. I want to sync both drives so they have the larger size files on both drives. There are far too many to sort out and I'd like to be able to use a file transfer like rsync to mirror the drives with the larger files only.

Help me Obiwan-Kenobe...

[WharfRat]

Check your source/destination wiht the -i option rsync -nai ./setclock ../script/setclock

I added a blank line to ./setclock and

Code: Select all

[bill@XPS] ~/junk $ rsync -nai  ./setclock ../script/setclock 
>f.st...... setclock
[bill@XPS] ~/junk $ 

[Flemur]

Unfortunately, the full size originals are on one drive and the downsized copies are on another, with the same file names.

Couldn't you just copy the originals over the resized ones? Or is the directory structure too weird for that?

[Arch_Enemy]

Unfortunately, the full size originals are on one drive and the downsized copies are on another, with the same file names.

Couldn't you just copy the originals over the resized ones? Or is the directory structure too weird for that?

There are a LOT of files in basic directories, but I want to sync them so the larger file appears on both mirrors. Also there are lots of directory levels in places.
Some of them have the small version on one mirror and the large version on the other, and vice-versa.

[Arch_Enemy]

Check your source/destination wiht the -i option rsync -nai ./setclock ../script/setclock

I added a blank line to ./setclock and

Code: Select all

[bill@XPS] ~/junk $ rsync -nai  ./setclock ../script/setclock 
>f.st...... setclock
[bill@XPS] ~/junk $ 

Is that -i as in ignore times?
Also, while I was a DOS bat file wizard, I don't get the setclock command in this use.

[WharfRat]

The -i option outputs a change-summary for all updates, i.e. an itemized list of what action is being taken and for what reason.

The setclock script was modified in the current folder so it then shows that it will be synced with (copied to) the script of that name in ../script

The >f.st...... setclock means that the flle is being transferred to the destination due to both a size and time change.

[rene]

So, is it correctly specified to be "copy all files from under ./source/ to the same path under ./dest/ unless the destination already exists and has a larger filesize"? (note, FILEsize; if you are talking about image sizes this may or may not be the same thing and would need a different test). If so, I don't believe you can hang this off rsync in any direct manner; would need something "manual" such as

Code: Select all

find source/ -type f -exec /bin/bash -c 'SRC="$1"; DST="dest/${SRC##source/}"; if [ ! -e "$DST" ] || [ $(stat -c %s "$SRC") -ge $(stat -c %s "$DST") ]; then echo cp -v "$SRC" "$DST"; fi' -- \{} \;

Remove the "echo" once satisfied this does what you want. Which it may not; this assumes the directory structure under ./source/ and ./dest/ equal. Supposedly tweakable as per actual need?

[Arch_Enemy]

So, is it correctly specified to be "copy all files from under ./source/ to the same path under ./dest/ unless the destination already exists and has a larger filesize"? (note, FILEsize; if you are talking about image sizes this may or may not be the same thing and would need a different test). If so, I don't believe you can hang this off rsync in any direct manner; would need something "manual" such as

Code: Select all

find source/ -type f -exec /bin/bash -c 'SRC="$1"; DST="dest/${SRC##source/}"; if [ ! -e "$DST" ] || [ $(stat -c %s "$SRC") -ge $(stat -c %s "$DST") ]; then echo cp -v "$SRC" "$DST"; fi' -- \{} \;

Remove the "echo" once satisfied this does what you want. Which it may not; this assumes the directory structure under ./source/ and ./dest/ equal. Supposedly tweakable as per actual need?

Yeah, I want them to be mirrors with the same files/size in both. They have the same dir names. And the difference in file sizes is significant. The smaller ones are in the 100-400K region while the original files are 1.5~5MB area.

[rene]

Yeah, I want them to be mirrors with the same files/size in both.

A from the above evolved version as a script is below. DO NOT RUN this on you actual to-be-synchronized directories for now; I may have misinterpreted the intention and even if not, I may have simply screwed something up; in (ba)sh it's real easy to overlook something silly when not specifically testing with actual input. This potentially overwrites files (i.e., by keeping the larger sized one only) and you should therefore first test this on copies of said to-be-synchronized directories.

Save as, say, ~/bin/sync.sh, make executable (chmod +x ~/bin/sync.sh) and run as sync.sh dir1/ dir2/. Should synchronize the two directories, keeping the larger sized file only in case of conflict.

Code: Select all

#!/bin/bash

copy() {
    find "$1" -type f -print0 | while read -d $'\0' SRC; do
        DST="$2${SRC##$1}"
        mkdir -pv $(dirname "$DST") && touch "$DST"
        [ $(stat -c %s "$DST") -ge $(stat -c %s "$SRC") ] && continue
        cp -v "$SRC" "$DST"
    done
}

[ -d "$1" ] && [ -d "$2" ] || exit

copy "$1" "$2"
copy "$2" "$1"

[Arch_Enemy]

Yeah, I want them to be mirrors with the same files/size in both.

A from the above evolved version as a script is below. DO NOT RUN this on you actual to-be-synchronized directories for now; I may have misinterpreted the intention and even if not, I may have simply screwed something up; in (ba)sh it's real easy to overlook something silly when not specifically testing with actual input. This potentially overwrites files (i.e., by keeping the larger sized one only) and you should therefore first test this on copies of said to-be-synchronized directories.

Save as, say, ~/bin/sync.sh, make executable (chmod +x ~/bin/sync.sh) and run as sync.sh dir1/ dir2/. Should synchronize the two directories, keeping the larger sized file only in case of conflict.

Code: Select all

#!/bin/bash

copy() {
    find "$1" -type f -print0 | while read -d $'\0' SRC; do
        DST="$2${SRC##$1}"
        mkdir -pv $(dirname "$DST") && touch "$DST"
        [ $(stat -c %s "$DST") -ge $(stat -c %s "$SRC") ] && continue
        cp -v "$SRC" "$DST"
    done
}

[ -d "$1" ] && [ -d "$2" ] || exit

copy "$1" "$2"
copy "$2" "$1"

I saw what you did there.

Yeah, that looks like it might work. I need to pick up a new drive soon, so before I load an OS on it I will probably clone the drive with the smaller files to it, and then try this out.

Thanks!

Is there a way to run it and just have it print what it WOULD do to std out?

[rene]

Is there a way to run it and just have it print what it WOULD do to std out?

Not generally/nicely; the mkdir && touch line is explicitly there so as to not have to surround the next stat call with confusing guards; needs to run for functionality. Of course, non-generally, if it is actually the case that your files and directory structure under dir1/ and dir2/ fully match except only for possible size differences, that line does nothing interesting anyway so simply sticking an echo in front of the (mkdir, the touch and the) cp would in that case work.

But the better approach is cp -rv dir1 dir1.copy, cp -rv dir2 dir2.copy, sync.sh dir1.copy/ dir2.copy/. The script is purposefully chatty...

[phd21]

HI Arch_Enemy,

I just read your post and the good replies to it. Here are my thoughts on this as well.

There are many file renamers that can easily change the original files in their folder or the modified files in their folder to have a suffix (or prefix), like "something-orig.xxx" or "something-mod.xxx". Look for "rename" in the "Synaptic Package Manager (SPM)", like Krename, Gwenrename, pyrenamer, gprename, etc...


Hope this helps ...

krename_FoldersFiles1.jpg

.

krename_Destination1.jpg

.

krename_Filename1.jpg

[Arch_Enemy]

HI Arch_Enemy,

I just read your post and the good replies to it. Here are my thoughts on this as well.

There are many file renamers that can easily change the original files in their folder or the modified files in their folder to have a suffix (or prefix), like "something-orig.xxx" or "something-mod.xxx". Look for "rename" in the "Synaptic Package Manager (SPM)", like Krename, Gwenrename, pyrenamer, gprename, etc...


Hope this helps ...

krename_FoldersFiles1.jpg
.
krename_Filename1.jpg

With all the work I have with photos, I have a couple good renamers. A-B Rename (Krename?) is a really good one, can do one file or a whole batch.
I think there is also an EXIF plugin that set the date to the original shoot date as well, which is really good when I copy files and it creates the file using TODAY'S date overwriting the original shoot date. That's a bummer, because I can remember when I shot the photo, but not the name since the camera names them according to Nikon's naming scheme. I rather sort on shoot date because I can generally remember the day I took it, and if not, at least I remember what year it was.

Memory's not that shot-out yet!

[phd21]

HI Arch_Enemy,

"Krename" is one of the best file renamers with many options from simple to complex that can run on batches of folders and files. "Gwenrename" is pretty great too.

In fact, after running this little example, I accidentally renamed those images with the "-orig" suffix, which I then had to remove that suffix which was easy using the find and replace option in the Filename Advanced tab, find "-orig", replace with nothing.

With all the work I have with photos, I have a couple good renamers. A-B Rename (Krename?) is a really good one, can do one file or a whole batch. I think there is also an EXIF plugin that set the date to the original shoot date as well, which is really good when I copy files and it creates the file using TODAY'S date overwriting the original shoot date. That's a bummer, because I can remember when I shot the photo, but not the name since the camera names them according to Nikon's naming scheme. I rather sort on shoot date because I can generally remember the day I took it, and if not, at least I remember what year it was.

You are correct, "krename" has many options including EXIF and other date and time options which you can add to the filename. You may have to search the "Synaptic Package Manager (SPM)" for "exif" and install those packages before they are available in apps like "krename".

FYI: There are other applications for actually modifying or updating the exif data for images.

Hope this helps ...

krename_Plugins1.jpg

.

krename_Filename_Simple.jpg

.

krename_Filename_Advanced.jpg

[rene]

Mmm. While writing I had some (debug) lines accompanying the actual cp but now that it's the only line anyway, a bit of pedantic cleaning:

Code: Select all

#!/bin/bash

copy() {
    find "$1" -type f -print0 | while read -d $'\0' SRC; do
        DST="$2${SRC##$1}"
        mkdir -pv $(dirname "$DST") && touch "$DST"
        [ $(stat -c %s "$DST") -ge $(stat -c %s "$SRC") ] || cp -v "$SRC" "$DST"
    done
}

[ -d "$1" ] && [ -d "$2" ] || exit

copy "$1" "$2"
copy "$2" "$1"

[Petermint]

Another option is to compare the directories using Meld and use the Meld copy on each file that is different. This works where there are many files and only a few are different. Meld does not display the contents of directories where the contents are identical. If every directory/file is different, Meld melts.

[Arch_Enemy]

Another option is to compare the directories using Meld and use the Meld copy on each file that is different. This works where there are many files and only a few are different. Meld does not display the contents of directories where the contents are identical. If every directory/file is different, Meld melts.

I think that got installed with the build-essentials package.

I'll have to try it.

[BG405]

Interesting thread, has given me a few ideas. I suspect a lot of us will have good uses for these and similar scripts. Bookmarked for reference. Thanks!