I have a dual part system with windows and linux mint 13.I also have a NTFS data partition which contains all my documents etc.This data folder has several subdirectores and sub-sub-directories containing several thousand files (most of them text).I often sync my USB fat32 stick with my ntfs data folder using an excellent freeware windows tool called winmerge.The developer is working on a winmerge version for Linux.Until then whenever i'm in Linux i've been using Komparator to sync the usb stick (which i can take to work) with the Data folder on my windows NTFS partition.The problem that i've encountered of late is that Komparator keeps crashing in Cinnamon.I don't know why that is and it was becoming quite a pain in the ass to keep booting back into windows to run winmerge.So i decided to create a bash script which runs recursively through all the comparison folder and sub-folders,picking out files and folders that are different (on the basis of filenames and filesize),and allowing you to "re-align" or sync them.
If your files all reside on a linux ext3/ext4 partition then this script may be redundant as you may be able to get away with using simple linux tools such as rsync.I can't be sure because i haven't tried it.All i can say is that rsync isn't the tool to use if you're trying to sync folders/sub-folders/files between FAT32 and NTFS partitions.This is because it uses file meta-information encoded on ext3/ext4 partitions not NTFS.
THE SCRIPT:
I ended up creating the following script.Its very crude and amateurish but it gets the job done for what i intended it to do.
It runs in two parts.You realign the folder structure in the first part.After doing that you run the script again to sync the files.It runs a CASE SENSITIVE directory and file comparison.Files with different filenames are identified or files with the same filename but different sizes are picked out (these are Windows text files ending in .txt .doc or .rtf extensions).It does NOT calculate a checksum of the files so if you have the file across the two comparison folders with the exact content but named differently it will identify it as two different files.Another limitation it has is that whenever it picks out sub-folders that are exclusive to one comparison directory or the other it gives you the option of either deleting them ALL or copying them ALL from from one folder to the other.You do not have the option of selecting which individual folder you want to copy or delete.
Before running it you need to install the zenity package.Also i suggest that you first test it on some "test directories" that you don't mind losing incase something goes wrong!
OK..so here's the script.The first shebang line is critical and needs to point to /bin/bash rather than /bin/sh because the script uses process substitution which doesn't work in /bin/sh.Read through and modify it to your heart's content! :
Code: Select all
#!/bin/bash
#This script "differ" runs in two parts.You realign the folder structure in the first part.After doing that you run
#differ again to sync the files.
#FIRST_DIR is your FAT32 directory
#SECOND_DIR is your windows NTFS directory
#Define function copy_same_to_fat32.This copies the files that you select to FAT32.These files have same filenames
#in both windows and fat32 but are of different files sizes:
copy_same_to_fat32(){
echo "Calling from inside the copy_same_to_fat32 function.The arguments you entered are: "$@
echo "The number of arguments you entered are: "$#
echo
for i in $@
do
directory=$(dirname "${COMMON_ARRAY[i]}")
cp "${SECOND_DIR}""${COMMON_ARRAY[i]}" "${FIRST_DIR}""$directory"
echo "Copied file to FAT32: ""${COMMON_ARRAY[i]}"
done
}
#Define function copyall_same_to_fat32.This copies all the files to FAT32.These files have same filenames
#in both windows and fat32 but are of different files sizes:
copyall_same_to_fat32(){
for (( i = 0 ; i < ${#COMMON_ARRAY[@]} ; i++ ))
do
directory=$(dirname "${COMMON_ARRAY[i]}")
cp "${SECOND_DIR}""${COMMON_ARRAY[i]}" "${FIRST_DIR}""$directory"
done
echo "ALL common files in WINDOWS have been copied to FAT32"
}
#Define function copy_same_to_win.This copies the files that you select to WINDOWS from FAT32.These files have same #filenames in both windows and fat32 but are of different files sizes:
copy_same_to_win(){
echo "Calling from inside the copy_same_to_win function.The arguments you entered are: "$@
echo "The number of arguments you entered are: "$#
echo
for i in $@
do
directory=$(dirname "${COMMON_ARRAY[i]}")
cp "${FIRST_DIR}""${COMMON_ARRAY[i]}" "${SECOND_DIR}""$directory"
echo "Copied file to WINDOWS: ""${COMMON_ARRAY[i]}"
done
}
#Define function copyall_same_to_win.This copies all the files from FAT32 to WINDOWS.These files have same
#filenames in both windows and fat32 but are of different files sizes:
copyall_same_to_win(){
for (( i = 0 ; i < ${#COMMON_ARRAY[@]} ; i++ ))
do
directory=$(dirname "${COMMON_ARRAY[i]}")
cp "${FIRST_DIR}""${COMMON_ARRAY[i]}" "${SECOND_DIR}""$directory"
done
echo "ALL common files in FAT32 have been copied to Windows"
}
#Define function copy_exclusive_to_fat32.This copies files exclusive in windows to fat32 that you select:
copy_exclusive_to_fat32(){
echo "Calling from inside the copy_exclusive_to_fat32 function.The arguments you entered are: "$@
echo "The number of arguments you entered are: "$#
echo
echo "========================================================================"
for i in $@
do
directory=$(dirname "${IN_SECOND_ONLY_ARRAY[i]}")
if [ -d "${FIRST_DIR}""$directory" ];then
echo "File[$i]:"
#cp -v "${SECOND_DIR}""${IN_SECOND_ONLY_ARRAY[i]}" "${FIRST_DIR}""$directory"
rsync -v --progress "${SECOND_DIR}""${IN_SECOND_ONLY_ARRAY[i]}" "${FIRST_DIR}""$directory"
#echo "Copied file to FAT32: ""${IN_SECOND_ONLY_ARRAY[i]}"
echo "========================================================================"
fi
done
}
#Define function copyall_exclusive_to_fat32.This copies ALL files exclusive in windows to fat32:
copyall_exclusive_to_fat32(){
echo "========================================================================"
for (( i = 0 ; i < ${#IN_SECOND_ONLY_ARRAY[@]} ; i++ ))
do
directory=$(dirname "${IN_SECOND_ONLY_ARRAY[i]}")
if [ -d "${FIRST_DIR}""$directory" ];then
#cp -v "${SECOND_DIR}""${IN_SECOND_ONLY_ARRAY[i]}" "${FIRST_DIR}""$directory"
echo "File[$i]:"
rsync -v --progress "${SECOND_DIR}""${IN_SECOND_ONLY_ARRAY[i]}" "${FIRST_DIR}""$directory"
echo
echo "========================================================================"
fi
done
echo "ALL exclusive files in WINDOWS have been copied to FAT32"
}
#Define function deleteall_exclusive_to_windows.This deletes all exclusive files in Windows:
deleteall_exclusive_to_windows(){
for (( i = 0 ; i < ${#IN_SECOND_ONLY_ARRAY[@]} ; i++ ))
do
rm -f "${SECOND_DIR}""${IN_SECOND_ONLY_ARRAY[i]}"
done
echo "ALL exclusive files in WINDOWS have been deleted"
}
#Define function deleteall_exclusive_to_fat32.This deletes ALL exclusive files on fat32:
deleteall_exclusive_to_fat32(){
for (( i = 0 ; i < ${#IN_FIRST_ONLY_ARRAY[@]} ; i++ ))
do
rm -f "${FIRST_DIR}""${IN_FIRST_ONLY_ARRAY[i]}"
done
echo "ALL exclusive files in FAT32 have been deleted"
}
#Define function delete_exclusive_to_fat32.This deletes exclusive files on fat32 that you select:
delete_exclusive_to_fat32(){
echo "Calling from inside the delete_exclusive_to _fat32 function.The arguments you entered are: "$@
echo "The number of arguments you entered are: "$#
echo
for i in $@
do
rm -f "${FIRST_DIR}""${IN_FIRST_ONLY_ARRAY[i]}"
echo "Deleted file in FAT32: ""${IN_FIRST_ONLY_ARRAY[i]}"
done
}
#Define function copyall_fat32_to_win.This copies ALL exclusive files on fat32 to windows:
copyall_fat32_to_win(){
echo "========================================================================"
for (( i = 0 ; i < ${#IN_FIRST_ONLY_ARRAY[@]} ; i++ ))
do
directory=$(dirname "${IN_FIRST_ONLY_ARRAY[i]}")
if [ -d "${SECOND_DIR}""$directory" ];then
#cp -v "${FIRST_DIR}""${IN_FIRST_ONLY_ARRAY[i]}" "${SECOND_DIR}""$directory"
echo "File[$i]:"
rsync -v --progress "${FIRST_DIR}""${IN_FIRST_ONLY_ARRAY[i]}" "${SECOND_DIR}""$directory"
echo
echo "========================================================================"
fi
done
echo "ALL exclusive files in FAT32 have been copied to WINDOWS"
}
#Define function copy_fat32_to_win.This copies exclusive files on fat32 that you select to windows:
copy_fat32_to_win(){
echo "Calling from inside the copy_fat32_to_win function.The arguments you entered are: "$@
echo "The number of arguments you entered are: "$#
echo
for i in $@
do
directory=$(dirname "${IN_FIRST_ONLY_ARRAY[i]}")
cp "${FIRST_DIR}""${IN_FIRST_ONLY_ARRAY[i]}" "${SECOND_DIR}""$directory"
echo "Copied file to WINDOWS: ""${IN_FIRST_ONLY_ARRAY[i]}"
done
}
#Define function exclusive_dirs_in_fat32.This lists all exclusive folders in fat32 (FIRST_DIR).It then gives you the
#option to either copy them all to windows (SECOND_DIR) or delete them all:
exclusive_dirs_in_fat32(){
echo "EXCLUSIVE FOLDERS in FIRST_DIR (FAT32):"
echo
while read IN_FIRST_FILE
do
echo "$IN_FIRST_FILE"
IN_FIRST_ONLY_ARRAY[IN_FIRST_ONLY]="$IN_FIRST_FILE"
((IN_FIRST_ONLY++))
done < <(comm -23 <(find "$FIRST_DIR" -type d | sed "s|$FIRST_DIR||" | sort ) <(find "$SECOND_DIR" -type d | sed "s|$SECOND_DIR||" | sort ))
echo
echo "There is/are $IN_FIRST_ONLY folder(s) EXCLUSIVE to the FIRST directory (i.e:FAT32)"
echo
if [ $IN_FIRST_ONLY -gt 0 ];then
echo "Do you want to delete ALL the exclusive directory(ies) in FAT32?Type y or n followed by [ENTER]:"
read RESPONSE
if [ $RESPONSE = y ];then
for (( i = 0 ; i < ${#IN_FIRST_ONLY_ARRAY[@]} ; i++ ))
do
rm -rf "${FIRST_DIR}""${IN_FIRST_ONLY_ARRAY[i]}"
done
echo "ALL exclusive folders in FAT32 deleted.Please run differ again"
exit
fi
fi
echo
if [ $IN_FIRST_ONLY -gt 0 ];then
echo "Do you want to COPY ALL the exclusive directory(ies) in FAT32 to WINDOWS?Type y or n followed by [ENTER]:"
read RESPONSE
if [ $RESPONSE = y ];then
for (( i = 0 ; i < ${#IN_FIRST_ONLY_ARRAY[@]} ; i++ ))
do
mkdir -p "${SECOND_DIR}""${IN_FIRST_ONLY_ARRAY[i]}"
done
echo "ALL exclusive folders in FAT32 copied to WINDOWS.Please run differ again"
exit
fi
fi
}
#Define function exclusive_dirs_in_windows.This lists all exclusive folders in Windows (SECOND_DIR).It then gives
#you the option to either copy them all to fat32 (FIRST_DIR) or delete them all:
exclusive_dirs_in_windows(){
echo "EXCLUSIVE FOLDERS in SECOND_DIR (WINDOWS):"
echo
while read IN_SECOND_FILE
do
echo "$IN_SECOND_FILE"
IN_SECOND_ONLY_ARRAY[IN_SECOND_ONLY]="$IN_SECOND_FILE"
((IN_SECOND_ONLY++))
done < <(comm -13 <(find "$FIRST_DIR" -type d | sed "s|$FIRST_DIR||" | sort ) <(find "$SECOND_DIR" -type d | sed "s|$SECOND_DIR||" | sort ))
echo
echo "There is/are $IN_SECOND_ONLY folder(s) EXCLUSIVE to the SECOND directory (i.e:WINDOWS)"
echo
#Give option to copy ALL the exclusive directories in Windows to FAT32:
if [ $IN_SECOND_ONLY -gt 0 ];then
echo "Do you want to copy ALL the exclusive directory(ies) from WINDOWS to FAT32?Type y or n followed by [ENTER]:"
read RESPONSE
if [ $RESPONSE = y ];then
for (( i = 0 ; i < ${#IN_SECOND_ONLY_ARRAY[@]} ; i++ ))
do
mkdir -p "${FIRST_DIR}""${IN_SECOND_ONLY_ARRAY[i]}"
done
echo "ALL exclusive folders in WINDOWS copied to FAT32.Please run differ again"
exit
fi
fi
echo
#Give option to delete ALL the exclusive directories in Windows:
if [ $IN_SECOND_ONLY -gt 0 ];then
echo "Do you want to DELETE ALL the exclusive directory(ies) in WINDOWS?Type y or n followed by [ENTER]:"
read RESPONSE
if [ $RESPONSE = y ];then
for (( i = 0 ; i < ${#IN_SECOND_ONLY_ARRAY[@]} ; i++ ))
do
rm -rf "${SECOND_DIR}""${IN_SECOND_ONLY_ARRAY[i]}"
done
echo "ALL exclusive folders in WINDOWS have been deleted.Please run differ again"
exit
fi
fi
}
#Get your directory paths.Following commands used in GNOME:
#FIRST_DIR is your FAT32 directory.The following command opens a select directory dialog box for you to choose the
#FIRST_DIR fat32 directory:
FIRST_DIR=`zenity --file-selection --directory --title="Please choose FIRST_DIR (FAT32)......"`
#Second directory is your medicine folder on the NTFS hard drive:
SECOND_DIR=`zenity --file-selection --directory --title="Please choose SECOND_DIR #(WINDOWS)......"`
COMMON=0
IN_FIRST_ONLY=0
IN_SECOND_ONLY=0
echo "======================================================================"
exclusive_dirs_in_fat32
echo "======================================================================"
exclusive_dirs_in_windows
echo "========================================================================"
echo "File(s) COMMON to both directories but differ in size (Bytes):"
echo
#UNIQUE_IN_FIRST list is based on presence of modifiable files present in FIRST_DIR but not
#in the SECOND_DIR or based on the presence of modifiable files with the same filenames
#present in both directories but have different filesizes:
UNIQUE_IN_FIRST=`(comm -23 <(find "$FIRST_DIR" \( -iname "*.txt" -o -iname "*.doc" -o -iname "*.rtf" \) -type f -exec ls -l {} \; | sed "s|$FIRST_DIR||" | tr -s ' ' | cut -d' ' -f5,9- | sort ) <(find "$SECOND_DIR" \( -iname "*.txt" -o -iname "*.doc" -o -iname "*.rtf" \) -type f -exec ls -l {} \; | sed "s|$SECOND_DIR||" | tr -s ' ' | cut -d' ' -f5,9- | sort )) | sort -k2`
#UNIQUE_IN_SECOND list is based on presence of modifiable files present in SECOND_DIR but not
#in the FIRST_DIR or based on the presence of modifiable files with the same filenames present
#in both directories but have different filesizes:
UNIQUE_IN_SECOND=`(comm -13 <(find "$FIRST_DIR" \( -iname "*.txt" -o -iname "*.doc" -o -iname "*.rtf" \) -type f -exec ls -l {} \; | sed "s|$FIRST_DIR||" | tr -s ' ' | cut -d' ' -f5,9- | sort ) <(find "$SECOND_DIR" \( -iname "*.txt" -o -iname "*.doc" -o -iname "*.rtf" \) -type f -exec ls -l {} \; | sed "s|$SECOND_DIR||" | tr -s ' ' | cut -d' ' -f5,9- | sort )) | sort -k2`
#UNIQUE_LIST=`echo -e "${uniqueFirst}\n""${uniqueSecond}" | sort | uniq`
#echo "Unique List is:"
#echo "$UNIQUE_LIST"
#POPULATE the COMMON_ARRAY:Read the common named files present in both directories(filtered to doc,txt or rtf) BUT with
#different files sizes to the while loop:
while read COMMON_FILE
do
FIRST=`echo "$UNIQUE_IN_FIRST" | grep -i "$COMMON_FILE"`
SECOND=`echo "$UNIQUE_IN_SECOND" | grep -i "$COMMON_FILE"`
if [ -n "$FIRST" ] && [ -n "$SECOND" ];then
COMMON_ARRAY[COMMON]="$COMMON_FILE"
((COMMON++))
fi
done < <(comm -12 <(find "$FIRST_DIR" \( -iname "*.txt" -o -iname "*.doc" -o -iname "*.rtf" \) -type f | sed "s|$FIRST_DIR||" | sort ) <(find "$SECOND_DIR" \( -iname "*.txt" -o -iname "*.doc" -o -iname "*.rtf" \) -type f | sed "s|$SECOND_DIR||" | sort ))
echo
echo "There is/are $COMMON common filenames but with different sizes"
echo
if [ $COMMON -gt 0 ];then
#Read through the COMMON_ARRAY:
echo "I am reading back COMMON_ARRAY:"
echo
for (( i = 0 ; i < ${#COMMON_ARRAY[@]} ; i++ ))
do
SIZE_FAT32=`ls -l "${FIRST_DIR}""${COMMON_ARRAY[i]}" | tr -s ' ' | cut -d' ' -f5`
SIZE_WIN=`ls -l "${SECOND_DIR}""${COMMON_ARRAY[i]}" | tr -s ' ' | cut -d' ' -f5`
echo index["$i"]: "${COMMON_ARRAY[i]} FAT32(bytes):$SIZE_FAT32 WIN(bytes):$SIZE_WIN"
done
echo
echo "There is/are $i common filenames but with different sizes"
echo
echo "To COPY ALL the common file(s) from WINDOWS ------------> to FAT32 enter 1"
echo "To SELECT the common file(s) to copy from WINDOWS ------> to FAT32 enter 2"
echo
echo "To COPY ALL the common file(s) from FAT32 ------------> to WINDOWS enter 3"
echo "To SELECT the common file(s) to copy from FAT32 ------> to WINDOWS enter 4"
echo
echo "To DO NOTHING ---------------------------------------------------- enter 5"
read RESPONSE
case $RESPONSE in
1)
copyall_same_to_fat32
;;
2)
echo "Enter indexes of the files you want copied seperated by spaces and then press the ENTER key"
read SELECTED
copy_same_to_fat32 $SELECTED
;;
3)
copyall_same_to_win
;;
4)
echo "Enter indexes of the files you want copied seperated by spaces and then press the ENTER key"
read SELECTED
copy_same_to_win $SELECTED
;;
5)
;;
*)
echo "You entered an option other than 1,2,3,4,or 5.NOTHING IS MODIFIED"
;;
esac
fi
echo "=============================================================="
echo
echo "EXCLUSIVE FILES to FIRST_DIR (FAT32):"
echo
IN_FIRST_ONLY=0
while read IN_FIRST_FILE
do
IN_FIRST_ONLY_ARRAY[IN_FIRST_ONLY]="$IN_FIRST_FILE"
((IN_FIRST_ONLY++))
done < <(comm -23 <(find "$FIRST_DIR" -type f | sed "s|$FIRST_DIR||" | sort ) <(find "$SECOND_DIR" -type f | sed "s|$SECOND_DIR||" | sort ))
#Read through initialised IN_FIRST_ONLY_ARRAY:
for (( i = 0 ; i < ${#IN_FIRST_ONLY_ARRAY[@]} ; i++ ))
do
echo $i ${IN_FIRST_ONLY_ARRAY[i]}
echo
done
echo
echo "There is/are $IN_FIRST_ONLY file(s) EXCLUSIVE to the FIRST directory (i.e:FAT32)"
echo
if [ $IN_FIRST_ONLY -gt 0 ];then
echo "To DELETE ALL the exclusive file(s) from FAT32 enter 1"
echo "To SELECT the exclusive file(s) to be deleted from FAT32 enter 2"
echo
echo "To COPY ALL the exclusive file(s) from FAT32 to WINDOWS enter 3"
echo "To SELECT the exclusive file(s) to be copied from FAT32 to WINDOWS enter 4"
echo "To DO NOTHING enter 5"
read RESPONSE
case $RESPONSE in
1)
deleteall_exclusive_to_fat32
;;
2)
echo "Enter indexes of the files you want deleted seperated by spaces and then press the ENTER key"
read SELECTED
delete_exclusive_to_fat32 $SELECTED
;;
3)
copyall_fat32_to_win
;;
4)
echo "Enter indexes of the files you want copied seperated by spaces and then press the ENTER key"
read SELECTED
copy_fat32_to_win $SELECTED
;;
5)
;;
*)
echo "You entered an option other than 1,2,3,4,or 5.NOTHING IS MODIFIED"
;;
esac
fi
echo "==============================================================="
echo
echo "EXCLUSIVE FILES to SECOND_DIR (WINDOWS):"
echo
IN_SECOND_ONLY=0
while read IN_SECOND_FILE
do
IN_SECOND_ONLY_ARRAY[IN_SECOND_ONLY]="$IN_SECOND_FILE"
((IN_SECOND_ONLY++))
done < <(comm -13 <(find "$FIRST_DIR" -type f | sed "s|$FIRST_DIR||" | sort ) <(find "$SECOND_DIR" -type f | sed "s|$SECOND_DIR||" | sort ))
#Read through initialised IN_SECOND_ONLY_ARRAY:
for (( i = 0 ; i < ${#IN_SECOND_ONLY_ARRAY[@]} ; i++ ))
do
echo $i ${IN_SECOND_ONLY_ARRAY[i]}
done
#Give option to copy ALL the exclusive files in Windows to FAT32:
echo
echo "There is/are $IN_SECOND_ONLY file(s) EXCLUSIVE to the SECOND directory (i.e:WINDOWS)"
echo
if [ $IN_SECOND_ONLY -gt 0 ];then
echo "To COPY ALL the exclusive file(s) from WINDOWS --------> to FAT32 enter 1"
echo "To SELECT the exclusive file(s) to copy from WINDOWS --> to FAT32 enter 2"
echo "To DELETE ALL the exclusive file(s) in WINDOWS enter 3"
echo "To DO NOTHING ---------------------------------------------------- enter 4"
read RESPONSE
case $RESPONSE in
1)
copyall_exclusive_to_fat32
;;
2)
echo "Enter indexes of the files you want copied seperated by spaces and then press the ENTER key"
read SELECTED
copy_exclusive_to_fat32 $SELECTED
;;
3)
deleteall_exclusive_to_windows
;;
4)
;;
*)
echo "You entered an option other than 1,2,3 or 4.NOTHING IS MODIFIED"
;;
esac
fi
