Setting Up an External Hard Drive for backup

[cheZ]

i just purchased a new Toshiba 500gb usb external Hard Drive for simple backup purposes and had some questions which perhaps some kind soul / Mint expert user can answer. in searching on both google and on these forums there didn't seem to be a concise yet simple explanatory guide to walk a beginner / semi advanced Mint user from points A thru Z the best way to set up a new external usb hard drive for the purposes of backup ( details of best initial formatting , simple partitioning , etc. ). there is a lot of sketchy and somewhat confusing information out there related to this topic , but nothing of a concise yet simple detailed guide , which im sure many of us could benefit from.
my own particular simple situation is that i have Lenovo laptop ( about 5 years old ) with a 250gb internal hard drive running Linux Mint 11 ( 64 bit ) that contains approx 140gb of personal data. my thoughts are that i would possibly format & partition this new external 500gb usb hard drive to accommodate in one partition a simple drag & drop of my computer's home folder with all my personal data , and then possibly in the other created partition , have a boot-able backup / clone of the laptop's Mint 11 system & all user data/home folder intact. it does not have to be this way , and frankly the bootable clone is an after thought as my main concern is backing up my personal data/home folder... these were just my initial thoughts/ideas as i just purchased this new 500gb external HD.
i realize from reading this forum over the last few years of using Mint , that there are many very advanced users out there who are quite comfortable and at ease doing many exotic things via the terminal ( and explaining things based on one's using of the terminal ) , however there are quite a few of us out there who feel more comfortable with a gui to guide us , without the need for multiple ( or complicated ) terminal commands. i myself do sometimes use the terminal for relatively easy & simple things that don't require much expertise ...and when possible , i try to use applications / utilities with a gui interface to ease the level of difficulty whenever possible. having very limited time with raising / supporting a family and working 2 jobs , i rarely if at all have the time for any steep learning curves when it comes to having a safe and stable computer situation , and my sincere hope is that there is someone out there who can talk me thru some options and simplify the process ( of both the A to Z of setting up the external HD , and making a simple problem free backup ) as i have no advanced ability or intention of re-inventing the wheel. the reason i use Linux Mint some might ask if i am not a terminal person ? well , the answer is quite simple ...it is because Linux Mint simply works , wonderfully any help is very very much appreciated , thanks all !

[kyphi]

Backing up is vital, especially with such a massive (compared to my own) home directory.

For backing up I use "rsync" which is already installed and a part of your system. Since you prefer to use the GUI, you can install "Grsync" which is rsync with a GUI.

Grsync lets you specify the source and destination of your files and is easily configured even to the extent of excluding files to be backed up.

A USB external drive is a plug and play device which will automount. You can re-format the drive to ext 4 but this is not essential. If you do not want to use the whole drive for backups then you can use "gparted" to divide it into partitions.

Creating a partition for a bootable installation of LinuxMint is not possible since a bootable system will require the whole drive.

The problem with using an external drive for backups is a procedural one in that you have to connect it to use it and then activate grsync. The first backup will take a long time (140 GB). Subsequent backups will take seconds or minutes.

If you need further information, just yell.

[cheZ]

thank you most kind kyphi , i truly appreciate you taking the time and effort to write a detailed reply in a way that can be easily understood by those of us with limited experience. i agree backing up is essential and probably the most important thing a user should know. i will follow all that you outlined , and i will install and investigate gparted for formatting and partitioning of this new 500gb external hard drive i purchased. i ran across partitionmanager while researching , and wondered if this was essentially the same as gparted ? or which was better. also , considering that i am running Linux Mint 11 64bit system , is there anything additional i need to consider when formatting the external HD or backing-up my data ? also considering that i have approx. 140gb of personal data to backup , would you have any recommendations on the appropriate size of say 2 partitions on the 500gb drive ? you had mentioned about re-formatting the drive to ext 4 , and i understand why , and that this is a format that is only recognized by Linux , but i wondered if it was possible to format say half of the drive with ext 4 ( for my data backup ) , and the other half with fat 32 so that this other half can be recognized by other systems and i can store pics , music and movies on this fat 32 partition. just a thought and i was curuous. and one last simple question : if i want to install these utilities you mentioned via the terminal ( gparted or partitionmanager , grsync ) because the Software Manager is a bit problematic ( it tends to freeze ) on my system ; is there a terminal command for install for each of these utilities ? much thanks again for all of your advice and expertise , it is sincerely appreciated

[kyphi]

Not a problem cheZ.

You can install gparted if it is not already installed, with this command:

Code: Select all

sudo apt-get install gparted

similarly, to install grsync:

Code: Select all

sudo apt-get install grsync

"partitionmanager" relies on the KDE libraries and "gparted" on the Gnome libraries. Both are partition managers.

I have several hard drives installed in my system and keep my operating systems on separate drives. Among these drives, one is a 500 GB hard drive segmented into four primary partitions. In this case, I considered it good practice to make the partitions of different sizes so that they can more easily be identified. You can imagine the confusion if all four partitions were of the same size. You can, of course, write them all down - they all have identifiers (/dev/sdd1, sdd2, sdd3, sdd4).

Yes, you can format each partition in a different format. Pictures, movies and music can be stored on ext 4.

[cheZ]

thanks again my friend i am downloading these 2 utilities / programs now , and will attempt to do a backup today based on your most helpful advice. question : if i format the entire external 500gb drive in ext 4 , and then create 2 different size partitions ( one for Mint backup , one for pictures movies ,etc ) ; will i be able to also access the movies , pictures from another computer that is running a different operating system ( i.e. Mac or Windows ) ? and also will grsync allow me to backup my Firefox prefs & bookmarks , and my Tomboy Notes ? thanks again for all your help with this !

[kyphi]

Windows cannot read ext 4 and will describe those partitions as "unknown partitions". For both Windows and Linux to be able to read the partitions they should be FAT 32.
For LinuxMint backup I would use ext 4, for storing images and movies use FAT 32 if you want to access these from another operating system.

I am not familiar with Macs but this will help: http://osxdaily.com/2012/04/22/format-d ... ompatible/

Firefox Bookmarks and Thunderbird emails will all be backed up via grsync because they are in your home directory and it is your home directory that contains the results of all your activities.

My external drive is formatted to ext 4 and is for Linux use only.

[catweazel]

i just purchased a new Toshiba 500gb usb external Hard Drive for simple backup purposes and ... there didn't seem to be a concise yet simple explanatory guide

That's because backing up all comes down to one question that has a different answer for everyone. How important is your data?

( details of best initial formatting , simple partitioning , etc. )

There's no real point partitioning your drive. If the drive dies then your data is gone, no matter what partition it's on. If your data is very important then get another external drive and backup your backups to that. As for format, make it FAT or FAT32 so you can plug the drives into Windows machines as well as linux machines. This is a safety mechanism in case your Linux machine explodes and you need to get at your data quickly but don't have another Linux machine or boot CD/USB drive available with Linux on it

My data is extremely important so I run a 2TB hardware RAID10 with spare drives, plus two 2TB USB drives. I copy the data from the hardware RAID to the first 2TB drive once every month. I copy the data from the first 2TB drive to the 2nd 2TB drive once every 3 months.

I recommend you develop a backup & recovery plan. The text below is from a site I used to run so I don't need to credit myself for the quote because I own the copyright

Develop a Backup & Recovery Plan

How important is your data?

If you rely on the data on your computer to earn your living then you ought to consider a robust backup regimen.

If you use the computer for 'surfing the web' and sending short emails to relatives then a single disk image and ad hoc backups every few weeks might be sufficient.

How often is the data updated?

The more important the data and the more often it changes, the more robust your backup plan should be.

Do you have the necessary hardware?

Depending on the type of backup you want, you will need one or both of:

A second hard disk
A DVD or CD writer

How will you manage your future needs?

Over time, your personal preferences are likely to change and your installed base of software is likely to grow. If you are constantly fiddling and making changes to your system then you should weigh up the risk of not taking another disk image before you fiddle. Fiddling with your system includes installing unproven beta software and hacking around in the OS when you don't really know what you're doing.

Create a Backup Plan

Scenario 1

In this scenario, it is assumed that you prefer not to reinstall the operating system and that your data is reasonably important, consequently you decide to take an image and do a full backup once a week.

Code: Select all

    Install the operating system and your favourite applications
    
    Take a disk image and burn it to CD, DVD or store it on a second hard disk
    
    Every Friday, run the backup software and perform a full backup of
    selected files, saving the backup file to a second disk
    
    Every second Friday, burn the backup file to CD or DVD

Scenario 2

In this scenario, it is assumed that you don't mind reinstalling the operating system and that your data is reasonably important, consequently you decide to do a full backup once a week.

Code: Select all

    Every Friday, run the backup software and perform a full backup of
    selected files, saving the backup file to a second disk
    
    Every second Friday, burn the backup file to CD or DVD

Your backup plan all comes down to one question; How important is your data?

[cheZ]

thank you both kyphi & TehGhodTrole , i truly appreciate all the time and effort you sent in responding and explaining , your advice is enormously informative and helpful and you are both very kind as i think this tread will be helpful to many Mint users who are not so very advanced. while i am neither as advanced as either of you , i do feel that my data is indeed important , and i am certainly trying to save myself from some anguish and grief by doing a proper backup. i followed the instructions you both had suggested , and i formatted and partitioned the external usb 500gb hard drive accordingly ( 1 partition in ext 4 of 199gb for a Mint backup using gsync , and 1 partition approx. 300gb in FAT for pictures and movies ). i opened gsync once i ran complete updates on the Mint system , and full maintenance on the computer/system ( bleachbit in root , and as user ) after restarting a couple of times. once i opened gsync i was somewhat confused by the " Basic " , " Advanced " and " Extras " panels in the open window of the app , and was unsure what if anything i should be checking off in the options for each of these in order to do a simple , but proper , backup. by default when using the first time this is what shows up >
Basic

Advanced

Extras

can you please specify what exactly i should be choosing / checking off to insure that i create a backup that will function properly ( and includes everything ) , that is , if i am ever in a jam and actually need to use this backup. also , if possible , perhaps you can offer a step by step procedure ( of using gsync to create a backup ) so as not to create something that is corrupted or incorrect in some way. once every thing is set in the user prefs ( that is , if i actually need to change the default user prefs ) , and set & checked off in these 3 specific panels ( Basic , Advanced , Extras ) , i am assuming that i simply click on the icon ( shaped like a gear ) that states " make a full run (go!) " , correct ? thanks for any detailed clarification that you can offer.

[bkidwell]

This is a bit more advanced then what your asking for and would require the HDD to always be attached if your never shutting the computer off and wish this to happen automatically, but hopefully you'll play with it, and see it's potential. The awesome thing about this is it saves a Backup every night at 2am, and makes a 30 day running achive that only records files that have changed. So for example, today you work on a spreadsheet and save it. Tomorrow you open the same spreadsheet and edit it again. Day 3 you open the same spreadsheet and edit it again, and you do this for the next 3 days. So now every night a new copy of the spreadsheet is saved in the current, and the current from last night is moved to a dated achive. So... if on day 7, you have screwed it up so bad, and you know you want it from three days ago... you simply browse to that day in the archives, and copy paste it back to your home folder. Because it only saves the changes you will see very little difference on the backup drive as far as space used is concerned. Unless of course you are manipulating sevral gigs of data each day. N E way... check it out. Try it out on a simple folder with a few pictures you don't care about till you understand it. This is one of the first things i learned about when I began my trip down Linux Lane, and this has saved my butt more then once! You'll be glad you learned it as a noob.

So this script will require you to have a direcotry called /BACKUP. On my server that is a mounted HDD. I save this script then make it executeable.
Open a terminal on the computer you wish to backup and enter this code.

Code: Select all

sudo mkdir /BACKUP

Now that you have the folder we need to mount the HDD to that folder. You can do this by attaching the drive and then type the following code in a terminal.

Code: Select all

sudo fdisk -l

this will list all the attached drives, look for the description and size of the one you just attached. Look for the part that says something to the effect of /dev/sdb1.

Code: Select all

sudo mount /dev/YOURDEVICE /BACKUP

Hopefully no errors occur. Now you can browse in nautilus <---- the file browser or in the terminal to /BACKUP and look around to make sure it is the usb drive you attached. The commands for the terminal would be this.

Code: Select all

cd /BACKUP

Code: Select all

ls -la

that should list everything at the top of that drive.
!! IF YOU INTEND TO DISCONNECT THIS DRIVE, YOU MUST FIRST ISSUE THIS COMMAND BEFORE UNPLUGGING IT!!!
!!!!!

Code: Select all

sudo umount /BACKUP

!!!!!
!! IF YOU INTEND TO DISCONNECT THIS DRIVE, YOU MUST FIRST ISSUE THIS COMMAND BEFORE UNPLUGGING IT!!!

otherwise, if your still with me... carry on!

Code: Select all

gedit /homebackup

copy paste this script and save.

Code: Select all

#!/bin/bash
# User Variables
PROTECT=/home/					#What to backup
BACKUPDIR=/BACKUP/homebackup	  	#Backup Parent Directory
CURRENT=$BACKUPDIR/current 			#Most recent backup 
OLD=$BACKUPDIR/archive				#Archived backups
DATE=`date +%m-%d-%Y`           	        #Date in variable
TIME=`date +%H:%M`              	                #Time in variable
NOW=`date +%m-%d-%Y_%H:%M`		#Archive folder name by date & time
NOW=$OLD/$NOW				        #Archive folder directory
mkdir -p $CURRENT;					#Make DIR if not present
mkdir -p $OLD;					        #Make DIR if not present
mkdir -p $NOW;					        #Make DIR if not present
cp -al $CURRENT/. $NOW				#Copy current backup to archives with links
chown -R root:root $NOW				#Change owner !!!CHANGE GROUP TO YOUR GROUP!!! owner:group
chmod -R 750 $NOW					#Change Permissions
echo ">>><<<>>><<<>>><<<>>><<<>>><<<>>><<>>><<<>>><<<>>><<<>>><<<>>><<<>>><<<"
echo "--------------$DATE $TIME files backed up to current--------------"
echo ">>><<<>>><<<>>><<<>>><<<>>><<<>>><<>>><<<>>><<<>>><<<>>><<<>>><<<>>><<<"
rsync -rltv --delete --exclude-from '/exclude' $PROTECT $CURRENT			#Update current backup
chown -R root:root $CURRENT					#change owner and group !!!CHANGE GROUP TO YOUR GROUP!!!
chmod -R 750 $CURRENT					#change file permissions
touch $CURRENT						#Update time stamp of current backup
echo ">>><<<>>><<<>>><<<>>><<<>>><<<>>><<>>><<<>>><<<>>><<<>>><<<>>><<<>>><<<"
echo "--------------$DATE $TIME files removed from archive--------------"
echo ">>><<<>>><<<>>><<<>>><<<>>><<<>>><<>>><<<>>><<<>>><<<>>><<<>>><<<>>><<<"
for file in $(find $OLD -maxdepth 1 -type d -mtime +30); #Find files older then -mtime in days
do 							 #example; "-mtime +10" =10 days	old
rm -rf --preserve-root $file				 #recursive force removal of find
echo "$file removed" 					 #print removed
done
echo ">>><<<>>><<<>>><<<>>><<<>>><<<>>><<>>><<<>>><<<>>><<<>>><<<>>><<<>>><<<"
echo "----------------------------------END----------------------------------"
echo ">>><<<>>><<<>>><<<>>><<<>>><<<>>><<>>><<<>>><<<>>><<<>>><<<>>><<<>>><<<"
exit 0

Code: Select all

sudo chmod +x /homebackup

now use crontab to make it execute on a regular basis

Code: Select all

sudo crontab -e 

and i use this in the crontab so that it execues every night at 2am

Code: Select all

0 2 * * * /homebackup

Feel free to ask more questions!

[cheZ]

hi bkidwell ~ well thank you kindly for all that , it is much appreciated i totally understand the concept , as i use a mac at work and time machine does incremental backups consistently. and when i was using a mac for personal use ( my last system before linux mint couple / few years back ) i simply made an exact clone ( using super-duper or carbon cloner ) of the entire system every week or so onto an external raid drive , to save me in a jam. as far as needing incremental backups now , i really dont need as i always save important personal docs to gmail consistently ...frankly the only thing i am concerned about is having a solid backup to what exists now on this computer ( all the older data from the past couple of years using mint ) ...it really does not have to be synced up on a daily basis the way you outlined , although i truly appreciate what you are explaining. what i wish to do here is basically the same thing as i did with the mac i had , and that method seemed to be fine for my limited purposes. with this computer running mint 11 , i would simply like to manually do a complete functional backup every week or 2 onto this external hard drive , as i will not be keeping this external usb hard drive plugged in all the time to the computer. i know this probably sounds very basic , but running terminal commands and shell scripts was frankly what i was hoping to avoid , as i know myself well enough to know i am not very good with all that. just hoping to have a solid backup on this external hard drive in the event of an emergency. hope that all makes sense. thanks for all of your time and effort explaining things , i sincerely do appreciate it.

[bkidwell]

No problems!! I get it, and to each his or her own! So how bout this:
When you connect the drive it probably automounts and opens up a browser folder yes?
In the navigation bar of that browser it gives you the name of that drive. That drive is likely (if you press the little arrow to the left of the name) in the media folder.

So copy this script, and the line that says Backup Parent Directory change to /media/YOURDRIVE.

Code: Select all

sudo gedit /homebackup

copy paste and change what you want to backup and where to back it up... be warned... THERE BE DRAGONS AHEAD!!!
this will DELETE EVERYTHING on the Backup Parent Directory that is not in the PROTECT directory... simply remove the "--delete" if this scares you. But the --delete is required when you want it to be an exact copy every time you run this in the future.

Code: Select all

    #!/bin/bash
    # User Variables
    PROTECT=/home               #What to backup
    BACKUPDIR=/media/YOURDRIVE/        #Backup Parent Directory
    rsync -rltv --delete $PROTECT $BACKUPDIR         #Make your backup
   exit 0

Still save it as /homebackup.

Code: Select all

sudo chmod +x /homebackup

Now we will make this a menu item, so you can browse to it and launch. No terminal required once this is done.
Right click the menu, and select edit menu, select Other, or whatever menu you want it under...
Select 'new item'

Type: application in terminal
Name:what ever you want
Command: /homebackup
Comment: blah blah

so now... every time you connect the drive, you can browse to the menu item you just made... run it and then eject the drive. Just remember, what ever is on that drive will be an exact mirror of the folder you decided to backup, so anything extra on that drive will be removed.
Hope this helps!!

[kyphi]

Plug in your USB drive.

When the drive automounts, click on "File" and "Create New Folder". Call the new folder "Backup" (without the quotes).

In grsync, on the first screen (Basic Options) you will have to enter the source and destination pathways. I recommend leaving the check boxes in their default stage at the moment = Preserve Time, Verbose, Show transfer progress.

The source will be your home directory = /home/cheZ/
The destination will be /media/xxxxxx/Backup/

The xxxxxx information you can see when you plug in your drive and it automounts. The numbers and letters will present at the top of the screen.

The second screen (Advanced Options) you can leave as default. Here you can enter an "exclude" command to stop some things from being backed up. If you need the information as to how to do this, just yell.
The third screen (Extra Options) you can leave as default. You can change any of these parameters as you become more familiar with their functions.

Now you are ready to click on the gear. The process will be in "verbose" mode which means that you can watch the letters and numerals (if you can read that fast). Everything in your home directory will now be backed up to your USB storage drive.