How to backup your stuff UNIX-style

Write tutorials for Linux Mint here
More tutorials on https://github.com/orgs/linuxmint/discu ... /tutorials and (archive) on https://community.linuxmint.com/tutorial
Forum rules
Don't add support questions to tutorials; start your own topic in the appropriate sub-forum instead. Before you post read forum rules
scorp123
Level 8
Level 8
Posts: 2272
Joined: Sat Dec 02, 2006 4:19 pm
Location: Switzerland

How to backup your stuff UNIX-style

Post by scorp123 »

Foreword:

This "How to" here involves a lot of shell commands. And you can't just copy & paste them 1:1 ... chances are your system is different from mine, so you need to think before you copy my commands.

People looking for a nice "Point and click" tutorial should move on. There is no pointing and clicking here. Simply because I don't believe in "pointing and clicking" and I also don't know any reliable GUI backup method ... "reliable" in the sense: We can get our stuff back after a disaster. So for a "GUI point and click backup How-To" I am simply the wrong person.

So we're going to use the shell here. And then plenty of it. So why should you be interested?

1. You could learn a lot.
2. You could learn a lot.
3. You could learn even more.
4. You could gain some confidence with the shell and your new Linux OS

5. This tutorial here works, GUI or not.

You can and you will get your data back if you do it right. But again: THINK. No tutorial can replace your brain and your intelligence. And did I mention that you could learn a lot ... ? :wink:

IMPORTANT: All the commands here need to be executed as super-user "root". So either put a "sudo" in front of each command that I mention or open a root shell via "sudo su -" and execute the commands directly in there.

Also: I strongly advise against using UUID's in /etc/fstab . UUID's make matters unnecessarily complicated by being "unique". So this means that if something goes wrong and you have to restore from a backup onto a new disk the UUID's will be different ... and that will cause problems. So it would be better to get rid of the UUID's in /etc/fstab right away (e.g. right after you installed your system) and use the more traditional device names in /etc/fstab ... Example:

Code: Select all

# /etc/fstab: static file system information.
#
#<file sys>          <mount point>     <type>   <options>   <dump>   <pass>
/dev/hda1    /                 ext3    rw,errors=remount-ro  0    1
/dev/hda5    none              swap    rw        0    0
none         /proc             proc    defaults  0    0
/dev/hda9    /data             ext3    rw        0    2
/dev/hda8    /home             ext3    rw        0    2
/dev/hda6    /tmp              ext3    rw        0    2
/dev/hda7    /var              ext3    rw        0    2
/dev/hdb1    /data2            ext3    rw        0    2
Device names are "dumb" and they don't care if they are on your old harddisk or a new one ... for as long as the harddisk and its partitions exist your system will boot. With UUID's: Nope. You'd have to find out each UUID for each disk (costs you a lot of time) and those UUID's are usually so long and so cryptic that there is a high chance that you will screw up. As I said: Get rid of them, change your /etc/fstab so that it uses device names!

Edit: Also, don't forget to edit /boot/grub/menu.lst !! There is a section which defines the default kernel options, e.g. mine looks like this:

Code: Select all

# kopt=root=UUID=8499b6f9-2196-4134-bf75-0599c17658e2 ro
Leave it commented out (that's perfectly OK; the setting *IS* *ACTIVE* nontheless! Also: the documentation says not to uncomment it, so I guess we just leave it like that!) but change that rear part! So if you change /etc/fstab and rid yourself of those UUID entries you also have to do it here. So you'd replace above string "kopt=root=UUID....." with the device name for your " / " mount point. In my case that would be:

Code: Select all

 kopt=root=/dev/sda5 ro 
Once you've changed this setting you need to activate it. Execute this as "root": update-grub .... It will rewrite your GRUB and make sure all entries in GRUB point to the right " / " partition. Or else those entries might still point to your old UUID= entry ... and that could cause booting to stop unexpectedly. If that happens (e.g. you forgot to edit menu.lst): You can temporarily "edit" the boot entry in GRUB's menu at boot time. Just make sure the menu gets displayed (e.g. by pressing 'ESC' if it's not shown per default) and then select the entry you wish to temporarily edit and then hit "e" (it's all displayed on-screen). Check that kernel line ... if it says "root=UUID....." you can change it to "root=/dev/sda5" (whatever the right setting for your " / " partition is) and then hit the 'ENTER' key to boot your system with the temporarily altered settings. It should work. Once you're in you can still change menu.lst and do the needed changes.


On the shoulders of giants

I of course am not the "inventor" of those backup methods ... Duh. As Einstein once put it: "I am standing on the shoulder of giants". Very huge giants at that. So credits go to the people who taught me UNIX and Linux back in the 90's, to all my bosses and superior managers who kicked my a** if I did stuff wrong (very important learning experience ...), to the people who ridiculed me on IRC back when I was a "n00b" (important learning experiences too ...). Last but not least, this book taught me a lot:

UNIX Backup & Recovery

I highly recommend it despite its age (1999). So nothing here is any "big secret". All the stuff shown here are standard UNIX shell commands that are most likely present out of the box on any UNIX-like operating system. The "big secret" (if it even is one) is to put these commands to work and have them backup our system.


Preparations: What do we use?

First things first: Every backup is only as good as the restore. And my work as UNIX admin has taught me some nasty lessons.

In a professional environment you'd use some form of tapes to write your backups to. Simply because they are very reliable and can hold large amounts of data.

But most home users probably can't afford a 4000$ DLT drive or a 10'000$ tape robot. So what's left?

I personally am not really a fan of making backups to CD's or DVD's. CD's are usually too small and if you span your backups across multiple CD's chances are you might lose one disk of this volume set ... and then your restore won't work anymore and your backup is worth nothing. This has happened to me. DVD's are bigger, yes. But both CD's and DVD's have serious drawbacks when it comes to reliability. Take a DVD+RW disk for example: "Can survive up to 1000 rewrites" they say. Right. What they don't tell you: Those 1000 rewrites are reached way faster than you think. If you want reliability don't count on DVD's ...

So I personally have settled on large external USB harddisks:
- they are cheap if you compare their price per GB with other options.
- these days it's really hard to find a machine that doesn't have USB.
- hence: USB harddisks most likely will work anywhere + anytime.
- reliability: I only turn on those disks when I really really want to use them for backups ... this greately increases their lifespan.
- speed: Let's face it: No DVD or CD could ever hope to beat a harddisk when it comes to read/write speeds.


As for software:

All software that we would need is already there in your OS. The only other thing I would recommend is to have a Linux Live CD ready, either your Mint Installer disk, or Ubuntu Live, or Knoppix, or whatever, for as long as it has:

- USB support
- tar on-board
- gzip on-board

you should already have all the software you'd need. As an extra and "just to be safe" I'd highly recommend the SuperGrub Disk ... Please use Google to find it. It's a live CD that is specialised on getting boot blocks back (hence the name and the relation with the "GRUB" boot loader).


Backup: What do we backup?

In short: Everything that matters. I'm a rather paranoid person and bad experience has taught me to really backup everything + the kitchen sink. It's better to have too much stuff in your backup and not need it than the other way round: Need too much stuff but not have it in any backup. That sucks. And it can seriously ruin your day :twisted:

So really -- what do we backup?

1. the Master Boot Record (MBR) + the partition table
2. we make an exact copy of /etc so we have a copy of all the config files that matter
3. we backup our package selections (can be used for cloning purposes too)
4. we pack all our partitions into tar.gz archives
5. except /home: we make a 1:1 copy of it so we can access all the files if needed
6. Putting it all together



Step 1: The Master Boot Record

For this we need the dd utility. If you want to learn more about dd please read the manual: man dd

In this example I will assume that /dev/hda is the first harddisk (for SATA or SCSI users this would most likely be /dev/sda ... as I said: THINK ... you can't just copy & paste here without thinking!).

We tell dd to read the relevant number of bytes from the first harddisk device (yes, really: the device --- and *not* any partition like /dev/hda1 or /dev/hda2 or anything like that ... ) and put that into a binary file ... if anything goes wrong we could restore the MBR from there (e.g. with the help of a Live CD):

Code: Select all

dd if=/dev/hda of=/tmp/hda-mbr.bin bs=512 count=1
Explanation: dd will read the first 512 bytes (= MBR + partition table) from the first harddisk /dev/hda and place it into the file I specify ... here: /tmp/hda-mbr.bin ... that path and filename could be adjusted to your taste and requirements if needed. Obviously it would be intelligent to have this file e.g. on an external disk or USB stick ... just in case.

How to restore your MBR

If you've destroyed your MBR somehow than this means that your PC won't boot anymore from the harddisks ... so to get it back to work you'd need a Live CD now. So in order to get our MBR back (there are other methods too but I will now focus on this one):
- you'd boot your Live CD
- mount that USB stick or USB harddisk where you put that MBR-backup on to
- open a terminal
- cd into the right directory
- restore the MBR with the "dd" command

Code: Select all

sudo su -
cd /path/to/where/MBR/backup/is/stored
dd if=hda-mbr.bin of=/dev/hda bs=446 count=1
Please note the difference here: We're only restoring 446 bytes ... not the full 512 we backed up earlier. Why? The last 64 bytes contain the partition table. Chances are you don't want to ruin your disk partitions so it's wiser to only restore 446 bytes ... not the full 512.

Please be advised that brainless tampering with the dd command while being "root" can lead to serious data loss and a really ruin your day.

Just think before you hit that enter key. If in doubt: Please come to the IRC channel and/or the forum and ask, OK?

How do we know it worked? dd itself won't give you much info. The only way to tell if it did work is to reboot. Ideally your GRUB menu should reappear again ... :D


Step 2: Copying /etc

That's simple. Just find a good location and then as "root" recursively copy your /etc there ... this has the advantage that if you need to ask for help or if anything goes wrong you have a copy of all your config files. Example:

Code: Select all

sudo su -
cd /where/we/will/store/our/stuff
cp -a -v -u /etc .
Explanation: "cp -a -v -u" copies everything in /etc recursively to the current location (= "." as target) plus it keeps the permissions intact.

How to restore your /etc

Simple: just check what's missing and then copy the single config files back to the /etc of your system. Let's suppose you've ruined your xorg.conf file somehow:

Code: Select all

sudo su -
cd /where/we/placed/the/backup/etc
cd X11
cp xorg.conf /etc/X11/
It's simple, it's stupid ... and it works.


Step 3: Our package selections

1. Backup /etc/apt/sources.list ... or copy it to one of your other systems (in case you want to have the same packages everywhere)

2. On system #1 (the one you want to backup) issue this command:
sudo dpkg --get-selections > packagelist.txt

=> This will write your package selections and the status of all packages into that text file "packagelist.txt". Now copy that file to your other system(s) or put it onto a USB stick or USB harddisk.

How to restore your package selections

1. You need to restore the /etc/apt/sources.list file (see section above about copying /etc recursively ...) and then you need to tell dpkg to read your package list:
sudo dpkg --set-selections < packagelist.txt

=> The system will now take over the package selections that were previously defined on the original or previous system.

2. All you need to do now is to execute the install process:
sudo apt-get dselect-upgrade

=> Result: You now will have the same packages installed like before on "System #1"

"Armed" with these commands you can also clone installations (e.g. produce identical systems with identical software selections) in a couple of minutes. Just make sure you get the /etc/apt/sources.list right (e.g. it should be the same on all systems).


Step 4: Packing your partitions into *.tar.gz archives

Again, you need to think a little and take a look at your /etc/fstab ... it doesn't make sense if you copy & paste my commands here, especially if you don't have the same mountpoints (e.g. separate /usr, /var and /opt partitions ... or not). The sense behind those commands is that you get an archived copy of all your partitions. The advantages you get:

- you could 100% restore a system into the state it had when the archives were made
- you could use the archives to make a 1:1 clone of your system
- you could also choose to only restore selected parts of your system (useful if you e.g. switch distros and e.g. don't need the /usr partition anymore from the previous distro ...)

This example here assumes that you have a " / " root filesystem and separate /boot, /var, /usr and /home filesystems (this needs to be changed according to your system settings):

Code: Select all

sudo su -
cd /path/to/USB/disk
tar --one-file-system -pzcvf BootFS_backup.tar.gz /boot
tar --one-file-system --exclude=/tmp/* --exclude=/dev/* --exclude=/proc/* -pzcvf RootFS_backup.tar.gz /
tar --one-file-system --exclude=/var/tmp/* -pzcvf VarFS_backup.tar.gz /var
tar --one-file-system -pzcvf UsrFS_backup.tar.gz /usr
tar --one-file-system -pzcvf HomeFS_backup.tar.gz /home
The result here would be that all these archive files get written to the path you specified above (here: "/path/to/USB/disk") ... and if that would be your USB disk you could in the future also fully restore your system from there again.

How to restore your partitions via the archives

So let's suppose that due to some serious misfortune your harddisk went to Nirvana -- you were forced to replace it. In such a case you could either re-install everything from scratch or you could restore the archives -- I mean that's precisely what we have them for, right?

So, first step would obviously be to remove the dead disk from the system and put in the new one. I will assume you know how to do it ... if not please Google around. It should be easy to find guides that will tell you how to replace a dead harddisk.

When this is done you now have a shiny new harddisk in your system. And it's totally empty and there is nothing on it .... yet.

So what we need now is to boot the system with a Live CD. The Mint installer disk should suffice, but an Ubuntu, Kubuntu or Knoppix Live CD should do just as fine. What we need to do now is to re-create the partitions as they were on the old disk -- chances are that your Live CD will have some form of partitioning program, either gparted for the GUI or fdisk for the command line. Also: These days chances are that a new harddisk will be bigger than a previous one, so don't worry about the sizes: It's OK if you make the partitions on the new harddisk bigger than they used to be on the old system. They don't have to be absolutely the same size. Just bear in mind that you can't restore e.g. a 4 GB /usr partition if you make the new one only half as big ...

IMPORTANT: What really is important here is that you create the same amount of partitions and in the same order as they used to be on the old harddisk. So if you used to have a /dev/hda5, /dev/hda6, and /dev/hda7 on your old disk then it would be wise to have that too on your new system. If you don't do this then it is imperative that you remember the new disk setup and edit your /etc/fstab before you reboot!

Also: I seriously hope you followed my advice above and got rid of the UUID's in your /etc/fstab ... when it comes to backup and especially when it comes to a restore those UUID's can be a serious pain.

So ... back on track:

- You get rid of the old harddisk, install the new one into your system
- you boot the Live CD
- you partition your new harddisk so that it resembles the old disk's partitioning scheme
- you format each of the new partitions, e.g. sudo mkfs -t ext3 /dev/hdaX
- you attach your USB harddisk (where our backups are) to the Live system
- the live system should have noticed the presence of your USB disk and it should have opened it
- you open a terminal and cd into the right directory on the USB disk so that you can access the archives

Once you got that far i'ts time for some thinking and planning. We are about to restore gigabytes and gigabytes of data ... so it better be the right disk and stuff be in the right order.

First, we need to check: What will our root partition " / " be? ==> Here Step #2 comes in handy. If you did copy your /etc as suggested above you could just open a second terminal and cd into the directory where you copied your /etc into and simply take a look at your old /etc/fstab ...

So let's assume that your " / " filesystem was on /dev/hda5 ... As this will be the " / " filesystem again we should recreate that one first.

Into a new terminal, type this (needs to be adjusted to reflect your own system settings):

Code: Select all

sudo su -
cd /mnt
mkdir new-root
mount /dev/hda5 /mnt/new-root
If everything went right these steps should finish without error.

If you check the contents of your "new-root" chances are that it is empty; there will likely be a "lost+found" directory but that's it.

So as a first step we need to recreate the various mount points again!

Code: Select all

mkdir /boot /usr /opt /home /var
Once that is done we can mount all sub-sequent partitions and attach it to our "new-root". So I will make some assumptions here, e.g. that /dev/hda6 is your /usr, /dev/hda7 is your /var, and so on. This needs to be adjusted, obviously:

Code: Select all

cd /mnt/new-root
mount /dev/hda1 boot
mount /dev/hda6 usr
mount /dev/hda7 var
mount /dev/hda8 opt
mount /dev/hda9 home  
With this we have recreated the structures of our harddisk ... so now we just need to fill the disk again!

There should still be an open terminal window where you cd'd to the archives, right? Into that type this command: pwd
This should spit out the current path, e.g. maybe something like /media/USBdisk/backups/archives ... Highlight it with your mouse (so that it gets copied into the clipboard).

In your other terminal window inside the "new-root" directory it's time for unpacking. Let's start with /boot ... being the smallest partition it's an excellent candidate for doing tests as it won't take too long to find out if it worked or not (if you don't have a separate /boot you can skip this ... scroll down where we restore the " / " filesystem).

Type this: tar -pzxvf and now hit your middle mouse button ... It should have inserted the path above from the clipboard into your command line so you don't have to type this path more than unnecessary. All that you need to do now is to add the name of the archive and then hit the enter key. So in the end it should look something like this:

Code: Select all

 cd /mnt/new-root
tar -pzxvf /media/USBdisk/where/the/backups/are/BootFS_Backup.tar.gz 
If everything worked the contents of your original /boot should have been restored to your current "new-root/boot" directory. If you do a "ls -al /mnt/new-root/boot" you should be able to see the files.

So if this works we can restore the rest too, e.g. the root filesystem " / ":

Code: Select all

 cd /mnt/new-root
tar -pzxvf /media/USBdisk/where/the/backups/are/RootFilesystem_Backup.tar.gz
tar -pzxvf /media/USBdisk/where/the/backups/are/UsrFilesystem_Backup.tar.gz
tar -pzxvf /media/USBdisk/where/the/backups/are/VarFilesystem_Backup.tar.gz
tar -pzxvf /media/USBdisk/where/the/backups/are/OptFilesystem_Backup.tar.gz
tar -pzxvf /media/USBdisk/where/the/backups/are/HomeFilesystem_Backup.tar.gz
Please note that if you preferred not to make a tar.gz of your home filesystem and instead used the "cp" command then you'd obviously restore your home folder as described in that section (Step 5: Restoring your /home).

If everything went as it should have you should have restored the content of your partitions to your new disk ... or your new system ... or a clone.

What's still missing is the boot loader. You could try restoring that one via restoring the MBR or with a tool like the Super Grub-Disk (please use Google ... it's a live CD specialised on restoring dead boot loaders).

Once everything is done it's time to umount and reboot the system .... which hopefully will reboot as if nothing had happened.

Code: Select all

cd /mnt/new-root
sync
umount boot
umount usr
umount var
umount opt
cd /mnt
umount new-root
If everything is OK there should be no errors. What could happen is that you get an error message about "mount point busy" so that a partition can't be unmounted. Chances are you still got a terminal open which is accessing something on that partition? Close everything and try again. Even if you still get errors it would probably safe to reboot now. Ideally your system should now come up and boot as if nothing had happened ... If it doesn't it could be that the boot block is still not 100% OK, so it would be a good idea to e.g. use the SuperGrub Disk and have the boot block rewritten again. But after that it should really work (I do the absolutely same thing here and I therefore know that this should work!).


Step 5: Copying /home

In the example above we packed /home into a *.tar.gz archive. That may or may not be a good idea. If you got tons of movies and music files in your /home directory then further compressing /home won't do no good ... it will just be a hell of a slow operation and take forever. In such circumstances it would be better to just make a copy onto an external harddrive as we did with /etc:

Code: Select all

sudo su -
cd /where/we/will/store/our/stuff
cp -a -v -u /home .
... This is basically the same command as above.

How to restore /home

In case something went wrong or if you had to reinstall your PC you can most likely just copy all the contents of /home back onto your new installation. Most stuff like your desktop settings, e-mails, bookmarks etc. should be tip top preserved as if nothing had happened. Example:

Code: Select all

 sudo su -
cd /path/to/where/we/backed/up/home
cp -a -v -u yourusername /home/ 
Please mind the syntax. The position of those slashes "/" is very important. If everything went right and the copy process finishes then all your files should be back in your /home directory and when you login everything should be in its place as if nothing had happened. I did that already several times and I can almost guarantee that if you execute the commands right you have zero data loss ... you can switch distros, uninstall, reinstall, experiment around and if done right you will hopefully never lose your files.


BONUS -- Step 6: Putting it all together

So this is a copy of my own personal backup script ... yes, no joking, I really use this monster and it does its job well. What it does:

- all of the above
- creates sub-directories per date
- which means I can go back to specific dates and get my system the way it was that day (provided I did run a backup then)
- hopefully it's not too hard to adapt it to your needs.
- due to some limitations in my knowledge and/or bash some parts of the script repeat over and over again ... sorry about that, but I haven't yet found a way around that.

Important: The script assumes that you are already "root" and that you are already on your USB drive and in the correct location where you want to store your backup, e.g. somewhere in e.g. /media/USBdrive/backup/wherever ...

Here we go:

Code: Select all

#! /bin/bash

export OSNAME="Mint_3.0"
export TODAYSBACKUPDATE=`date +%d-%b-%Y`
export HOSTNAME=`hostname`
export RELEASE=`uname -r`

mkdir $TODAYSBACKUPDATE
cd $TODAYSBACKUPDATE
dd if=/dev/hda of=$HOSTNAME"_hda-MBR_"$TODAYSBACKUPDATE".backup" bs=512 count=1

cp -a -v -u /etc .
cp -a -v -u /home .
sync

export MOUNTPOINT="BootFS"
export ARCHIVEFILENAME=$HOSTNAME"_"$OSNAME"_Kernel_"$RELEASE"_"$TODAYSBACKUPDATE"_"$MOUNTPOINT".tar.gz"
tar --one-file-system -pzcvf $ARCHIVEFILENAME /boot
sync

export MOUNTPOINT="RootFS"
export ARCHIVEFILENAME=$HOSTNAME"_"$OSNAME"_Kernel_"$RELEASE"_"$TODAYSBACKUPDATE"_"$MOUNTPOINT".tar.gz"
tar --one-file-system --exclude=/tmp/* --exclude=/proc/* --exclude=/dev/* --exclude=/sys/* -pzcvf $ARCHIVEFILENAME /
sync

export MOUNTPOINT="UsrFS"
export ARCHIVEFILENAME=$HOSTNAME"_"$OSNAME"_Kernel_"$RELEASE"_"$TODAYSBACKUPDATE"_"$MOUNTPOINT".tar.gz"
tar --one-file-system -pzcvf $ARCHIVEFILENAME /usr
sync

export MOUNTPOINT="OptFS"
export ARCHIVEFILENAME=$HOSTNAME"_"$OSNAME"_Kernel_"$RELEASE"_"$TODAYSBACKUPDATE"_"$MOUNTPOINT".tar.gz"
tar --one-file-system -pzcvf $ARCHIVEFILENAME /opt
sync

export MOUNTPOINT="VarFS"
export ARCHIVEFILENAME=$HOSTNAME"_"$OSNAME"_Kernel_"$RELEASE"_"$TODAYSBACKUPDATE"_"$MOUNTPOINT".tar.gz"
tar --one-file-system --exclude=/var/tmp/* -pzcvf $ARCHIVEFILENAME /var
sync

dpkg --get-selections > $HOSTNAME"_dpkg-PackageSelection.txt"
sync

unset MOUNTPOINT
unset ARCHIVEFILENAME 

That's it from here. Hope this was useful ... :D
.
.
.
.
.
- typos corrected! + section about "menu.lst" and UUID's added.
- bad, bad, bad typo removed. :(
Last edited by scorp123 on Fri Nov 16, 2007 10:42 pm, edited 5 times in total.
carlos
Level 4
Level 4
Posts: 470
Joined: Thu Dec 14, 2006 4:30 pm
Location: USA
Contact:

Post by carlos »

I think this needs to be in the next Mint magazine. Really excellent write-up Scorp. Definitely learned a lot here. :D
____________ ________ ______ ____ ___ __ __ _
http://www.designdrifter.com
The Purpose of Life is for DNA to reproduce.
User avatar
Sorensei
Level 4
Level 4
Posts: 257
Joined: Tue Jan 16, 2007 8:24 am
Location: Switzerland

Post by Sorensei »

Yes! :D :D :D

Thank you...
I don't have a solution, but I do admire the problem.
User avatar
linuxviolin
Level 8
Level 8
Posts: 2081
Joined: Tue Feb 27, 2007 6:55 pm
Location: France

Post by linuxviolin »

Thank you very much scorp! Great tutorial! As carlos I learned a lot.

Thanks.
Last edited by linuxviolin on Wed Aug 01, 2007 10:22 am, edited 1 time in total.
K.I.S.S. ===> "Keep It Simple, Stupid"
"Simplicity is the ultimate sophistication." (Leonardo da Vinci)
"Everything should be made as simple as possible, but no simpler." (Albert Einstein)
scorp123
Level 8
Level 8
Posts: 2272
Joined: Sat Dec 02, 2006 4:19 pm
Location: Switzerland

Post by scorp123 »

newW2 asked something via a private message ... I am going to answer here.
Can I repartition my drive to match the backup
You don't need to. If you just have one single " / " filesystem then you just need this line to backup -- the resulting archive already contains everything:

Code: Select all

tar --one-file-system --exclude=/tmp/* --exclude=/dev/* --exclude=/proc/* -pzcvf RootFS_backup.tar.gz / 
If you have a separate /home too ... well then: just add the line for the /home partition too:

Code: Select all

tar --one-file-system -pzcvf HomeFS_backup.tar.gz /home
The other lines about backing up /usr, /var and so on would be redundant in your case ... first of all your " / " backup already covers those locations if you only have a single " / " partition.

This does not prevent you from creating an elaborate partitioning scheme and then follow the restore procedure as I suggested where you'd mount all the partitions in their right locations: Because in the moment you unpack your single " / " backup archive all your files in there would then automatically fall into the right places onto their new proper partitions (if you do it right ... and I hope I wrote my instructions clear enough).

What you need to take care about: /etc/fstab .... the one that you restore won't yet have the right mount-point infos so you'd need to make sure that whatever way you re-partition your harddisks to is also reflected in /etc/fstab .... so you will need to edit this before you reboot! And you will probably need Super Grub Disk .... now that I think of it rebooting will fail too at first (because the info in the master boot record and e.g. your new /boot partition don't match ... ) and it would be handy to either have Super Grub Disk ready or to follow Husse's instructions in the wiki ( http://www.linuxmint.com/wiki/index.php ... _your_grub )

So, again, hand-holding and step-by-step:

- I assume you have one single " / " partition
- to backup you'd just use the line relevant to " / " partitions

Code: Select all

tar --one-file-system --exclude=/tmp/* --exclude=/dev/* --exclude=/proc/* -pzcvf RootFS_backup.tar.gz / 
- then follow my "How to restore" instructions above, except that you don't "re-create the partitions as they were on the old disk" ... you partition stuff the way you want

When done: Again follow the restore procedure and mount those things in their right (new) locations:
So let's assume that your " / " filesystem was on /dev/hda5 ... As this will be the " / " filesystem again we should recreate that one first.

Into a new terminal, type this (needs to be adjusted to reflect your own system settings):
Code:
sudo su -
cd /mnt
mkdir new-root
mount /dev/hda5 /mnt/new-root
If everything went right these steps should finish without error.

If you check the contents of your "new-root" chances are that it is empty; there will likely be a "lost+found" directory but that's it.

So as a first step we need to recreate the various mount points again!
Code:
mkdir /boot /usr /opt /home /var
Once that is done we can mount all sub-sequent partitions and attach it to our "new-root". So I will make some assumptions here, e.g. that /dev/hda6 is your /usr, /dev/hda7 is your /var, and so on. This needs to be adjusted, obviously:
Code:
cd /mnt/new-root
mount /dev/hda1 boot
mount /dev/hda6 usr
mount /dev/hda7 var
mount /dev/hda8 opt
mount /dev/hda9 home
With this we have recreated the structures of our harddisk ... so now we just need to fill the disk again!
As for filling the disk ... Unpack your single " / " archive now ... all the files inside of it should now fall into place onto their new partition locations!

Code: Select all

cd /mnt/new-root
tar -pzxvf /media/USBdisk/where/the/backups/are/RootFilesystem_Backup.tar.gz 
Dont forget to edit /etc/fstab !! Then follow the rest when you're done: "Once everything is done it's time to umount and reboot the system ..."

But it won't boot in your case (because infos in the MBR and on your new partitioning scheme are different now) so you hopefully have that Super Grub Disk ready ..... :wink:

But after "repairing" GRUB it then really should reboot again as if nothing had happened. I've done this before as well and I am sure it works. Even if it doesn't: you still got all your stuff inside that backup archive file, right? :D
.
.
.
.
typos corrected!!
Last edited by scorp123 on Wed Aug 01, 2007 6:07 am, edited 2 times in total.
Chinaski

Re: How to backup your stuff UNIX-style

Post by Chinaski »

What is "--one-filesystem" When I try and backup, I get "tar: unrecognized option `--one-filesystem'
Try `tar --help' or `tar --usage' for more information."
Please forgive me for my ignorance. :wink:
scorp123
Level 8
Level 8
Posts: 2272
Joined: Sat Dec 02, 2006 4:19 pm
Location: Switzerland

Re: How to backup your stuff UNIX-style

Post by scorp123 »

Chinaski wrote:What is "--one-filesystem"
A typo. Thanks for catching that!! It should be --one-file-system .... please note the dash between "file" and "system". I corrected my posting above, seems I was not fully awake when I wrote that. Thanks again. :D
Chinaski

Re: How to backup your stuff UNIX-style

Post by Chinaski »

scorp123 wrote:
Chinaski wrote:What is "--one-filesystem"
A typo. Thanks for catching that!! It should be --one-file-system .... please note the dash between "file" and "system". I corrected my posting above, seems I was not fully awake when I wrote that. Thanks again. :D
You welcome. It works great now :) .
Raphael
Level 1
Level 1
Posts: 15
Joined: Sun Jul 15, 2007 4:22 am

Re: How to backup your stuff UNIX-style

Post by Raphael »

scorp123 wrote:
Step 3: Our package selections

1. Backup /etc/apt/sources.list ... or copy it to one of your other systems (in case you want to have the same packages everywhere)

2. On system #1 (the one you want to backup) issue this command:
sudo dpkg --get-selections > packagelist.txt

=> This will write your package selections and the status of all packages into that text file "packagelist.txt". Now copy that file to your other system(s) or put it onto a USB stick or USB harddisk.

How to restore your package selections

1. You need to restore the /etc/apt/sources.list file (see section above about copying /etc recursively ...) and then you need to tell dpkg to read your package list:
sudo dpkg --set-selections < packagelist.txt

=> The system will now take over the package selections that were previously defined on the original or previous system.

2. All you need to do now is to execute the install process:
sudo apt-get dselect-upgrade

=> Result: You now will have the same packages installed like before on "System #1"

"Armed" with these commands you can also clone installations (e.g. produce identical systems with identical software selections) in a couple of minutes. Just make sure you get the /etc/apt/sources.list right (e.g. it should be the same on all systems).
I did this "for real" now for the first time when I switched from Celena to Daryna. Worked fine and safed me a lot of hassle. Thank you.
sc123

Re: How to backup your stuff UNIX-style

Post by sc123 »

muskratmx wrote:Removal of UUIDs degrade the preformance of your system.
Nonsense!
Husse

Re: How to backup your stuff UNIX-style

Post by Husse »

sc123 wrote:
muskratmx wrote:Removal of UUIDs degrade the preformance of your system.
Nonsense!
Is this scorp?
Fred

Re: How to backup your stuff UNIX-style

Post by Fred »

There is not a lot to be found on the issue of legacy /dev/sdxx designations and UUIDs relative to system speed. I did find a couple exchanges on the MySQL forums indicating that UUIDs enabled faster database access in some situations, but it was disputed and contentious. I haven't been able to find anything that I would be willing to hang my hat on one way or the other, otherwise.

My suggestion would be to use labels as a compromise between UUIDs and legacy. Labels have a distinct maintenance advantage over UUIDs and still allow for a unique ID for name collision purposes.

Fred
sc123

Re: How to backup your stuff UNIX-style

Post by sc123 »

Fred wrote: There is not a lot to be found on the issue of legacy /dev/sdxx designations and UUIDs relative to system speed.
And you want to know why?? Because there is nothing to find. Whether you use the legacy /dev/sdXX method, or UUID's or disk labels ... The *ONLY* purpose of all these mechanisms is to tell the system via /etc/fstab which disk it needs to mount onto which directory. That's it! Once the disks are mounted to their locations the way by which they got mounted is totally irrelevant (unless you do crazy stuff to mounted filesystems?), all speed or lack thereof solely depends on your hardware (e.g. SAS disk vs. USB 1.1 ... :D ) and the overall performance of your system.

To claim that using the legacy notation in /etc/fstab would downgrade the overall system performance is downright stupid. If I am wrong here then someone please correct me and post stuff that proves the contrary! :evil:
Fred wrote: I did find a couple exchanges on the MySQL forums indicating that UUIDs enabled faster database access in some situations, but it was disputed and contentious.
And rightly so!
Fred wrote: My suggestion would be to use labels as a compromise between UUIDs and legacy.
For as long as you know why and what you do what you do you can mix all methods. The suggestion to remove UUIDs was based on a few observations:
- it confuses the hell out of users, especially newbies
- for the purpose of replacing a dead disk UUID's can get badly into one's way, especially if you forgot to get the volume ID of the new disk and forgot to edit the /etc/fstab when you restored your backup ... most users might react with panic and frustration (... "This #@!!!! backup doesn't work !!!" ...) because they won't realise in time that all they'd need to do is to change the UUID to get their restored system to boot again ...

So why bother with UUID's at all then? For the purpose of doing system recoveries and getting your data back (even onto new disks and new systems if you need to do that) they only offer disadvantages and are in fact rather hasslesome (write down your old UUID, get the new UUID, don't forget to edit /etc/fstab and /boot/grub/menu.lst before you reboot ... ) whereas with the legacy "/dev/sdXX" method you could just swap a dead disk for a new disk ... and voila: you're done and can focus on other things.

The disk label method could work as well here .. thing is you'd have to remember to give your new disk the same label as the old one if you have to swap disks ...

As I said above: If you know what you do and why you do it ... fine. :twisted:
User avatar
GrayWizardLinux
Level 6
Level 6
Posts: 1232
Joined: Wed Sep 12, 2007 5:47 pm
Location: Anywhere I Am!

Re: How to backup your stuff UNIX-style

Post by GrayWizardLinux »

Husse, I also think this is scorp. He coculd have picked TomH or BettyK or laptop57... :D

welcome back Scorp!
Linux Mint - Pure Bliss!
Fred

Re: How to backup your stuff UNIX-style

Post by Fred »

sc123,

Greetings sir,

I read your post, looking for something I could debate with you, but to no avail. I am afraid you said nothing I couldn't agree with. :-)

As far as my post goes, it is what it is. I stand by it.

EDIT: If you are interested in why speed concerns might even be questioned, you might want to look into inode addressing issues at the kernel level.

sc123 wrote:
As I said above: If you know what you do and why you do it ... fine.
Again, I think I we can agree here also. In fact, I would suggest that might apply to quite a few things in life, not just the structure of an fstab file. :-)

Fred

PS: By-the-way, it's nice to meet you. Your reputation of competence precedes you.
Last edited by Fred on Thu May 15, 2008 2:25 pm, edited 1 time in total.
User avatar
newW2
Level 5
Level 5
Posts: 821
Joined: Fri Apr 06, 2007 10:24 am
Location: USA

Re: How to backup your stuff UNIX-style

Post by newW2 »

The writing style is / seems un-mistakable scorp.
Husse

Re: How to backup your stuff UNIX-style

Post by Husse »

If it is - welcome back
deadguy
Level 5
Level 5
Posts: 707
Joined: Wed Aug 15, 2007 9:41 pm
Location: Illinois USA

Re: How to backup your stuff UNIX-style

Post by deadguy »

Fred wrote:My suggestion would be to use labels as a compromise between UUIDs and legacy. Labels have a distinct maintenance advantage over UUIDs and still allow for a unique ID for name collision purposes.

Fred

I agree with Fred here!

and, if that is indeed Scorp, glad to see him back :D

cheers,

Deadguy
sofasurfer

Re: How to backup your stuff UNIX-style

Post by sofasurfer »

Scorp123.
I'm making good progress and learning a lot. But I don't understand one thing. How do you run a script such as yours at the beginning of this thread?
kpswalin

Re: How to backup your stuff UNIX-style

Post by kpswalin »

This a great walk through, and I am customising it for my specific partitioning scheme and system. I have however hit a snag and was hoping for some guidance from someone more experienced.

When I perform the following command,

Code: Select all

 tar --one-file-system --exclude=/tmp/* --exclude=/dev/* --exclude=/proc/* --exclude=/mnt/* -pzcvf /home/'My Home Directory'/Desktop/RootFS_backup.tar.gz /
I get this when the action completes.

Code: Select all

/srv/
/dev/
tar: /dev/: file is on a different filesystem; not dumped
/sys/
tar: /sys/: file is on a different filesystem; not dumped
I assume it has something to do with '/proc/mounts' based on some Google searches, but I am not experienced enough to solve it or change my command.

Any guidance is appreciated.

Linux Mint Rocks!!!

Ok, so I see the '--one-file-system' keeps tar from backing up anything that is not within the local file system, i.e. remote mounts. But why is /dev (even though it is excluded) and /sys on a remote file system. It appears to me they are local?

Code: Select all

~ $ ls -lh /
total 704K
drwxr-xr-x   2 root root 4.0K 2008-12-17 19:49 bin
drwxr-xr-x   6 root root 4.0K 2009-01-02 09:07 boot
lrwxrwxrwx   1 root root   11 2008-11-28 19:32 cdrom -> media/cdrom
drwxr-xr-x  15 root root  14K 2009-01-04 21:25 dev
drwxr-xr-x 147 root root  12K 2009-01-04 21:25 etc
drwxrwxr-x   8 root root 4.0K 2008-11-28 22:21 Gallery_Remote
drwxr-xr-x   4 root root 4.0K 2008-12-01 20:41 home
lrwxrwxrwx   1 root root   32 2009-01-02 09:05 initrd.img -> boot/initrd.img-2.6.27-9-generic
lrwxrwxrwx   1 root root   32 2008-11-28 19:42 initrd.img.old -> boot/initrd.img-2.6.27-7-generic
drwxr-xr-x  16 root root 4.0K 2008-12-20 16:32 lib
drwx------   2 root root  16K 2008-11-28 19:31 lost+found
drwxr-xr-x   3 root root 4.0K 2009-01-03 14:24 media
drwxr-xr-x   4 root root 4.0K 2009-01-04 14:30 mnt
drwxr-xr-x   2 root root 4.0K 2008-10-29 16:53 opt
dr-xr-xr-x 153 root root    0 2009-01-03 09:14 proc
drwxr-xr-x  31 root root 4.0K 2008-12-26 23:43 root
drwxr-xr-x   2 root root 4.0K 2009-01-02 09:04 sbin
drwxr-xr-x   2 root root 4.0K 2008-10-29 16:53 srv
drwxr-xr-x  12 root root    0 2009-01-03 09:14 sys
-rw-r--r--   1 root root 615K 2009-01-02 09:16 tftpboot
drwxrwxrwt  17 root root 4.0K 2009-01-04 21:44 tmp
drwxr-xr-x  13 root root 4.0K 2008-12-26 22:57 usr
drwxr-xr-x  14 root root 4.0K 2008-11-28 22:17 var
lrwxrwxrwx   1 root root   29 2009-01-02 09:05 vmlinuz -> boot/vmlinuz-2.6.27-9-generic
lrwxrwxrwx   1 root root   29 2008-11-28 19:42 vmlinuz.old -> boot/vmlinuz-2.6.27-7-generic
~ $ 
Here is my fstab too

Code: Select all

# /etc/fstab: static file system information.
#
# <file system> <mount point>   <type>          <options>                       <dump>  <pass>
proc            /proc           proc            defaults                        0       0
/dev/sda2       /               ext3            relatime,errors=remount-ro      0       1
/dev/sda1       /boot           ext3            relatime                        0       2
/dev/sdb1       /home           ext3            relatime                        0       2
/dev/sdb2       none            swap            sw                              0       0
/dev/scd0       /media/cdrom0   udf,iso9660     user,noauto,exec,utf8           0       0
nas:/share      /mnt/nas        nfs             defaults                        0       0
I thought I was getting better at this, but I guess I have not graduated from noob yet. :mrgreen:
Post Reply

Return to “Tutorials”