How to backup your stuff UNIX-style

Write tutorials for Linux Mint here
More tutorials on https://github.com/orgs/linuxmint/discu ... /tutorials and (archive) on https://community.linuxmint.com/tutorial
Forum rules
Don't add support questions to tutorials; start your own topic in the appropriate sub-forum instead. Before you post read forum rules
kpswalin

Re: How to backup your stuff UNIX-style

Post by kpswalin »

Can I just safely exclude /dev and /sys directories. I have seen on the web where others are suggesting that. What are the implications? If there are none I assume my new command would be something like this

Code: Select all

tar --one-file-system --exclude=/tmp/* --exclude=/dev --exclude=/sys --exclude=/proc --exclude=/mnt/* -pzcvf /home/'My Home Directory'/Desktop/RootFS_backup.tar.gz /
But that may require that I recreate the /dev and /sys directories when restoring a system. I am wondering if scorp123 just added the "--exclude=/dev/*" so that the /dev directory would be recreated during a restore?

Why didn't scorp123 exclude the /sys directory if it is in fact not needed?
calibre97

Re: How to backup your stuff UNIX-style

Post by calibre97 »

AARGH! I am sooo close. Here's my setup:
Dell D630 with an internal 250GB drive, and a 120GB external USB drive.
/dev/sda1 XP (I back it up after booting to Acronis TrueImage Home 11 and connecting a USB drive)
/dev/sda5 Data1 (ext3, 100GB)
/dev/sda6 Data2 (ext3, 50GB)
/dev/sda7 M7Home (ext3, 5GB, for Mint 7 "/home")
/dev/sda8 M7Root (ext3, 10GB, for Mint 7 "/")
/dev/sda9 swap
/dev/sda10 M6Home (ext3, 5GB, for Mint 6 "/home")
/dev/sda11 M6Root (ext3, 10GB, for Mint 6 "/")
/dev/sda12 (ext4 so no label, 5GB for Kubuntu 9.04 "/home")
/dev/sda13 (ext4 so no label, 10GB for Kubuntu 9.04 "/")

I'm trying out the techniques described in this thread, but backing up to Data1 for now (I'll rsync that partition to the external USB drive). I boot up with RescueCD v1.1.7 and create directories under "/A/" for all points. For example, /A/K9Home, and so on, and then mount them.
I can use the tar command for /A/K9Home to /A/Data1/K904/K9home_backup.tar.gz and /A/K9Root/boot to /A/Data1/K904/K9boot_backup.tar.gz. I'm stuck trying to tar up "/" though. I keep getting kicked back with "zsh: no matches found: --exclude=/A/K9Root/tmp/*. Here's the command I'm using, straight from your examples:
root@sysresccd /A/Data1/K904 % tar --one-file-system --exclude=/A/K9Root/tmp/* --exclude=/A/K9Root/dev/* --exclude=/A/K9Root/proc/* -pzcvf K904root_backup.tar.gz /A/K9Root

I've also tried that with a trailing slash at the end for "/A/K9Root/".
I run ls on /A/K9Root/tmp and it's there so I don't understand why the --exclude is choking on it. I have a lot of partitions so I can try out various versions of Mint and Kubuntu. I'm about to blow away the Mint6 stuff, but I'd like a backup just in case. I'd also of course like a backup of everything else. Previously, I was booting the rescueCD and rsyncing each partition to a like partition (with Data1 and Data2 not being the same size of course) on the external 120GB USB drive. I thought I'd try this technique, creating TAR balls on Data1 so I could just rsync Data1 to the external drive and have the stuff handy in case a partition went tits up.

Can you explain why I'm having so much trouble backing up "/"? Is it because it's ext4 perhaps? I'll try doing it with the Mint6 root partition, which is ext3, and see how that goes.

UPDATE: Nope. Doesn't work for Mint 6 either. If I remove the tmp option, it chokes on /dev. I try --exclude=/tmp/* and --exclude=/A/M6Root/tmp/* (thinking since I'm booted to the RescueCD, I need to give the explicit path to the /tmp directory in the Mint 6 partition. But no go.

I'm officially stumped. I guess I'll go back to rsyncing all of the individual partitions to the external drive for now.
User avatar
eiver
Level 5
Level 5
Posts: 568
Joined: Fri Jan 23, 2009 5:51 am

Re: How to backup your stuff UNIX-style

Post by eiver »

I have read the above thread and also the very interesting: http://www.halfgaar.net/backing-up-unix
I decided to use dar instead of tar after reading the link above.
The only question is - should I include /dev in my backup. I saw some people arguing for, but also some tutorials suggesting to exclude it.
When I include it I get some errors while restoring /dev to a new HDD. Also the system reports some errors during boot. The system boots however and is usable.

What should I do to get the file system as close to the original as possible (including all metadata like permissions, hard and soft links, etc.)
Include or ignore /dev?

Consider a scenario: HDD failed and was replaced, but all other hardware is still there
Consider a second scenario: The whole machine is fried and the system must be restored to a different machine with different hardware
webb1976

Re: How to backup your stuff UNIX-style

Post by webb1976 »

Thanks for this.

I just took a beginner's Linux class and what you posted MADE A TON OF SENSE. It is amazing what you can do when you know the basics of how things work. If this were Windows people would be making restore points and all other craziness.
kwevej

Re: How to backup your stuff UNIX-style

Post by kwevej »

This tutorial is way too complicated.
That's what i use:
(system and /home partitions separated)

system backup (you can run this while working with your system):

Code: Select all

#!/bin/bash
DEV=/dev/sda2
mount -o remount,acl,user_xattr $DEV
BPATH='./sys/back_sda2_'$(date '+%F')'.fsa'
echo 'saving fs'
nice ./fsarchiver -A -o -z7 savefs $BPATH $DEV -e /lost\+found -e /media -e /mnt -e /cdrom -e /proc -e /dev -e /tmp -e /sys 
echo 'remounting fs'
mount -o remount $DEV
echo 'done'
system restore (run from liveCD):

Code: Select all

#!/bin/bash
DEV=/dev/sda2
MNT=/media/sysrmnt
nice ./fsarchiver restfs $1 id=0,dest=$DEV
mkdir -p $MNT
mount $DEV $MNT
mkdir $MNT/tmp
chmod 777 $MNT/tmp
mkdir $MNT/sys
mkdir $MNT/proc
mkdir $MNT/dev
mkdir $MNT/media
mkdir $MNT/mnt
(gpg encryption left out of the scripts)

for home backup i use incremental backup https://help.ubuntu.com/community/DuplicityBackupHowto
Last edited by kwevej on Mon Mar 21, 2011 8:19 am, edited 1 time in total.
steev

Re: How to backup your stuff UNIX-style

Post by steev »

I think the correct permissions for /tmp are 1777 (this prevents users from deleting files that don't belong to them)
Otherwise the script looks fine.
agrandir

Re: How to backup your stuff UNIX-style

Post by agrandir »

A very useful and educational set of instructions which I personally have found very useful, especially in regard to saving MBR and lpartition tables, as well as list of packages installed, which are always things one tends to forget and on re-installation it becomes guesswork for those with a poor memory as to what to put where. However, as this Howto was written 3 years ago and Grub2 has now replaced the legacy versions used at this time, then might I suggest a modification so that newbies, like myself, don't screw up their system in trying to modify UUIDs. menu.lst no longer exists in the /boot/grub directory.

I don't feel confident enough for suggesting how to go about changing UUIDs so if someone more competent happens to read this ... :?:
oldfart

Re: How to backup your stuff UNIX-style

Post by oldfart »

You don't need to worry about the UUID. There is no need to change the fstab. If you are going to put your backed up system on a different device all you need do is change the UUID, of the device you are going to write to, to the UUID that you backed up from. To do this one uses the appropriate filesystem utility. Something like tune2fs or tunefs.reiserfs. It's a simple command. Read the man pages or do a google search to learn more.

Also if your MBR gets thrashed you failed to follow the advice to THINK before acting. So one shouldn't need a backup of their MBR. Also it can be rewritten a number of different ways depending on what is the OS that wrote it. In windows it's fdisk /mbr and with a grub system just reinstall grub with grub-install or input lilo for a lilo boot loaded system. You do need to be chrooted to do that but it's not hard.

Now a days hard drives are cheap. I bought a 2 Terabyte drive on sale for $80. ( The first hard drive I had was ten megs and cost hundreds of dollars. ) So I use a very large drive in an enclosure and just copy and paste it in some sort of file manager. Don't forget your dot files.

This was a slightly interesting idea but it's over kill when I can do the same thing with a copy paste. Yes your ego might get a stroke or two from using the command line, I got over it you can too, but that is no longer needed to do a simple backup.

Time marches on. March with it or be left behind.
Umbra Polaris

Re: How to backup your stuff UNIX-style

Post by Umbra Polaris »

an update for UEFI/GPT machine will be appreciated :D
crossroads

Re: How to backup your stuff UNIX-style

Post by crossroads »

change your /etc/fstab so that it uses device names!


That stopped me in my tracks. Could not find anything in the forum directly pointing to how to change device names. Internet searches brought me to a couple of interesting places ... one being this ... http://pclosmag.com/html/Issues/200709/page07.html "How to edit and understand /etc/fstab - 1.1. Which assumes that I already know how to mount file systems and partitions with the mount commands, specific syntax, file types, etc.

Guess for now, I'll stick to the "appropriate file system utilities". Thanks anyway. Maybe later, I'll come back to this tutorial when I have a bit more experience under my belt.
whitelight999

Re: How to backup your stuff UNIX-style

Post by whitelight999 »

Thanks for that
Just finished first back up of system,

I used to work in a place years ago where we had dozens of HP-UX machines.
We used to back them up once a month to a tape machine with that tar -pcvf command.
I was in Hardware maintenance at the time.
Cannot remember how big were the tapes or what we backed up, Was it whole root or just the equivalent of the home drive.
We never had to restore anyway,

What would be Linux equivalent of a System Repair disk in windows? is it just a bootable ISO on a USB drive?
User avatar
SaintDanBert
Level 4
Level 4
Posts: 440
Joined: Sat Feb 20, 2010 4:04 pm
Location: Covington, LA USA

Re: How to backup your stuff UNIX-style

Post by SaintDanBert »

cldx3000 wrote:This is a crazy awesome tutorial, actually i was looking for the one which explains how to set up partitions in a smart way since i just started with Mint but this definetely is one of the most insight tuts ive ever read, ill copy it to my USB Drive alongside my Backups for future reference for sure. I was wondering if the same procedure would be working on an external Harddrive thats connected to my Router, or would that be on the stupid side of things to do?
(grinning) partition plans are a common source of interesting discussions that last forever...

Here is what I do:
win-doze and Linux on separate drives where possible

linux drive file systems:
  • /boot ........ files needed for linux boot
  • win-doze and Linux on separate drives where possible
  • swap space
  • /home ....... all per-user files
  • /var .......... logs are system run-time files
  • / .......... /etc, /usr, /opt, /dev, root-user $HOME, and such go here
  • /wrk ......... "landfill" of free disk space
win-doze drive file systems
  • win-doze wants two(2) partitions for OS parts having nothing to do with drive letters
  • drive-D, E, F .......... all per-user files (NOTE -- Linux can access NTFS just fine)
  • drive-Z .......... default DVD/CD/BluRay for win-doze use
  • drive-U, V,W,X .......... external drives and such for win-doze use
I said file system instead of partition because what you do depends on the size of the native drive and how you separate the parts.
Also, there are various ways to create logical combinations of space using LVM and multi-drive techniques.

As a system runs, things write to /var for all sorts of reasons. If something runs amok, a separate /var will prevent most damage to important system parts.

Keeping /home separate makes your per-user files safer. I try to use part of a separate physical drive when I can. Typically, my /home will be part of the same physical drive with /wrk. I've used multi-drive to extend /wrk many times.

The /boot file system holds all of the kernel images and the compressed root-file-system images. I find that a separate file system keeps me out of trouble when one of these gets modified during updates. NOTE -- Your physical drive Master Boot Record (MBR) or GUID Partition Table (GPT) will point at your boot loader -- typically grub2. Grub2 will then use the files in /boot to start Linux kernel and supporting
processes.

You still need swap space even in these days of cheap physical RAM. Both of the system-suspend features -- suspend-to-rem (SLEEP) and suspend-to-disk(HIBERNATE) -- use the swap space to do their job. If you really, REALLY load your sustem with processes or if you run really large graphics applications, you will use swap space for run-time memory page and process management.
User avatar
SaintDanBert
Level 4
Level 4
Posts: 440
Joined: Sat Feb 20, 2010 4:04 pm
Location: Covington, LA USA

Re: How to backup your stuff UNIX-style

Post by SaintDanBert »

kwevej wrote:This tutorial is way too complicated.
That's what i use:
(system and /home partitions separated)

system backup (you can run this while working with your system):

Code: Select all

#!/bin/bash
DEV=/dev/sda2
mount -o remount,acl,user_xattr $DEV
BPATH='./sys/back_sda2_'$(date '+%F')'.fsa'
echo 'saving fs'
nice ./fsarchiver -A -o -z7 savefs $BPATH $DEV -e /lost\+found -e /media -e /mnt -e /cdrom -e /proc -e /dev -e /tmp -e /sys 
echo 'remounting fs'
mount -o remount $DEV
echo 'done'
system restore (run from liveCD):

Code: Select all

#!/bin/bash
DEV=/dev/sda2
MNT=/media/sysrmnt
nice ./fsarchiver restfs $1 id=0,dest=$DEV
mkdir -p $MNT
mount $DEV $MNT
mkdir $MNT/tmp
chmod 777 $MNT/tmp
mkdir $MNT/sys
mkdir $MNT/proc
mkdir $MNT/dev
mkdir $MNT/media
mkdir $MNT/mnt
(gpg encryption left out of the scripts)

for home backup i use incremental backup https://help.ubuntu.com/community/DuplicityBackupHowto
What are these fsarchive, savefs and restfv commands that you mention? Are they scripts or programs that you wrote?

Thanks,
~~~ 0;-Dan
DaveMarshall

Re: How to backup your stuff UNIX-style

Post by DaveMarshall »

I've been looking all over for an in-depth explanation on backing up Unix-style so this was a much needed read through. Hopefully now I can get to grips with getting this working on my new system.
whitelight999

Re: How to backup your stuff UNIX-style

Post by whitelight999 »

How would you do the first part if you had a GPT instead of MBR ?
manintveld

Re: How to backup your stuff UNIX-style

Post by manintveld »

Well I feel a bit like Charlie Brown lying back on the mound looking up at the clouds with Lucy and Linus

and Lucy says "if you use your imagination you can see lots of things in the cloud formations ... what do you think you see, Linus?" and Linus says "Well those clouds up there look to me like the map of the British Honduras on the Caribbean ... That cloud up there looks a little like the profile of Thomas Eakins, the famous painter and sculptor ... and that group of clouds over there gives me the impression of the stoning of Stephen ...

and Lucy says "Uh huh .. that's very good what do you see in the clouds, Charlie Brown?

"Well I was going to say I saw a Ducky and a Horsie but I changed my mind."



But I don't care :)

My backup (which I worked out myself) is, first accessing each of the disks I'm not quite sure why but it needs it, then:

chmod +x /home/manintveld/BatchBackup.sh
./BatchBackup.sh

with ./BatchBackup.sh as:

#!/bin/bash
rsync -auv /home/manintveld/Desktop /media/manintveld/Data/Desktopbackup/
rsync -auv /home/manintveld/.thunderbird /media/manintveld/Data/Desktopbackup/
rsync -auv /home/manintveld/Music /media/manintveld/Data/Desktopbackup/
rsync -auv /manintiveld/WindowsDesktop/Users/manintveld/Desktop /media/manintveld/Data/Desktopbackup/WindowsDesktop
rsync -auv /media/manintveld/Data/ /media/manintveld/"Backup disk"/WeeklyComputerbackup/
rsync -auv /media/manintveld/Data/ /media/manintveld/Elements/WeeklyComputerbackup/

which backs up my desktop and e-mails and music and Windows desktop files to my Data, Backup and backup backup (Elements) disks

What could go wrong?

No, seriously, it seems to work nicely but what could go wrong?
whitelight999

Re: How to backup your stuff UNIX-style

Post by whitelight999 »

Regarding the first part of the tutorial

So if you wanted a script which had to run on both MBR and GPT drives
sudo gdisk -l /dev/sda | grep "MBR only"
sudo gdisk -l /dev/sda | grep "GPT present"


If #MBR
then
dd if=/dev/hda of=/tmp/hda-mbr.bin bs=512 count=1
elseif #GPT
???
fi

Anyone know what to put in for ??? to solve this?
typematrix

Re: How to backup your stuff UNIX-style

Post by typematrix »

whitelight999 wrote:Regarding the first part of the tutorial
So if you wanted a script which had to run on both MBR and GPT drives
sudo gdisk -l /dev/sda | grep "MBR only"
sudo gdisk -l /dev/sda | grep "GPT present"
If #MBR
then
dd if=/dev/hda of=/tmp/hda-mbr.bin bs=512 count=1
elseif #GPT
???
fi
Anyone know what to put in for ??? to solve this?
# ??? in bash

echo "GPT partition table detected "
sudo dd if=/dev/hda of=/tmp/hda-gpt.bin bs=512 skip=1 count=33

I can show you how to detect GPT and MBR as well and what file drive the file system is on.
Post Reply

Return to “Tutorials”