Expanding /home (mdadm RAID 1)

All Gurus once were Newbies
Forum rules
There are no such things as "stupid" questions. However if you think your question is a bit stupid, then this is the right place for you to post it. Please stick to easy to-the-point questions that you feel people can answer fast. For long and complicated questions prefer the other forums within the support section.
Before you post please read how to get help
Post Reply
5hades0f6rey
Level 1
Level 1
Posts: 26
Joined: Mon Jun 14, 2010 10:35 pm

Expanding /home (mdadm RAID 1)

Post by 5hades0f6rey »

Let me start by saying that I'm sure there are things I'm forgetting or wasn't the clearest about in the text below... I'm pretty sleep deprived ATM and sincerely beg your patience. :oops: Tell me if there's anything you need to know, and I'll do my best to fill in the blanks ASAP.


Presently I had two 500GB drives formatted as ext4 in a RAID 1 array using dmraid that constituted my /home directory. I created a new array with two additional 1TB drives in another RAID 1 array using mdadm and used rsync to effectively move /home before deleting the old array. Now I want to take the two old drives and somehow add their capacity to /home. Hopefully doing so without having to necessarily take /home off-line. Or at least shuffling /home the way I already have around during the process that wouldn't necessarily mean absolutely having to have to backup/restore /home. I don't have the storage necessary to backup up home while adding the other drives... Or at least I don't think so. I understand that I could build a RAID 1 array already degraded so that when a second disk is added, the mirror is built. But I'm not sure what the best strategy might be for the overall process. Otherwise I'm looking at a whole mess load of DVDs and finding some way to backup /home to them. I've done some searching on this topic and so far I'm finding the amount of info out there a bit daunting (and some rather dated). I was hoping more experienced users could give me some advice on how I might go about doing this. I would also be willing to entertain other suggestions on how to expand the capacity of /home that might be a bit more novel. I probably should mention that I have a few users on this system that want to be able to encrypt their /home directories. Presently that means eCryptfs. It may have been lazy of me but the expediency of the "Encrypt home folder to protect sensitive data" option during user creation was really attractive... Anyway, just keep in mind that encryption of some sort will be in the mix.

BTW, I'm also entertaining the idea of switching file systems. Particularly if I ended up absolutely having to make an off-line backup of /home. However, the last time I seriously tried switching to Linux full time, I settled on the still somewhat 'experimental' reiserfs and had all kinds of problems. This time I stuck with the install CD/DVD default of ext4, figuring that I could always switch later. It looks like this may be my opportunity to do so. Thing is, I'm not sure which file system to use. Should I stick with ext4 or try something different? I plan of having a mix of predominantly large files like video files & disk images (i.e. VirtualBox images, ISOs, Windows based laptop drive images) and smaller files like word processing documents, spreadsheets, digital pictures, music, etc. Which file systems would you all recommend? I'd also like fairly reliable file system. Again, my experience with reiser was in a world, "painful."
5hades0f6rey
Level 1
Level 1
Posts: 26
Joined: Mon Jun 14, 2010 10:35 pm

Re: Expanding /home (mdadm RAID 1)

Post by 5hades0f6rey »

My last post was not a very well constructed one. Insomnia is a real <expletive deleted>. Anyway, here's hoping that this one actually garners some response.

Put simply, I'd like to know how I can or should best go about increasing the capacity of my /home directory.

I started with two 500GB drive in a fakeRAID RAID 1 array using dmraid. Since then, I decided I wanted to make this Linux system more of a file server in addition to being a shared desktop. To that end, I purchased two 2TB drives (yes, 2TB not 1TB, typo in the OP). My original plan was to create a separate RAID 1 array using mdadm - the switch from mdadm was twofold, one it allows the RAID to be 'portable' and it seems its a better supported software RAID solution - with the 2TB drives and create a new mount point representing the entire 2TB as /mnt/Data. Which I've already done.

However, after further consideration, I came to the conclusion that it made far more sense to simply merge /mnt/Data with /home. Problem is, I'm not 100% sure how to do this and do this safely... Or rather not quite as recklessly/dangerously as someone who has rally only casually dabbled with Linux. I can read manual pages and I usually get the desired result, but in this case I'd appreciate some experienced guidance then trust in my luck. Particularly after reading the mdadm man pages a bit more closely.

I now know I can grow a RAID 1 array. What I'm not sure of is if I can add two pairs of drives of different capacities into one aggregate 2.5GB RAID 1 array. The man pages are a bit vague as to how the addition and removal of drives will affect capacity other than to say, "Normally when an array is build [sic] the "size" it taken from the smallest of the drives. If all the small drives in an arrays are, one at a time, removed and replaced with larger drives, then you could have an array of large drives with only a small amount used. In this situation, changing the "size" with "GROW" mode will allow the extra space to start being used. If the size is increased in this way, a "resync" process will start to make sure the new parts of the array are synchronised. " So I'm a bit unsure if this in fact means that the array can grow from 2TB to 2.5TB with the addition of the 500GB drives or if it only means that I could grow a 500GB RAID 1 array to a 2TB RAID 1 array by first degrading the 500GB array and adding larger capacity drives.

Another wrinkle is the possible use of LVM to accomplish my goal. As far as I understand it, this would allow my to marry the 500 and 2TB arrays as one. Thing is, I'm having a hard time weeding through all the literature (some of which is possibly dated) and I'm not exactly sure of LVM is available with the Linux Mint 9 kernel. I didn't mind going in blind with experimenting with moving form dmraid to mdadm. After all, I was able to leave my original /home directory on the old array while I experimented with the new array. Now though I'm a bit short on spare drives I can keep a copy my original data on. How dangerous would it be to grow my /home directory using LVM without a full backup? Would using LVM negate the 'portability' of using mdamd? Can I simply add LVM without a re-install? Are there any other considerations that should be taken into account? What other advice and/or resources would users here recommend to me before I proceed?

There are two other issues I have to consider as well. One is the issue of encryption. As I said originally, I didn't give much thought on how to handle encryption. I just used the option available when I created those user accounts that needed/wanted encryption. I now know that that encryption is provided by eCryptfs. However, I don't know much more than that ATM. In what ways might the eCryptfs encrypted data be affected by any attempts to increase the capacity of /home? Meaning, are there any best practices to be considered to minimize problems? I'm also curious what other here think of eCryptfs and whether I should add any other encryption methods in place of eCryptfs or in combination with it? Might there be any other suggestions about encryption methods or policies anyone here would recommend?

The other, which may or may not be possible given my stated goals and limitations, is choice of file system. As I said before, I went with the default ext4 during installation because I figured it was the safest option. Now I'd like to at least entertain using something that might be better suited to my needs. My concern is that in an earlier stint with Linux I did experiment with reiserfs and got burned. I lost some irreplaceable data (my fault for not having a backup). I don't want to repeat that experience again. This would probably mean I would have to make an off-line backup of /home while I transitioned to a different file system. So the benefits would have to be significant. So is ext4 good enough, or should I really consider something else? Particularly if I'm using RAID 1 arrays?


BTW, I could use some advice on backups. I know relying on a RAID array is no guarantee agaisnt catastrophic events. So at some point in the near future I intend to backup at least /home and /etc. But after splurging on the two 2TB hard drive, I'm not in a position to afford another drive to act as an external off-line backup. So that leaves me with optical media... In my case record-able or re-writable DVD. Which has has lengthy list of drawbacks, most of which is the raw capacity of hard disk storage. Sure there will be a lot of files that remain pretty much static that could be copied/moved to an off-line medium. I'm not exactly sure how much dynamic data resides in a users /home directory and what is absolutely necessary to backup and what can be safely disregarded (i.e. .bash_history). So, are there any tools and methods users here would recommend for performing backups of /etc & /home to optical media?
User avatar
thetank
Level 1
Level 1
Posts: 29
Joined: Sun Nov 09, 2008 1:16 pm
Location: Cornwall, Ontario
Contact:

Re: Expanding /home (mdadm RAID 1)

Post by thetank »

Ubuntu to Richard Stallman
5hades0f6rey
Level 1
Level 1
Posts: 26
Joined: Mon Jun 14, 2010 10:35 pm

Re: Expanding /home (mdadm RAID 1)

Post by 5hades0f6rey »

Perhaps you misunderstood what I wrote. I was speaking of data backup as akin to tape backup, not the ripping of an audio CD or video DVD™. In those cases where I discuss backing up music and video files I never alluded to those being obtained in an illegal manner. There is plenty such content available under permissive licenses such as the Creative Commons Licenses, among others; https://secure.wikimedia.org/wikipedia/ ... Free_music, https://secure.wikimedia.org/wikipedia/ ... en_content. Not to mention that in some jurisdictions it is perfectly legal for a consumer of CDs and DVDs to make personal archival copies. For you to jump to the conclusion that I'm discussing anything illegal is unfair and, in my opinion, a bit insulting.
User avatar
Lolo Uila
Level 5
Level 5
Posts: 576
Joined: Thu Mar 15, 2007 3:40 am
Location: Kapolei, Hawaii

Re: Expanding /home (mdadm RAID 1)

Post by Lolo Uila »

The simplest way to merge your 500GB RAID-1 array into your /home directory would be mount it somewhere, then create a symlink.

So let's say you want to use the 500GB array to store music. You mount it in, for example /mnt/DATA with fstab, create a folder on that (/mnt/DATA/music), then create a symlink in your /home directory.
  • ln -s /mnt/DATA/music /home/{your_username}/Music
So now anything you read/write in your Music folder will be on the 500GB array.

Note: you could just mount it directly to /home/{your_username}/Music, but it's messier that way (permissions, lost+found folder, etc.).

Anyway... that's the easy way.

Aloha, Tim
5hades0f6rey
Level 1
Level 1
Posts: 26
Joined: Mon Jun 14, 2010 10:35 pm

Re: Expanding /home (mdadm RAID 1)

Post by 5hades0f6rey »

Lolo Uila wrote:The simplest way to merge your 500GB RAID-1 array into your /home directory would be mount it somewhere, then create a symlink.

So let's say you want to use the 500GB array to store music. You mount it in, for example /mnt/DATA with fstab, create a folder on that (/mnt/DATA/music), then create a symlink in your /home directory.
  • ln -s /mnt/DATA/music /home/{your_username}/Music
So now anything you read/write in your Music folder will be on the 500GB array.

Note: you could just mount it directly to /home/{your_username}/Music, but it's messier that way (permissions, lost+found folder, etc.).

Anyway... that's the easy way.

Aloha, Tim
Having been (primarily) a Windows user for so long, I never really got the hang of links. I learned early on that whatever benefits there were to links under Windows: they were easily negated by the hassles they could cause. I know I shouldn't expect Linux to be Windows. That's one of the reasons I'm making a concerted effort to switch to Linux now. But I guess my aversion to links has become reflexive.

Anyway, I'll read up some more on links and give your suggestion a try. One question you might be able to answer before I do so is this. Might I have any issues with eCrypftfs depending on where I place the symbolic link(s) to the second array for those users with encrypted home directories? I'm still learning how eCryptfs works exactly. But I do know that regardless of whatever encryption methods used, if I execute things poorly now I could inadvertently lock away user data permanently.
5hades0f6rey
Level 1
Level 1
Posts: 26
Joined: Mon Jun 14, 2010 10:35 pm

eCryptfs

Post by 5hades0f6rey »

Lolo Uila wrote:The simplest way to merge your 500GB RAID-1 array into your /home directory would be mount it somewhere, then create a symlink.

So let's say you want to use the 500GB array to store music. You mount it in, for example /mnt/DATA with fstab, create a folder on that (/mnt/DATA/music), then create a symlink in your /home directory.
  • ln -s /mnt/DATA/music /home/{your_username}/Music
So now anything you read/write in your Music folder will be on the 500GB array.
Following this advice was simple enough, except for the fact that files at the new location aren't encrypted. So, I've been trying to figure out eCryptfs in order to encrypt those user folders that desire/require encryption and then auto mount those folders when those users login. I have gotten close following the instructions in /usr/share/doc/ecryptfs-utils/README.gz, but I've hit a snag. Under the "PAM MODULE" heading it states:

Code: Select all

Finally, add this to the appropriate /etc/pam* file (e.g.,
/etc/pam.d/login) after the call to pam_unix.so:

---
auth required pam_ecryptfs.so
---
Well, /etc/pam.d/login doesn't have a call to pam_unix.so. Instead it has a bunch of "@include" statements to files where either pam_unix.so or pam_ecryptfs.so, or both are called. I'm not familiar enough with PAM to know where this line should go or if it's necessary in this case. For example, I have no ideas what the difference between "auth required pam_ecryptfs.so" and "session optional pam_ecryptfs.so unwrap" really means. What I do know is that attempting to mount a test mount point, result in a "mount: No such file or directory" error. Which in turn looks like it's a result of the mount command not being able find the valid File Name Encryption Key in fstab:

From kernel.log

Code: Select all

Oct 27 09:51:19 Shared kernel: [213473.796823] Could not find key with description: [################]
Oct 27 09:51:19 Shared kernel: [213473.796834] process_request_key_err: No key
Oct 27 09:51:19 Shared kernel: [213473.796839] Could not find valid key in user session keyring for sig specified in mount option: [################]
Oct 27 09:51:19 Shared kernel: [213473.796846] One or more global auth toks could not properly register; rc = [-2]
Oct 27 09:51:19 Shared kernel: [213473.796851] Error parsing options; rc = [-2]
Manually adding a passphrase using ecryptfs-add-passphrase --fnek makes it possible to mount the new location. But, while I might not mind the manual approach, some of my other users certainly will. I really need this to be automatic upon login.

I also looked at the instructions in /usr/share/doc/ecryptfs-utils/ecryptfs-pam-doc.txt.gz, hoping it might help. Instead it further muddies the waters making reference to the non-existent /etc/pam.d/system-auth file. I'm beginning to wonder if encryption is really such a good idea...

I'd appreciate some help figuring out how to make this work. Better yet, suggestion on an easier method to add and auto mount eCryptfs 'volumes' (other than .Private) altogether would be nice. I had hoped that I could modify ecryptfs-mount-private, but since it relies on the binary mount.ecryptfs-private_private, that isn't an option.
Post Reply

Return to “Newbie Questions”