LVM: to stripe or not to stripe

[niowluka]

So far my record on this forum has been absolutely pathetic. I have not had a response to any of my problems, so I'm not really having any hopes on this one. I'll throw it in there anyway though, you never know...

Background

So, I have a PC with 3 drives:
1) 128GB Samsung SSD with a standard partition and JFS with Mint system
2) 2T WD Green HDD with 1 LV (entire drive, JFS) used for movies, music, pictures, etc (let's call it 'storage')
3) 2T WD Black HDD with 2 LVs: around 350G /home, used for downloads, temp files, etc, and the rest merged with 'storage' on disk 2). All using JFS. The 2 LVs are spread over 4 partitions and also use a legacy, recovered partition table. All due to historical reasons.

From what I gather (with LVM you don't 'know'). most data is on disk 2) and I'm slowly losing performance in terms of write speed from 'downloads' to 'storage' (as I'm reading from and writing to the same drive 3)).

Altogether I use just under 2T of data and I'm the only user. SSD is used for Mint system only at the moment, I might add /home and my hidden folders, but it's not at all important, it's basically out of equation, we focus on disk 2) and 3) only.

I'm about to borrow a 2T drive from a friend to tidy up my setup, and I'm considering 2 approaches:

Approach 1

• Remove all partions on both drives and create 1 massive VG with striping (probably 2 x 4K stripes to match the JFS file system)
  Split it into 2 LVs: 1 for downloads (and maybe /home) and 2nd for 'storage'

Approach 2

• Remove all partitions and create 1 VG and 1 LV on drive 2) first, with no striping.
  Copy as much data back as possible.
  Then expand VG and LV to include drive 3)
  Copy the rest of data.

Approach 1 will most likely produce a setup with best read speeds (i.e. I'll be readng from 2 drives simultaneously), while approach 2 will produce a setup with highest write speeds (I will always be writing from disk 2) to disk 3)).

I would like to hear from real live experiences, what actually makes the biggest difference and what makes the biggest impact in 'real life'. I've never had a problem with reads and I do notice write speeds (e.g. saving an 18GB movie). Will striping make a lot of difference in terms of read speeds ? Or should I focus on writes ?

[mintybits]

https://access.redhat.com/documentation ... lumes.html
I don't have experience of LVM striping but I do of raid0 and raid 10 and if they are similar I would expect both your read and write performance to be much improved. I'm not sure why you think approach 2 would do much for you.
Whenever using striping you pay a penalty in reliability so make sure you back this stuff up regularly. Or get a couple of more disks and use raid 10.

Oh I see now. In approach 2 you are planning to fill the slower disk up first so that newer writes and reads use the faster disk. I doubt the black is twice the speed of the green for large files but I haven't checked the specs. So I think striping would be faster but you should do a read speed test on each disk to compare. You can use the Disk utility for this or dd or whatever.

[niowluka]

The disks are very similar.

I'm worried that with striping and all my data on one VG spanned across just 2 disks, if I have to move data between LVs I will be doing both reading and writing to the same disks. In other words I wonder what is faster:

• copy from striped VG to the same VG (different LVs)
  copy from disk A to disk B (I can arrange LVs so that any new files will be on one drive, and almost all free storage will be on another)

Whenever using striping you pay a penalty in reliability

Just to be clear, you talk about the risks if a drive fails ? Or do you mean that striping increases risk of a drive failure ?

[eanfrid]

My own experience is not to stripe with LVM and not to stripe (RAID 0) at all for your data. Use mdadm for RAID 0 on your system disk if you do not have a SSD (but you have a SSD) and mdadm RAID 1, 5 or 10 for your data. Then you can put some LVM volumes on RAIDed volumes. LVM striping is not reliable and adds a layer of complexity where mdadm brings better performance and reliability.

When only one single disk/volume fails or is corrupted on a RAID 0, you will say goodbye to your data. Mathematically, failure risks with a N-disks RAID-0 is much higher than the sum of each one.

[niowluka]

Thanks eanfrid

That's very interesting. I would need to get another drive for raid5 and 10 though...

How does extending work on raid5 ? Can I add a disk or do I need 3 every time ?

As for raid0, my mobo supports it, I can set it up in bios ... I might go for that.

I only really care about losing music and photos, currently just under 400G. I could get a cheap 500G backup disk and keep it offline.

I don't have any backups atm (go on, release the hounds...)

[mintybits]

• copy from striped VG to the same VG (different LVs)
  copy from disk A to disk B (I can arrange LVs so that any new files will be on one drive, and almost all free storage will be on another)

Of course the fastest scheme to move files from one folder to another is to have both folders in the same filesystem: that's almost instant. I'm not sure what your usage model is...you are downloading files to one volume and then moving them to another?
If you are copying the files you probably want the source and destination filesystems on different disks for maximum speed. Using two disks striped may not be any faster because the speed is limited by the slowest disks seek time.

Whenever using striping you pay a penalty in reliability

Just to be clear, you talk about the risks if a drive fails ? Or do you mean that striping increases risk of a drive failure ?

As eanfrid says, striping (and LVM spanning two disks for that matter) is less reliable because if one disk fails the other is unusable. AFAIK if your LVM spans multiple disks and you lose a disk, the whole of the LVM volume becomes unusable. RAID 5 is a bit dodgy but RAID 1 & 10 are robust and RAID 6 is ok.

[mintybits]

As for raid0, my mobo supports it, I can set it up in bios ... I might go for that.

Probably less complicated in linux to not use mobo raid. It is really a Windows raid enabler (it isnt hardware raid...it is just some mobo features to facilitate Windows software raid). I think mdadm may support it now but I'm not sure. In any case it is really easy to make a linux raid array using the Disk utility.

Having a separate backup disk for critical data is a good precaution against accidentally erasing something. I use a WD Elements USB drive.

[niowluka]

I'm not sure what your usage model is...you are downloading files to one volume and then moving them to another?

It's an old habit from Shwindows times, to prevent data getting fragmented. I use a dedicated volume for downloading torrents, mp3s etc, and then move to separate storage (if they are worthy )

As eanfrid says, striping (and LVM spanning two disks for that matter) is less reliable because if one disk fails the other is unusable. AFAIK if your LVM spans multiple disks and you lose a disk, the whole of the LVM volume becomes unusable.

I know there are risks with LVM without striping. I'm not sure if it becomes completely unusable, but there is nothing stopping LVM from splitting a file across disks (it's just that without striping it's not written in parallel), so some files will definitely get lost.

I just want to be sure that using striping, be it raid 0 or lvm, doesn't actually increase a risk that a drive will fail. In 15 years I had one complete disk failure, but it was my own fault (the disk was failing for some time, but I did nothing about it) and I managed to have all data recovered ... Of course, with striping that will no longer be possible.

RAID 5 is a bit dodgy but RAID 1 & 10 are robust and RAID 6 is ok.

I would need more disk(s) for those... I just got the 2T WD a month or two ago.

[niowluka]

Probably less complicated in linux to not use mobo raid. It is really a Windows raid enabler (it isnt hardware raid...it is just some mobo features to facilitate Windows software raid).

Life-saver ! I thought it's some kind of true, native raid support. How naive of me... Just reading about 'fakeraid'.

[eanfrid]

Some good readings about RAID on Linux... https://raid.wiki.kernel.org/index.php/Linux_Raid : forget about "fakeraid"

FWIW, my servers use SSDs for system/boot/programs and mdadm RAID1 HDDs pairs for everything else (maybe I will try one day or another some RAID10 with 2 disks ) with LVM on top where it is useful.

[niowluka]

Thanks for all the advice. I had given it a lot of thought, and I think I'm going to 'invest' in another 2T WD Green (it's not a big difference between that and cheapest disks), and setup a raid 0 with the other Green for 'storage'. I will split the 2T WD Black into 2 regular partitions, one for downloads and other for backup of most important files.

Thoughts ?

I was seriously considering a raidz, but I'm worried the faster Black will cause problems with 2 Greens. And it's a bit much for my needs tbh...

[mintybits]

Why don't you put the download folder on your storage filesystem? Then moving files into storage proper would be instant.
How much space does the download folder need? I don't think fragmentation is a real issue with the linux ext filesystem...it's rather more sophisticated than NTFS.

[niowluka]

Why don't you put the download folder on your storage filesystem? Then moving files into storage proper would be instant.
How much space does the download folder need? I don't think fragmentation is a real issue with the linux ext filesystem...it's rather more sophisticated than NTFS.

I guess it's psychological, scarred for life by hours of defragging of my xp years ago

I like to keep it quite small, currently 350G. That forces me to tidy the files regularily. Otherwise it would just grow to one big dir called 'EVERYTHING'.

I understand that fragmentation on JFS is nowhere near as bad as NTFS or FAT, but it's not non-existent. On the other hand though, with raid 0 that it will be implicitly fragmented, so... Need to think about it

Thanks for all the suggestions.

[eanfrid]

and setup a raid 0 with the other Green for 'storage'.

O_o

[niowluka]

and setup a raid 0 with the other Green for 'storage'.

O_o

Thanks eanfrid for this insightful comment

I am about to setup my raid 0 array, and I have no idea what chunk size to use. I found conflicting advice on google. Some websites say that for big files the chunk should be small, to utilize the bandwidth better, while others say it should be proportional, bigger for big files, smaller for small files. So which one is it ?
Vast majority of my data will be comprised of HD movies, high quality mp3 (well, most of them anyway), HD images, very few documents, i.e. large files.

Intuition tells me it's the former, so I was thinking about matching the chunk with the drives' physical block size of 4k, which also matches block size of JFS which I will use.

[eanfrid]

I would choose 4K too. However, RAID 0 is the worst choice for data safety and reliability: your are trading the security of your data for probably a rather worthless speed increase. A secure choice is RAID 1 (mirror) or RAID 10 (mirrored stripes).... the keyword is "mirror"

[niowluka]

I would choose 4K too. However, RAID 0 is the worst choice for data safety and reliability: your are trading the security of your data for probably a rather worthless speed increase. A secure choice is RAID 1 (mirror) or RAID 10 (mirrored stripes).... the keyword is "mirror"

I get that, but for mirror I need to sacrifice 50% of disk space. I don't have the capacity, or need for that matter. I mean, am I going to be concerned if I lose my copy of Argo ? The answer is no. I'm not running a data centre here, so I'm happy to trade the risk. I backed up my important files on a separate drive, and for me that's good enough.

I've set it up last night with 4k chunks. Didn't really get a chance to test it much, but didn't seem to make a hell of a difference at first glance.