Sharing folders between Virtual Machines

Questions about other topics - please check if your question fits better in another category before posting here
Forum rules
Before you post please read this

Sharing folders between Virtual Machines

Postby gordon.cooke on Sat Mar 30, 2013 11:21 pm

Im in process of putting together a new machine for home and looking to make use of virtualization (Xen or KVM) (See post at: Home Server Ideas )

Planning to have the following guests VMs:
  • Linux Mint - my primary working desktop
  • Win 7
  • Media Server - most likely Plex to stream music and movies
  • Backup Server - most likely BackupPC to backup other machines in the house
  • File Server (NAS) - provide Samba and other shares to other computers in the house

I have a 160GB hard drive that I am using to install the OS on, and a seperate 3TB hard drive that I am going to dedicate as a storage drive for all my actual files (documents, music, video, etc) Still deciding where the VMs themselves will ive for backup reasones etc.

Question 1:
I need/want to be able to access files from multiple VMs. I.e. I want to use Mint to rip or edit movies in a directory of Videos, and the Media server also needs access to the same folder in order to serve them out. Should I dedicate the storage drive to the NAS VM and then access these shared folders vis the NAS as if it was a true seperate NAS somewhere else on the network -OR- can I just mount the shared drive, or parts of it, into the multiple VMs as a filesystem via the hypervisor? ANy other ideas for proiding a shared directory between multiple VMs at the same time?

Question 2:
What file system should I use on the storage drive? I dont want anything too exotic because if the system every crashes hard I want to be able to easily pull the drive and mount it from a live CD or another linux box easily. Ext3 or Ext 4 would meet that requirement well, but not sure if something else would serve better for the acess by multiple VMs described above. Something with automatic data depulication would be wonderful.

Note: I have not decided between Xen and KVM yet, so open to ideas for either or feedback from any experiences to try and make the above happen.

Thanks for the help.
------------------------------------------------------------------------
Mint 13 64bit Cinnamon
Asus U56E Laptop, Core i5-2410M, Intel graphics, 6GB RAM
Mint 9 64bit
Averatec 2573 Laptop, AMD Turion 64x2, ATI Graphics, Atheros wireless, 3GB RAM
User avatar
gordon.cooke
Level 3
Level 3
 
Posts: 151
Joined: Tue Dec 01, 2009 9:48 pm
Location: New York

Linux Mint is funded by ads and donations.
 

Re: Sharing folders between Virtual Machines

Postby catweazel on Sun Mar 31, 2013 1:29 am

Regarding Q2...

I use Oracle's closed source VirtualBox VM for a similar purpose. I put all my VMs and the files I need access to on a hardware RAID box with 5 disks running RAID 10 (4 disks + 1 hot spare) hanging of an eSata port multiplier. It's more than fast enough to keep up with several VMs running at once - around 90-100MB/s. The filesystem is ext4 (though it was NTFS until a few days ago). Sharing the files is done by the VM manager transparently over TCP/IP from the host to the guests.

No special software is needed because the hardware RAID is managed by the kernel so if the system crashes the drive is accessible. Linux sees it as just another standard hard disk.
Mint Testing Team & Mint Donor #3606
KDE 4.12.0, custom preemptive kernel 3.12.5,
Intel i7 4770K @ 4.7GHz, 16GB 2666MHz XMP,
4 Samsung 840 PRO 512GB SSDs in RAID0,
6TB HW RAID10, dual 24" Acer X243H,
Gigabyte nVidia GTX 680 Super Overclock
User avatar
catweazel
Level 7
Level 7
 
Posts: 1656
Joined: Fri Oct 12, 2012 9:44 pm

Re: Sharing folders between Virtual Machines

Postby powerhouse on Tue Apr 02, 2013 10:35 am

I just posted a link to my Xen how-to on your other thread.

Q1: I use Xen and have all my data drives under dom0. They are shared with both guests (e.g. Windows) and other PCs using Samba. Alternatively, you could try the following:
a. Install a dedicated NAS VM.
b. Pass through the SATA controller to the NAS VM (PCI passthrough) and share the storage (Samba, whatever). This way the NAS will have direct access to the SATA controller, for performance reasons.

I would start using a Samba shared drive under dom0 or another VM and see how that works.

Q2: Use LVM volumes for all partitions except /boot. Then use ext4 for Linux dom0. For the guests under Xen, use RAW (that is, don't format to anything under dom0, and use the guest VM to format the drive). For a Windows guest, use NTFS, never use the Microsoft equivalent to LVM (can't remember how it's called). The RAW file systems can then be mounted under dom0 using kpartx - see http://forums.linuxmint.com/viewtopic.php?f=42&t=111783.

For backup, you got several options:

1. Use the LVM snapshot feature and back up to a dedicated backup drive using simple Linux commands - this is what I use. If you need backup scripts, let me know and I post them here. The advantage of snapshots is that you can backup a running system. I personally don't backup my Windows guest while it is running, only when the guest is turned off, but I do backup my dom0 Linux host this way.

2. Use RAID1 or RAID10 or whatever you prefer - of course always software RAID ! (Never use the RAID feature in your motherboard BIOS!)

3. Using LVM you can also mirror volumes (disks).

I don't know how familiar you are with LVM, but let's put it this way: It's a shame that Linux Mint doesn't format disks per default to LVM, like Fedora. There are so many advantages to it and once you converted to LVM, you don't want to look back. For installing Linux Mint on LVM, see here: http://forums.linuxmint.com/viewtopic.php?f=42&t=108442.


Re Xen versus KVM:

KVM has gotten a lot of publicity and many people would probably default to KVM. When I built my current PC I was looking for a virtualization solution that would make dual-boot obsolete. I read the benchmarks over at Phoronix and thought that KVM would beat Xen any time. Well, I now see it differently. The Phoronix benchmarks are, in my opinion, superficial at best (I would call them flawed).

Performance wise I believe Xen to be at least as good as KVM. But Xen has one advantage over KVM: PCI and VGA passthrough is available now for a long time and there are dozens of user reports and how-tos one can follow. KVM is new to that arena, and although the developers are working hard on it, there are still some glitches. I also find the Xen documentation to much better than the KVM documentation.

Given the right hardware, and the right configuration, both options should work. I chose Xen and my experience over the past 8-9 months has been very positive. I've started out with LM 13 Mate and Xen 4.1.1 and run now Linux Mint 14 Mate and Xen 4.1.3. The installation of LM14 on top of LM13 went smooth, the Xen hypervisor was updated using the Update utility. I find no need in installing Xen 4.2, and can wait until it hits the repository.

CPU resource allocation between guest and host works wonderful. For example, running RAW photo conversion in Lightroom (Windows guest) while ripping a DVD with Handbrake under Linux dom0 gives me 200 fps for Handbrake (which is 50% of the maximum I get without running heavy stuff on Windows), and no detectable slowdown in Windows. When the Windows guest is idle, the CPU resources go to dom0 (Linux). By the way, I assigned 10 VCPUs to the Windows guest and 2 VCPUs to Linux dom0, as I do the heavyweight photo editing/conversion stuff in Windows. Here some Passmark benchmarks for reference (all benchmarks are from the Windows 7 VM): http://forums.linuxmint.com/viewtopic.php?f=42&t=112013&start=40#p702361.
Asus Sabertooth X79, i7 3930K CPU, 8x4GB Kingston DDR3 RAM, Noctua NH-D14 CPU cooler, Sapphire 7770 GPU, PNY Quadro 2000 GPU, Asus Xonar Essence STX, Sandisk Extreme 120GB SSD + various HD, Corsair 500R case, SeaSonic 660W Gold X PS
powerhouse
Level 5
Level 5
 
Posts: 645
Joined: Thu May 03, 2012 3:54 am

Re: Sharing folders between Virtual Machines

Postby gordon.cooke on Fri Apr 05, 2013 6:47 am

Thanks for all the great info powerhouse. Ive been following your XEN VGA passthrough thread all week. It is a great resource. I'll post an update on the general virtualization in my other thread.

My plan has been to have a NAS running in domU as a file and print server. But when I installed Mint dom0 I noticed the printer was auto detected and auto configured so I am reconsidering if it's really worth the hassel. I know I can easily use NFS from my other linux machines, and Samba is on and running in the default install.

I am really liking the File Server over at Turnkey Linux: http://www.turnkeylinux.org/fileserver
The AjaXplorer looks very interesting. But I couldnt get it installed under Xen properly. Tried unpacking the Xen tar ball and tried to install inside a VM from the iso.

I think I will go with using the dom0. Probably not worth the hassel of passing through all different hardware. KISS
------------------------------------------------------------------------
Mint 13 64bit Cinnamon
Asus U56E Laptop, Core i5-2410M, Intel graphics, 6GB RAM
Mint 9 64bit
Averatec 2573 Laptop, AMD Turion 64x2, ATI Graphics, Atheros wireless, 3GB RAM
User avatar
gordon.cooke
Level 3
Level 3
 
Posts: 151
Joined: Tue Dec 01, 2009 9:48 pm
Location: New York

Re: Sharing folders between Virtual Machines

Postby powerhouse on Fri Jun 21, 2013 12:04 pm

This comes a little late, but here some corrections and additions to what I wrote before:

1. My Windows VM (domU) uses two LVM data volumes, in addition to the system volume (C: drive under Windows). These data volumes are NTFS formated. In fact, one of the two volumes is a striped LVM volume using 2 disks for better (RAID0-like) performance. So I don't know what happened to me when I posted above that I use Samba for all data drives.

2. I do have several shared volumes (Samba) under dom0 for various purposes, for example my multimedia library.

3. I have access to my Windows NTFS drives from dom0 using kpartx. Here is the script I use:

Code: Select all
#!/bin/sh

# Change the group name and volume name (see under /dev/mapper/...):
lv="group_name-volume_name"
lv2=${lv}2

if grep -q $lv /proc/mounts; then
#   zenity --info --timeout=10 --text="Unmounting volume $lv." &
   umount /mnt/$lv
   sleep 2
   kpartx -dv /dev/mapper/$lv
   rc=$?
   if [ $rc -ne 0 ] ; then
      zenity --error --text=" Could not unmount volume $lv !!!" &
      exit 1
   else
      zenity --info --timeout=10 --text="Successfully unmounted volume $lv." &
      exit 0
   fi
else
# Change "win7" below to the name of your domU:
   if xm list | grep -q win7; then
#      zenity --info --timeout=10 --text="Windows 7 is running. Mounting volume $lv in read-only mode." &
      kpartx -av /dev/mapper/$lv
      sleep 1
      mount -t ntfs -o ro /dev/mapper/$lv2 /mnt/$lv
      rc=$?
      if [ $rc -ne 0 ] ; then
         zenity --error --text=" Could not mount volume $lv !!!" &
         # clean up, just in case
         kpartx -dv /dev/mapper/$lv
         exit 1
      else
         zenity --info --timeout=10 --text="Successfully mounted volume $lv in read-only mode." &
         exit 0
      fi
   else
#      zenity --info --timeout=10 --text="Mounting volume $lv in read-write mode." &
      kpartx -av /dev/mapper/$lv
      sleep 1
      mount -t ntfs -o rw /dev/mapper/$lv2 /mnt/$lv
      rc=$?
      if [ $rc -ne 0 ] ; then
         zenity --error --text="Could not mount volume $lv !!!" &
         # clean up, just in case
         kpartx -dv /dev/mapper/$lv
         exit 1
      else
         zenity --info --timeout=10 --text="Successfully mounted volume $lv in read-write mode." &
         exit 0
      fi
   fi
fi


You need to create a directory under /mnt named group_name-volume_name (where group_name is the name of the volume group, and volume_name is the name of the lv volume, as it appears under /dev/mapper/). You also need to change "win7" to the domain name of your domU. You may also need to install the zenity package.

The script checks if your domU is running - if yes it will mount the specified volume read-only, if not it will mount the volume read-write. The script assumes that your volume is NTFS formated - you need to change the mount command if you use another format. It also assumes you are using the xm toolstack.

Hope this is helpful.
Asus Sabertooth X79, i7 3930K CPU, 8x4GB Kingston DDR3 RAM, Noctua NH-D14 CPU cooler, Sapphire 7770 GPU, PNY Quadro 2000 GPU, Asus Xonar Essence STX, Sandisk Extreme 120GB SSD + various HD, Corsair 500R case, SeaSonic 660W Gold X PS
powerhouse
Level 5
Level 5
 
Posts: 645
Joined: Thu May 03, 2012 3:54 am


Return to Other topics

Who is online

Users browsing this forum: No registered users and 3 guests