HOW TO Software RAID(mdadm) with easy sSMTP Gmail allerts

Write tutorials and howtos in here
There are more tutorials here http://community.linuxmint.com/tutorial/welcome
Forum rules
Do not start a support topic here please. Before you post please read this

HOW TO Software RAID(mdadm) with easy sSMTP Gmail allerts

Postby Treefrog on Sun Aug 23, 2009 7:06 pm

RAID: Redundant Array Of Inexpensive Disks.

There are many types of RAID but Raid 5 is the one I'm interested in here.
[*][*]EDIT This How To is a collection of data pertinent to all software RAID configurations. But I am interested in RAID 5 here.
It requires at least 3 disks. The result is only 2 disks worth of storage space but if any one disk fails the data is not lost.
The following is scraped together from various places but I thought it warranted putting in one place. Its not complete as its a big topic, but I found myself running all over the place just trying to understand how to set up RAID. So this is a future reference for me as much as anything and it may as well be a reference for the next guy/gal too ;-)
Its become obvious that there are a few layers of decisions when setting up RAID. This is just the path I took.

Any disks will do for experimenting but the final result will be only as fast as the slowest disk and as big as the smallest disk. If it comes to purchasing disks then don't forget to get a spare. After all, the point of most RAID implementations is that you can pop a replacement disk in when one fails, and they should be the same size. I guess at a pinch the spare could be bigger than the rest.

Description of the types of RAID : http://en.wikipedia.org/wiki/RAID
Interesting argument of Software V Hardware RAID: http://linux.yyz.us/why-software-raid.html

RAID tools for Linux:
LVM........http://www.redhat.com/magazine/009jul05/features/lvm2/
dmraid....http://www.linuxmanpages.com/man8/dmraid.8.php
mdadm...http://www.linuxmanpages.com/man8/mdadm.8.php

LVM is used to manage partitions on RAID arrays.

dmraid is used to discover and use the so called "fakeRAID" hardware.
FakeRAID is so called because although there are some settings outside of the OS (in the motherboards BIOS or on a controller card), the OS is used to do the work. Real hardware RAID is implemented on an embedded chip so the OS never knows the disks are RAID disks. Full hardware RAID is usually expensive too.
Software RAID on the other hand is not expensive :-), and if implemented correctly, is not difficult and gives many benefits.
Setting up a fakeRAID has been covered in the Mint forums here http://forums.linuxmint.com/viewtopic.php?f=42&t=20170&p=178926&hilit=raid#p178926

mdadm is used to set up and manage software RAID

[*][*][*]EDIT START[*][*][*][*]
[*][*] I have since got back at looking at RAID while looking at eBox from eBox Platform
eBox has some useful information about RAID and seems to be able to do the work for you. Making this How To a little redundant :-(
All the same have a look at the documentation as it shows a use for LVM in expanding a RAID more easily and how to deal with the /boot partition on software RAID. There are some clear diagrams and at the the moment they use XFS partitions as its supposed to be possible to expand them while the system is live. I would love to see that. !!
There is another documentation section but this is where I found the good bits:
http://trac.ebox-platform.com/wiki/Document/Index

This link has RAID info:
http://trac.ebox-platform.com/wiki/Document/Documentation/ThePerfectEboxSetup/StorageConfiguration

Some info about using XFS:
http://trac.ebox-platform.com/wiki/Document/Documentation/ThePerfectEboxSetup
http://trac.ebox-platform.com/wiki/Document/Documentation/ThePerfectEboxSetup/Install
Resizing LVM and XFS:
http://trac.ebox-platform.com/wiki/Document/Documentation/ThePerfectEboxSetup/StorageLiveVolumeResize

Also I have since come to my own conclusion that for a simple home or even small office set-up RAID 1 (mirroring) is best.
Reasons:
-Less trouble in recovery as the data is all on either drive. (if its a business then an off site backup solution needs to be implemented anyway)
-The OS has redundancy (its not possible to have the /boot on RAID 5 but you can put it on RAID1)
-Speed is not likely to be an issue for a small setup. Speed is needed for database access by multiple (thousands) connections.
-Cost. Why buy 3 or 4 identical disks when you can use 2 for RAID1. If you need more you just get 2 more and not necessarily the same size as the originals. The extra space available when using RAID 5 is really only realized when using 5 or more disks. At that point the "redundant" disk space is a negligible part of the total disk space and cost. But for small configurations 2 hard drives and 1/2 the space used is close "ish" to 3 drives and 2/3 the space used.
Thats my current thinking anyway.

All that said, its good to play with good tools and build RAID 5 to learn.
[*][*][*EDIT END][*][*][*][*]

--setting up software RAID
I got my basic info. here: http://linux-raid.osdl.org/index.php/RAID_setup but found it needed cleaning up with a change or two for my Mint.
Near the end of writing I found this. Its for Software RAID in Ubuntu: http://bfish.xaedalus.net/2006/11/software-raid-5-in-ubuntu-with-mdadm/comment-page-2/ Note: in the linked example ext3 partitions are created on the disks and the RAID is made out of the partitions. This How To uses full disks without partitions.

--Getting started
With software RAID 5 it doesn't seem possible to boot from it. The best I can think of for OS redundancy is a backup of the OS with Remastersys. It is possible to boot from another mirror RAID. Alternatively put grub and /boot on another drive and let them point at the rest of the OS on the RAID array.
Anyway this RAID will just be for general data and for now I'll use another disk for the OS.
Boot from a LIVE Mint CD to get going :) . Because GRUB is installed during the OS install, I want to make sure it is going on the right disk and not on one of my RAID disks. One way to do this is make sure the OS disk is on the first primary SATA or IDE cable. (Its all very possible to do fancy things with grub but I really don't want to start messing with GRUB at the moment)
After the LIVE CD boot GParted is good for checking the first drive is right. For me its /dev/sda. I can tell by the size of the disks etc what disks are what.

Then on to the Mint install with whatever preferences you like. Just remember to stay clear of the RAID disks.
Do some Mint updates and then open a command prompt and make sure the Kernel supports software RAID with this (Mint supports it ):
Code: Select all
sudo modprobe raid5

See the results in the file by doing:
Code: Select all
cat /proc/mdstat

It should tell you that you have the right RAID personalities (eg. RAID modes) registered, and that no RAID devices are currently active. (This script is useful for debugging.)

--Installing mdadm
The original text I read said to use apt-get to install mdadm:
Code: Select all
sudo apt-get install mdadm

However Synaptic Package Manager makes it easier when managing the install. I change the included email tool postifix for sSMTP. Postfix can do plenty including sending an email when a disk fails. Postfix might be over kill for this purpose while sSMTP is easy to setup with Gmail and well do all we need. There are many other tools though.
In Synaptic Package Manager right click on mdadm button and go to the "Mark Recommended for Installation". If you select sSMTP here you will get sSMTP instead of Postfix with the mdadm install.
We will get back to sSMTP later. Did I mention its easy ;-)

--On to the disks
There is a choice to be made about how the RAID is implemented on the disks. Either putting a partition on each disk and then using partitions to build the RAID array or as whole disks without partitioning first. Have not found any major argument against using the whole disk! So not much for me to do here.
If using LVM then perhaps partitioning first would make be a good idea.

--Creating the RAID array
Note: drive or partition names need to be corrected for your system. These were right for mine but they may not be right for yours.
Gparted gives a nice view of a disk, its name and if there are any partitions on it. There are other, more CLI "correct" ways of doing things though. (feel free to comment )
This is the original format I found for the mdadm create command:
Code: Select all
mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdc1 /dev/sdd1 --spare-devices=1 /dev/sde1

I dont have the spare device attached so --spare-devices=1 /dev/sde1 is omitted from my script. Also I'm using full disks and not partitions so I have to change the names eg. /dev/sbd1 to /dev/sdb.
My script is like this:
Code: Select all
sudo mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sdb /dev/sdc /dev/sdd

It should kick off straight away.
The status can be checked with this:
Code: Select all
watch cat /proc/mdstat

Press CTRL+C to cancel the watch.

There's a count down displayed with "watch cat " e.g. "finish=200.5min"
Apparently its possible to format it and start using it immediately !
Anyway it should be possible to carry on with the configuration while its working away. Its generally recommended that even when its finished you "burn it in" and don't depend on it straight away. Put data on it and use it but keep another copy for a week or so.

--config of mdadm and mount points
When the RAID array is ready it's necessary to make sure the mdadm.conf file knows about it.
In a terminal type:
Code: Select all
sudo mdadm --detail --scan

The result will look something like
ARRAY /dev/md0 level=raid5 num-devices=3 metadata=00.90 UUID=0a32fea9:b84eeae7:7b509728:5b74e2ab

This line needs to be in the mdadm.conf file. I found it necessary to edit the mdadm.conf file with gedit:
Code: Select all
sudo gedit /etc/mdadm/mdadm.conf

and copy/past it in. I put it under:
#definitions of existing MD arrays
ARRAY /dev/md0 level=raid5 num-devices=3 metadata=0.90 UUID=0a32fea9:b84eeae7:7b509728:5b74e2ab

NOTE: There is a problem with how data is parsed somewhere so the an extra "0" gets added. Change "metadata=00.90" to "metadata=0.90"
Your UUID is going to be different to mine lol

Make a mount point for the new raid drive:
Code: Select all
sudo mkdir /media/raidmd0

As you can see I have decided to create a folder in "/media" called "/raidmd0" You can put yours anywhere and call it anything but "/media" seems a good place.
Make sure fstab.conf knows where to mount the drive so:
Code: Select all
sudo gedit /etc/fstab


The entry in fstab will be something like this
/dev/md0 /media/raidmd0 auto defaults 0 0

That's what my one looks like and it works for now.

I was looking at using LVM2 to manage the RAID drive but have not seen the advantage of it for a simple setup like this yet. If anyone can comment on how it may be useful and suggest a way to set it up it would be great :-)

Some useful mdadm commands
[*][*][*][*][*][*][*][*][*][*][*][*][*][*][*][*][*][*][*]
[*]--Removing RAID
[*]If for some reason you need to get rid of the RAID set-up entirely then this is what is needed.

[*]unmount any partitions on the array that might be mounted.

[*]Find out what disks are involved in the RAID:
Code: Select all
sudo mdadm --detail /dev/md0

[*]unmount the RAID array:
Code: Select all
sudo umount /dev/md0

[*]stop the RAID:
Code: Select all
sudo mdadm --stop /dev/md0

[*]Now zero the superblock FOR EACH disk:
Code: Select all
sudo mdadm --zero-superblock /dev/sda

[*]--Starting a RAID array
[*]Its also useful to know how to start a RAID array and this is one way:
Code: Select all
mdadm --assemble --scan

[*][*][*][*][*][*][*][*][*][*][*][*][*][*][*][*][*][*][*]

-- Formatting the new RAID drive
I'm interested in using ext3 as a file system but mke2fs has many parameters to look at. The "-j" option makes it ext3.
This will get it up and running:
Code: Select all
sudo mke2fs -j /dev/md0


There is a performance tip. You dont have to use this but apparently it helps quite a bit.
The stride and strip-width in the RAID can be set using
tune2fs -E stride=n,stripe-width=m /dev/mdx

Code: Select all
sudo modprobe raid5

and:
Code: Select all
cat /proc/mdstat

will tell us the chunk size on the created RAID.

My chunk size is 64 and there is a block size of 4096B or 4 kB mentioned when mke2fs ran.
The values for the stride and strip-width are calculated like this:
There are 16 blocks of 4kB in one array chunk of size 64. Therefor 16 is one stride.
Stripe-width is calculated by multiplying the stride=16 value with the number of data disks in the array ( a raid5 with n disks has n-1 data disks, one being reserved for parity)
My setup with 3 disks has 2 data disks. 16 X 2 gives a Stripe-width=32

So:
Code: Select all
tune2fs -E stride=16,stripe-width=32 /dev/md0


--setting up sSMTP
Probably good to uninstall postifix first if its installed and your not going to use it.
sSMTP may already be installed from the install of mdadm, but if not then its available in Synaptic Package Manager or with :
Code: Select all
sudo apt-get install ssmtp


sSMTP is confgured in one text file /etc/ssmtp/ssmtp.conf
Code: Select all
sudo gedit /etc/ssmtp/ssmtp.conf


There is already some commenting in the file but if you want to know more try this:
Code: Select all
man ssmtp

I often dont find man files easy but there are examples on the web. e.g. http://tombuntu.com/index.php/2008/10/21/sending-email-from-your-system-with-ssmtp/
As with the linked example I'm using Gmail. However since this setup requires putting a gmail account password in the file, I created a nice new Gmail account just for my computer to use. It sends system email to me from the new account.
There is a way to secure the .conf file: http://blogs.techrepublic.com.com/security/?p=440 but I'm leaving that for another day.

In the ssmtp.conf file set:
root=myemailaddress@whatever.com
(I'm the administrator so I wont to get the system emails. This could alternatively be set as the new Gmail account )
mailhub=smtp.gmail.com:587
(this is the mailhub info for the Gmail account )

And add these:
AuthUser=Thegmailusername
(This is the user name of the new Gmail account I want sSMTP to use)

AuthPass=Thegmailpassword
(This is the password of the new Gmail account I want sSMTP to use)

UseSTARTTLS=YES
(This means you dont have to explicitly set dedicated port number for SMTP)

Once the changes are saved this can be tested in a terminal:
Code: Select all
sendmail AnyEmailAddress@Anyservice.com
   to: AnyEmailAddress@Anyservice.com
   from: Thegmailusername
   Subject: First Test of sSMTP

   Hello. This is a test sSMTP email

To send the email press Crtl+D

Note the blank line after "Subject:" This is needed and everything below it is message body.
The line with the sendmail command can be

Check the email accounts and see what's in the sent items and in-boxes :-)

As you can see this is simple at least at this level and quite flexible.

--Configure mdadm monitor mode
First open the mdadm.conf file:
Code: Select all
sudo gedit /etc/mdadm/mdadm.conf

find the line
# instruct the monitoring daemon where to send mail alerts.

and set the next line to:


Now test the relationship between mdadm and sSMTP:
Code: Select all
sudo mdadm --monitor --scan --test


If the --scan finds an mdadm RAID it will seem to just sit there and not return the command prompt to you.
After 30 sec or so press Ctrl+C and check the email accounts again.

Now set the auto monitor mode:
Code: Select all
sudo dpkg-reconfigure mdadm

answer a few fairly logical questions and its all set.

Result is Software RAID 5 with an email sent when bad things happen :wink:

Enjoy,
Treefrog
--
Last edited by Treefrog on Mon Nov 07, 2011 12:44 am, edited 5 times in total.
Treefrog
Level 1
Level 1
 
Posts: 12
Joined: Fri Jun 12, 2009 12:38 pm

Linux Mint is funded by ads and donations.
 

Re: HOW TO Software RAID(mdadm) with easy sSMTP Gmail allerts

Postby Husse on Wed Aug 26, 2009 5:51 pm

Thank you for this extensive guide
It is probably also partly useful for any RAID level
Image
Don't fix it if it ain't broken, don't break it if you can't fix it
Husse
Level 21
Level 21
 
Posts: 19710
Joined: Sun Feb 11, 2007 7:22 am
Location: Near Borås Sweden

Re: HOW TO Software RAID(mdadm) with easy sSMTP Gmail allerts

Postby MadOCer on Wed Jan 20, 2010 7:48 pm

Found this guide through the search function and have to say "BIG THX"!

I'm quite new to using Linux and struggled getting Software Raid to run with a following Linux Mint install. Just too many new things at once. I tried to find good info on the matter, but postponed the problem and just made a temporrary test-install using debian and it's Software Raid function included in the installer.

Now good to see that I'll probably get it to run with Linux Mint, too. I really like the look and feel of Linux Mint, so it would be a shame to use something else just because of the missing mdadm support.
MadOCer
Level 1
Level 1
 
Posts: 8
Joined: Wed Jan 20, 2010 7:41 pm

Re: HOW TO Software RAID(mdadm) with easy sSMTP Gmail allert

Postby geomax on Mon Mar 29, 2010 11:04 pm

Thanks for the guide. It's been very helpful. I could set up the raid array (0) using Palimpsest but couldn't figure out how to get the array to start at boot so that the mount did not fail. Thanks agin
geomax
Level 1
Level 1
 
Posts: 2
Joined: Mon Mar 29, 2010 3:08 am

Re: HOW TO Software RAID(mdadm) with easy sSMTP Gmail allert

Postby Treefrog on Wed Apr 07, 2010 9:51 pm

Perhaps a little ego "testicle" of me to say it but I'm glad my How To is still working away for people.
I have since found an answer to the use of partitions to make the RAID out of and what LVM is all about.

I got back into this when looking at eBox from eBox Platform. It has a RAID config tool and the documentation about RAID on the site is quite informative.
http://trac.ebox-platform.com/wiki/Document/Index

There is an EDIT near the top of the original post for all this

For the moment though my original RAID 5 set up with Mint is still working just fine.. (I must backup the OS some time) I like the use of an extra desktop on the server for turrents and such. eBox has not got such a space to mess with. Or not yet.. anything is possible. ;-)
Treefrog
Level 1
Level 1
 
Posts: 12
Joined: Fri Jun 12, 2009 12:38 pm

Re: HOW TO Software RAID(mdadm) with easy sSMTP Gmail allert

Postby mstng_67 on Mon Jun 07, 2010 9:27 pm

Treefrog,

Great How To! However, you might correct the following error:

Also I have since concluded that for a simple home or even small office set-up RAID 0 (mirroring) is best.


This is just the first mentioning of RAID 0. There are others in the same paragraph. I'm sure you meant RAID 1 (Mirroring). RAID 0 is not really RAID by definition as there is no redundancy. RAID 0 is striping without parity. Just trying to help alleviate some confusion for those not as familiar with RAID.

To confirm this you could refer to the following:
http://www.google.com/url?q=http://simp ... d=0CBMQygQ

Thanks again,

Micheal
Last edited by mstng_67 on Thu Nov 25, 2010 10:02 pm, edited 1 time in total.
mstng_67
Level 3
Level 3
 
Posts: 104
Joined: Sat Mar 21, 2009 1:28 pm

Re: HOW TO Software RAID(mdadm) with easy sSMTP Gmail allert

Postby ifreearticle on Wed Jul 14, 2010 10:40 pm

Thanks for sharing
ifreearticle
Level 1
Level 1
 
Posts: 1
Joined: Wed Jul 14, 2010 10:12 pm

HOW TO Software RAID(mdadm) with easy sSMTP Gmail allerts

Postby Rooper3546 on Wed Nov 17, 2010 7:40 am

MoBo: Intel D865GBF Desktop Board
CPU: P4 3.0Ghz
RAM: 4x1GB G.Skill Sticks
VIDEO: FX5200 AGP card
IDE1: Seagate 250GB (ST380013A)
IDE2: Maxtor 250GB (Diamond Max Plus9)
SATA1: Western Digital 320GB (WD3200JD-22KLB0)
SATA2: Seagate 200GB (ST300822AS) – ***UBUNTU INSTALLED ON THIS ONE***

So I will be using IDE1,IDE2, & SATA1 for my RAID5 Array.

After a quick shout to the Ubuntu Community I found this wonderfully simple guide on setting up a RAID5 array using mdadm: http://bfish.xaedalus.net/?p=188

I did however run into one problem with the guide:

sudo mdadm –detail –scan >> /etc/mdadm/mdadm.conf

Which adds the following to my mdadm.conf file:

ARRAY /dev/md0 level=raid5 num-devices=3 metadata=00.90 UUID=13f95aef:9c364189:75b10d3a:87a53e2f

but after I’ve added that to the mdadm.conf and when I run the following:

sudo mdadm –examine –scan
mdadm: metadata format 00.90 unknown, ignored.
ARRAY /dev/md0 level=raid5 num-devices=3 UUID=13f95aef:9c364189:75b10d3a:87a53e2f

Notice: mdadm: metadata format 00.90 unknown, ignored.
So I had to remove “metadata=00.90” from my mdadm.conf file and I no longer get that message.
Anyone know why the metadata entry isn’t working?
Rooper3546
Level 1
Level 1
 
Posts: 1
Joined: Tue Nov 16, 2010 8:00 am

Re: HOW TO Software RAID(mdadm) with easy sSMTP Gmail allert

Postby Treefrog on Sat Nov 27, 2010 5:47 pm

@Rooper3546
take a "0" out so it reads metadata=0.90
There is a Note in the How To pointing this out.
Treefrog
Level 1
Level 1
 
Posts: 12
Joined: Fri Jun 12, 2009 12:38 pm

Re: HOW TO Software RAID(mdadm) with easy sSMTP Gmail allert

Postby Treefrog on Sat Nov 27, 2010 6:18 pm

I thought it relevant to mention that upgrading the OS and keeping the RAID 5 array was not as hard as I thought.
This RAID 5 setup was originally on Mint 7. Its now on Mint 9.
All I did was:
Backed up the installed software with mint backup to a CD.
Took a manual copy of any files I could remember needing.
Installed the new OS on a new Hard Disk so I would have the old one for reference if need be.
When the new OS was installed I just added back the packages with the mint backup tool.
Currently I have dmraid, dmsetup, libdevmapper, libmraid, mdadm, I cant say I know which tool did the trick for me as it was a while ago, but the RAID array was automatically discovered. That put a smile on my face I can tell you. I really didn't expect an upgrade to be easy. :D
Treefrog
Level 1
Level 1
 
Posts: 12
Joined: Fri Jun 12, 2009 12:38 pm

Re: HOW TO Software RAID(mdadm) with easy sSMTP Gmail allert

Postby wentty on Wed Apr 13, 2011 5:04 am

I had used raid5 for la little long time.(a friend told me that may be ssfer for my data). i remember i just engaged a few steps to partition raid 5. if there's anyone want to try raid, u can follow the steps below. it's very easy. here attached detail informations. :D
http://www.partition-tool.com/resource/ ... raid-5.htm
wentty
Level 1
Level 1
 
Posts: 1
Joined: Wed Apr 13, 2011 4:54 am

Re: HOW TO Software RAID(mdadm) with easy sSMTP Gmail allert

Postby Underpants on Mon May 30, 2011 9:27 pm

This is a very good tutorial. There are many on the internet like this; but this one is simple, clean, and decisive.

Future topics should include:
* How to add a drive and expand the array without losing data
* Rebuilding the array in the case of a drive failure

:P :P
Underpants
Level 1
Level 1
 
Posts: 8
Joined: Mon May 30, 2011 9:10 pm

Re: HOW TO Software RAID(mdadm) with easy sSMTP Gmail allert

Postby Treefrog on Mon Nov 07, 2011 1:06 am

Thank you Underpants. I know I'm not giving my RAID install or thread the attention they deserves.
I learned a lot from implementing RAID at home and still draw on that experience. It took me weeks to gather all the information, equipment and assimilate it all into a working result. Its still working away though and servers out music with Subsonic :D
I'm waiting for ZFS to get implemented nicely in Debian. Then I'll be switching over to try that I think.
I have become interested in RAID 10 also as it has performance closer to RAID 0. Speed being important when looking at iSCSI. Time though is the biggest thing I just dont have.
Treefrog
Level 1
Level 1
 
Posts: 12
Joined: Fri Jun 12, 2009 12:38 pm

Re: HOW TO Software RAID(mdadm) with easy sSMTP Gmail allert

Postby DeMus on Wed Dec 11, 2013 5:26 pm

Hi,

Also I am looking for a way to use RAID and so I came across this How-to.
Although I believe it to be a very good one, I do have one simple question:
Why does it need to be so hard to install a Linux distro onto RAID? Why is it necessary to do all this, type all these commands, change files, etc?
Why isn't it possible to have an installer which does it all for you? During install, when choosing where to mount /, /home, swap, etc why can't there be a question if you want to use RAID?

I believe things like this are why Linux is still being considered an OS for geeks. To be really honest, I can't blame people who say so.

I love to use RAID but I am afraid to start with it simply because there are so many things which can go wrong.

Maybe during the upcoming holidays I will give it a try, to see how far I get. Will let you know.
No more Windows and Gates since Ubuntu 8.04, used 9.04, 10.04, then switched to Kubuntu because of Unity and Gnome 3.
After that it is Mint, used 9 till 17 all in KDE version and loving it.
DeMus
Level 4
Level 4
 
Posts: 248
Joined: Sat Aug 15, 2009 5:11 pm

Re: HOW TO Software RAID(mdadm) with easy sSMTP Gmail allert

Postby ro55mo on Mon Mar 17, 2014 3:30 pm

Thank you for this post. Armed with this knowledge I mirrored my two old spare 500GB 7,200 RPM drives in minutes.
ro55mo
Level 1
Level 1
 
Posts: 16
Joined: Tue Dec 03, 2013 5:32 pm

Linux Mint is funded by ads and donations.
 

Return to Tutorials / Howtos

Who is online

Users browsing this forum: No registered users and 15 guests