Automate MDADM checking? [SOLVED]

[ralplpcr]

I've been asked to build what basically amounts to a NAS for a friend. This friend is *not* Linux-literate at all, and really is only user-level under Windows. So whatever I build will need to be as "plug & play/ low maintenance" as possible - both for my own sanity as well as for ease of use.

Since the goal is to keep it as low-cost as possible, I've decided to use an old core2 duo motherboard & case with several old HDDs put in a raid via MDADM. I'll likely install XCFE, but it'll run 99% of the time without a desktop just in terminal mode. I'll create a SAMBA share out of the raid, and then it'll basically be left alone - - only connecting to my friend's local LAN as storage for their Windows machines.

My question is whether it's possible to have some type of automatic check of the MDADM raid to verify that the disks haven't failed? Since this won't be using a dedicated raid card with indicator lights for each disk, it'll be necessary to check them from time to time. I know how to check sudo mdadm --detail /dev/md0 , but I'm not sure if there's a simple way to

• 1. Check the output
  2. If a disk has failed or the RAID is degraded, send an email.

I *could* do both tasks by installing PHP and setting a CRON job to run the command & direct output to a file, read/analyze the file, and then send email if the RAID isn't clean.... but that's a pretty clunky solution. Does anyone know if there is a simpler or more elegant way to do this?

[Laurent85]

See the configuration file /etc/mdadm/mdadm.conf. You can configure an email address to instruct the monitoring daemon where to send alerts.

[ralplpcr]

See the configuration file /etc/mdadm/mdadm.conf. You can configure an email address to instruct the monitoring daemon where to send alerts.

Thanks Laurent - I shall check that out. It would certainly be preferable to making my own!

[ralplpcr]

Well, it *seemed* like a good idea, but either I'm doing something terribly wrong, or my ISP has the port blocked somewhere.

What I've tried so far:

• Editing /etc/mdadm/mdadm.conf to add my email address
• Restarting the mdadm service
• Installing sendmail
• Setting SMTP passthrough on my router
• Manually creating a passthrough for port 25 on my router

After each of these, I attempted to send a test email to myself. Each time, the command appeared to complete.... but the email never came. (And yes, I did check my SPAM folder as well!)

Code I used to send a test email:

Code: Select all

sudo mdadm --monitor --scan --test -1
sudo mdadm --monitor --scan --test --mail=<my email address> -1

Unless someone can point out what I'm doing wrong, I think my next step will be to try the --program parameter. I certainly can't install this at my friend's if I can't verify that it works first. If I can successfully get it to run a bash script, then I may be able to work around that?

[ralplpcr]

Well, I have a partial solution? I've at least figured out how to get an email to send from a bash script. So now if I can get that pesky --program parameter working from mdadm, I should be able to get it to send an email when there's a failure in the RAID.

In order to get the email to send through my ISP, I had to install & configure mtmtp:

Code: Select all

sudo apt-get install msmtp
vi ~/.msmtprc

Then in the blank msmtprc file, I added the following for my Gmail account:

Code: Select all

account gmail
tls on
tls_certcheck off
auth on
host smtp.gmail.com
port 587
user user1@gmail.com
from user1@gmail.com
password yourgmailPassw0rd

Do a chmod 600 ~/.msmtprc to set the permissions so that only the owner can see it.
Then create a plain txt file as follows:

Code: Select all

echo -e "From: myEmail@gmail.com \n\
To: myEmail@yahoo.com \n\
Subject: Hello World 
Test
This email was sent using MSMTP via Gmail" >>sample_email.txt

Finally, enter the following into the terminal: cat sample_email.txt | msmtp --debug -a gmail myEmail@yahoo.com

And just like that, I have a new email in my Yahoo account.

Just one more reason I love Linux....

Selection_003.jpg

EDIT: made a few minor corrections due to typos

[lazarus]

Now that you've added your email address to /etc/mdadm/mdadm.conf I suggest you see if /etc/cron.daily/mdadm exists. That's part of the monitoring daemon and should already be installed/running. (I'm unsure whether it was created as part of setting up the initial mdadm array or just plain old default sw. No matter.)

In it's entirety, it contains:

Code: Select all

#!/bin/sh
#
# cron.daily/mdadm -- daily check that MD devices are functional
#
# Copyright © 2008 Paul Slootman <paul@debian.org>
# distributed under the terms of the Artistic Licence 2.0

# As recommended by the manpage, run
#      mdadm --monitor --scan --oneshot
# every day to ensure that any degraded MD devices don't go unnoticed.
# Email will go to the address specified in /etc/mdadm/mdadm.conf .
#
set -eu

MDADM=/sbin/mdadm
[ -x $MDADM ] || exit 0 # package may be removed but not purged

exec $MDADM --monitor --scan --oneshot

This is pretty much what you're trying to achieve, yes? It works for me, anyway.

How are you testing? Are you deliberately setting the array into a degraded state to generate/test the emails? 'Cos they're only sent on a fail/warn...
eg. a DegradedArray or FailSpare event. I don't get emails saying "Hey, everything is fine."

[ralplpcr]

Hi Lazarus,

I haven't had time to mess with this for the past few days, and will be away this weekend.... so probably won't get back to it until next week. I'll definitely check the mdadm.conf to see whether or not that's set up, and give it a try from there.

I've been testing using the code sudo mdadm --monitor --scan --test --mail=<my email address> -1, since my understanding is that this should send a simulated failure/confirmation mail. I'd rather not degrade the array intentionally if I don't have to - - I'm testing on a machine that has good data on that array. If that ends up being the only way to test, then I'll have to actually assemble the new array first.

I have found that my Gmail solution above only works for a limited time, though. It seems Gmail detects it as a security risk, and will block those emails after about 10 are sent. Good for security, but not helpful in this situation.

I'll let you all know how it progresses once I return next week & have a chance to try again. Thank you for the feedback!

[catweazel]

I haven't had time to mess with this for the past few days, and will be away this weekend.... so probably won't get back to it until next week.

For $US50 you can buy an ASR 6805T 6G SAS/PCIe 2 hardware RAID card with SAS-SATA cables from fleabay. Just set and forget. With a single minute of work you can install maxView Storage Manager and arcconf then grab a copy of the Adaptec RAID Controller Command Line Utility User's Guide and create a simple script to monitor and report on the array.

For $US50, your friend is in a much better position and the data is far safer than using software or fake RAID, and you're far less likely to get frantic calls in the middle of the night.

[ralplpcr]

For $US50 you can buy an ASR 6805T 6G SAS/PCIe 2 hardware RAID card ....

Actually, for under $20 I could get an AMCC 9650SE-8LPML SATA-II / SAS card that would do just as well. Believe me, I thought about it.... and if it were up to me, that's probably what I'd do. It's all in whether I want to donate $20 to the "cause".... since my friend seems to think that he shouldn't have to pay for it. Just a matter of how much effort I want to put forth in time vs. getting the card & being done with it all.

Either way, I'd still like to learn how to solve this issue. If nothing else, it'll increase my own knowledge of how to do stuff like this should a similar situation arise down the road.

[ralplpcr]

Sorry for the delay, but as I mentioned I was out of town for a few days, and hadn't had a chance to mess with it.

The good news is that I've got it sorted out now. It ended up being tricky, since the Gmail account wanted to block me after running several tests... but I found a different means of sending, and that seems to be OK. At least for now?

Here's what I ended up doing to make it work:

Start by installing ssmtp:

Code: Select all

sudo apt-get update
sudo apt-get install ssmtp

The edit the /etc/ssmtp/ssmtp.conf file to add the settings below - substituting your own info, of course:

Code: Select all

root=<email>@gmail.com
mailhub=smtp.gmail.com:465
FromLineOverride=YES
AuthUser=<email>@gmail.com
AuthPass=<password>
UseTLS=YES

Then, I created a simplistic bash file and set it to executable. The contents were simply

Code: Select all

echo "Testing...1...2...3" | ssmtp xxxxxx@gmail.com

Once all that was done, I typed in the following to simulate a failure:

Code: Select all

sudo mdadm --monitor /dev/md0 --test -1 --program=/home/ralplpcr/Desktop/mailtest.sh

To my surprise, not only did the bash file I created execute & send a test mail, but the built-in mdadm mail code *also* sent a test mail.

mails.jpg

I'll keep poking around, but I think overall this can be considered solved. If I can consistently get the mdadm monitoring to send email, then I'd rather just leave that one in place - - much cleaner than triggering it to run a bash script if I can avoid it!