Automate MDADM checking? [SOLVED]

Questions about applications and software
Forum rules
Before you post please read how to get help
Post Reply
ralplpcr
Level 5
Level 5
Posts: 630
Joined: Tue Jul 28, 2015 10:11 am

Automate MDADM checking? [SOLVED]

Post by ralplpcr » Tue Sep 18, 2018 12:47 pm

I've been asked to build what basically amounts to a NAS for a friend. This friend is *not* Linux-literate at all, and really is only user-level under Windows. So whatever I build will need to be as "plug & play/ low maintenance" as possible - both for my own sanity as well as for ease of use.

Since the goal is to keep it as low-cost as possible, I've decided to use an old core2 duo motherboard & case with several old HDDs put in a raid via MDADM. I'll likely install XCFE, but it'll run 99% of the time without a desktop just in terminal mode. I'll create a SAMBA share out of the raid, and then it'll basically be left alone - - only connecting to my friend's local LAN as storage for their Windows machines.

My question is whether it's possible to have some type of automatic check of the MDADM raid to verify that the disks haven't failed? Since this won't be using a dedicated raid card with indicator lights for each disk, it'll be necessary to check them from time to time. I know how to check sudo mdadm --detail /dev/md0 , but I'm not sure if there's a simple way to
  • 1. Check the output
    2. If a disk has failed or the RAID is degraded, send an email.
I *could* do both tasks by installing PHP and setting a CRON job to run the command & direct output to a file, read/analyze the file, and then send email if the RAID isn't clean.... but that's a pretty clunky solution. Does anyone know if there is a simpler or more elegant way to do this?
Last edited by ralplpcr on Tue Sep 25, 2018 11:47 am, edited 1 time in total.

Laurent85
Level 15
Level 15
Posts: 5696
Joined: Tue May 26, 2015 10:11 am

Re: Automate MDADM checking?

Post by Laurent85 » Tue Sep 18, 2018 2:14 pm

See the configuration file /etc/mdadm/mdadm.conf. You can configure an email address to instruct the monitoring daemon where to send alerts.
Image

ralplpcr
Level 5
Level 5
Posts: 630
Joined: Tue Jul 28, 2015 10:11 am

Re: Automate MDADM checking?

Post by ralplpcr » Tue Sep 18, 2018 2:20 pm

Laurent85 wrote:
Tue Sep 18, 2018 2:14 pm
See the configuration file /etc/mdadm/mdadm.conf. You can configure an email address to instruct the monitoring daemon where to send alerts.
Thanks Laurent - I shall check that out. It would certainly be preferable to making my own!

ralplpcr
Level 5
Level 5
Posts: 630
Joined: Tue Jul 28, 2015 10:11 am

Re: Automate MDADM checking?

Post by ralplpcr » Tue Sep 18, 2018 5:02 pm

Well, it *seemed* like a good idea, but either I'm doing something terribly wrong, or my ISP has the port blocked somewhere.

What I've tried so far:
  • Editing /etc/mdadm/mdadm.conf to add my email address
  • Restarting the mdadm service
  • Installing sendmail
  • Setting SMTP passthrough on my router
  • Manually creating a passthrough for port 25 on my router
After each of these, I attempted to send a test email to myself. Each time, the command appeared to complete.... but the email never came. (And yes, I did check my SPAM folder as well!)

Code I used to send a test email:

Code: Select all

sudo mdadm --monitor --scan --test -1
sudo mdadm --monitor --scan --test --mail=<my email address> -1
Unless someone can point out what I'm doing wrong, I think my next step will be to try the --program parameter. I certainly can't install this at my friend's if I can't verify that it works first. If I can successfully get it to run a bash script, then I may be able to work around that?

ralplpcr
Level 5
Level 5
Posts: 630
Joined: Tue Jul 28, 2015 10:11 am

Re: Automate MDADM checking?

Post by ralplpcr » Wed Sep 19, 2018 5:34 pm

Well, I have a partial solution? I've at least figured out how to get an email to send from a bash script. So now if I can get that pesky --program parameter working from mdadm, I should be able to get it to send an email when there's a failure in the RAID.

In order to get the email to send through my ISP, I had to install & configure mtmtp:

Code: Select all

sudo apt-get install msmtp
vi ~/.msmtprc
Then in the blank msmtprc file, I added the following for my Gmail account:

Code: Select all

account gmail
tls on
tls_certcheck off
auth on
host smtp.gmail.com
port 587
user user1@gmail.com
from user1@gmail.com
password yourgmailPassw0rd
Do a chmod 600 ~/.msmtprc to set the permissions so that only the owner can see it.
Then create a plain txt file as follows:

Code: Select all

echo -e "From: myEmail@gmail.com \n\
To: myEmail@yahoo.com \n\
Subject: Hello World 
Test
This email was sent using MSMTP via Gmail" >>sample_email.txt
Finally, enter the following into the terminal: cat sample_email.txt | msmtp --debug -a gmail myEmail@yahoo.com

And just like that, I have a new email in my Yahoo account. :)

Just one more reason I love Linux....
Selection_003.jpg
EDIT: made a few minor corrections due to typos

lazarus
Level 3
Level 3
Posts: 143
Joined: Mon Jul 02, 2018 11:36 pm
Location: Oberon, OZ

Re: Automate MDADM checking?

Post by lazarus » Thu Sep 20, 2018 9:44 pm

Now that you've added your email address to /etc/mdadm/mdadm.conf I suggest you see if /etc/cron.daily/mdadm exists. That's part of the monitoring daemon and should already be installed/running. (I'm unsure whether it was created as part of setting up the initial mdadm array or just plain old default sw. No matter.)

In it's entirety, it contains:

Code: Select all

#!/bin/sh
#
# cron.daily/mdadm -- daily check that MD devices are functional
#
# Copyright © 2008 Paul Slootman <paul@debian.org>
# distributed under the terms of the Artistic Licence 2.0

# As recommended by the manpage, run
#      mdadm --monitor --scan --oneshot
# every day to ensure that any degraded MD devices don't go unnoticed.
# Email will go to the address specified in /etc/mdadm/mdadm.conf .
#
set -eu

MDADM=/sbin/mdadm
[ -x $MDADM ] || exit 0 # package may be removed but not purged

exec $MDADM --monitor --scan --oneshot
This is pretty much what you're trying to achieve, yes? It works for me, anyway.

How are you testing? Are you deliberately setting the array into a degraded state to generate/test the emails? 'Cos they're only sent on a fail/warn...
eg. a DegradedArray or FailSpare event. I don't get emails saying "Hey, everything is fine." ;)
- Andy

I may be weird but I'm saving up to become eccentric.

ralplpcr
Level 5
Level 5
Posts: 630
Joined: Tue Jul 28, 2015 10:11 am

Re: Automate MDADM checking?

Post by ralplpcr » Fri Sep 21, 2018 5:04 am

Hi Lazarus,

I haven't had time to mess with this for the past few days, and will be away this weekend.... so probably won't get back to it until next week. I'll definitely check the mdadm.conf to see whether or not that's set up, and give it a try from there.

I've been testing using the code sudo mdadm --monitor --scan --test --mail=<my email address> -1, since my understanding is that this should send a simulated failure/confirmation mail. I'd rather not degrade the array intentionally if I don't have to - - I'm testing on a machine that has good data on that array. If that ends up being the only way to test, then I'll have to actually assemble the new array first.

I have found that my Gmail solution above only works for a limited time, though. It seems Gmail detects it as a security risk, and will block those emails after about 10 are sent. Good for security, but not helpful in this situation. ;)

I'll let you all know how it progresses once I return next week & have a chance to try again. Thank you for the feedback!

User avatar
catweazel
Level 17
Level 17
Posts: 7497
Joined: Fri Oct 12, 2012 9:44 pm
Location: Australian Antarctic Territory

Re: Automate MDADM checking?

Post by catweazel » Fri Sep 21, 2018 5:27 am

ralplpcr wrote:
Fri Sep 21, 2018 5:04 am
I haven't had time to mess with this for the past few days, and will be away this weekend.... so probably won't get back to it until next week.
For $US50 you can buy an ASR 6805T 6G SAS/PCIe 2 hardware RAID card with SAS-SATA cables from fleabay. Just set and forget. With a single minute of work you can install maxView Storage Manager and arcconf then grab a copy of the Adaptec RAID Controller Command Line Utility User's Guide and create a simple script to monitor and report on the array.

For $US50, your friend is in a much better position and the data is far safer than using software or fake RAID, and you're far less likely to get frantic calls in the middle of the night.
¡uʍop ǝpısdn sı buıɥʇʎɹǝʌǝ os ɐıןɐɹʇsnɐ ɯoɹɟ ɯ,ı

ralplpcr
Level 5
Level 5
Posts: 630
Joined: Tue Jul 28, 2015 10:11 am

Re: Automate MDADM checking?

Post by ralplpcr » Fri Sep 21, 2018 8:03 am

catweazel wrote:
Fri Sep 21, 2018 5:27 am
For $US50 you can buy an ASR 6805T 6G SAS/PCIe 2 hardware RAID card ....
Actually, for under $20 I could get an AMCC 9650SE-8LPML SATA-II / SAS card that would do just as well. Believe me, I thought about it.... and if it were up to me, that's probably what I'd do. It's all in whether I want to donate $20 to the "cause".... since my friend seems to think that he shouldn't have to pay for it. :( Just a matter of how much effort I want to put forth in time vs. getting the card & being done with it all.

Either way, I'd still like to learn how to solve this issue. If nothing else, it'll increase my own knowledge of how to do stuff like this should a similar situation arise down the road. :)

ralplpcr
Level 5
Level 5
Posts: 630
Joined: Tue Jul 28, 2015 10:11 am

Re: Automate MDADM checking? [SOLVED]

Post by ralplpcr » Tue Sep 25, 2018 11:46 am

Sorry for the delay, but as I mentioned I was out of town for a few days, and hadn't had a chance to mess with it.

The good news is that I've got it sorted out now. :) It ended up being tricky, since the Gmail account wanted to block me after running several tests... but I found a different means of sending, and that seems to be OK. At least for now?

Here's what I ended up doing to make it work:

Start by installing ssmtp:

Code: Select all

sudo apt-get update
sudo apt-get install ssmtp
The edit the /etc/ssmtp/ssmtp.conf file to add the settings below - substituting your own info, of course:

Code: Select all

root=<email>@gmail.com
mailhub=smtp.gmail.com:465
FromLineOverride=YES
AuthUser=<email>@gmail.com
AuthPass=<password>
UseTLS=YES
Then, I created a simplistic bash file and set it to executable. The contents were simply

Code: Select all

echo "Testing...1...2...3" | ssmtp xxxxxx@gmail.com
Once all that was done, I typed in the following to simulate a failure:

Code: Select all

sudo mdadm --monitor /dev/md0 --test -1 --program=/home/ralplpcr/Desktop/mailtest.sh
To my surprise, not only did the bash file I created execute & send a test mail, but the built-in mdadm mail code *also* sent a test mail. :)
mails.jpg
I'll keep poking around, but I think overall this can be considered solved. If I can consistently get the mdadm monitoring to send email, then I'd rather just leave that one in place - - much cleaner than triggering it to run a bash script if I can avoid it!

Post Reply

Return to “Software & Applications”