when run from cron, script fails with "broken pipe"

Forum rules
Before you post please read how to get help

when run from cron, script fails with "broken pipe"

Postby patrick.mooney on Sun Nov 20, 2011 5:22 pm

Hello,

I'm using a script to automate a backup of some of my data to a remote server. It compresses the selected folders, encrypts them with GPG, and ships them off with scp. When run from the command line, the script works fine ... when run as a cron job, though, the script fails, and the log file that it creates just ends with:

Code: Select all
tar: -: Cannot write: Broken pipe
tar: Error is not recoverable: exiting now


Here's the script:

Code: Select all
# !/bin/bash
tar c -PSvv --to-stdout --exclude=".thumbnails" --exclude-tag-under=".SkipMe" /home/patrick/Documents/school /home/patrick/Documents/writing 2> /home/patrick/Desktop/remote-backup.log | bzip2 -z 2>> /home/patrick/Desktop/remote-backup.log | gpg2 -r 505AB18E --batch -o "/tmp/Backup.tar.bz2.gpg" -e - > /home/patrick/Desktop/remote-backup.log && scp /tmp/Backup.tar.bz2.gpg patrickmooney@ustorage.ucsb.edu:backups 2>> /home/patrick/Desktop/remote-backup.log
if [ -d /media/Externa/backups ]; then
  cp /tmp/Backup.tar.bz2.gpg /media/Externa/backups/
fi
rm /tmp/Backup.tar.bz2.gpg
chown patrick:patrick /home/patrick/Desktop/remote-backup.log


The script is run as root (i.e., installed with "sudo crontab -e"). Here's the cron line:

Code: Select all
30 6 * * * /home/patrick/.scripts/backup-remote.sh


I've tried searching the Mint and Ubuntu forums, plus Google, but nothing useful has turned up. I've been tearing my hair out over this for a month now. Sure, I have a lot of hair, but still. Any help is appreciated.

Currently running Linux Mint Julia on an HP Pavilion dv6000, 2GB RAM, 2 GHz Intel Core Duo.
Current setup: Sony VAIO VGN-FZ250E, 2GB RAM, Intel Core Duo, 2.2GHz; Linux Mint Nadia
http://patrickbrianmooney.nfshost.com/~patrick/
patrick.mooney
Level 1
Level 1
 
Posts: 29
Joined: Sun Nov 20, 2011 5:07 pm
Location: Santa Barbara, California

Linux Mint is funded by ads and donations.
 

Re: when run from cron, script fails with "broken pipe"

Postby Pilosopong Tasyo on Mon Nov 21, 2011 1:32 am

patrick.mooney wrote:The script is run as root...

I can't seem to find the logic why the script has to be run as root. I don't see any sudo commands anywhere in that script. The script runs fine under the user account, so why not just set it under the user's cron table instead of root's? The script will run at the designated time anyway even if the user account is not logged in.
Image
o Give a man a fish and he will eat for a day. Teach him how to fish and he will eat for a lifetime!
o If an issue has been fixed, please edit your first post and add the word [SOLVED].
User avatar
Pilosopong Tasyo
Level 6
Level 6
 
Posts: 1190
Joined: Mon Jun 22, 2009 3:26 am
Location: Philippines

Re: when run from cron, script fails with "broken pipe"

Postby thaimann on Mon Nov 21, 2011 3:08 pm

As I said in a previous post, cron provides very few global variables and this can cause your script to fail.

But if you run it like the following, it should get your user globals and should work;

58 * * * * /bin/su -c /home/user/myscript
thaimann
Level 1
Level 1
 
Posts: 32
Joined: Thu Aug 18, 2011 9:36 am

Re: when run from cron, script fails with "broken pipe"

Postby thaimann on Mon Nov 21, 2011 3:10 pm

Sorry I said that wrong. It will run with root privileges.
thaimann
Level 1
Level 1
 
Posts: 32
Joined: Thu Aug 18, 2011 9:36 am

Re: when run from cron, script fails with "broken pipe"

Postby xenopeek on Mon Nov 21, 2011 4:04 pm

Probably this is cron run as root has restrictive $PATH setting. Try setting at the top of your script:
Code: Select all
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
User avatar
xenopeek
Level 21
Level 21
 
Posts: 15261
Joined: Wed Jul 06, 2011 3:58 am
Location: The Netherlands

Re: when run from cron, script fails with "broken pipe"

Postby Pilosopong Tasyo on Mon Nov 21, 2011 10:00 pm

thaimann wrote:cron provides very few global variables...

I looked at the code fragment again and I don't see any "$VARIABLEs" being used anywhere. Unless if you're referring to environment variables.

Anyway, I suggest the OP break up this portion of the code:

Code: Select all
tar c -PSvv --to-stdout --exclude=".thumbnails" --exclude-tag-under=".SkipMe" /home/patrick/Documents/school /home/patrick/Documents/writing 2> /home/patrick/Desktop/remote-backup.log | bzip2 -z 2>> /home/patrick/Desktop/remote-backup.log | gpg2 -r 505AB18E --batch -o "/tmp/Backup.tar.bz2.gpg" -e - > /home/patrick/Desktop/remote-backup.log && scp /tmp/Backup.tar.bz2.gpg patrickmooney@ustorage.ucsb.edu:backups 2>> /home/patrick/Desktop/remote-backup.log

into individual commands instead of putting everything as a single line of code, relying solely on piping "|" and redirection ">" as input to the next command.

You'd be amazed how easier it is to debug [programming] problems if you simplify code.
Image
o Give a man a fish and he will eat for a day. Teach him how to fish and he will eat for a lifetime!
o If an issue has been fixed, please edit your first post and add the word [SOLVED].
User avatar
Pilosopong Tasyo
Level 6
Level 6
 
Posts: 1190
Joined: Mon Jun 22, 2009 3:26 am
Location: Philippines

Re: when run from cron, script fails with "broken pipe"

Postby Habitual on Mon Nov 21, 2011 10:37 pm

add this to your code and see what happens to the log file?
Code: Select all
# !/bin/bash
set -x
...
What's a landing but a take off in reverse?
User avatar
Habitual
Level 8
Level 8
 
Posts: 2246
Joined: Sun Nov 21, 2010 8:31 pm
Location: LM17Q-Xfce

Re: when run from cron, script fails with "broken pipe"

Postby patrick.mooney on Tue Nov 22, 2011 3:53 am

Thanks for all the replies! I've modified the cron entry to try thaimann's idea (run with su -c) and the script should start again in a few minutes. I'll report back then.
Current setup: Sony VAIO VGN-FZ250E, 2GB RAM, Intel Core Duo, 2.2GHz; Linux Mint Nadia
http://patrickbrianmooney.nfshost.com/~patrick/
patrick.mooney
Level 1
Level 1
 
Posts: 29
Joined: Sun Nov 20, 2011 5:07 pm
Location: Santa Barbara, California

Re: when run from cron, script fails with "broken pipe"

Postby patrick.mooney on Wed Nov 30, 2011 3:46 pm

Sorry for the delay getting back to you. Internet connectivity in the grad student dorms here seems to be highly intermittent over a holiday break.

None of the suggested fixes actually resolves the problem. Running the script with su -c in the cron job produces the same problem ... same with prefixing the cron command with sudo. Adding set -x to the beginning of the script either produces no change to the log, or adds a few hundred bytes of zeros to the beginning of it.

I could certainly break up the primary command into subcommands, but (a) this would require several hundred MB of extra scratch space, plus requiring that commands be run sequentially instead of concurrently, and (b) what is this, DOS? everyone keeps telling me that one of the advantages of the Linux command line/scripting system is its flexibility. I'd like to take advantage of this flexibility when writing a script. Ideally, I'd just like to get the piping working.

any help is much appreciated.
Current setup: Sony VAIO VGN-FZ250E, 2GB RAM, Intel Core Duo, 2.2GHz; Linux Mint Nadia
http://patrickbrianmooney.nfshost.com/~patrick/
patrick.mooney
Level 1
Level 1
 
Posts: 29
Joined: Sun Nov 20, 2011 5:07 pm
Location: Santa Barbara, California

Re: when run from cron, script fails with "broken pipe"

Postby patrick.mooney on Sun Dec 11, 2011 3:50 pm

... and I notice now that executing the script from a user crontab, instead of the root crontab, seems to fix the problem. I'm scratching my head over why this is, but it seems to be working. Still, if anyone can explain why it would fail from the root user's account, I'd be interested in hearing. (=
Current setup: Sony VAIO VGN-FZ250E, 2GB RAM, Intel Core Duo, 2.2GHz; Linux Mint Nadia
http://patrickbrianmooney.nfshost.com/~patrick/
patrick.mooney
Level 1
Level 1
 
Posts: 29
Joined: Sun Nov 20, 2011 5:07 pm
Location: Santa Barbara, California

Re: when run from cron, script fails with "broken pipe"

Postby riffraff on Fri Dec 30, 2011 4:05 am

Hi there,

you should definetly change your first line to a shebang!

change this
Code: Select all
# !/bin/bash


to that

Code: Select all
#!/bin/bash


This is a big difference - just run

Code: Select all
file <scriptname>


on both variants and you will see that yours is a "ASCII-FILE" and, after correcting it, it will be a "Bourne-Again Shell Script".

Maybe that is your problem here at all ;-)
riffraff
Level 1
Level 1
 
Posts: 8
Joined: Fri Dec 30, 2011 3:52 am

Linux Mint is funded by ads and donations.
 

Return to Scripts & Bash

Who is online

Users browsing this forum: No registered users and 1 guest