automatic adjustment of rsync priority with renice SOLVED

All Gurus once were Newbies
Forum rules
There are no such things as "stupid" questions. However if you think your question is a bit stupid, then this is the right place for you to post it. Please stick to easy to-the-point questions that you feel people can answer fast. For long and complicated questions prefer the other forums within the support section.
Before you post please read how to get help
Post Reply
royston
Level 2
Level 2
Posts: 58
Joined: Fri Sep 28, 2018 6:27 pm

automatic adjustment of rsync priority with renice SOLVED

Post by royston » Sun Jun 02, 2019 9:50 pm

I run into the problem with rsync that it takes such complete control of my workstation that I can't get it to do anything else. I literally have to either walk away from my computer and go do something else for a while or force a shutdown by holding down the power button. I don't dare for either of these.

I have written a shell script that reliably renices running rsync processes. I have tried to make this "automatic" with cron and avoid password entry with sudoers. I don't think this is working and don't know why. The script renice_rsync.sh is a little clumsy but it works. FYI, I am running bash.

Here are the files I think relevant:

Code: Select all

roy@sartoris:~$ cat /etc/issue
Linux Mint 19 Tara \n \l
roy@sartoris:~$ whoami
roy
roy@sartoris:~$ groups
roy adm cdrom sudo dip plugdev lpadmin sambashare
roy@sartoris:~$
===========================================================

Code: Select all

roy@sartoris:~/bin$ ls -l renice_rsync.sh 
-rwxrw--w- 1 roy roy 994 May 17 20:12 renice_rsync.sh
roy@sartoris:~/bin$ cat renice_rsync.sh 
#!/bin/bash
#
# renice_rsync.sh
#
# assignment of variables
a=1234
n=12
m=10
line1=LINE1
line2=LINE2
line3=LINE3
line4=LINE4
#
# record rsync pids in rsync_file
ps -u root|grep rsync > rsync_file
#
# count number of rsync processes presently running
m=`wc -l rsync_file`
# m may become the counter for a loop
# if m=0 then exit 1
#
# read pid records in rsync_file into separated variables
File=./rsync_file
{
read line1 
echo "$line1">line1file
read line2 
echo "$line2">line2file
read line3 
echo "$line3">line3file
read line4 
echo "$line4">line4file
} < $File
#
# isolate pids
word1=`cut -c 1-5 line1file` 
word2=`cut -c 1-5 line2file`
word3=`cut -c 1-5 line3file`
word4=`cut -c 1-5 line4file`
#
#rm rsync_file
# renice running rsync processes
sudo renice 19 $word1 &>trashfile
sudo renice 19 $word2 &>trashfile
sudo renice 19 $word3 &>trashfile
sudo renice 19 $word4 &>trashfile
#
# cleanup
rm rsync_file
rm line1file
rm line2file
rm line3file
rm line4file 
rm trashfile
exit 0  

======================================================================

Code: Select all

roy@sartoris:~$ crontab -e
# Edit this file to introduce tasks to be run by cron.
#
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
#
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').#
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
#
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
#
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
#
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h  dom mon dow   command
5,35 * * * * /home/roy/bin/renice_rsync.sh
============================================================================

Code: Select all

roy@sartoris:~$ sudo cat /etc/sudoers
[sudo] password for roy:                
#
# This file MUST be edited with the 'visudo' command as root.
# preferences adjusted to make vim(tiny) default editor
# Please consider adding local content in /etc/sudoers.d/ instead of
# directly modifying this file.
#
# See the man page for details on how to write a sudoers file.
#
Defaults	env_reset
Defaults	mail_badpass
Defaults	secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin"
Defaults        timestamp_timeout=60

# Host alias specification

# User alias specification

# Cmnd alias specification

# User privilege specification
root	ALL=(ALL:ALL) ALL
roy ALL=(ALL) NOPASSWD: /home/roy/bin/renice_rsync.sh
# Members of the admin group may gain root privileges
%admin ALL=(ALL) ALL

# Allow members of group sudo to execute any command
%sudo	ALL=(ALL:ALL) ALL

# See sudoers(5) for more information on "#include" directives:

#includedir /etc/sudoers.d
roy@sartoris:~$ 
==========================================================
I will be grateful for any suggestions!
Last edited by royston on Thu Jun 20, 2019 2:22 pm, edited 2 times in total.

AscLinux
Level 4
Level 4
Posts: 342
Joined: Sat Oct 29, 2016 3:32 pm

Re: automatic adjustment of rsync priority with renice

Post by AscLinux » Sun Jun 02, 2019 10:10 pm

First, if there is no environment set for your cronjob there will be none. Meaning command won't work, you must use /full/path_to/command in your script. Even basic commands like ps won't be found without full path.
Second, use --bwlimit option with rsync to limit transfer rate instead of nice.

royston
Level 2
Level 2
Posts: 58
Joined: Fri Sep 28, 2018 6:27 pm

Re: automatic adjustment of rsync priority with renice

Post by royston » Mon Jun 03, 2019 11:10 am

Thank you. I am looking into this. I'll try to figure this out...

User avatar
Flemur
Level 17
Level 17
Posts: 7314
Joined: Mon Aug 20, 2012 9:41 pm
Location: Potemkin Village

Re: automatic adjustment of rsync priority with renice

Post by Flemur » Mon Jun 03, 2019 11:34 am

Wouldn't it make more sense to start rsync with "nice"?

Code: Select all

nice rsync ...
Please edit your original post title to include [SOLVED] if/when it is solved!
Your data and OS are backed up....right?

royston
Level 2
Level 2
Posts: 58
Joined: Fri Sep 28, 2018 6:27 pm

Re: automatic adjustment of rsync priority with renice

Post by royston » Tue Jun 04, 2019 12:17 am

Thanks to all. I have altered my renice_rsync.sh to use full paths:

# renice running rsync processes
sudo /usr/bin/renice 19 $word1 &>trashfile
sudo /usr/bin/renice 19 $word2 &>trashfile
sudo /usr/bin/renice 19 $word3 &>trashfile
sudo /usr/bin/renice 19 $word4 &>trashfile

and syslog indicates that the script is run but it still doesn't renice to 19.
I think that the renice idea is the wrong one anyway. I want to learn how to use --bwlimit, write an /etc/rsync.conf file and make the execution of rsync a cron job.
Does this seem like the way to go?

royston
Level 2
Level 2
Posts: 58
Joined: Fri Sep 28, 2018 6:27 pm

Re: automatic adjustment of rsync priority with renice

Post by royston » Wed Jun 05, 2019 5:09 pm

Ok, I have been doing RTFM and am going to abandon the renice idea. I have been reading my existing /etc/init.d/rsync and I think this starts and controls my rsync daemon which runs along just fine except it uses too much bandwidth and I don't know where the backedup files are located. These are significant problems. Here is my /etc/init.d/rsync file which is as standard installed except I have added a few comments. For clarity these are: I want to set --bwlimit=600 as a starting point and see how that works. I also don't know where to put this --bwlimit. At present I have no /etc/rsyncd.conf file.

roy@sartoris:~$ cat /etc/init.d/rsync
#! /bin/sh

### BEGIN INIT INFO
# Provides: rsyncd
# Required-Start: $remote_fs $syslog
# Required-Stop: $remote_fs $syslog
# Should-Start: $named autofs
# Default-Start: 2 3 4 5
# Default-Stop:
# Short-Description: fast remote file copy program daemon
# Description: rsync is a program that allows files to be copied to and
# from remote machines in much the same way as rcp.
# This provides rsyncd daemon functionality.
#=====================================================================
# This is the existing file on my workstation. This line is added. Want to alter to limit bandwidth.
# --bwlimit=600 # 600 is a guess and where do I put this line?
#=====================================================================
### END INIT INFO

set -e

# /etc/init.d/rsync: start and stop the rsync daemon

DAEMON=/usr/bin/rsync
RSYNC_ENABLE=false
RSYNC_OPTS=''
RSYNC_DEFAULTS_FILE=/etc/default/rsync
RSYNC_CONFIG_FILE=/etc/rsyncd.conf
RSYNC_PID_FILE=/var/run/rsync.pid
RSYNC_NICE_PARM=''
RSYNC_IONICE_PARM=''

test -x $DAEMON || exit 0

. /lib/lsb/init-functions

if [ -s $RSYNC_DEFAULTS_FILE ]; then
. $RSYNC_DEFAULTS_FILE
case "x$RSYNC_ENABLE" in
xtrue|xfalse) ;;
xinetd) exit 0
;;
*) log_failure_msg "Value of RSYNC_ENABLE in $RSYNC_DEFAULTS_FILE must be either 'true' or 'false';"
log_failure_msg "not starting rsync daemon."
exit 1
;;
esac
case "x$RSYNC_NICE" in
x[0-9]|x1[0-9]) RSYNC_NICE_PARM="--nicelevel $RSYNC_NICE";;
x) ;;
*) log_warning_msg "Value of RSYNC_NICE in $RSYNC_DEFAULTS_FILE must be a value between 0 and 19 (inclusive);"
log_warning_msg "ignoring RSYNC_NICE now."
;;
esac
case "x$RSYNC_IONICE" in
x-c[123]*) RSYNC_IONICE_PARM="$RSYNC_IONICE";;
x) ;;
*) log_warning_msg "Value of RSYNC_IONICE in $RSYNC_DEFAULTS_FILE must be -c1, -c2 or -c3;"
log_warning_msg "ignoring RSYNC_IONICE now."
;;
esac
fi

export PATH="${PATH:+$PATH:}/usr/sbin:/sbin"

rsync_start() {
if [ ! -s "$RSYNC_CONFIG_FILE" ]; then
log_failure_msg "missing or empty config file $RSYNC_CONFIG_FILE"
log_end_msg 1
exit 0
fi
# See ionice(1)
if [ -n "$RSYNC_IONICE_PARM" ] && [ -x /usr/bin/ionice ] &&
/usr/bin/ionice "$RSYNC_IONICE_PARM" true 2>/dev/null; then
/usr/bin/ionice "$RSYNC_IONICE_PARM" -p$$ > /dev/null 2>&1
fi
if start-stop-daemon --start --quiet --background \
--pidfile $RSYNC_PID_FILE --make-pidfile \
$RSYNC_NICE_PARM --exec $DAEMON \
-- --no-detach --daemon --config "$RSYNC_CONFIG_FILE" $RSYNC_OPTS
then
rc=0
sleep 1
if ! kill -0 $(cat $RSYNC_PID_FILE) >/dev/null 2>&1; then
log_failure_msg "rsync daemon failed to start"
rc=1
fi
else
rc=1
fi
if [ $rc -eq 0 ]; then
log_end_msg 0
else
log_end_msg 1
rm -f $RSYNC_PID_FILE
fi
} # rsync_start


case "$1" in
start)
if "$RSYNC_ENABLE"; then
log_daemon_msg "Starting rsync daemon" "rsync"
if [ -s $RSYNC_PID_FILE ] && kill -0 $(cat $RSYNC_PID_FILE) >/dev/null 2>&1; then
log_progress_msg "apparently already running"
log_end_msg 0
exit 0
fi
rsync_start
else
if [ -s "$RSYNC_CONFIG_FILE" ]; then
[ "$VERBOSE" != no ] && log_warning_msg "rsync daemon not enabled in $RSYNC_DEFAULTS_FILE, not starting..."
fi
fi
;;
stop)
log_daemon_msg "Stopping rsync daemon" "rsync"
start-stop-daemon --stop --quiet --oknodo --pidfile $RSYNC_PID_FILE
log_end_msg $?
rm -f $RSYNC_PID_FILE
;;

reload|force-reload)
log_warning_msg "Reloading rsync daemon: not needed, as the daemon"
log_warning_msg "re-reads the config file whenever a client connects."
;;

restart)
set +e
if $RSYNC_ENABLE; then
log_daemon_msg "Restarting rsync daemon" "rsync"
if [ -s $RSYNC_PID_FILE ] && kill -0 $(cat $RSYNC_PID_FILE) >/dev/null 2>&1; then
start-stop-daemon --stop --quiet --oknodo --pidfile $RSYNC_PID_FILE || true
sleep 1
else
log_warning_msg "rsync daemon not running, attempting to start."
rm -f $RSYNC_PID_FILE
fi
rsync_start
else
if [ -s "$RSYNC_CONFIG_FILE" ]; then
[ "$VERBOSE" != no ] && log_warning_msg "rsync daemon not enabled in $RSYNC_DEFAULTS_FILE, not starting..."
fi
fi
;;

status)
status_of_proc -p $RSYNC_PID_FILE "$DAEMON" rsync
exit $? # notreached due to set -e
;;
*)
echo "Usage: /etc/init.d/rsync {start|stop|reload|force-reload|restart|status}"
exit 1
esac

exit 0
#==========================================================
Suggestions?

royston
Level 2
Level 2
Posts: 58
Joined: Fri Sep 28, 2018 6:27 pm

Re: automatic adjustment of rsync priority with renice

Post by royston » Wed Jun 05, 2019 5:24 pm

I think I found my backedup files: they are in /var/backups. Sorry, I'm slow...

royston
Level 2
Level 2
Posts: 58
Joined: Fri Sep 28, 2018 6:27 pm

Re: automatic adjustment of rsync priority with renice

Post by royston » Wed Jun 05, 2019 8:09 pm

I have created a very basic /etc/rsync.conf file:
roy@sartoris:/etc$ cat rsyncd.conf
# remarked out anything I did not understand
# "global-only" options
# syslog facility =local5

# global options which may also be defined
# in modules
#use chroot = yes
#uid = nobody
#gid = nobody
#max connections = 20
#timeout = 600
$read only = yes
bwlimit=600

& did this:
roy@sartoris:/etc$ /etc/init.d/rsync start
[ ok ] Starting rsync (via systemctl): rsync.service.
& got this:
Jun 5 16:56:51 sartoris systemd[1]: Started fast remote file copy program daemon.
Jun 5 16:56:51 sartoris rsyncd[10734]: Unknown Parameter encountered: "$read only"
Jun 5 16:56:51 sartoris rsyncd[10734]: IGNORING unknown parameter "$read only"
Jun 5 16:56:51 sartoris rsyncd[10734]: Unknown Parameter encountered: "bwlimit"
Jun 5 16:56:51 sartoris rsyncd[10734]: IGNORING unknown parameter "bwlimit"
so my rsyncd.conf is not working.
What did I do wrong? How can I fix it?

Jun 5 16:56:51 sartoris rsyncd[10734]: rsyncd version 3.1.2 starting, listening on port 873

royston
Level 2
Level 2
Posts: 58
Joined: Fri Sep 28, 2018 6:27 pm

Re: automatic adjustment of rsync priority with renice

Post by royston » Thu Jun 20, 2019 2:19 pm

Mark as resolved!

Post Reply

Return to “Newbie Questions”