scp does not work in crontab, works only on command line!

[magnetor1000]

I put the following in my crontab (via crontab -e ):

Code: Select all

#copy performance.log from root home in server to local home directory
* * * * * scp root@111.11.111.11:~/performance.log  ~

#copy ip.txt from root home in server to local home directory
* * * * * scp root@111.11.111.11:~/ip.txt  ~/

#copy hi.log from local home directory to root's home directory on server
* * * * * scp ~/hi.log root@111.11.111.11:~/

All of the above commands work PERFECT from the command line, no issues at all. But the moment I put them in crontab, nothing works. It's as if nothing is there.

And to be clear, I delete the relevant files on each receiving side prior to checking whether it works. Because if I delete the files, then within 1-2 minutes it should appear there again, no?

Why won't this work?

[Coggy]

I can think of two problems, though there may be more.

1) You don't give the full path to the scp executable. Crontab doesn't have the same environment as a logged in user. You should specify the full path in all scripts: /usr/bin/scp
2) Actually, this is also just an incomplete path. ~ doesn't expand properly in scripts. Specify /home/whoever instead.
3) I'm guessing you are using public key authentication? If not you will have to deal with providing a password as well.
4) It seems I can't count.

[magnetor1000]

I can think of two problems, though there may be more.

1) You don't give the full path to the scp executable. Crontab doesn't have the same environment as a logged in user. You should specify the full path in all scripts: /usr/bin/scp
2) Actually, this is also just an incomplete path. ~ doesn't expand properly in scripts. Specify /home/whoever instead.
3) I'm guessing you are using public key authentication? If not you will have to deal with providing a password as well.
4) It seems I can't count.

I took care of the full path issue, I outlined it out explicitly and fully:

Code: Select all

* * * * * scp root@111.11.111.11:/root/performance.log  /home/jay
* * * * * scp root@111.11.111.11:/root/ip.txt  /home/jay
* * * * * scp /home/jay/hi.log root@111.11.111.11:/root/

Still no dice.

Regarding number 3, yes there is that public key authentication with a password. I put in the password now which is just to unlock it on my end. Still doesn't work. Is that the whole issue?

If it is an issue, wouldn't that be a security risk in the future if I were to remove this requirement for a password? (I am just starting to learn how to use SSH, I am still a beginner)

[1000]

With regard " 1) "
Example how find full path of scp

Code: Select all

$ locate scp | grep bin
...
/usr/bin/scp

So , try /usr/bin/scp instead just scp

You can add at the end of command

Code: Select all

>> /var/log/mydebug.cron.log 2>&1

like this ( edited: IP is hidden for you - )
( I suggest replacing private IP with a name and not entering it on the forum )

Code: Select all

* * * * * /usr/bin/scp root@PASTE_IP_ADDRESS:/root/performance.log  /home/jay >> /var/log/mydebug.cron.log 2>&1

and check file /var/log/mydebug.cron.log

[Coggy]

Actually, I have only ever used scp with fully password-less configuration : my private key does not require a password to make an outgoing connection, annd the public key means the far end does not ask for a password. It may be worth trying without a password on the local private key, just to prove whether or not that is the problem, even if you decide to add a password requirement again later on.

It would also be worth doing as 1000 suggests, and piping stdout and stderr to a file that you can examine later to see what it complains about.

[magnetor1000]

With regard " 1) "
Example how find full path of scp

Code: Select all

$ locate scp | grep bin
...
/usr/bin/scp

So , try /usr/bin/scp instead just scp

You can add at the end of command

Code: Select all

>> /var/log/mydebug.cron.log 2>&1

like this ( edited: IP is hidden for you - )
( I suggest replacing private IP with a name and not entering it on the forum )

Code: Select all

* * * * * /usr/bin/scp root@PASTE_IP_ADDRESS:/root/performance.log  /home/jay >> /var/log/mydebug.cron.log 2>&1

and check file /var/log/mydebug.cron.log

Thank you for the advice on hiding the IP. Btw it is the server's IP address, not my own private IP. It's the one that I am renting out specifically just to learn about SSH, no private files are on there or anything. And the moment I am done I am deleting it. Is there still any security risk in exposing the IP (even if I were to keep it)?

Oh I realized that I did not put the full path of the command, you are right about that!

The way I searched for it is: whereis scp or type -a scp (the 2nd one gives two results, but the top one they both give):

Code: Select all

scp is /usr/bin/scp
scp is /bin/scp

So now I put in:

Code: Select all

* * * * * /usr/bin/scp root@111.11.111.11:/root/performance.log  /home/jay >> /var/log/mydebug.cron.log 2>&1
* * * * * /usr/bin/scp root@111.11.111.11:/root/ip.txt  /home/jay
* * * * * /usr/bin/scp /home/jay/hi.log root@111.11.111.11:/root/

I also tried /bin/scp when the first one did not work, but then I switched it back to the above after that did not work either.

I put the log file only for one of them, just so we can have a solid starting point and we will know which one it is.

However, when I do /var/log/mydebug.cron.log this is what I get:
cat: /var/log/mydebug.cron.log: No such file or directory

Just to be clear, I am doing this on my own private computer's side, not the server's (don't know if that makes a difference).

Actually, I have only ever used scp with fully password-less configuration : my private key does not require a password to make an outgoing connection, annd the public key means the far end does not ask for a password. It may be worth trying without a password on the local private key, just to prove whether or not that is the problem, even if you decide to add a password requirement again later on.

It would also be worth doing as 1000 suggests, and piping stdout and stderr to a file that you can examine later to see what it complains about.

I tried this as well. This is how I did that:

Code: Select all

cd /etc/ssh
nano +25 ssh_config
#PasswordAuthentication no

Saved and exited. Then restarted the ssh with sudo systemctl restart ssh as well as the crontab with service crond restart. Then bash to restart bash, closed the terminal, and even logged out and in of my system. (Yes, I did all that seriously)

Logged back in, and it required a password to connect!! But then I checked the box to "not require passwords when I am logged in".

This still doesn't work.

[Coggy]

Is the private key on the client (the machine with the crontab) password protected? Remember that there are two points where you may need to supply a password:
1: The server may ask for a password if you don't supply an acceptable key. With PasswordAuthentication set to no, it will simply drop the connection instead.
2: The private key file on the client might be password protected. I really don't know how to cope with that requirement in a cron job.

Who is the cron job running as? Did you use crontab -e to edit it? If so, the job is running as you and probably doesn't have permission to write to /var/log.]
If it's running as root, then it won't know to look at your private key file in /home/jay/.ssh, and probably wouldn't like the differing ownership if you gave it -i filename to point it to the right file.

On which machine did you do systemctl restart ssh? Confusingly, restart ssh actually restarts the sshd daemon, the ssh listening service. This is something you would do on the server, not the client. But /etc/ssh_config affects the ssh client only: /etc/sshd_config is for the daemon running on the server.

[magnetor1000]

So I semi-solved this (sort of). I opened up a new server with a private key, and the key was not password protected like my previous key was for the previous server (the one in question this entire post until this response here).

Interestingly enough, when editing the cron jobs (via crontab -e) the syntax required was different for the scp command itself, and is as follows:

Code: Select all

#COPYING FROM CLIENT TO SERVER
* * * * * scp /home/jay/hi.log root@111.111.111.111:/root/

#COPYING FROM SERVER TO CLIENT
* * * * * /usr/bin/scp root@111.111.111.111:/root/performance.log  /home/jay
* * * * * /usr/bin/scp root@111.111.111.111:/root/ip.txt  /home/jay

From the client to server: scp by itself sufficed. Cron job ran with no issues. It probably would run with the full path as well, but interesting to point out that it didn't require it like when the copying direction was switched.

From the server to the client: /usr/bin/scp was required for the files to get copied. Putting scp by itself did not run the cron job.

Why would there be a difference in requirements for the cron job to run depending on the direction of the copying? Both commands run in the same environment, or lack thereof, it's the same context. So what happened here?

Is the private key on the client (the machine with the crontab) password protected? Remember that there are two points where you may need to supply a password:
1: The server may ask for a password if you don't supply an acceptable key. With PasswordAuthentication set to no, it will simply drop the connection instead.

This certainly seems to have been part of the problem. Even after I lifted the password requirement, nothing worked at all. Except when running command line directly, but not as a cron job.

2: The private key file on the client might be password protected. I really don't know how to cope with that requirement in a cron job.

Yeah, me neither. I do not know almost anything about SSH (literally started learning about 3 days ago for the first time). But as a general question, perhaps you know, if the only way to do cron jobs is to not have a password on the private SSH key, wouldn't that be a potential security risk of some sort? For example, if someone gained unauthorized access the computer that the private key is on.

I am just trying to build good security habits from the get-go, but I want this all to work too! haha

Who is the cron job running as? Did you use crontab -e to edit it? If so, the job is running as you and probably doesn't have permission to write to /var/log.]
If it's running as root, then it won't know to look at your private key file in /home/jay/.ssh, and probably wouldn't like the differing ownership if you gave it -i filename to point it to the right file.

Yes, I used crontab -e on my local machine. What does this part of your answer mean as far as "...wouldn't like differing ownership.."? I did not understand that part or what the -i filename refers to or means.

On which machine did you do systemctl restart ssh? Confusingly, restart ssh actually restarts the sshd daemon, the ssh listening service. This is something you would do on the server, not the client. But /etc/ssh_config affects the ssh client only: /etc/sshd_config is for the daemon running on the server.

Very possible that I did the wrong command, this is literally my first attempt at working with SSH.

[Coggy]

For password protection of the private key, I can't think of a way to automate that. However convoluted you make it, the cron script will have to be able to get that password in order to use the key. If you put the password in an encrypted keystore, you have to give the script the password to the keystore, and so on. It's elephants all the way down. So an attacker would just have to replicate what the cron job does. I don't see any way round it, so you might as well just not password protect the key. Unless someone else knows better?

I don't see you using a different syntax for each direction. You seem to be copying a specific file to a specific folder, in both directions there. It maight look cleaner if you also gave the destination filename instead of just the destination folder (as you would have to do if you were renaming the file).

I don't understand why you think you need to specify /usr/bin/scp in one direction but not the other. It looks to me as though in both cases thge cron job is running on the client (the machine where user jay has his home directory), and connecting to the server 111.111.111.111 to either push hi.log or to pull performance.log. It's the same executable running under the same account (jay's account I presume) on the client.