Bash script executing commands like a babboon's ___ [SOLVED]

About writing shell scripts and making the most of your shell
Forum rules
Topics in this forum are automatically closed 6 months after creation.
Locked
User avatar
linx255
Level 5
Level 5
Posts: 668
Joined: Mon Mar 17, 2014 12:43 am

Bash script executing commands like a babboon's ___ [SOLVED]

Post by linx255 »

My code is supposed to detect if a directory exists and take action depending on whether or not it does.
Let's call them 'Action 1' and 'Action 2'. When action 2 is taken, I expect the script to subsequently pause there ( read command ) before proceeding with the rest of the script.
However, that is not what happens. Instead it continues with the next line which moves file1 to file2... While paused at the read command.

Code: Select all

if [ "$(ls -d /directory)" ]
  then
    echo "Action 1"
  else
    echo "Action 2"; read
fi
# rest of program...
mv file1 file2
So the file gets moved when it's not supposed to, when I intend for it to be done only after ENTER is pressed to resume the script ( in the case of Action 2 ). This has resulted in data loss. I've tried debugging this and I can't figure out why it continues down the script when it's supposed to stop at 'read' command. Clues?

Thanks

EDIT: Just want to add, when I extract these commands out of my script ( which has other stuff in it ) and run in the terminal or in a script of its own it works. And I'm not certain it fails like this every time. And by the way, these commands are nested in a function, not that it should make any difference.
Last edited by LockBot on Wed Dec 28, 2022 7:16 am, edited 3 times in total.
Reason: Topic automatically closed 6 months after creation. New replies are no longer allowed.
- I'm running Mint 18 Mate 64-bit
- 4.15.0-34-generic x86_64
- All my bash scripts begin with #!/bin/bash
User avatar
smurphos
Level 18
Level 18
Posts: 8501
Joined: Fri Sep 05, 2014 12:18 am
Location: Irish Brit in Portugal
Contact:

Re: Bash script executing commands like a babboon's ___

Post by smurphos »

Don't know - I just tried this as a quick test and it works fine (pauses after Action 2 until enter is pressed).

Have you tried running the entire script through Shellcheck?

Code: Select all

#!/bin/bash
STUFF=true
if ! $STUFF
  then
    echo "Action 1"
  else
    echo "Action 2"; read
fi
# rest of program...
echo "next"
For custom Nemo actions, useful scripts for the Cinnamon desktop, and Cinnamox themes visit my Github pages.
User avatar
linx255
Level 5
Level 5
Posts: 668
Joined: Mon Mar 17, 2014 12:43 am

Re: Bash script executing commands like a babboon's ___

Post by linx255 »

I'm not sure how it would know what my intentions are but it passes shellcheck at any rate.

I'm not getting any error messages; it's just plain malfunctioning.

Will add debug functions if problem persists and post back...
- I'm running Mint 18 Mate 64-bit
- 4.15.0-34-generic x86_64
- All my bash scripts begin with #!/bin/bash
rene
Level 20
Level 20
Posts: 12240
Joined: Sun Mar 27, 2016 6:58 pm

Re: Bash script executing commands like a babboon's ___

Post by rene »

Only possibility would seem to be input being available on stdin at the time of the "read". That is, try the following and hit enter twice rather than once:

Code: Select all

sleep 5 && echo Waiting for enter && read
You'll note "read" to return immediately; the second enter that was input even before "read" was called to immediately satisfy it. You should therefore probably look before this specific bit, at a part of your script leaving input available on stdin.

Other, non-consequential issue: you may want [ -d /directory ] instead of [ "$(ls -d /directory)" ]; man bash, 1440g (or man test) for available test expressions.
User avatar
linx255
Level 5
Level 5
Posts: 668
Joined: Mon Mar 17, 2014 12:43 am

Re: Bash script executing commands like a babboon's ___

Post by linx255 »

Enter was not pressed prior to nor after read command, neither once nor twice. It's like it executed read and the following command together in parallel.

Could not reproduce or understand your scenario. For your code, one enter press to execute the commands results in:
1) sleep
2) echo
3) read
4) exit with enter ( first time since pressing enter to execute the code )

Executing the commands with one additional enter does the same, but as expected, the second enter press is passed to read and program exits.

With subsequent enters, nothing changes; it just delivers more prompts:
~ $
~ $

What's happening in my case is no keys are pressed, read command occurs and waits for input, while in the background it continues executing the following line.

I'll have to test more. It's not doing it every time because the script works most of the time. It could be a part of a bigger problem where something isn't right when my machine boots up and bash scripts don't work properly. I have noticed this from time to time. Probably 1-2x a year scripts don't work properly at all, and there is no rhyme or reason despite vigorous debugging efforts. Can't explain it! :x

And these aren't very complex scripts-- they do not depend on system conditions or external factors in the directory structure to be consistent. In fact, I'm only checking to see if this directory exists because for some unexplained reason it keeps disappearing from time to time. No pattern to it that I can tell.
- I'm running Mint 18 Mate 64-bit
- 4.15.0-34-generic x86_64
- All my bash scripts begin with #!/bin/bash
rene
Level 20
Level 20
Posts: 12240
Joined: Sun Mar 27, 2016 6:58 pm

Re: Bash script executing commands like a babboon's ___

Post by rene »

linx255 wrote: Thu May 10, 2018 2:57 pm What's happening in my case is [ ... ] read command occurs and waits for input [ ... ]
If the "While paused at the read command." bit of your original description above was already present when I typed the above reply I missed it. The point of the line I posted was showing that "read" will exit immediately if the enter was available on stdin even before "read" gets executed, i.e., during the "sleep 5". That "read" does not flush stdin and that as such a previous part of the script leaving an enter on stdin could be the culprit, would cause read to be "skipped".

In any case, if that suggestion was not valid or enough I rather expect we're going to need the actual script and method of calling it. Certainly scripts do not parallelise themselves; if by your use of the word "function" you refer to an external subscript then depending on method of launching it this could indeed create a parallel process causing the sort of race which you seem to be seeing but without the full script to look at it does not seem to make much sense.
User avatar
xenopeek
Level 25
Level 25
Posts: 29507
Joined: Wed Jul 06, 2011 3:58 am

Re: Bash script executing commands like a babboon's ___

Post by xenopeek »

First, you don't need to run ls in a subshell just to test if something is a directory. See help test for summary of file testing options you have. Instead of:
if [ "$(ls -d /directory)" ]
Use this to test the file exists and is a directory:
if [[ -d "/directory" ]]
See http://mywiki.wooledge.org/BashFAQ/031 as to why you should be using [[ instead of [.

As to the problem at hand, could it be you would have spuriously pressed Enter before the script reached the read command? Anything typed since the previous read command (or command invocation, whichever was most recent) would be stdin and read would process that. So if there is an Enter already in stdin before reaching the read command, it would immediately continue.

For example run sleep 10; read test; echo "read: $test" and after starting it immediately type something short and press Enter, then wait for the 10 second sleep to run out? The read command will not wait for input but immediately put what you typed in the test variable and continue.

To work around that you must clear stdin before the read command. There is not a standard command to clear stdin but you can use a command like this:
while read -r -t 0; do read -r; done
Probably most clear to put that in a function "clear_stdin" or something and call that where you need it. Like in your example:

Code: Select all

  else
    echo "Action 2"; clear_stdin; read
There are other ways to clear stdin but this one is reliable and works well.
Image
User avatar
linx255
Level 5
Level 5
Posts: 668
Joined: Mon Mar 17, 2014 12:43 am

Re: Bash script executing commands like a babboon's ___

Post by linx255 »

Thank you rene, I understand what you're saying now. The commands are just within a function {}, no calls to external scripts or anything. I haven't posted the full script because it's just a handful of simple unrelated commands, a call to the function, the function, and exit, and it works 99% of the time.

As far as stdin goes, my observation was that the read command did not complete ( was still waiting for input ) and the command after still executed ( apparently ). I created this function to inform me of the directory's absense so I'd have time to check the directory structure for myself before continuing the script. And that part of the script worked because the read function paused the script. After checking things out, I pressed ENTER and the directory to be moved was missing-- however, I just now realized, and I feel extremely :oops: about it, there are actually two malfunctions going on, and I'm sorry I didn't catch this earlier and confused the two problems after a long day with a deep-fried sleep-deprived brain..... Again, really sorry! :oops: And sorry this is so complex!

So the first malfunction, as already mentioned, is the directory ( a mounted /media device ) didn't exist when it was supposed to but now I see that it's not a huge surprise given that I frequently have errors and problems with USB devices and mounting. ( Need to post about that too! My machine really does not like USB devices, will occassionally fail to mount them or straight up freeze my machine even when ejecting them safely. )

The real problem is: a file that was also supposed to be present when that read command executed was missing. That file mysteriously went missing and I can't for the life of me trace how it disappeared at that point in the script. It's only supposed to be created and destroyed at specific times and conditions in the script, and 99% of the time it works like clockwork. ( The script opens the file for decryption on startup and would bark if it was absent. In this case, it opened up and decrypted fine, but when I "closed it out" for the encryption procedure the file was missing and the device wasn't mounted so the succeeding backup operation failed. )

The file is supposed to be moved first and then recreated by openssl encryption operation. The only way the file could disappear AFAIK is if the encryption operation failed and never completed for whatever reason, but the script is programmed to retry encryption infinitely until done. Normally, killing the script prematurely or a machine freeze requiring reboot could explain this, however, I would've known if etiher of these were the case because my script didn't die, I didn't have to retry encryption, and it had opened the file to begin with as it was supposed to.

With the above realization I'd need to spend some time analyzing the process flow of the script and study contigencies of various outcomes and failures to understand how I should better keep track of the file and ensure its existence and absense in the right circumstances. So for now it's a bit of a mystery that will require work on my part to set traps for debugging and I'm sorry for getting confused and wasting your time. Now I do know for a fact, at least on my machine, bash does rarely but surely malfunction and behave extremely bizarrely from time to time and that remains a possibility. It's definitely some kind of quirk; when it happens I have to reboot and then the script works fine. ( Even commands that don't reference files or variables fail to execute properly when this occurs. ) BUT now I know what I need to do to further debug and make sure it's not me.

The problem is I'm running out of time to mess with this, but next time this problem occurs I'll know how to start debugging and will post back if I hit a brick wall.

Thanks so much, everybody! And thanks xenopeek for the stdin insights; I was actually needing that anyway! 8)
- I'm running Mint 18 Mate 64-bit
- 4.15.0-34-generic x86_64
- All my bash scripts begin with #!/bin/bash
lmuserx4849

Re: Bash script executing commands like a babboon's ___

Post by lmuserx4849 »

- For debugging, at the top of your script use: set -x
I like to always use set -u

- Any place needed put

declare -p varname

Where varname is a variable name (no dollar sign)
It a quick way to see the value of a variable.

- Check out inotifywait.
May need to install package inotify-tools.

man page:
inotifywait efficiently waits for changes to files using Linux's inotify(7) interface. It is suitable for
waiting for changes to files from shell scripts. It can either exit once an event occurs, or continually
execute and output events as they occur.
Example: inotifywait -m -r ~/directory

- Consider a simple logging every time you execute a command in your script.

Code: Select all

  declare -- tmpFile=$(mktemp --tmpdir=${TMPDIR} "${0##*/}-$$-XXXXXXXX")
  ...
  echo 'command - some message' >"${tmpFile}"
- Definitely change[ "$(ls -d /directory)" ] to [[ -d directory ]]

- Maybe change your read to:

Code: Select all

  REPLY=
  while [[ "${REPLY:-N}" != [yY] ]]; do
    read -r -p 'Continue [y|N]: '
  done
User avatar
linx255
Level 5
Level 5
Posts: 668
Joined: Mon Mar 17, 2014 12:43 am

Re: Bash script executing commands like a babboon's ___

Post by linx255 »

Thanks lmuserx4849!

Looks like I have a bit of homework :shock:.
I'll shout back asap.
- I'm running Mint 18 Mate 64-bit
- 4.15.0-34-generic x86_64
- All my bash scripts begin with #!/bin/bash
lmuserx4849

Re: Bash script executing commands like a babboon's ___

Post by lmuserx4849 »

lmuserx4849 wrote: Fri May 11, 2018 4:05 pm - For debugging, at the top of your script use: set -x
I like to always use set -u to catch unused variables (maybe typos).

- Any place needed put

declare -p varname

Where varname is a variable name (no dollar sign)
It's a quick way to see the value of a variable.

- Check out inotifywait.
May need to install package inotify-tools.

man page:
inotifywait efficiently waits for changes to files using Linux's inotify(7) interface. It is suitable for
waiting for changes to files from shell scripts. It can either exit once an event occurs, or continually
execute and output events as they occur.
Example: inotifywait -m -r ~/directory

- Consider a simple logging every time you execute a command in your script.

Code: Select all

  declare -- tmpFile=$(mktemp --tmpdir=${TMPDIR} "${0##*/}-$$-XXXXXXXX")
  ...
  echo 'command - some message' >>"${tmpFile}"
- Definitely change[ "$(ls -d /directory)" ] to [[ -d directory ]]

- Maybe change your read to:

Code: Select all

  REPLY=
  while [[ "${REPLY:-N}" != [yY] ]]; do
    read -r -p 'Continue [y|N]: '
  done
User avatar
linx255
Level 5
Level 5
Posts: 668
Joined: Mon Mar 17, 2014 12:43 am

Re: Bash script executing commands like a babboon's ___

Post by linx255 »

spuriously
Kudos for snazzy word, xenopeek! You must read a lot or I don't read enough; I don't think I've ever heard this word used anywhere. Now I can go impress my friends with this word and look smart. :D
- I'm running Mint 18 Mate 64-bit
- 4.15.0-34-generic x86_64
- All my bash scripts begin with #!/bin/bash
User avatar
Termy
Level 12
Level 12
Posts: 4254
Joined: Mon Sep 04, 2017 8:49 pm
Location: UK
Contact:

Re: Bash script executing commands like a babboon's ___

Post by Termy »

xenopeek wrote: Thu May 10, 2018 5:40 pm First, you don't need to run ls in a subshell just to test if something is a directory. See help test for summary of file testing options you have. Instead of:
if [ "$(ls -d /directory)" ]
Use this to test the file exists and is a directory:
if [[ -d "/directory" ]]
See http://mywiki.wooledge.org/BashFAQ/031 as to why you should be using [[ instead of [.
There's no need to use [[ form here, from what I can see. Interesting link, by the way. The only advantage I can think of here, and something I somehow never knew, is that [[ with file testing doesn't need to have the filename protected with quotes, yet you've chosen to quote it anyway. I personally use the least necessary when it comes to tests, and a couple of quotes, per the norm, seems trivial to me. I imagine it uses more code to process [[ than it does [. Just how I approach it anyway. There's no "should" about it, is what I'm really getting at.
I'm also Terminalforlife on GitHub.
User avatar
linx255
Level 5
Level 5
Posts: 668
Joined: Mon Mar 17, 2014 12:43 am

Re: Bash script executing commands like a babboon's ___

Post by linx255 »

I'm definitely no true Bash master. My scripting is a last resort time torcher to save time in the long run, so I just grab and modify existing code that seems to work without learning the ins and outs of Bash or checking too much for proper form, until something doesn't work as intended. Unfortunately I don't have time to read all the amazing guides cover to cover, otherwise I totally would. Most of the time a web search is sufficient, and occasionally I pop in here with questions.
- I'm running Mint 18 Mate 64-bit
- 4.15.0-34-generic x86_64
- All my bash scripts begin with #!/bin/bash
User avatar
Termy
Level 12
Level 12
Posts: 4254
Joined: Mon Sep 04, 2017 8:49 pm
Location: UK
Contact:

Re: Bash script executing commands like a babboon's ___

Post by Termy »

linx255 wrote: Thu May 24, 2018 4:29 am I'm definitely no true Bash master. My scripting is a last resort time torcher to save time in the long run, so I just grab and modify existing code that seems to work without learning the ins and outs of Bash or checking too much for proper form, until something doesn't work as intended. Unfortunately I don't have time to read all the amazing guides cover to cover, otherwise I totally would. Most of the time a web search is sufficient, and occasionally I pop in here with questions.
Not a problem. As long as it works for you and you get something out of it. :)
I'm also Terminalforlife on GitHub.
Paul_Pedant

Re: Bash script executing commands like a babboon's ___

Post by Paul_Pedant »

Termy said: "I imagine it uses more code to process [[ than it does [."

Not so. It takes about half a million instructions to process [ and a couple of hundred to process [[.

[ is (usually) implemented by calling an external process called /usr/bin/[ which is a copy of /usr/bin/test. So that will:
.. fork a child process of the current shell
.. search all directories in PATH until it finds [
.. load that binary from disk into RAM, and fix up its arguments, environment, and stack.
.. queue it for scheduler to start when the current process finished its time slice
.. [ will fetch its arguments and do the test specified
.. [ process will exit
.. kernel will send a SIGCHLD to your shell (parent of [)
.. shell wlll get scheduled next time a processor is free
.. shell will issue a kernel call to fetch the process status
.. kernel will garbage-collect the [ process and clean up the process table
.. shell will (finally) use the process status as a boolean to decide which branch of the test to take.
That's a long way round to fetch 1 bit of data.

[[ is a built-in shell command. Shell will just do the same test that [ did itself, without running a whole complete process to do a trivial task.

As [[..]] is a built-in, it also gets all its options properly syntax-checked by the shell, and you get way better diagnostics if you get it wrong.

Pretty much the same with the "$( ls .. )". The $(..) runs a child process just to return a copy of its parameter in most cases, otherwise it puts a message you don't want out to stderr.

There is very much a "should" about using the [[..]] form for tests, and the ((..)) form for expression evaluation. Apart from slowing your script, it steals CPU, scheduler slots, memory, disc access, and cache from other processes.
gm10

Re: Bash script executing commands like a babboon's ___ [SOLVED]

Post by gm10 »

Paul_Pedant wrote: Thu Sep 26, 2019 11:17 am There is very much a "should" about using the [[..]] form for tests, and the ((..)) form for expression evaluation. Apart from slowing your script, it steals CPU, scheduler slots, memory, disc access, and cache from other processes.
[ is also a built-in, rendering your entire argument moot. In addition, double brackets are a bashism and bash is the slowest shell, making it even more incorrect.

Also best don't revive long dead threads.
copperhead wrote: Thu Sep 26, 2019 11:43 am Thanks for all the replies y'all. I forgot to mark this as solved; I think the [[ took care of it. The main topic name seems to be locked so I can't mark it solved.
This isn't even your thread, in fact it's your first post in this thread. :shock:
Locked

Return to “Scripts & Bash”