(Solved)Need Help with a Script

About writing shell scripts and making the most of your shell
Forum rules
Topics in this forum are automatically closed 6 months after creation.
Locked
Pecu1iar
Level 1
Level 1
Posts: 39
Joined: Sun Nov 01, 2020 6:14 pm

(Solved)Need Help with a Script

Post by Pecu1iar »

I have a script that I'm having some trouble with. The script uses a file and copies it multiple times renaming it with a new file name from a list each time until it has been copied and renamed with every name in the list.

This is the script:

#!/bin/bash
cd /home/tony/working
while IFS= read -r name; do cp -f "MyFile.gz" /home/tony/active/"$name NewFile.gz"; done < active
while IFS= read -r name; do cp -f "MyOtherFile.gz" /home/tony/expired/"$name NewFile.gz"; done < expired

Attached is a copy of the two files used in this process.

This issue that I am having is that the script is not using the last name in the list from either file. This is happening with with both files.

I'm not sure why this is happening as I've used this same script on other files to perform the same thing.

I'm not sure if this has anything to do with the problem, but both files are generated from a google sheets spreadsheet.

Everyone has been great help in the past. I'm hoping that you all can come through for me again.

Thanks in advance
Attachments
active.txt
(143 Bytes) Downloaded 40 times
expired.txt
(43 Bytes) Downloaded 52 times
Last edited by LockBot on Wed Dec 28, 2022 7:16 am, edited 2 times in total.
Reason: Topic automatically closed 6 months after creation. New replies are no longer allowed.
1000
Level 6
Level 6
Posts: 1023
Joined: Wed Jul 29, 2020 2:14 am

Re: Need Help with a Script

Post by 1000 »

1, Better section viewforum.php?f=213

2. A little help.

Code: Select all

  while IFS= read -r NAME ; do echo "/home/tony/active/$NAME NewFile.gz"; done < text1.active
/home/tony/active/Trial 1 NewFile.gz
/home/tony/active/Tom Tucker 3 NewFile.gz
/home/tony/active/Mike Melon 5 NewFile.gz
/home/tony/active/Rick Ralston 6 NewFile.gz
/home/tony/active/Taylor Swift 8 NewFile.gz
/home/tony/active/Carl Sagan 9 NewFile.gz
/home/tony/active/George Carlin 10 NewFile.gz
/home/tony/active/Tim Tebow 11 NewFile.gz
/home/tony/active/Gene Simmons 12 NewFile.gz
/home/tony/active/John Sno 13 NewFile.gz

Code: Select all

$ while IFS= read -r NAME ; do echo "$NAME" ; done < text1
Trial 1
Tom Tucker 3
Mike Melon 5
Rick Ralston 6
Taylor Swift 8
Carl Sagan 9
George Carlin 10
Tim Tebow 11
Gene Simmons 12
John Sno 13
A) Do you see now how to diagnose the problem?
B) Do you see now that the variable needs to be filtered?
C) Can you try it alone?
You don't need to write a " while loop " on one line.
wwfwng
Level 3
Level 3
Posts: 158
Joined: Tue Jul 24, 2012 9:19 pm

Re: Need Help with a Script

Post by wwfwng »

I am a C Shell guy myself so I cannot offer great coding advice. But my first thought was that your script doesn't like or doesn't know what to do with the spaces. Is it possible to substitute "TomTucker" For "Tom Tucker" or use another character such as "-" instead of a space?
User avatar
Moem
Level 22
Level 22
Posts: 16226
Joined: Tue Nov 17, 2015 9:14 am
Location: The Netherlands
Contact:

Re: Need Help with a Script

Post by Moem »

Mod note:
Topic has been moved, carry on.
Image

If your issue is solved, kindly indicate that by editing the first post in the topic, and adding [SOLVED] to the title. Thanks!
Pecu1iar
Level 1
Level 1
Posts: 39
Joined: Sun Nov 01, 2020 6:14 pm

Re: Need Help with a Script

Post by Pecu1iar »

1000 wrote: Wed Jan 06, 2021 12:02 pm 1, Better section viewforum.php?f=213

2. A little help.

Code: Select all

  while IFS= read -r NAME ; do echo "/home/tony/active/$NAME NewFile.gz"; done < text1.active
/home/tony/active/Trial 1 NewFile.gz
/home/tony/active/Tom Tucker 3 NewFile.gz
/home/tony/active/Mike Melon 5 NewFile.gz
/home/tony/active/Rick Ralston 6 NewFile.gz
/home/tony/active/Taylor Swift 8 NewFile.gz
/home/tony/active/Carl Sagan 9 NewFile.gz
/home/tony/active/George Carlin 10 NewFile.gz
/home/tony/active/Tim Tebow 11 NewFile.gz
/home/tony/active/Gene Simmons 12 NewFile.gz
/home/tony/active/John Sno 13 NewFile.gz

Code: Select all

$ while IFS= read -r NAME ; do echo "$NAME" ; done < text1
Trial 1
Tom Tucker 3
Mike Melon 5
Rick Ralston 6
Taylor Swift 8
Carl Sagan 9
George Carlin 10
Tim Tebow 11
Gene Simmons 12
John Sno 13
A) Do you see now how to diagnose the problem?
B) Do you see now that the variable needs to be filtered?
C) Can you try it alone?
You don't need to write a " while loop " on one line.
I'm sorry, I don't see how to get this to work. I need actual copies of the source file with the new file names. I'm not sure how to incorporate the echo command in such a way that I can accomplish this.

If it shed's any light on my situation. I'm a complete noob at all of this.

Thanks
User avatar
deck_luck
Level 7
Level 7
Posts: 1577
Joined: Mon May 27, 2019 6:57 pm
Location: R-4808 North

Re: Need Help with a Script

Post by deck_luck »

Pecu1iar wrote: Wed Jan 06, 2021 5:36 am ...
This issue that I am having is that the script is not using the last name in the list from either file. This is happening with with both files.
...
Your source files are not line terminating the last text item.

Code: Select all

$ od -c expired 
0000000    T   o   m       C   l   a   n   c   y       2  \r  \n   F   r
0000020    e   d       S   a   v   a   g   e       4  \r  \n   T   o   m
0000040    m   y       F   r   a   n   k   s       7                    
0000053

$
In this case the last item is "To my Franks 7", however it does not have the "\r" carriage return and "\n" newline suffix characters. In your case the bash read command requires the item ending with a "\n" new line character. You can correct the situation by adding a new line to the end of the file as follows:

Code: Select all

cp expired expired.save
echo >> expired

Code: Select all

$ od -c expired
0000000    T   o   m       C   l   a   n   c   y       2  \r  \n   F   r
0000020    e   d       S   a   v   a   g   e       4  \r  \n   T   o   m
0000040    m   y       F   r   a   n   k   s       7  \n                
0000054
$
Notice the "\n" new line character after the last text item.
🐧Linux Mint 20.3 XFCE (UEFI - Secure Boot Enabled) dual boot with Windows 11

Give a friend a fish, and you feed them for a day. Teach a friend how to fish, and you feed them for a lifetime. ✝️
User avatar
AndyMH
Level 21
Level 21
Posts: 13583
Joined: Fri Mar 04, 2016 5:23 pm
Location: Wiltshire

Re: Need Help with a Script

Post by AndyMH »

A list of filenames in test.txt:

Code: Select all

one
two
three
four 
five
six
A script, test.sh:

Code: Select all

#!/bin/bash
testvar=$(cat ./test.txt)
for i in $testvar # or you could dispense with testvar and have for i in $(cat ./test.txt)
do
    echo $i
    # or in your case replace echo $i with
    # cp basefilename $i
done
Output:

Code: Select all

andy@T431 ~/test $ ./test.sh
one
two
three
four
five
six
#!/bin/bash is called a shebang and tells linux what interpreter to use, in this case bash. As long as you make the script executable you don't have to run it with bash test.sh.

testvar=$(cat ./test.txt) cat is a command that echoes a file's contents to the screen, in this case the output is assigned to a variable testvar.

./ means look in the current folder for test.txt.

for i in $testvar assigns each element in testvar to i and repeats the code between do and done until it runs out of elements in testvar.

# means a comment in bash.

Don't put spaces in the filenames (contents of test.txt), it complicates the script.
Thinkcentre M720Q - LM21.3 cinnamon, 4 x T430 - LM21.3 cinnamon, Homebrew desktop i5-8400+GTX1080 Cinnamon 19.0
Pecu1iar
Level 1
Level 1
Posts: 39
Joined: Sun Nov 01, 2020 6:14 pm

Re: Need Help with a Script

Post by Pecu1iar »

Thank You everyone for the suggestions. I'm still trying to figure this all out.

I use this same script to perform the same action with the same type of file. The only difference is that I manually maintain the "list" of names.

The two lists attached to this post are actually the product of a google sheets project. The list's are downloaded from google sheets which might explain the format problem.
Also, when I can get it to copy once for each name on the list, the resulting file names are on two line, which is also kind of annoying, so that it appears:

Person's Name 1
New File.gz

I would like it to appear: Person's Name 1 New File.gz

In any event, I thank you all for all your comments.

If anyone else has anything to offer, I welcome any advice and help.
User avatar
deck_luck
Level 7
Level 7
Posts: 1577
Joined: Mon May 27, 2019 6:57 pm
Location: R-4808 North

Re: Need Help with a Script

Post by deck_luck »

Code: Select all

$ od -c expired
0000000    T   o   m       C   l   a   n   c   y       2  \r  \n   F   r
0000020    e   d       S   a   v   a   g   e       4  \r  \n   T   o   m
0000040    m   y       F   r   a   n   k   s       7  \n                
0000054
$ 
Notice the "\r" represents a carriage return.

Your text file has both the "\r" carriage return and "\n" newline characters. This is common for DOS or Windows text files.

In Linux or UNIX the text file lines only use the new line delimiter. You need to remove the pesky "\r" carriage returns from the file, otherwise they will be part of your new file names.

Please use the following commands to create a new expired file without the carriage returns:

Code: Select all

cp expired expired.orig
tr -d '\r' <expired.orig >expired

Code: Select all

$ od -c expired
0000000    T   o   m       C   l   a   n   c   y       2  \n   F   r   e
0000020    d       S   a   v   a   g   e       4  \n   T   o   m   m   y
0000040        F   r   a   n   k   s       7  \n                        
0000052
$ 
Notice, the "\r" carriage returns are no longer present.
🐧Linux Mint 20.3 XFCE (UEFI - Secure Boot Enabled) dual boot with Windows 11

Give a friend a fish, and you feed them for a day. Teach a friend how to fish, and you feed them for a lifetime. ✝️
Pecu1iar
Level 1
Level 1
Posts: 39
Joined: Sun Nov 01, 2020 6:14 pm

Re: Need Help with a Script

Post by Pecu1iar »

deck_luck wrote: Wed Jan 06, 2021 11:57 pm

Code: Select all

$ od -c expired
0000000    T   o   m       C   l   a   n   c   y       2  \r  \n   F   r
0000020    e   d       S   a   v   a   g   e       4  \r  \n   T   o   m
0000040    m   y       F   r   a   n   k   s       7  \n                
0000054
$ 
Notice the "\r" represents a carriage return.

Your text file has both the "\r" carriage return and "\n" newline characters. This is common for DOS or Windows text files.

In Linux or UNIX the text file lines only use the new line delimiter. You need to remove the pesky "\r" carriage returns from the file, otherwise they will be part of your new file names.

Please use the following commands to create a new expired file without the carriage returns:

Code: Select all

cp expired expired.orig
tr -d '\r' <expired.orig >expired

Code: Select all

$ od -c expired
0000000    T   o   m       C   l   a   n   c   y       2  \n   F   r   e
0000020    d       S   a   v   a   g   e       4  \n   T   o   m   m   y
0000040        F   r   a   n   k   s       7  \n                        
0000052
$ 
Notice, the "\r" carriage returns are no longer present.
Thank you. I read your comment about the carriage return and figured that is why my new file name is is two lines.
I used "sed -i 's/\r//g' active" to remove the carriage returns. My files are now one line.
1000
Level 6
Level 6
Posts: 1023
Joined: Wed Jul 29, 2020 2:14 am

Re: Need Help with a Script

Post by 1000 »

1. Code

Code: Select all

#!/bin/bash

# Loop - Reading file line by line
while IFS= read -r  LINE ; do

    # Read $LINE for debug
    echo "DEBUG: LINE = $LINE" ;
    
    # Counting words in a line to avoid errors
    # From example we know that every line should have 3 words
    CHECK_1=$(wc -w <<< "$LINE")
    if [ "$CHECK_1" = 3 ] ; then
        # Then parse $LINE to get "the last name"
        SURNAME=$(echo "$LINE" | cut -d' ' -f2)
        echo " DEBUG: SURNAME = $SURNAME"
        echo "    IF is correct, now you can use cp command"
    elif [ "$CHECK_1" = 0 ] ; then
        echo "  Error: Line is empty."
    else
        echo "  Error: The number of words is different than 3."
    fi

done < active.txt
Output

Code: Select all

DEBUG: LINE = Trial 1
  Error: The number of words is different than 3.
DEBUG: LINE = Tom Tucker 3
 DEBUG: SURNAME = Tucker
    IF is correct, now you can use cp command
DEBUG: LINE = Mike Melon 5
 DEBUG: SURNAME = Melon
    IF is correct, now you can use cp command
DEBUG: LINE = Rick Ralston 6
 DEBUG: SURNAME = Ralston
    IF is correct, now you can use cp command
DEBUG: LINE = Taylor Swift 8
 DEBUG: SURNAME = Swift
    IF is correct, now you can use cp command
DEBUG: LINE = Carl Sagan 9
 DEBUG: SURNAME = Sagan
    IF is correct, now you can use cp command
DEBUG: LINE = George Carlin 10
 DEBUG: SURNAME = Carlin
    IF is correct, now you can use cp command
DEBUG: LINE = Tim Tebow 11
 DEBUG: SURNAME = Tebow
    IF is correct, now you can use cp command
DEBUG: LINE = Gene Simmons 12
 DEBUG: SURNAME = Simmons
    IF is correct, now you can use cp command
DEBUG: LINE = John Sno 13
 DEBUG: SURNAME = Sno
    IF is correct, now you can use cp command

Edit:

Don't forget to add a condition with "cp". ( If the file already exists then display an error )

2. Read this
viewtopic.php?f=213&t=336790
User avatar
deck_luck
Level 7
Level 7
Posts: 1577
Joined: Mon May 27, 2019 6:57 pm
Location: R-4808 North

Re: Need Help with a Script

Post by deck_luck »

Pecu1iar wrote: Thu Jan 07, 2021 1:12 am ...
Thank you. I read your comment about the carriage return and figured that is why my new file name is is two lines.
I used "sed -i 's/\r//g' active" to remove the carriage returns. My files are now one line.
Cool beans. :wink:
🐧Linux Mint 20.3 XFCE (UEFI - Secure Boot Enabled) dual boot with Windows 11

Give a friend a fish, and you feed them for a day. Teach a friend how to fish, and you feed them for a lifetime. ✝️
Pecu1iar
Level 1
Level 1
Posts: 39
Joined: Sun Nov 01, 2020 6:14 pm

Re: Need Help with a Script

Post by Pecu1iar »

So, With all of your help, I've managed to get my script to do more in line of what I want.

I now have a one line file name.

I've adjusted my google sheet formula so that I have a file name more in line with what I like.

The only problem I have now, which is not really noticeable to the outcome of the process is I have a multiple of extra empty lines.

When I run the od -c active command, I can see them all.

The downfall to this is a little more processing time as the script wants to process those lines and constantly overwrite with the same "blank" filename. The end result is that I end up with one extra file, not a big deal.

If there is a way to remove these extra empty lines at the end of the list, I could add it to my script. I've found a few things online, but nothing of what I've found does the trick.

I intend to read the articles that some of you have suggested. I thank you for all of your help. You guys are awesome!!!!
User avatar
deck_luck
Level 7
Level 7
Posts: 1577
Joined: Mon May 27, 2019 6:57 pm
Location: R-4808 North

Re: Need Help with a Script

Post by deck_luck »

Pecu1iar wrote: Thu Jan 07, 2021 4:45 am ...
The only problem I have now, which is not really noticeable to the outcome of the process is I have a multiple of extra empty lines.
...
If there is a way to remove these extra empty lines at the end of the list, I could add it to my script. I've found a few things online, but nothing of what I've found does the trick.
...
You can use the grep command to remove blank lines. The following expire text file contains some blank lines.

Code: Select all

$ od -c expired_blank_lines 
0000000    T   o   m       C   l   a   n   c   y       2  \n  \n   F   r
0000020    e   d       S   a   v   a   g   e       4  \n  \n   T   o   m
0000040    m   y       F   r   a   n   k   s       7  \n  \n            
0000055
$
The following grep command will remove the blank lines during output:

Code: Select all

$ grep -v '^$' expired_blank_lines 
Tom Clancy 2
Fred Savage 4
Tommy Franks 7
$ 
🐧Linux Mint 20.3 XFCE (UEFI - Secure Boot Enabled) dual boot with Windows 11

Give a friend a fish, and you feed them for a day. Teach a friend how to fish, and you feed them for a lifetime. ✝️
Pecu1iar
Level 1
Level 1
Posts: 39
Joined: Sun Nov 01, 2020 6:14 pm

Re: Need Help with a Script

Post by Pecu1iar »

You can use the grep command to remove blank lines. The following expire text file contains some blank lines.

Code: Select all

$ od -c expired_blank_lines 
0000000    T   o   m       C   l   a   n   c   y       2  \n  \n   F   r
0000020    e   d       S   a   v   a   g   e       4  \n  \n   T   o   m
0000040    m   y       F   r   a   n   k   s       7  \n  \n            
0000055
$
The following grep command will remove the blank lines during output:

Code: Select all

$ grep -v '^$' expired_blank_lines 
Tom Clancy 2
Fred Savage 4
Tommy Franks 7
$ 
[/quote]

I changed the parameters of my google sheets.
I've tried the the grep command, along with several other things I've found on the net to remove the excessive \n blank new lines, but none work.
I've attached the new file.
Any ideas?

Thanks.
Attachments
new.txt
(192 Bytes) Downloaded 28 times
User avatar
deck_luck
Level 7
Level 7
Posts: 1577
Joined: Mon May 27, 2019 6:57 pm
Location: R-4808 North

Re: Need Help with a Script

Post by deck_luck »

Code: Select all

$ od -c new.txt
0000000   T   r   i   a   l       0  \n   F   r   e   d       S   a   v
0000020   a   g   e       3  \n   R   i   c   k       R   a   l   s   t
0000040   o   n       5  \n   T   o   m   m   y       F   r   a   n   k
0000060   s       6  \n   T   a   y   l   o   r       S   w   i   f   t
0000100       7  \n   T   r   o   y       M   a   l   i   n       8  \n
0000120   G   e   o   r   g   e       C   a   r   l   i   n       9  \n
0000140   T   i   m       T   e   b   o   w       1   0  \n   G   e   n
0000160   e       S   i   m   m   o   n   s       1   1  \n   J   o   h
0000200   n       S   n   o       1   2  \n   G   r   e   g       A   b
0000220   b   o   t   t       1   3  \n      \n      \n      \n      \n
0000240      \n      \n      \n      \n      \n      \n      \n      \n
*
0000300
$
You actually have some lines containing only a space. :) You can use grep to remove lines only containing one or more white spaces using the following grep command:

Code: Select all

$ grep -v '^\s*$' new.txt
Trial 0
Fred Savage 3
Rick Ralston 5
Tommy Franks 6
Taylor Swift 7
Troy Malin 8
George Carlin 9
Tim Tebow 10
Gene Simmons 11
John Sno 12
Greg Abbott 13
$ 
🐧Linux Mint 20.3 XFCE (UEFI - Secure Boot Enabled) dual boot with Windows 11

Give a friend a fish, and you feed them for a day. Teach a friend how to fish, and you feed them for a lifetime. ✝️
User avatar
deck_luck
Level 7
Level 7
Posts: 1577
Joined: Mon May 27, 2019 6:57 pm
Location: R-4808 North

Re: Need Help with a Script

Post by deck_luck »

Clarification:
The grep -v '^\s*$' command will remove an empty (blank) lines as well as lines only containing zero or more white spaces.
🐧Linux Mint 20.3 XFCE (UEFI - Secure Boot Enabled) dual boot with Windows 11

Give a friend a fish, and you feed them for a day. Teach a friend how to fish, and you feed them for a lifetime. ✝️
Locked

Return to “Scripts & Bash”