[SOLVED] Tar with dialog not working

About writing shell scripts and making the most of your shell
Forum rules
Topics in this forum are automatically closed 6 months after creation.
Locked
RavenLX

[SOLVED] Tar with dialog not working

Post by RavenLX »

I have this:

Code: Select all

#!/bin/bash

exec<"myfiles.txt"
while read line
do
(tar -pPrvf - $line | pv -n -s $(du -sb $line | awk '{print $1}') > myarchive.tar) 2>&1 \
| dialog --gauge "Backing up: $line" 7 70 0
done
Problem is, it shows in the dialog the file it's backing up. But, the progress bar doesn't move, things go fast and the resulting myarchive.tar is 0 bytes. The myfiles.txt contains 152 entries and some of the entries are files, and some are directories. Some directories and/or files are a bit large so they should have triggered the dialog progress bar.

What am I doing wrong here? Can anyone walk me through the correct commands?

[UPDATE] I solved it. Please see my post farther down.
Last edited by LockBot on Wed Dec 28, 2022 7:16 am, edited 2 times in total.
Reason: Topic automatically closed 6 months after creation. New replies are no longer allowed.
RavenLX

Re: Tar with dialog not working

Post by RavenLX »

Ideally, I would rather get the list of files with something like 'locate keyword' instead of reading a list. to get the list I did a 'locate keyword > myfiles.txt' . So if I could skip that step and do it all in the batch file, that'd be even better!
Habitual

Re: Tar with dialog not working

Post by Habitual »

why --progress AND pv?

Seems easier to debug if you had fewer "options" involved. but that's just my opinion.

Code: Select all

(tar -pPrvf - $line | pv -n -s $(du -sb $line | awk '{print $1}') > myarchive.tar) 2>&1
is just wrong.
tar <options> <archive.tar> <files>

IF it were me, I'd try this, or a variation on it until it mostly worked...and assuming myfiles.txt has the complete path to the files needed to backup...

Code: Select all

for i in `cat myfiles.txt` ; do tar -pPrvf  myarchive.tar $i ; done


Once that works, then I'd make that a function utillized like so:

Code: Select all

#!/bin/bash
do_work ()
{
for i in `cat myfiles.txt` ; do tar -pPrvf /path/to/myarchive.tar $i ; done
}

touch /tmp/$$
( ( echo 1 ; while [ -f /tmp/$$ ] ; do sleep 1 ; done ; echo 100 ) | dialog --gauge 7 70 0) & do_work
rm /tmp/$$
#EOF
Personally, I use yad for this stuff, and the dialog with gauge is kind of "iffy".
If you choose to try yad, here's the code I wrote for a similar routine:

Code: Select all

touch /tmp/$$
( ( echo 1 ; while [ -f /tmp/$$ ] ; do sleep 1 ; done ; echo 100 ) | yad --progress --pulsate --auto-close --text="Retrieving Snapshots..." --width=150 --title="" --undecorated --no-buttons) & do_work
rm /tmp/$$
Good luck.
RavenLX

Re: Tar with dialog not working

Post by RavenLX »

Thanks but none of these will do what I need: I take a list of files (hopefully directly from 'locate keyword' for example or a text file), tarball it while showing a progress bar or percentage of the tarball operation (so I'm not either seeing a stream of files scrolling on the screen or dots scrolling on the screen or waiting and waiting and wondering if it hung).

Your first example does not show any progress bar.
Your second example scrolls on the terminal screen (even with the echo commands removed), which is what I'm trying to avoid.
I can't find Yad in the repositories and it's not installed in my system.

BTW, I never used any --progress switch in my code?

I am still looking for a solution...
DrHu

Re: Tar with dialog not working

Post by DrHu »

http://superuser.com/questions/168749/is-there-a-way-to-see-any-tar-progress-per-file
RavenLX

Re: Tar with dialog not working

Post by RavenLX »

Thanks, Dr. Hu. I already looked at that page and that's where I found out about 'dialog'. Thing is, I can't get it to work if I'm using a variable ($line).
Habitual

Re: Tar with dialog not working

Post by Habitual »

RavenLX wrote:Thanks but none of these will do what I need...
BTW, I never used any --progress switch in my code?
Better re-read that again. I included a snippet "....yad --progress ..." in the example I cited as my source for the snippets I gave to you to try out.
I didn't tell you to remove any echoes, nor did I say they were there to prevent accidental data loss.

What you "need" is to learn how to use tar correctly and "How do I test my tar files?"
Humor me, try

Code: Select all

cat myfiles.txt` ; do tar -pPrf  myarchive.tar $i ; done
wrt:
RavenLX wrote:myarchive.tar is 0 bytes
Test it with:

Code: Select all

tar -tf archive.tar
Does it look about correct judging by the output? based on the input myfiles.txt file?

Is the archive.tar 0 bytes? Yes, start over.
Double check your work.

Don't stick in your script and blindly expect it to work. Use it in terminal>
It's a piece of the puzzle.

Sentences like "Doesn't work" or "I can't get it to work" don't mean much.

http://www.tldp.org/LDP/abs/html/internal.html#READPIPEREF
I'm done. I fear I have spent more time on this than you have.

Good luck.
RavenLX

Re: Tar with dialog not working

Post by RavenLX »

You asked "why --progress AND pv?" At the top of your response to my first post. Therefore I thought you thought I used that when I didn't. I can't find yad in the respository.

I know you didn't "tell" me to take out the echoes from your snippets. I tried with the echoes and also without. You did give a script to try as well, and I tried that (after all I assumed it was a script since you put !#/bin/bash in the code snippet). The other stuff was tried at the terminal.

Here's what I got in the terminal when trying your snippet (I'm including the prompt as I'm copy/pasting):

Code: Select all

$ cat myfiles.txt` ; do tar -pPrf  myarchive.tar $i ; done
>

So I noted that you forgot a backtick at the start of myfiles.txt. I tried again:

Code: Select all

$ cat `myfiles.txt` ; do tar -pPrf  myarchive.tar $i ; done
bash: syntax error near unexpected token `do'

I humored you. It doesn't work. I'm sorry but none of your snippets is giving me any information that I can use to resolve my problem. As I have stated in the first post I'm looking for a progress indicator to work while tarballing files that are from a list of files (or from locate 'keyword') and not something that will scroll the screen with information.

I'm sorry if you don't want to spend more time on this. You don't have to. Thank you for trying anyway. I'll keep watching this and seeing if someone else has a solution.
RavenLX

[SOLVED] Re: Tar with dialog not working

Post by RavenLX »

I found the problem! I want to thank all those who tried to help anyway.

Here's the solution:

Code: Select all

#!/bin/bash

IFS=$'\r\n' line=($(cat myfiles.txt))
arrlen=${#line[@]}
for (( i=1; i<${arrlen}; i++ ));
do
  (tar -pPrf myarchive.tar ${line[$i]} | pv -n -s $(du -sb ${line[$i]} | awk '{print $1}') ) 2>&1 | dialog --gauge "Backing up: ${line[$i]}" 7 70 0
done
Explaination

First off, I needed to iterate through each line in the mygiles.txt in order to get each file or directory path to feed pv, du and awk (which gets the information needed for dialog to show on the progress bar).

The second problem was putting the tar filename just before the last parenthesis (as in > myarchive.tar). Tar needs it right after the 'f' parameter.

These things fixed, it now works! My files weren't large enough though to get the progress to go past 0%. So I'm still not sure if that will work or not. But at least this is a start. Theoretically it should work.
Habitual

Re: Tar with dialog not working

Post by Habitual »

RavenLX wrote:You asked "why --progress AND pv?"
...
I'm sorry if you don't want to spend more time on this. You don't have to. Thank you for trying anyway.

I was short with you and I apologize. I forget that I once couldn't code my way out of a wet paper bag.

Have you tested the tars?

Code: Select all

tar -tf archive.tar | less
RavenLX

Re: [SOLVED] Tar with dialog not working

Post by RavenLX »

No problem. :) I admit even though I've grown used to the GNU/Linux command line I still don't know all that can be done with any particular command. In fact, I don't even know a fraction of all that can be done with it.

I've tested the tar in the code that didn't work and it didn't have anything in it. I tested the tar in the code that did work and everything was there as it should be.

Also, I used this code to compress the tar:

Code: Select all

(pv -n myarchive.tar | gzip -> myarchive.tgz ) 2>&1 | dialog --backtitle "$title"  --gauge "Compressing myarchive.tar" 7 70 0
if [ -f "myarchive.tgz" ]; then
  rm -rf myarchive.tar
fi
For anyone that may be reading this and wonder why I do it this way; you can't add files to a compressed tar (ie. a .tar.gz or .tgz). So I had to add them into a plain .tar. Then when done adding all the files, I can compress it with gzip and remove the tar. gzip could have done this but since we need to put this through a dialog, I used the code above instead.
Habitual

Re: [SOLVED] Tar with dialog not working

Post by Habitual »

Well, you're making progress and that's what counts.
Scripts can always be made "better", faster, more efficient after the basic task works.

Have fun!
RavenLX

Re: [SOLVED] Tar with dialog not working

Post by RavenLX »

Here is some of what I've learned. These are code snippets that I hope will be useful to some people. Edit it to suit your needs.

Also I've found if you are concatenating stuff with directories and they show a date string of 0/0:00 in the archive, you can avoid seeing the errors by doing this:

$ tar -xzf myarchive.tgz 2>/dev/null

The archive should still extract ok.

Code: Select all

!#/bin/bash

# This is example code. It does not work as-is.
# Edit the code to suit your purpose.

clear
dialog --backtitle "$title" --infobox \
"Scanning for Files..." 3 46 1>&2
# Scan for all 'file_name' files in all of the system and add it to the list.
# This adds the entire full path.
(find / -type f ! -path "$HOME/*" ! -path "/proc/*" 2>&1  | grep file_name) > files.lst

clear
dialog --backtitle "$title" --infobox \
# Scan for user config for 'file_name and add it to the list.
# This adds the entire full path.
"Scanning for User Configuration..." 3 46 1>&2
(find $HOME/.file_name -type f) 2>&1  >> files.lst

# Back up all the files into a tar archive
clear
IFS=$'\r\n' line=($(cat files.lst))
arrlen=${#line[@]}
for(( i=0; i<${arrlen}; i++ ));
do
  clear
  (pv -n ${line[$i]} | tar pPrf myarchive.tar ${line[$i]} ) 2>&1 \
  | dialog --backtitle "$title" --gauge "Backing up file $i of $arrlen" 7 70 0
done

clear
(pv -n eclipse-backup.tar | gzip -> myarchive.tgz ) 2>&1 \
| dialog --backtitle "$title" --gauge "Compressing Archive." 7 70 0
if [ -f "myarchive.tgz" ]; then
  rm -rf myarchive.tar
fi

# Backing up one full directories into one archive
mydir="$HOME/MyDirectory"

clear
(tar cpPzf - ~/.one_of_my_config_dirs | pv -n -s $(du -sb ~/.one_of_my_config_dirs \
| awk '{print $1}') > myotherarchive.tgz) 2>&1 \
| dialog --backtitle "$title" --gauge "Backing up My Data..." 7 70 0

clear
(tar cpPzf - $mydir | pv -n -s $(du -sb $mydir \
| awk '{print $1}') > temp.tgz) 2>&1 \
| dialog --backtitle "$title" --gauge "Backing up More Data..." 7 70 0

clear
(tar -Ppz --concatenate --file=myotherarchive.tgz temp.tgz \
| pv -n temp.tgz > /dev/null) 2>&1 \
| dialog --backtitle "$title" --gauge "Compressing archive." 7 70 0
rm -rf temp.tgz
Locked

Return to “Scripts & Bash”