ogg downloader

Forum rules
Before you post please read this

ogg downloader

Postby passstab on Tue Oct 11, 2011 2:58 pm

I'm trying to make a script that will download all the ogg files on a specific IA album
this is what i have so far it doesn't work yet
the site should be the http link (e.g for http://www.archive.org/details/ball_and ... 1_librivox use http://ia600408.us.archive.org/2/items/ ... _librivox/)
Code: Select all
rm -f ./index.html
rm -f ./sindex.html
echo 'please type name of site'
read SITE
FN=$(basename $SITE )
wget $SITE -O ./index.html
echo 'please type file extension'
read EXT
grep .$EXT ./index.html | cut -d "\"" -f 2 > ./itemnames
chmod a+wrx ./itemnames
cat ./itemnames | parallel -i -- "mkdir -p -m 777 ./$FN/{}  wget  -O -B $SITE -i  {} >> ./$FN/{}"
rm ./index.html
rm ./itemnames

i'm sure this already exists but I'm trying to learn bash
anyone know what's wrong with it?
Last edited by passstab on Tue Oct 11, 2011 3:11 pm, edited 1 time in total.
proud supporter of Israel
User avatar
passstab
Level 1
Level 1
 
Posts: 27
Joined: Thu May 05, 2011 5:29 pm

Linux Mint is funded by ads and donations.
 

Re: ogg downloader

Postby xenopeek on Tue Oct 11, 2011 3:09 pm

passstab wrote:this is what i have so far it doesn't work yet

I'm no BASH guru, but where does it go wrong? Would help if you can point out what part is working as it should, and where it breaks. (I usually add some echo statements inbetween lines to find out if things are going as they should.)

You stated you wanted to learn BASH, but for anybody wanting to have a website downloader, HTTrack for linux works very good in downloading websites or sets of files from websites.
User avatar
xenopeek
Level 21
Level 21
 
Posts: 14994
Joined: Wed Jul 06, 2011 3:58 am
Location: The Netherlands

Re: ogg downloader

Postby passstab on Tue Oct 11, 2011 3:20 pm

thanks
the error i get is
Code: Select all
sh: cannot create ./ball_and_cross_1001_librivox/{}: Directory nonexistent

i added one echo in the only place i see it could go
Code: Select all
rm -f ./index.html
rm -f ./sindex.html
echo 'please type name of site'
read SITE
FN=$(basename $SITE )
wget $SITE -O ./index.html
echo 'please type file extension'
read EXT
grep .$EXT ./index.html | cut -d "\"" -f 2 > ./itemnames
echo $(cat ./itemnames)
chmod a+wrx ./itemnames
cat ./itemnames | parallel -i -- "mkdir -p -m 777 ./$FN/{}  wget  -O -B $SITE -i  {} >> ./$FN/{}"
rm ./index.html
rm ./itemnames
proud supporter of Israel
User avatar
passstab
Level 1
Level 1
 
Posts: 27
Joined: Thu May 05, 2011 5:29 pm

Re: ogg downloader

Postby xenopeek on Tue Oct 11, 2011 3:36 pm

Then this is your error:
cat ./itemnames | parallel -i -- "mkdir -p -m 777 ./$FN/{} wget -O -B $SITE -i {} >> ./$FN/{}"

As you can see from the error you get, the red statement above resolves to the script trying to create the {} directory in your ./ball_and_cross_1001_librivox directory. As the latter doesn't exist yet, this fails. What are the {} for?
User avatar
xenopeek
Level 21
Level 21
 
Posts: 14994
Joined: Wed Jul 06, 2011 3:58 am
Location: The Netherlands

Re: ogg downloader

Postby passstab on Tue Oct 11, 2011 5:28 pm

Code: Select all
rm -f ./index.html
rm -f ./itemnames
echo 'please type name of site'
read SITE
FN=$(basename $SITE )
wget $SITE -O ./index.html
echo 'please type file extension'
read EXT
grep .$EXT ./index.html | cut -d "\"" -f 2 > ./itemnames
echo $(cat ./itemnames)
chmod a+wrx ./itemnames
cat ./itemnames | xargs -I {} mkdir -p -m 777 ./$FN/{}  wget  -O -B $SITE -i  {} >> ./$FN/{}
rm ./index.html
rm ./itemnames

{} is supposed to be the xargs input (ditched parallel cause it froze my computer)
http://linux.die.net/man/1/xargs
mkdir has a -p option (parent) that i think should make that not a issue
https://secure.wikimedia.org/wikipedia/ ... r#Examples
but this is what i get
Code: Select all
lease type name of site
http://ia600408.us.archive.org/2/items/ball_and_cross_1001_librivox/
--2011-10-11 17:32:28--  http://ia600408.us.archive.org/2/items/ball_and_cross_1001_librivox/
Resolving ia600408.us.archive.org... 207.241.227.218
Connecting to ia600408.us.archive.org|207.241.227.218|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `./index.html'

    [ <=>                                                                                                                ] 8,999       --.-K/s   in 0.08s   

2011-10-11 17:32:28 (104 KB/s) - `./index.html' saved [8999]

please type file extension
ogg
ballandcross_01_chesterton.ogg ballandcross_02_chesterton.ogg ballandcross_03_chesterton.ogg ballandcross_04_chesterton.ogg ballandcross_05_chesterton.ogg ballandcross_06_chesterton.ogg ballandcross_07_chesterton.ogg ballandcross_08_chesterton.ogg ballandcross_09_chesterton.ogg ballandcross_10_chesterton.ogg ballandcross_11_chesterton.ogg ballandcross_12_chesterton.ogg ballandcross_13_chesterton.ogg ballandcross_14_chesterton.ogg ballandcross_15_chesterton.ogg ballandcross_16_chesterton.ogg ballandcross_17_chesterton.ogg ballandcross_18_chesterton.ogg ballandcross_19_chesterton.ogg ballandcross_20_chesterton.ogg
../../adfgsa.sh: line 12: ./ball_and_cross_1001_librivox/{}: No such file or directory

proud supporter of Israel
User avatar
passstab
Level 1
Level 1
 
Posts: 27
Joined: Thu May 05, 2011 5:29 pm

Re: ogg downloader

Postby xenopeek on Wed Oct 12, 2011 5:37 am

Perhaps add as first line to your script:
Code: Select all
#!/bin/bash

Instead of your single not working line:
cat ./itemnames | xargs -I {} mkdir -p -m 777 ./$FN/{} wget -O -B $SITE -i {} >> ./$FN/{}

You can do something like the following (not sure I've put your commands in properly):
Code: Select all
ITEMNAMES=$(cat ./itemnames)
for ITEMNAME in $ITEMNAMES; do
   mkdir -p -m 777 ./$FN/$ITEMNAME
   wget  -O -B $SITE -i $ITEMNAME >> ./$FN/$ITEMNAME
done

Yes, more lines of code. But six months from now you'll still understand what this does and how it works. At least I've found that I can really make obfuscated one-liners :wink: Six months later, heck even six weeks later, I'd wished I programmed it a little clearer and added some comments.
User avatar
xenopeek
Level 21
Level 21
 
Posts: 14994
Joined: Wed Jul 06, 2011 3:58 am
Location: The Netherlands

Re: ogg downloader

Postby passstab on Mon Oct 17, 2011 11:30 am

got it to work!
Code: Select all
#!/bin/bash
rm -f ./webpage
rm -f ./basenames
echo 'please type name of site'
read URL
wget $URL -O ./webpage
echo 'please type file extension'
read EXTENSION
grep .$EXTENSION ./webpage | cut -d "\"" -f 2 > ./basenames
wget -x -nH -B $URL -i ./basenames
rm ./webpage
rm ./basenames

the wget -x -nH options do something close enough to what i wanted
(ideally it would only make the last two directories e.g "/items/ball_and_cross_1001_librivox/" instead of /2/items/ball_and_cross_1001_librivox/)
i think i was giving it values ware it wanted filenames (your suggestion didn't work i think for the same reason)
thanks for the help
proud supporter of Israel
User avatar
passstab
Level 1
Level 1
 
Posts: 27
Joined: Thu May 05, 2011 5:29 pm

Re: ogg downloader

Postby xenopeek on Mon Oct 17, 2011 11:37 am

Yeah, wget has a lot of options :D
User avatar
xenopeek
Level 21
Level 21
 
Posts: 14994
Joined: Wed Jul 06, 2011 3:58 am
Location: The Netherlands

Linux Mint is funded by ads and donations.
 

Return to Scripts & Bash

Who is online

Users browsing this forum: No registered users and 5 guests