ogg downloader

Forum rules
Before you post please read how to get help
User avatar
passstab
Level 1
Level 1
Posts: 26
Joined: Thu May 05, 2011 5:29 pm

ogg downloader

Postby passstab » Tue Oct 11, 2011 2:58 pm

I'm trying to make a script that will download all the ogg files on a specific IA album
this is what i have so far it doesn't work yet
the site should be the http link (e.g for http://www.archive.org/details/ball_and ... 1_librivox use http://ia600408.us.archive.org/2/items/ ... _librivox/)

Code: Select all

rm -f ./index.html
rm -f ./sindex.html
echo 'please type name of site'
read SITE
FN=$(basename $SITE )
wget $SITE -O ./index.html
echo 'please type file extension'
read EXT
grep .$EXT ./index.html | cut -d "\"" -f 2 > ./itemnames
chmod a+wrx ./itemnames
cat ./itemnames | parallel -i -- "mkdir -p -m 777 ./$FN/{}  wget  -O -B $SITE -i  {} >> ./$FN/{}"
rm ./index.html
rm ./itemnames

i'm sure this already exists but I'm trying to learn bash
anyone know what's wrong with it?
Last edited by passstab on Tue Oct 11, 2011 3:11 pm, edited 1 time in total.
proud supporter of Israel

User avatar
xenopeek
Level 23
Level 23
Posts: 19024
Joined: Wed Jul 06, 2011 3:58 am
Location: The Netherlands

Re: ogg downloader

Postby xenopeek » Tue Oct 11, 2011 3:09 pm

passstab wrote:this is what i have so far it doesn't work yet

I'm no BASH guru, but where does it go wrong? Would help if you can point out what part is working as it should, and where it breaks. (I usually add some echo statements inbetween lines to find out if things are going as they should.)

You stated you wanted to learn BASH, but for anybody wanting to have a website downloader, HTTrack for linux works very good in downloading websites or sets of files from websites.
Image

User avatar
passstab
Level 1
Level 1
Posts: 26
Joined: Thu May 05, 2011 5:29 pm

Re: ogg downloader

Postby passstab » Tue Oct 11, 2011 3:20 pm

thanks
the error i get is

Code: Select all

sh: cannot create ./ball_and_cross_1001_librivox/{}: Directory nonexistent

i added one echo in the only place i see it could go

Code: Select all

rm -f ./index.html
rm -f ./sindex.html
echo 'please type name of site'
read SITE
FN=$(basename $SITE )
wget $SITE -O ./index.html
echo 'please type file extension'
read EXT
grep .$EXT ./index.html | cut -d "\"" -f 2 > ./itemnames
echo $(cat ./itemnames)
chmod a+wrx ./itemnames
cat ./itemnames | parallel -i -- "mkdir -p -m 777 ./$FN/{}  wget  -O -B $SITE -i  {} >> ./$FN/{}"
rm ./index.html
rm ./itemnames
proud supporter of Israel

User avatar
xenopeek
Level 23
Level 23
Posts: 19024
Joined: Wed Jul 06, 2011 3:58 am
Location: The Netherlands

Re: ogg downloader

Postby xenopeek » Tue Oct 11, 2011 3:36 pm

Then this is your error:
cat ./itemnames | parallel -i -- "mkdir -p -m 777 ./$FN/{} wget -O -B $SITE -i {} >> ./$FN/{}"

As you can see from the error you get, the red statement above resolves to the script trying to create the {} directory in your ./ball_and_cross_1001_librivox directory. As the latter doesn't exist yet, this fails. What are the {} for?
Image

User avatar
passstab
Level 1
Level 1
Posts: 26
Joined: Thu May 05, 2011 5:29 pm

Re: ogg downloader

Postby passstab » Tue Oct 11, 2011 5:28 pm

Code: Select all

rm -f ./index.html
rm -f ./itemnames
echo 'please type name of site'
read SITE
FN=$(basename $SITE )
wget $SITE -O ./index.html
echo 'please type file extension'
read EXT
grep .$EXT ./index.html | cut -d "\"" -f 2 > ./itemnames
echo $(cat ./itemnames)
chmod a+wrx ./itemnames
cat ./itemnames | xargs -I {} mkdir -p -m 777 ./$FN/{}  wget  -O -B $SITE -i  {} >> ./$FN/{}
rm ./index.html
rm ./itemnames

{} is supposed to be the xargs input (ditched parallel cause it froze my computer)
http://linux.die.net/man/1/xargs
mkdir has a -p option (parent) that i think should make that not a issue
https://secure.wikimedia.org/wikipedia/ ... r#Examples
but this is what i get

Code: Select all

lease type name of site
http://ia600408.us.archive.org/2/items/ball_and_cross_1001_librivox/
--2011-10-11 17:32:28--  http://ia600408.us.archive.org/2/items/ball_and_cross_1001_librivox/
Resolving ia600408.us.archive.org... 207.241.227.218
Connecting to ia600408.us.archive.org|207.241.227.218|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `./index.html'

    [ <=>                                                                                                                ] 8,999       --.-K/s   in 0.08s   

2011-10-11 17:32:28 (104 KB/s) - `./index.html' saved [8999]

please type file extension
ogg
ballandcross_01_chesterton.ogg ballandcross_02_chesterton.ogg ballandcross_03_chesterton.ogg ballandcross_04_chesterton.ogg ballandcross_05_chesterton.ogg ballandcross_06_chesterton.ogg ballandcross_07_chesterton.ogg ballandcross_08_chesterton.ogg ballandcross_09_chesterton.ogg ballandcross_10_chesterton.ogg ballandcross_11_chesterton.ogg ballandcross_12_chesterton.ogg ballandcross_13_chesterton.ogg ballandcross_14_chesterton.ogg ballandcross_15_chesterton.ogg ballandcross_16_chesterton.ogg ballandcross_17_chesterton.ogg ballandcross_18_chesterton.ogg ballandcross_19_chesterton.ogg ballandcross_20_chesterton.ogg
../../adfgsa.sh: line 12: ./ball_and_cross_1001_librivox/{}: No such file or directory

proud supporter of Israel

User avatar
xenopeek
Level 23
Level 23
Posts: 19024
Joined: Wed Jul 06, 2011 3:58 am
Location: The Netherlands

Re: ogg downloader

Postby xenopeek » Wed Oct 12, 2011 5:37 am

Perhaps add as first line to your script:

Code: Select all

#!/bin/bash

Instead of your single not working line:
cat ./itemnames | xargs -I {} mkdir -p -m 777 ./$FN/{} wget -O -B $SITE -i {} >> ./$FN/{}

You can do something like the following (not sure I've put your commands in properly):

Code: Select all

ITEMNAMES=$(cat ./itemnames)
for ITEMNAME in $ITEMNAMES; do
   mkdir -p -m 777 ./$FN/$ITEMNAME
   wget  -O -B $SITE -i $ITEMNAME >> ./$FN/$ITEMNAME
done

Yes, more lines of code. But six months from now you'll still understand what this does and how it works. At least I've found that I can really make obfuscated one-liners :wink: Six months later, heck even six weeks later, I'd wished I programmed it a little clearer and added some comments.
Image

User avatar
passstab
Level 1
Level 1
Posts: 26
Joined: Thu May 05, 2011 5:29 pm

Re: ogg downloader

Postby passstab » Mon Oct 17, 2011 11:30 am

got it to work!

Code: Select all

#!/bin/bash
rm -f ./webpage
rm -f ./basenames
echo 'please type name of site'
read URL
wget $URL -O ./webpage
echo 'please type file extension'
read EXTENSION
grep .$EXTENSION ./webpage | cut -d "\"" -f 2 > ./basenames
wget -x -nH -B $URL -i ./basenames
rm ./webpage
rm ./basenames

the wget -x -nH options do something close enough to what i wanted
(ideally it would only make the last two directories e.g "/items/ball_and_cross_1001_librivox/" instead of /2/items/ball_and_cross_1001_librivox/)
i think i was giving it values ware it wanted filenames (your suggestion didn't work i think for the same reason)
thanks for the help
proud supporter of Israel

User avatar
xenopeek
Level 23
Level 23
Posts: 19024
Joined: Wed Jul 06, 2011 3:58 am
Location: The Netherlands

Re: ogg downloader

Postby xenopeek » Mon Oct 17, 2011 11:37 am

Yeah, wget has a lot of options :D
Image


Return to “Scripts & Bash”

Who is online

Users browsing this forum: No registered users and 1 guest