Wget and many consoles at once time?

Forum rules
Before you post please read how to get help
User avatar
Level 1
Level 1
Posts: 46
Joined: Thu Jan 31, 2013 9:06 am
Location: Paris

Wget and many consoles at once time?

Postby Anubisone » Thu May 09, 2013 3:43 pm

I have a huge website I need to download. It will take me one week, but I want to do it in one day, mayby 8 hours etc.

Since now I used:

Code: Select all

wget --random-wait -r -p -e robots=off -U --mirror -p --html-extension --convert-links mozilla -c http://www.cia.org

I know wget uses wildcards only for ftp, not for http, but maybe if I run the same task in 10 consoles, each console will know that another console downloaded something successfully?

Another thing is, in my language there are some foreign signs in http like "ś" or "ó", but the downloaded file has been replaced by some %82 etc.

Can I replace it somehow before I download it in the command line, cause there will be about 90.000 files ...

Thanks for help!!

Level 1
Level 1
Posts: 10
Joined: Sat Mar 08, 2014 1:58 pm

Re: Wget and many consoles at once time?

Postby tigrezno » Wed Mar 19, 2014 6:41 pm

Have you tried the "-nc" option?

Code: Select all

-nc, --no-clobber              skip downloads that would download to existing files (overwriting them).

Return to “Scripts & Bash”