Wget and many consoles at once time?

Forum rules
Before you post please read how to get help

Wget and many consoles at once time?

Postby Anubisone on Thu May 09, 2013 3:43 pm

I have a huge website I need to download. It will take me one week, but I want to do it in one day, mayby 8 hours etc.

Since now I used:

Code: Select all
wget --random-wait -r -p -e robots=off -U --mirror -p --html-extension --convert-links mozilla -c http://www.cia.org

I know wget uses wildcards only for ftp, not for http, but maybe if I run the same task in 10 consoles, each console will know that another console downloaded something successfully?

Another thing is, in my language there are some foreign signs in http like "ś" or "ó", but the downloaded file has been replaced by some %82 etc.

Can I replace it somehow before I download it in the command line, cause there will be about 90.000 files ...

Thanks for help!!
User avatar
Level 1
Level 1
Posts: 46
Joined: Thu Jan 31, 2013 9:06 am
Location: Paris

Linux Mint is funded by ads and donations.

Re: Wget and many consoles at once time?

Postby tigrezno on Wed Mar 19, 2014 6:41 pm

Have you tried the "-nc" option?
Code: Select all
-nc, --no-clobber              skip downloads that would download to existing files (overwriting them).
Level 1
Level 1
Posts: 10
Joined: Sat Mar 08, 2014 1:58 pm

Return to Scripts & Bash

Who is online

Users browsing this forum: No registered users and 2 guests