Possible to automate data DLs during offpeak periods? [SOLVED]

All Gurus once were Newbies
Forum rules
There are no such things as "stupid" questions. However if you think your question is a bit stupid, then this is the right place for you to post it. Please stick to easy to-the-point questions that you feel people can answer fast. For long and complicated questions prefer the other forums within the support section.
Before you post please read how to get help
Post Reply
lazarus
Level 3
Level 3
Posts: 153
Joined: Mon Jul 02, 2018 11:36 pm
Location: Oberon, OZ

Possible to automate data DLs during offpeak periods? [SOLVED]

Post by lazarus »

Linux Mint 18.3 Cinnamon X64

As we live in the outback, all comms are satellite. The best deal we can get gives gives me 60GB/month peak & 190GB/month off-peak. (Offpeak is only 1AM-5AM daily.) Being our main form of entertainment, 60GB blows through very fast. Kids. :(

Naturally, I'd like to move as many DLs to the off-peak period as possible, but I gotta sleep sometimes. My dev days are long in the past.
  • Is it possible to set it up so Update Manager, etc. only pull files during off-peak? I'd like to refresh the list manually, select possible upgrades and then let it wait 'til the wee hours...
  • I imagine that I can use cron to download from a txt file of predetermined URLs, so we can grab available movies, etc. How would I go about this? ie. is there an all-purpose downloader I can start from crontab?
  • I further imagine that youtube wouldn't be accessible this way? What with copyright issues, blah, blah?
FWIW, I'm comfortable in CON: if not currently particularly confident. I loved Linux in my Uni days - when RedHat was still new kid on the block - but the last decade or so had Windows forced on me for work reasons. With a recent RAID loss due to a #$%^# WinUpdate I finally decided to go hunt down a linux distro and see what's new. I think I'm keeping it :wink:
Last edited by lazarus on Thu Jul 05, 2018 2:53 am, edited 1 time in total.
- Andy

I may be weird but I'm saving up to become eccentric.
User avatar
jimallyn
Level 18
Level 18
Posts: 8959
Joined: Thu Jun 05, 2014 7:34 pm
Location: Wenatchee, WA USA

Re: Possible to automate data DLs during offpeak periods?

Post by jimallyn »

wget can be used to download files. Enter man wget in terminal to learn how to use it. It does have a lot of options, but you won't need most of them. Yes, you could use chron to run wget. I've never quite figured chron out, so I often use at instead. With at you can specify commands to be run whenever you want, and it accepts a bunch of different time formats, like 01:30, or 11:00AM, or tomorrow, or now + 3 hours, or midnight. And of course man at will give you all the details.
Image

“If the government were coming for your TVs and cars, then you'd be upset. But, as it is, they're only coming for your sons.” - Daniel Berrigan
User avatar
jimallyn
Level 18
Level 18
Posts: 8959
Joined: Thu Jun 05, 2014 7:34 pm
Location: Wenatchee, WA USA

Re: Possible to automate data DLs during offpeak periods?

Post by jimallyn »

I'm not sure if there's a way to script Update Manager. You could probably run apt or apt-get from a script to accomplish the same thing. Downloading youtube videos is easy with youtube-dl. The version in the repos is usually old, so get the most recent version from here:

https://rg3.github.io/youtube-dl/download.html

There is also a GUI for youtube-dl, called youtube-dlg. But it is missing a lot of the options available from the command line.

Which reminds me there is another downloader command, curl. I used it once to download a complete website. Of course, curl has a man page, too, and I'm sure it could also be run with chron or at. I think curl and wget both have options to limit the download rate, which you might find convenient.
Image

“If the government were coming for your TVs and cars, then you'd be upset. But, as it is, they're only coming for your sons.” - Daniel Berrigan
User avatar
catweazel
Level 19
Level 19
Posts: 9884
Joined: Fri Oct 12, 2012 9:44 pm
Location: Australian Antarctic Territory

Re: Possible to automate data DLs during offpeak periods?

Post by catweazel »

lazarus wrote:
Tue Jul 03, 2018 12:29 am
As we live in the outback, all comms are satellite. The best deal we can get gives gives me 60GB/month peak & 190GB/month off-peak. (Offpeak is only 1AM-5AM daily.)
XDM download manager has a scheduler, and it works well.

Code: Select all

sudo add-apt-repository ppa:noobslab/apps
sudo apt-get update
sudo apt-get install xdman
https://sourceforge.net/projects/xdman/

No need to use the command line.
"There is, ultimately, only one truth -- cogito, ergo sum -- everything else is an assumption." - Me, my swansong.
gm10
Level 20
Level 20
Posts: 10999
Joined: Thu Jun 21, 2018 5:11 pm

Re: Possible to automate data DLs during offpeak periods?

Post by gm10 »

lazarus wrote:
Tue Jul 03, 2018 12:29 am
  • Is it possible to set it up so Update Manager, etc. only pull files during off-peak? I'd like to refresh the list manually, select possible upgrades and then let it wait 'til the wee hours...
  • I imagine that I can use cron to download from a txt file of predetermined URLs, so we can grab available movies, etc. How would I go about this? ie. is there an all-purpose downloader I can start from crontab?
  • I further imagine that youtube wouldn't be accessible this way? What with copyright issues, blah, blah?
- Remove the update manager from the startup applications, set up a cron job for mintupdate-cli instead to run whenever you want it to.
- as explained above, a cron job for wget is one way to get arbitrary downloads done. But you may be more comfortable with a GUI download manager that allows you to schedule downloads. I can recommend uget, it's in the official repository
- uget can also download from youtube. but if you prefer CLI downloads, the best tool is youtube-dl

Copyright issues I'll let you sort out with your conscience and/or your lawyer.
User avatar
jimallyn
Level 18
Level 18
Posts: 8959
Joined: Thu Jun 05, 2014 7:34 pm
Location: Wenatchee, WA USA

Re: Possible to automate data DLs during offpeak periods?

Post by jimallyn »

catweazel wrote:
Wed Jul 04, 2018 4:53 am
No need to use the command line.
Spoilsport! Good find, actually. I just installed it and will try it out in the next day or two.
Image

“If the government were coming for your TVs and cars, then you'd be upset. But, as it is, they're only coming for your sons.” - Daniel Berrigan
User avatar
catweazel
Level 19
Level 19
Posts: 9884
Joined: Fri Oct 12, 2012 9:44 pm
Location: Australian Antarctic Territory

Re: Possible to automate data DLs during offpeak periods?

Post by catweazel »

jimallyn wrote:
Wed Jul 04, 2018 5:33 am
catweazel wrote:
Wed Jul 04, 2018 4:53 am
No need to use the command line.
Spoilsport!
:mrgreen:
"There is, ultimately, only one truth -- cogito, ergo sum -- everything else is an assumption." - Me, my swansong.
User avatar
catweazel
Level 19
Level 19
Posts: 9884
Joined: Fri Oct 12, 2012 9:44 pm
Location: Australian Antarctic Territory

Re: Possible to automate data DLs during offpeak periods?

Post by catweazel »

jimallyn wrote:
Wed Jul 04, 2018 5:33 am
Good find, actually. I just installed it and will try it out in the next day or two.
There are XDM add-ons available to allow downloading from various browsers.
"There is, ultimately, only one truth -- cogito, ergo sum -- everything else is an assumption." - Me, my swansong.
User avatar
jimallyn
Level 18
Level 18
Posts: 8959
Joined: Thu Jun 05, 2014 7:34 pm
Location: Wenatchee, WA USA

Re: Possible to automate data DLs during offpeak periods?

Post by jimallyn »

catweazel wrote:
Wed Jul 04, 2018 5:38 am
There are XDM add-ons available to allow downloading from various browsers.
I installed the add-on for Firefox, and it's telling me there's a video available to download. Pretty cool.
Image

“If the government were coming for your TVs and cars, then you'd be upset. But, as it is, they're only coming for your sons.” - Daniel Berrigan
User avatar
trytip
Level 13
Level 13
Posts: 4928
Joined: Tue Jul 05, 2016 1:20 pm

Re: Possible to automate data DLs during offpeak periods?

Post by trytip »

when i was on limited data, i would always select youtube low quality 240 is fine and used clipgrab if needed download. disable any autostart updates or checking i do anyways. weather widgets also connect to internet quite often. i used conky to monitor how much is downloaded/uploaded but system monitor can also tell you that
Image
lazarus
Level 3
Level 3
Posts: 153
Joined: Mon Jul 02, 2018 11:36 pm
Location: Oberon, OZ

Re: Possible to automate data DLs during offpeak periods?

Post by lazarus »

Thanks fellas, now I know how to move in the right direction.

I've installed XDM to give the GUI-deprived members of the household something to play with; the provided install info was a tad outdated (a 2015 version) but it got the ball rolling.

In the meantime I'll try fiddling with a few of the other suggestions, even if only to get a feel for what's possible and refresh long disused grey cells. I suspect I'll end up with a wget/at/curl + cron set up for my own usage.
gm10 wrote:
Wed Jul 04, 2018 4:53 am
- Remove the update manager from the startup applications, set up a cron job for mintupdate-cli instead to run whenever you want it to.
This! :) This is where I was stuck as I couldn't see a way to script Update Mgr. beyond it's own in-built "every x min/hour/days." Thanks again.
- Andy

I may be weird but I'm saving up to become eccentric.
Post Reply

Return to “Newbie Questions”