Using wget on https

[butiflauta]

Hi friends, I'm trying to download ts files (from a live concert webcast) with wget in https website using the instruction:

Code: Select all

wget «--no-check-certificate» https://thenameofwebsite/{01..643}.ts

I'm trying also with:

Code: Select all

wget --secure-protocol=auto https://thenameofwebsite/{01..643}.ts

In both cases appears error. The problem is about the certificate of the website. Anybody can help? I prefer not to mention the website but I can do it if necessary.

[gm10]

use --no-check-certificate

[Flemur]

I almost always use my own 'Wget' ...

cat ~/bin/Wget

Code: Select all

wget -c --no-check-certificate "$@"

[butiflauta]

use --no-check-certificate

Using this appear ERROR: 403 Forbidden

[butiflauta]

I almost always use my own 'Wget' ...

cat ~/bin/Wget

Code: Select all

wget -c --no-check-certificate "$@"

Literally I write

Code: Select all

wget -c --no-check-certificate "$@" https://thenameofthewebsite/{01..643}.ts

The result is ERROR: 403 Forbidden

[Flemur]

[

Literally I write

Code: Select all

wget -c --no-check-certificate "$@" https://thenameofthewebsite/{01..643}.ts

The result is ERROR: 403 Forbidden

I showed the script file that you'd run, in a terminal like this:

Code: Select all

Wget URL

But your problem seems to be: https://thenameofthewebsite/ Perhaps you need the real name of the website.

Edit: https isn't a problem, e.g.:

Code: Select all

wget https://forums.linuxmint.com

[gm10]

use --no-check-certificate

Using this appear ERROR: 403 Forbidden

sounds like you've got the wrong URL or the server doesn't allow you to download just like that (referrer check, etc.).

[phd21]

Hi butiflauta,

I just read your post and the good replies to it. Here are my thoughts on this as well.

Can you please give us the exact URL so we can try it as well. Also, in Linux Mint 19.x there is a "wget2".

Why not try "youtube-dl" to download the video files?

Not sure if installing this may help, but I always install it.

Code: Select all

sudo apt-get install apt-transport-https

How to Download a File on Ubuntu Linux using the Command Line
- various options including wget.
https://vitux.com/how-to-download-a-fil ... mand-line/

Top 10 command line tools for downloading in Linux | Unixmen
https://www.unixmen.com/top-10-command- ... ing-linux/

I know that when I try to download BBC concerts or shows I have had to use "iplayer" and sometimes (most of the time) I have to be connected a VPN server in the UK first.

Download YouTube Videos in Linux Command Line (youtube-dl)
https://itsfoss.com/download-youtube-linux/

Program to download BBC shows
unixpkg · get-iplayer/get_iplayer Wiki
https://github.com/get-iplayer/get_iplayer/wiki/unixpkg

giPlayer - desktop GUI for get-iplayer, but I usually use the console terminal program.
https://sourceforge.net/projects/giplayer/files/

FYI Note: I just installed and tested this Nuvola BBC iPlayer app and it works, I had to connect to a UK VPN server and register with a UK postal code, and say yes to "do you have a license to watch tv?"??? FYI: postcode of the University of Roehampton in London is "SW15 5PU". Then with the app closed, I went to the BBC iPlayer website in my Chrome browser and it worked as well without installing the Nuvola BBC iPlayer app. So I started to play a TV show from their website, copied the URL and used "youtube-dl" to download the show and after a bit of "negotiating", it started to download the video (I was still connected to the UK VPN server).

Install BBC iPlayer in Ubuntu 18.04 LTS Bionic Beaver • Nuvola Apps
https://nuvola.tiliado.eu/app/bbc_iplay ... tu/bionic/

Install Nuvola Apps Runtime on Ubuntu 18.04 LTS Bionic Beaver • Nuvola Apps including BBC iplayer app.
https://nuvola.tiliado.eu/nuvola/ubuntu/bionic/

Hope this helps ...

[butiflauta]

Hi butiflauta,

I just read your post and the good replies to it. Here are my thoughts on this as well.

Can you please give us the exact URL so we can try it as well. Also, in Linux Mint 19.x there is a "wget2".

Why not try "youtube-dl" to download the video files?

Not sure if installing this may help, but I always install it.

Code: Select all

sudo apt-get install apt-transport-https

How to Download a File on Ubuntu Linux using the Command Line
- various options including wget.
https://vitux.com/how-to-download-a-fil ... mand-line/

Top 10 command line tools for downloading in Linux | Unixmen
https://www.unixmen.com/top-10-command- ... ing-linux/

I know that when I try to download BBC concerts or shows I have had to use "iplayer" and sometimes (most of the time) I have to be connected a VPN server in the UK first.

Download YouTube Videos in Linux Command Line (youtube-dl)
https://itsfoss.com/download-youtube-linux/

Program to download BBC shows
unixpkg · get-iplayer/get_iplayer Wiki
https://github.com/get-iplayer/get_iplayer/wiki/unixpkg

giPlayer - desktop GUI for get-iplayer, but I usually use the console terminal program.
https://sourceforge.net/projects/giplayer/files/

FYI Note: I just installed and tested this Nuvola BBC iPlayer app and it works, I had to connect to a UK VPN server and register with a UK postal code, and say yes to "do you have a license to watch tv?"??? FYI: postcode of the University of Roehampton in London is "SW15 5PU". Then with the app closed, I went to the BBC iPlayer website in my Chrome browser and it worked as well without installing the Nuvola BBC iPlayer app. So I started to play a TV show from their website, copied the URL and used "youtube-dl" to download the show and after a bit of "negotiating", it started to download the video (I was still connected to the UK VPN server).

Install BBC iPlayer in Ubuntu 18.04 LTS Bionic Beaver • Nuvola Apps
https://nuvola.tiliado.eu/app/bbc_iplay ... tu/bionic/

Install Nuvola Apps Runtime on Ubuntu 18.04 LTS Bionic Beaver • Nuvola Apps including BBC iplayer app.
https://nuvola.tiliado.eu/nuvola/ubuntu/bionic/

Hope this helps ...

Thanks for your answer. What I want is to download some of the videos of http://vod.graspop.be/nl, for example the first one is Carcass concert. If you try with the url that appears on browser (http://vod.graspop.be/nl/39018) the result is 404 Not Found. Finally I found the url https://paille1.streaming.proximustv.be ... ll/01-7.ts which 01-7.ts is only one of the files to download of 1,22MB that starts in this one and ends at 01-643.ts

I guess the only way to download this or other concerts is download ts files and join after downloaded.

[phd21]

Hi butiflauta,

You are welcome...

I guess the only way to download this or other concerts is to download ts files and join after downloaded.

That would depend on the website and the content provider. If they only provide the videos in segments (pieces), then yes, you would probably have to join them after downloading them which can easily be done using video editors, losslesscut, vidcutter, various console terminal commands, other apps.

If you can play the videos through your browser, then you might consider installing "XDM" Xtreme Download Manager (has Linux software and browser components), or "uGet", etc... these can download videos and or audio files as well.

I tried "youtube-dl" with the links you provided but they did not work, perhaps I have to be connected to a VPN server in that area first. Are you located in the same country as that website?

FYI: It is not necessary to keep the text when you reply to someone's post or reply unless you are making "point for point" comments, so to reduce duplication, please delete the quoted text before you submit your reply.

Hope this helps ...

[butiflauta]

If you can play the videos through your browser, then you might consider installing "XDM" Xtreme Download Manager (has Linux software and browser components), or "uGet", etc... these can download videos and or audio files as well.

I don't know about Xtreme Download Manager, I search on Firefox complements and nothing appears. I will try installing from http://xdman.sourceforge.net/

I tried "youtube-dl" with the links you provided but they did not work, perhaps I have to be connected to a VPN server in that area first. Are you located in the same country as that website?

No, I'm in a different country

[gm10]

Thanks for your answer. What I want is to download some of the videos of http://vod.graspop.be/nl, for example the first one is Carcass concert. If you try with the url that appears on browser (http://vod.graspop.be/nl/39018) the result is 404 Not Found. Finally I found the url https://paille1.streaming.proximustv.be ... ll/01-7.ts which 01-7.ts is only one of the files to download of 1,22MB that starts in this one and ends at 01-643.ts

I guess the only way to download this or other concerts is download ts files and join after downloaded.

Well, then it's simple, this would download your video:

Code: Select all

wget -nv --no-check-certificate https://paille1.streaming.proximustv.be/session/27f681a6-b20d-11e9-8195-005056bd90b1/867r6f/shares/export/mounthttp/festivals/gmm19-concert-carcass-full/01-{1..643}.ts --output-document=output.ts

Note the {1..643} defining the parts you want to download and the output filename --output-document=output.ts. Also looking at that URL you'll likely need to request a new session id first (i.e. grab a new URL via your browser - could be automated if you're doing this often but w/e).

PS: You can add --quiet so you do not get spammed 643 times by wget.

[gm10]

Not sure if installing this may help, but I always install it.

Code: Select all

sudo apt-get install apt-transport-https

I don't know what that's got to do with anything (apt is the package management system), but just FYI, that package's description says:

Code: Select all

 This is a dummy transitional package - https support has been moved into
 the apt package in 1.5. It can be safely removed.

In other words, installing this does exactly nothing.

[phd21]

Hi butiflauta,

You did not state whether or not you could view the video in your browser. I could not access the video until I connected to VPN server in the Netherlands (NL) which I did use a browser VPN extension add-on to do that, then the website popped right up and I could play the video.

The video you want for the heavy metal band "Carcass" also appears to be one video 1 hour and 4 minutes long.

Highly recommend installing these packages using the command below or from the "Synaptic Package Manager (SPM)" if you have not already done so.

Code: Select all

sudo apt install ffmpeg x264 x264 h264enc mencoder mplayer smplayer mpv mediainfo mediainfo-gui openjdk-8-jre openjfx

Make Java 8 the default Java runtime using the command below

Code: Select all

sudo update-alternatives --config java

"XDM" had no problem downloading the video, see screenshots below and provided various resolution options (higher size number usually means higher resolution) and you have an option to convert the original ".ts" to another format like MP4 using XDM (download was fast but conversion part was very slow on my ancient underpowered computer system) or keep the original .ts video format and convert that afterward which was much faster. FYI: "MKVToolNix GUI" converted this "Graspop Metal Meeting - livestream_1.ts" widescreen (16:9) video file (1280x720 at 25fps, 805.7mb) into an "Graspop Metal Meeting - livestream_1.mkv" (705.5mb) video file in less than 60 seconds.

MKVToolNix Downloads – Matroska tools for Linux/Unix and Windows
https://mkvtoolnix.download/downloads.html

XDM is not hard to install and works great. For the browser component install an extension add-on "XDM 2018" or newer not the older one, and maybe restart browser.
How to install and use Xtreme download manager on Ubuntu 17.04 | LinuxHelp Tutorials
https://www.linuxhelp.com/how-to-instal ... untu-17-04
.
.
@gm10, I tried your wget command and it did not work for me. I will try it again after connecting to a VPN in NL.

And I agree that installing the apt-transport-https was not necessary in this scenario although recommended in many other software installation instructions. I was actually answering more than one member's requests while answering this one and one of those recommended installing this.

Ubuntu Manpage: apt-transport-https - APT transport for downloading via the HTTP Secure protocol (HTTPS)
http://manpages.ubuntu.com/manpages/bio ... tps.1.html

Hope this helps ...
.
Animated Gif video of XDM browser webpage video download - click image to view larger size

.

[gm10]

@gm10, I tried your wget command and it did not work for me. I will try it again after connecting to a VPN in NL.

Did you get your own session ID? As I mentioned, that URL isn't copy & pasteable.

And I agree that installing the apt-transport-https was not necessary in this scenario although recommended in many other software installation instructions. I was actually answering more than one member's requests while answering this one and one of those recommended installing this.

All outdated instructions then, as I said, that package has no content on bionic / LM 19.x. Also irrelevant for LM 18.x and LMDE3 because the package is pre-installed there.

[butiflauta]

Also looking at that URL you'll likely need to request a new session id first (i.e. grab a new URL via your browser - could be automated if you're doing this often but w/e).

Sorry but I don't understand exactly what you mean, the website don't need identification. Maybe you mean new ip? or delete cookies?

PS: You can add --quiet so you do not get spammed 643 times by wget.

I try the orders what you mention and the result is with --quiet ERROR: 403 Forbidden and without just a ts files of 0 bytes, nothing

And... do you think maybe it will work with a live stream? This week will be live broadcast that I'm interest in this website https://www.magenta-musik-360.de/ and if would like to capture. It's german website, I'm located in Catalonia (Spain) but I can full access.

Sorry about my misunderstanding but english it's not my mother language but I try my best.

Thank you.

[gm10]

Also looking at that URL you'll likely need to request a new session id first (i.e. grab a new URL via your browser - could be automated if you're doing this often but w/e).

Sorry but I don't understand exactly what you mean, the website don't need identification. Maybe you mean new ip? or delete cookies?

No, just refresh the website in your browser and grab a new URL because it's only a temporary link that you've been given. That's why you're getting the 403, your URL isn't valid anymore.

I cannot tell you whether that future livestream will use the same technology, I've never been to that site.

[butiflauta]

No, just refresh the website in your browser and grab a new URL because it's only a temporary link that you've been given. That's why you're getting the 403, your URL isn't valid anymore.

Understood, I thought it was same url all the time. The thing is that I can download a single ts file with:

Code: Select all

wget -nv --quiet --no-check-certificate https://marais1.streaming.proximustv.be/session/7ae73dbe-b2d1-11e9-a9eb-0050569e6b3d/867r6f/shares/export/mounthttp/festivals/gmm19-concert-carcass-full/01-0.ts --output-document=carcass.ts

But not all the files with one order for example:

Code: Select all

wget -nv --quiet --no-check-certificate https://marais1.streaming.proximustv.be/session/7ae73dbe-b2d1-11e9-a9eb-0050569e6b3d/867r6f/shares/export/mounthttp/festivals/gmm19-concert-carcass-full/{01-0..01-653}.ts --output-document=carcass.ts

I suppose it's not the correct order but I don't know which is the correct.

[gm10]

but not all the files with one order for example:

Code: Select all

wget -nv --quiet --no-check-certificate https://marais1.streaming.proximustv.be/session/7ae73dbe-b2d1-11e9-a9eb-0050569e6b3d/867r6f/shares/export/mounthttp/festivals/gmm19-concert-carcass-full/{01-0..01-653}.ts --output-document=carcass.ts

I suppose it's not the correct order but I don't know which is the correct.

You just mistyped the enumerator, you have:
.../{01-0..01-653}.ts --output-document=carcass.ts
but it should be like in my example:
.../01-{1..653}.ts --output-document=carcass.ts
You must only put the number that need to be counted up into the {} brackets.

[butiflauta]

Hi butiflauta,

You did not state whether or not you could view the video in your browser. I could not access the video until I connected to VPN server in the Netherlands (NL) which I did use a browser VPN extension add-on to do that, then the website popped right up and I could play the video.

I don't need vpn to access to the videos in my case.

Certainly xdm works perfect. I installed and can download the full concert. I installed MKVToolnix to convert to mkv, thank you for your help and other friends of the forum.
I will try this week if xdm works on live stream that will be on https://www.magenta-musik-360.de/, I post if it works