Can ffmpeg record video from this kind of URL?
I would like to record MSNBC news streams for later viewing, but I cannot figure out how I should make that happen using ffmpeg.... Here is a webpage URL I found by googling and on which there is a video player streaming the MSNBC video: https://www.livenewsmag.com/msnbc-news-live-stream/ I have tried to use F12 in FireFox in order to be able to extract a video url, but failed to find anything (not really used to using the F12 view). About half a year ago I had a similar problem and then someone here came up with this to be used in a Linux script: "ffmpeg -hide_banner -user_agent \"Mozilla\" -i $VIDEOURL -t $CAPTURETIME -c copy $TARGETFILE" The variables VIDEOURL , CAPTURETIME and TARGETFILE are preset to the URL extracted via F12 on the webpage that streamed the video, the length of the extraction and the output file. And the extracted video URL used looked like this (obfuscated domain): https://hlss.somedomain.com/hls/186379.m3u8 This worked until they changed that webpage and removed the player... So now I would like to get the same system going on the newly found webpage shown above. Any ideas? -- Bo Berglund Developer in Sweden
On 2021-08-29T17:10:29+0200, Bo Berglund <bo.berglund@gmail.com> wrote:
https://www.livenewsmag.com/msnbc-news-live-stream/
I have tried to use F12 in FireFox in order to be able to extract a video url, but failed to find anything (not really used to using the F12 view). This one?
$ xidel -s "https://www.livenewsmag.com/msnbc-news-live-stream/" -e ' parse-json( //script/extract(.,"playerInstance.setup\((.+)\)",1,"s"), {"liberal":true()} ) ' { "file": "https://1420543146.rsc.cdn77.org/V8uAdAW-VTHZeQ0E-rxccA==,1630271176/LS-ATL-...", "autostart": false, "width": "100%", "aspectratio": "16:10", "androidhls": false, "hlsjsdefault": true, "logo": { "file": "/wp-content/uploads/2016/10/live-news-logo-1.jpg", "hide": "true", "link": "http://www.livenewson.com/" }, "abouttext": "LiveNewsOn", "aboutlink": "http://www.livenewson.com", "cast": {} }
From where I am I don't think I'm allowed to access the stream.
$ ffmpeg -hide_banner -i "$(xidel -s "https://www.livenewsmag.com/msnbc-news-live-stream/" -e 'parse-json(//script/extract(.,"playerInstance.setup\((.+)\)",1,"s"),{"liberal":true()})/file')" [https @ 04f49bc0] HTTP error 403 Forbidden https://1420543146.rsc.cdn77.org/V8uAdAW-VTHZeQ0E-rxccA==,1630271176/LS-ATL-...: Server returned 403 Forbidden (access denied) -- Reino
On Sun, 29 Aug 2021 17:41:49 +0200, Reino Wijnsma <rwijnsma@xs4all.nl> wrote:
From where I am I don't think I'm allowed to access the stream.
Yes, it seems like you are only allowed to *view* the video through your browser embedded into the full page... That is not good for me due to the very big time difference, I need to download for later viewing. So I have to wait until the original stream I have used is fixed. Thanks. -- Bo Berglund Developer in Sweden
On Mon, 30 Aug 2021 08:20:38 +0200 Bo Berglund <bo.berglund@gmail.com> wrote:
On Sun, 29 Aug 2021 17:41:49 +0200, Reino Wijnsma <rwijnsma@xs4all.nl> wrote:
From where I am I don't think I'm allowed to access the stream.
Yes, it seems like you are only allowed to *view* the video through your browser embedded into the full page...
You just need to set "Referer" header in your http request to url of page that embeds that video. ffmpeg -referer "https://www.livenewsmag.com/msnbc-news-live-stream" -i "https://1420543146.rsc.cdn77.org/J2SMCND-re6j5PPYbi2Hiw==,1630371979/LS-ATL-..." -c copy msnbc-rec-1.mp4
On 2021-08-30T21:56:49+0200, Anatoly <anatoly@kazanfieldhockey.ru> wrote:
You just need to set "Referer" header in your http request to url of page that embeds that video. ffmpeg -referer "https://www.livenewsmag.com/msnbc-news-live-stream" -i "https://1420543146.rsc.cdn77.org/J2SMCND-re6j5PPYbi2Hiw==,1630371979/LS-ATL-..." -c copy msnbc-rec-1.mp4 I can confirm that this works. Thanks.
-- Reino
On Mon, 30 Aug 2021 22:56:49 +0300, Anatoly <anatoly@kazanfieldhockey.ru> wrote:
On Mon, 30 Aug 2021 08:20:38 +0200 Bo Berglund <bo.berglund@gmail.com> wrote:
On Sun, 29 Aug 2021 17:41:49 +0200, Reino Wijnsma <rwijnsma@xs4all.nl> wrote:
From where I am I don't think I'm allowed to access the stream.
Yes, it seems like you are only allowed to *view* the video through your browser embedded into the full page...
You just need to set "Referer" header in your http request to url of page that embeds that video. ffmpeg -referer "https://www.livenewsmag.com/msnbc-news-live-stream" -i "https://1420543146.rsc.cdn77.org/J2SMCND-re6j5PPYbi2Hiw==,1630371979/LS-ATL-..." -c copy msnbc-rec-1.mp4
THANK YOU VERY MUCH for this comment! The -referer argument solved the problem of allowing access to the m3u8 stream. Related question: ----------------- How do I extract the proper m3u8 URL from the webpage in the best/safest way? Some of these pages display a scrolling long list of items when one presses the F12 function key in FireFox and it is not obvious to me what I should try. And there are also URL:s ending with .ts in that scrolling list, but these change all the time so they are probably just short sections of the video, right? Is there a command line call I can make to extract the m3u8 URL automatically so it can be used in a script only supplied the main page URL? -- Bo Berglund Developer in Sweden
Related question: ----------------- How do I extract the proper m3u8 URL from the webpage in the best/safest way? Some of these pages display a scrolling long list of items when one presses the F12 function key in FireFox and it is not obvious to me what I should try. And there are also URL:s ending with .ts in that scrolling list, but these change all the time so they are probably just short sections of the video, right?
Right, you need m3u8. F12->Network->Enter "m3u8" in the textbox above the list to filter out.
Is there a command line call I can make to extract the m3u8 URL automatically so it can be used in a script only supplied the main page URL? To get url of the first m3u8 from the page (this page actually has only one) curl "https://www.livenewsmag.com/msnbc-news-live-stream/" | grep -o -e "https://.\+m3u8" | head -n 1
-- Bo Berglund Developer in Sweden
_______________________________________________ ffmpeg-user mailing list ffmpeg-user@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-user
To unsubscribe, visit link above, or email ffmpeg-user-request@ffmpeg.org with subject "unsubscribe".
On Wed, 1 Sep 2021 16:30:00 +0300, anatoly@kazanfieldhockey.ru wrote:
Is there a command line call I can make to extract the m3u8 URL automatically so it can be used in a script only supplied the main page URL? To get url of the first m3u8 from the page (this page actually has only one) curl "https://www.livenewsmag.com/msnbc-news-live-stream/" | grep -o -e "https://.\+m3u8" | head -n 1
Thanks, this works inasmuch as it shows the URL on screen from where I can copy the URL manually. However it looks like this when executed: $ curl "https://www.livenewsmag.com/msnbc-news-live-stream/" | grep -o -e "https://.\+m3u8" | head -n 1 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 86975 100 86975 0 0 375k 0 --:--:-- --:--:-- --:--:-- 374k https://1420543146.rsc.cdn77.org/tsOQo0dAAnGr5jKtXe_k5g==,1630526780/LS-ATL-... So curl is outputting stuff during its probe that is also on the terminal output... But changing it to this: $ curl -s "https://www.livenewsmag.com/msnbc-news-live-stream/" | grep -o -e "https://.\+m3u8" | head -n 1 removes all that fluff and I get exactly the end result! I have now cereated a script, which updates the URL-file with the current items for the given URL. This can be called before actually using the video URL for download making sure the m3u8 URL is current. Thanks again! -- Bo Berglund Developer in Sweden
On Wed, 01 Sep 2021 19:00:52 +0200 Bo Berglund <bo.berglund@gmail.com> wrote:
On Wed, 1 Sep 2021 16:30:00 +0300, anatoly@kazanfieldhockey.ru wrote:
Is there a command line call I can make to extract the m3u8 URL automatically so it can be used in a script only supplied the main page URL? To get url of the first m3u8 from the page (this page actually has only one) curl "https://www.livenewsmag.com/msnbc-news-live-stream/" | grep -o -e "https://.\+m3u8" | head -n 1
Thanks, this works inasmuch as it shows the URL on screen from where I can copy the URL manually.
However it looks like this when executed:
$ curl "https://www.livenewsmag.com/msnbc-news-live-stream/" | grep -o -e "https://.\+m3u8" | head -n 1 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 86975 100 86975 0 0 375k 0 --:--:-- --:--:-- --:--:-- 374k https://1420543146.rsc.cdn77.org/tsOQo0dAAnGr5jKtXe_k5g==,1630526780/LS-ATL-...
So curl is outputting stuff during its probe that is also on the terminal output... Extra stuff is otput to stderr, you see it in terminal, but it isn't subject to '>' redirection or `` capture. But changing it to this:
$ curl -s "https://www.livenewsmag.com/msnbc-news-live-stream/" | grep -o -e "https://.\+m3u8" | head -n 1
removes all that fluff and I get exactly the end result!
I have now cereated a script, which updates the URL-file with the current items for the given URL. Actually, you may do all in one script like this #!/bin/sh msnurl="https://www.livenewsmag.com/msnbc-news-live-stream/" recdate=`date "+%Y%m%d-%H%M"` m3u8url=`curl "https://www.livenewsmag.com/msnbc-news-live-stream/" | grep -o -e "https://.\+m3u8" | head -n 1` ffmpeg -referer $msnurl -i $m3u8url -c copy "msnbc-${recdate}.mp4"
On Thu, 2 Sep 2021 11:38:06 +0300, Anatoly <anatoly@kazanfieldhockey.ru> wrote:
I have now cereated a script, which updates the URL-file with the current items for the given URL. Actually, you may do all in one script like this #!/bin/sh msnurl="https://www.livenewsmag.com/msnbc-news-live-stream/" recdate=`date "+%Y%m%d-%H%M"` m3u8url=`curl "https://www.livenewsmag.com/msnbc-news-live-stream/" | grep -o -e "https://.\+m3u8" | head -n 1` ffmpeg -referer $msnurl -i $m3u8url -c copy "msnbc-${recdate}.mp4"
I have designed my script so it writes the page URL and m3u8-URL into the url-file read by my main download script. This main script does a number of other things too, like reformatting the video to a fixed resolution and quality. In order to secure the validity of the m3u8 I have set up a cron job that runs the script a few ninutes before the download script starts. So it uses a fresh version of the URL.... It happens a bit too often that one source goes bad but I have found 3 stream pages with the same type of data so I can change one argument to my script to switch the main URL (all 3 are present in the script), but right now that only works if I detect that a problem has arisen and edit the crontab entry.. -- Bo Berglund Developer in Sweden
On Wed, 1 Sep 2021 16:30:00 +0300, anatoly@kazanfieldhockey.ru wrote:
Right, you need m3u8. F12->Network->Enter "m3u8" in the textbox above the list to filter out.
Is there a command line call I can make to extract the m3u8 URL automatically so it can be used in a script only supplied the main page URL? To get url of the first m3u8 from the page (this page actually has only one) curl "https://www.livenewsmag.com/msnbc-news-live-stream/" | grep -o -e "https://.\+m3u8" | head -n 1
Coming back to this issue again since the command you suggested has stopped working... I am trying this command and I get an empty result: $ curl -s "https://msnbcdailyshows.com/" | grep -o -e "https://.\+m3u8" | head -n 1 The page above does have an embedded video player but your command for extraction of the m3u8 URL returns an *empty* output... But it does work on other such pages like this: $ curl -s "https://www.livenewsmag.com/msnbc-news-live-stream" | grep -o -e "https://.\+m3u8" | head -n 1 https://1420543146.rsc.cdn77.org/Z5XPzYjFisC9SrZHwPgDUg==,1633615580/LS-ATL-... Why is the command working on one URL but not the other? -- Bo Berglund Developer in Sweden
On Thu, 07 Oct 2021 10:27:51 +0200 Bo Berglund <bo.berglund@gmail.com> wrote:
On Wed, 1 Sep 2021 16:30:00 +0300, anatoly@kazanfieldhockey.ru wrote:
Right, you need m3u8. F12->Network->Enter "m3u8" in the textbox above the list to filter out.
Is there a command line call I can make to extract the m3u8 URL automatically so it can be used in a script only supplied the main page URL? To get url of the first m3u8 from the page (this page actually has only one) curl "https://www.livenewsmag.com/msnbc-news-live-stream/" | grep -o -e "https://.\+m3u8" | head -n 1
Coming back to this issue again since the command you suggested has stopped working... I am trying this command and I get an empty result:
$ curl -s "https://msnbcdailyshows.com/" | grep -o -e "https://.\+m3u8" | head -n 1
The page above does have an embedded video player but your command for extraction of the m3u8 URL returns an *empty* output...
But it does work on other such pages like this:
$ curl -s "https://www.livenewsmag.com/msnbc-news-live-stream" | grep -o -e "https://.\+m3u8" | head -n 1 https://1420543146.rsc.cdn77.org/Z5XPzYjFisC9SrZHwPgDUg==,1633615580/LS-ATL-...
Why is the command working on one URL but not the other? Because as Moritz Barsnick <barsnick@gmx.net> Wed, 1 Sep 2021 15:37:21 +0200 already told you: "There's often a lot of magic involved in extracting the video URLs from webpages - parsing JavaScript, downloading and parsing JSON files, re-using cookies and referrers, and so on."
So this is very hard, nearly to impossible to create a tool or receipe that will automatically extract video urls on *any* arbitrary page. Last time I just took a quick look at www.livenewsmag.com and found a simple solution for you, because that page is simple enough. Now let's do the same: #!/bin/sh #resolution here may be: prem, 720, 480, 240 resolution="480" sdate=`date "+%Y%m%d-%H%M"` filename="rec-msnbc1-${sdate}.mp4" msn1url="https://msnbcdailyshows.com/" ggid=`curl $msn1url | grep -o -e "https://goodgame\.ru/player?[0-9]\+" | grep -o -e "[0-9]\+"` m3u8url="https://hls.goodgame.ru/hls/${ggid}_${resolution}.m3u8" ffmpeg -user_agent "Mozilla" -i $m3u8url -c copy $filename Also this page contains url of youtube stream of same program, which can be extracted with curl "https://msnbcdailyshows.com/" | grep -o -e "https://youtu.be/[0-9a-zA-Z]\+" and then fed to youtube-dl
On Sat, 9 Oct 2021 03:23:52 +0300, Anatoly <anatoly@kazanfieldhockey.ru> wrote:
Why is the command working on one URL but not the other?
Because as Moritz Barsnick <barsnick@gmx.net> Wed, 1 Sep 2021 15:37:21 +0200 already told you: "There's often a lot of magic involved in extracting the video URLs from webpages - parsing JavaScript, downloading and parsing JSON files, re-using cookies and referrers, and so on."
So this is very hard, nearly to impossible to create a tool or receipe that will automatically extract video urls on *any* arbitrary page. Last time I just took a quick look at www.livenewsmag.com and found a simple solution for you, because that page is simple enough. Now let's do the same:
#!/bin/sh #resolution here may be: prem, 720, 480, 240 resolution="480" sdate=`date "+%Y%m%d-%H%M"` filename="rec-msnbc1-${sdate}.mp4" msn1url="https://msnbcdailyshows.com/" ggid=`curl $msn1url | grep -o -e "https://goodgame\.ru/player?[0-9]\+" | grep -o -e "[0-9]\+"` m3u8url="https://hls.goodgame.ru/hls/${ggid}_${resolution}.m3u8" ffmpeg -user_agent "Mozilla" -i $m3u8url -c copy $filename
Also this page contains url of youtube stream of same program, which can be extracted with curl "https://msnbcdailyshows.com/" | grep -o -e "https://youtu.be/[0-9a-zA-Z]\+" and then fed to youtube-dl
Oh boy! Sorry to be back here again! Now I am caught again! They have changed the format of the stream such that my tools fail again. There appears to be no more any m3u8 stream.... So it is no longer possible to extract by looking for m3u8... If I hit F12 when I display the webpage https://msnbcdailyshows.com/ I see a lot of stuff streaming by on the Networking tab and it seems to be instead of m3u8 some <changing_text>.dash, like this: https://vsd79.mycdn.me/dash/stream_1146644400798/stream.manifest/sig/RcfEI7w... Is it possible to find a working URL for ffmpeg to use in such a situation too? I have tried to check the site using this command: youtube-dl -F https://msnbcdailyshows.com/ After some thinking it returns: [info] Available formats for 2125507010206: format code extension resolution note hls-109 mp4 864x488 109k hls-175 mp4 640x360 175k hls-485 mp4 428x240 485k hls-782 mp4 1280x720 782k (best) So I tried to use youtube-dl to download the video: youtube-dl -f hls-175 -o test.mp4 https://msnbcdailyshows.com/ And it seems like it is doing what it should as if it was a regular youtube streaming video, however I cannot stop it successfully like I can with normal youtube streams by using ctrl-C. And Youtube-dl does not have any argument to set the duration of the download either as far as I can tell. On normal streaming videos youtube-dl closes the video download on receiving Ctrl-C and then exits. And the interrupted video is fixed so it can be played. So I am using a this command structure in my at job to download a specific video: timeout --signal=2 64m youtube-dl -f hls-175 -o xxx.mp4 some-youtube-stream-url And this causes youtube-dl to run for 64 minutes from when it was started and then cleanly close the video and exit. Not so with this one site, here youtube-dl will just abort the download and the xxx.part file remains and cannot be viewed. Is there any possibility to use ffmpeg itself to download from this kind of stream and if so what would the argument for the stream be? -- Bo Berglund Developer in Sweden
On 2021-11-04T01:00:52+0100, Bo Berglund <bo.berglund@gmail.com> wrote:
https://vsd79.mycdn.me/dash/stream_1146644400798/stream.manifest/sig/RcfEI7w...
I hope you realize your IP-address is now out in the open!
youtube-dl -F https://msnbcdailyshows.com/ After some thinking it returns:
[info] Available formats for 2125507010206: format code extension resolution note hls-109 mp4 864x488 109k hls-175 mp4 640x360 175k hls-485 mp4 428x240 485k hls-782 mp4 1280x720 782k (best)
[...]
Is there any possibility to use ffmpeg itself to download from this kind of stream and if so what would the argument for the stream be?
$ ffmpeg -i "$(youtube-dl -g https://www.livenewsmag.com/msnbc-news-live-stream/)" -c copy [...] or with xidel: $ ffmpeg -i "$(xidel -s https://msnbcdailyshows.com/ -f '//iframe/@src' -e 'parse-json(//@data-options)/flashvars/parse-json(metadata)/hlsMasterPlaylistUrl')" -map 0:p:0 -c copy [...] -- Reino
On 2021-11-04T01:40:07+0100, Reino Wijnsma <rwijnsma@xs4all.nl> wrote:
$ ffmpeg -i "$(youtube-dl -g https://www.livenewsmag.com/msnbc-news-live-stream/)" -c copy [...]
That obviously must've been: $ ffmpeg -i "$(youtube-dl -g https://msnbcdailyshows.com/)" -c copy [...] My mistake. -- Reino
On Thu, 4 Nov 2021 01:59:10 +0100, Reino Wijnsma <rwijnsma@xs4all.nl> wrote:
On 2021-11-04T01:40:07+0100, Reino Wijnsma <rwijnsma@xs4all.nl> wrote:
$ ffmpeg -i "$(youtube-dl -g https://www.livenewsmag.com/msnbc-news-live-stream/)" -c copy [...]
That obviously must've been: $ ffmpeg -i "$(youtube-dl -g https://msnbcdailyshows.com/)" -c copy [...]
I assume you mean like this (replace [...] with actual file name): ffmpeg -i "$(youtube-dl -g https://msnbcdailyshows.com/)" -t 60 -c copy test35.mp4 or like this (since it is now possible to get a link to the video by rightclicking the player on the page): ffmpeg -i "$(youtube-dl -g https://ok.ru/video/2125507010206)" -t 60 -c copy test35.mp4 However both commands results in this after the usual ffmpeg (or maybe youtube-dl) output: [https @ 0x55f0277f11c0] HTTP error 403 Forbidden https://vsd79.mycdn.me/hls/1146644400798.m3u8/sig/2d23M3cZ2y0/expires/163609...: Server returned 403 Forbidden (access denied) -- Bo Berglund Developer in Sweden
On 2021-11-04T08:54:40+0100, Bo Berglund <bo.berglund@gmail.com> wrote:
I assume you mean like this (replace [...] with actual file name):
Yes
However both commands results in this after the usual ffmpeg (or maybe youtube-dl) output: [https @ 0x55f0277f11c0] HTTP error 403 Forbidden https://vsd79.mycdn.me/hls/1146644400798.m3u8/sig/2d23M3cZ2y0/expires/163609...: Server returned 403 Forbidden (access denied)
Probably some piece of user-agent code in Youtube-dl that causes "CHROME" to appear in the video-url. The Xidel command I posted earlier returns "UNKNOWN" instead of "CHROME" (and another host) and appears to work. -- Reino
youtube-dl
While youtube-dl still works for many things, there hasn't been a release since June 2021. If you run into issues, you might have better luck with a fork like yt-dlp [1]. Cheers, Rob [1]: https://github.com/yt-dlp/yt-dlp
On Thu, 04 Nov 2021 01:00:52 +0100 Bo Berglund <bo.berglund@gmail.com> wrote:
On normal streaming videos youtube-dl closes the video download on receiving Ctrl-C and then exits. And the interrupted video is fixed so it can be played. So I am using a this command structure in my at job to download a specific video:
timeout --signal=2 64m youtube-dl -f hls-175 -o xxx.mp4 some-youtube-stream-url
And this causes youtube-dl to run for 64 minutes from when it was started and then cleanly close the video and exit.
Not so with this one site, here youtube-dl will just abort the download and the xxx.part file remains and cannot be viewed.
Is there any possibility to use ffmpeg itself to download from this kind of stream and if so what would the argument for the stream be?
This stream comes from "Odnoklassniki" Russian social network, youtube-dl has support for this. And youtube-dl anyway will just execute ffmpeg in turn like: ffmpeg -y -headers <some_http_request_heahers> -i <some_url> -c copy -f mp4 -bsf:a aac_adtstoasc file:<some_name>.part So "xxx.part" file is normal mp4 file. If your file manager/player doesnt know how to open it, being confused by extension, just fix extension to xxx.mp4 manually. On Thu, 4 Nov 2021 12:29:17 +0100 Reino Wijnsma <rwijnsma@xs4all.nl> wrote:
On 2021-11-04T08:54:40+0100, Bo Berglund <bo.berglund@gmail.com> wrote:
I assume you mean like this (replace [...] with actual file name):
Yes
However both commands results in this after the usual ffmpeg (or maybe youtube-dl) output: [https @ 0x55f0277f11c0] HTTP error 403 Forbidden https://vsd79.mycdn.me/hls/1146644400798.m3u8/sig/2d23M3cZ2y0/expires/163609...: Server returned 403 Forbidden (access denied)
Probably some piece of user-agent code in Youtube-dl that causes "CHROME" to appear in the video-url. The Xidel command I posted earlier returns "UNKNOWN" instead of "CHROME" (and another host) and appears to work.
Yes, with version 2021.04.17 of youtube-dl which I have by hand, it works like this: #!/bin/sh url=`youtube-dl -g -f hls-178 "https://msnbcdailyshows.com/"` echo "URL IS: $url" hdrs=`printf "User-Agent: Mozilla/5.0 (Windows NT 10.0; \ Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) \ Chrome/72.0.3626.42 Safari/537.36\r\n\ Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\n\ Accept: text/html,application/xhtml+xml,\ application/xml;q=0.9,*/*;q=0.8\r\n\ Accept-Encoding: gzip, deflate\r\n\ Accept-Language: en-us,en;q=0.5\r\n"` echo "HEADERS: $hdrs" echo "STARTING FFMPEG" /usr/local/bin/ffmpeg -headers "$hdrs" -i "$url" -c copy msn2.mp4
On Sat, 9 Oct 2021 03:23:52 +0300, Anatoly <anatoly@kazanfieldhockey.ru> wrote:
"There's often a lot of magic involved in extracting the video URLs from webpages - parsing JavaScript, downloading and parsing JSON files, re-using cookies and referrers, and so on."
So this is very hard, nearly to impossible to create a tool or receipe that will automatically extract video urls on *any* arbitrary page. Last time I just took a quick look at www.livenewsmag.com and found a simple solution for you, because that page is simple enough. Now let's do the same:
#!/bin/sh #resolution here may be: prem, 720, 480, 240 resolution="480" sdate=`date "+%Y%m%d-%H%M"` filename="rec-msnbc1-${sdate}.mp4" msn1url="https://msnbcdailyshows.com/" ggid=`curl $msn1url | grep -o -e "https://goodgame\.ru/player?[0-9]\+" | grep -o -e "[0-9]\+"` m3u8url="https://hls.goodgame.ru/hls/${ggid}_${resolution}.m3u8" ffmpeg -user_agent "Mozilla" -i $m3u8url -c copy $filename
Thanks for this example, I have now extracted parts of the code and put it into my download script as a way in order to find the m3u8 URL at the beginning of the download: VIDEOURL="https://msnbcdailyshows.com/" ggid=`curl $VIDEOURL | grep -o -e "https://goodgame\.ru/player?[0-9]\+" | grep -o -e "[0-9]\+"` VIDEOSTR="https://hlss.goodgame.ru/hls/${ggid}.m3u8" And then I have used these for ffmpeg as follows: CMD="ffmpeg -hide_banner -user_agent \"Mozilla\" -i $VIDEOSTR -vf scale=w=-4:h=480 -c:v libx264 -preset fast -crf 26 -c:a copy -t $CAPTURETIME $TARGETFILE" eval $CMD I could not use the resolution flag you showed because it did not work properly for me. But the re-formatting code was already done and working so I kept that instead. Thanks again! -- Bo Berglund Developer in Sweden
On Sat, 9 Oct 2021 03:23:52 +0300, Anatoly <anatoly@kazanfieldhockey.ru> wrote:
On Thu, 07 Oct 2021 10:27:51 +0200 Bo Berglund <bo.berglund@gmail.com> wrote:
On Wed, 1 Sep 2021 16:30:00 +0300, anatoly@kazanfieldhockey.ru wrote:
Right, you need m3u8. F12->Network->Enter "m3u8" in the textbox above the list to filter out.
Is there a command line call I can make to extract the m3u8 URL automatically so it can be used in a script only supplied the main page URL? To get url of the first m3u8 from the page (this page actually has only one) curl "https://www.livenewsmag.com/msnbc-news-live-stream/" | grep -o -e "https://.\+m3u8" | head -n 1
Coming back to this issue again since the command you suggested has stopped working... I am trying this command and I get an empty result:
$ curl -s "https://msnbcdailyshows.com/" | grep -o -e "https://.\+m3u8" | head -n 1
The page above does have an embedded video player but your command for extraction of the m3u8 URL returns an *empty* output...
But it does work on other such pages like this:
$ curl -s "https://www.livenewsmag.com/msnbc-news-live-stream" | grep -o -e "https://.\+m3u8" | head -n 1 https://1420543146.rsc.cdn77.org/Z5XPzYjFisC9SrZHwPgDUg==,1633615580/LS-ATL-...
Why is the command working on one URL but not the other?
Because as Moritz Barsnick <barsnick@gmx.net> Wed, 1 Sep 2021 15:37:21 +0200 already told you: "There's often a lot of magic involved in extracting the video URLs from webpages - parsing JavaScript, downloading and parsing JSON files, re-using cookies and referrers, and so on."
So this is very hard, nearly to impossible to create a tool or receipe that will automatically extract video urls on *any* arbitrary page. Last time I just took a quick look at www.livenewsmag.com and found a simple solution for you, because that page is simple enough. Now let's do the same:
#!/bin/sh #resolution here may be: prem, 720, 480, 240 resolution="480" sdate=`date "+%Y%m%d-%H%M"` filename="rec-msnbc1-${sdate}.mp4" msn1url="https://msnbcdailyshows.com/" ggid=`curl $msn1url | grep -o -e "https://goodgame\.ru/player?[0-9]\+" | grep -o -e "[0-9]\+"` m3u8url="https://hls.goodgame.ru/hls/${ggid}_${resolution}.m3u8" ffmpeg -user_agent "Mozilla" -i $m3u8url -c copy $filename
Also this page contains url of youtube stream of same program, which can be extracted with curl "https://msnbcdailyshows.com/" | grep -o -e "https://youtu.be/[0-9a-zA-Z]\+" and then fed to youtube-dl
I return again to this subject because the extraction I have used for a long time now has failed again, probably because part of it is a russian site so they changed to something else... With the war in Ukraine I would very much like to record the MSNBC stream nightly using ffmpeg and it seems impossible now. I have tried my two ffmpeg formats using the page url: https://msnbcdailyshows.com/ and the m3u8 URL I find if I use F12 in the browser when it plays: https://cdn.plt.one/cdn/premium327/chunks.m3u8 But my ffmpeg command does not work with these... Example (on one line): ffmpeg -hide_banner -referer \"https://msnbcdailyshows.com/\" -i https://cdn.plt.one/cdn/premium327/chunks.m3u8 -vf scale=w=-4:h=480 -c:v libx264 -preset fast -crf 26 -c:a copy -t 120 test1.mp4 It fails to actually get any data, so much tghat the timeout does not even trigger... What can be done to get this working again? -- Bo Berglund Developer in Sweden
On 3/4/22, Bo Berglund <bo.berglund@gmail.com> wrote:
On Sat, 9 Oct 2021 03:23:52 +0300, Anatoly <anatoly@kazanfieldhockey.ru> wrote:
On Thu, 07 Oct 2021 10:27:51 +0200 Bo Berglund <bo.berglund@gmail.com> wrote:
On Wed, 1 Sep 2021 16:30:00 +0300, anatoly@kazanfieldhockey.ru wrote:
Right, you need m3u8. F12->Network->Enter "m3u8" in the textbox above the list to filter out.
Is there a command line call I can make to extract the m3u8 URL automatically so it can be used in a script only supplied the main page URL? To get url of the first m3u8 from the page (this page actually has only one) curl "https://www.livenewsmag.com/msnbc-news-live-stream/" | grep -o -e "https://.\+m3u8" | head -n 1
Coming back to this issue again since the command you suggested has stopped working... I am trying this command and I get an empty result:
$ curl -s "https://msnbcdailyshows.com/" | grep -o -e "https://.\+m3u8" | head -n 1
The page above does have an embedded video player but your command for extraction of the m3u8 URL returns an *empty* output...
But it does work on other such pages like this:
$ curl -s "https://www.livenewsmag.com/msnbc-news-live-stream" | grep -o -e "https://.\+m3u8" | head -n 1 https://1420543146.rsc.cdn77.org/Z5XPzYjFisC9SrZHwPgDUg==,1633615580/LS-ATL-...
Why is the command working on one URL but not the other?
Because as Moritz Barsnick <barsnick@gmx.net> Wed, 1 Sep 2021 15:37:21 +0200 already told you: "There's often a lot of magic involved in extracting the video URLs from webpages - parsing JavaScript, downloading and parsing JSON files, re-using cookies and referrers, and so on."
So this is very hard, nearly to impossible to create a tool or receipe that will automatically extract video urls on *any* arbitrary page. Last time I just took a quick look at www.livenewsmag.com and found a simple solution for you, because that page is simple enough. Now let's do the same:
#!/bin/sh #resolution here may be: prem, 720, 480, 240 resolution="480" sdate=`date "+%Y%m%d-%H%M"` filename="rec-msnbc1-${sdate}.mp4" msn1url="https://msnbcdailyshows.com/" ggid=`curl $msn1url | grep -o -e "https://goodgame\.ru/player?[0-9]\+" | grep -o -e "[0-9]\+"` m3u8url="https://hls.goodgame.ru/hls/${ggid}_${resolution}.m3u8" ffmpeg -user_agent "Mozilla" -i $m3u8url -c copy $filename
Also this page contains url of youtube stream of same program, which can be extracted with curl "https://msnbcdailyshows.com/" | grep -o -e "https://youtu.be/[0-9a-zA-Z]\+" and then fed to youtube-dl
I return again to this subject because the extraction I have used for a long time now has failed again, probably because part of it is a russian site so they changed to something else...
With the war in Ukraine I would very much like to record the MSNBC stream nightly using ffmpeg and it seems impossible now. I have tried my two ffmpeg formats using the page url:
and the m3u8 URL I find if I use F12 in the browser when it plays:
https://cdn.plt.one/cdn/premium327/chunks.m3u8
But my ffmpeg command does not work with these... Example (on one line):
ffmpeg -hide_banner -referer \"https://msnbcdailyshows.com/\" -i https://cdn.plt.one/cdn/premium327/chunks.m3u8 -vf scale=w=-4:h=480 -c:v libx264 -preset fast -crf 26 -c:a copy -t 120 test1.mp4
It fails to actually get any data, so much tghat the timeout does not even trigger...
What can be done to get this working again?
It is premium content for users that pay.
-- Bo Berglund Developer in Sweden
_______________________________________________ ffmpeg-user mailing list ffmpeg-user@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-user
To unsubscribe, visit link above, or email ffmpeg-user-request@ffmpeg.org with subject "unsubscribe".
This is not an ffmpeg solution, but the Internet Archive has a large repository of TV News content that includes 24/7 MSNBCfeeds: https://archive.org/details/tv?q=MSNBC For your consideration. Best, - Dan On Fri, Mar 4, 2022 at 2:44 PM Paul B Mahol <onemda@gmail.com> wrote:
On 3/4/22, Bo Berglund <bo.berglund@gmail.com> wrote:
On Sat, 9 Oct 2021 03:23:52 +0300, Anatoly <anatoly@kazanfieldhockey.ru> wrote:
On Thu, 07 Oct 2021 10:27:51 +0200 Bo Berglund <bo.berglund@gmail.com> wrote:
On Wed, 1 Sep 2021 16:30:00 +0300, anatoly@kazanfieldhockey.ru wrote:
Right, you need m3u8. F12->Network->Enter "m3u8" in the textbox above the list to filter out.
Is there a command line call I can make to extract the m3u8 URL automatically so it can be used in a script only supplied the main page URL? To get url of the first m3u8 from the page (this page actually has only one) curl "https://www.livenewsmag.com/msnbc-news-live-stream/" | grep -o -e "https://.\+m3u8" | head -n 1
Coming back to this issue again since the command you suggested has stopped working... I am trying this command and I get an empty result:
$ curl -s "https://msnbcdailyshows.com/" | grep -o -e "https://.\+m3u8" | head -n 1
The page above does have an embedded video player but your command for extraction of the m3u8 URL returns an *empty* output...
But it does work on other such pages like this:
$ curl -s "https://www.livenewsmag.com/msnbc-news-live-stream" | grep -o -e "https://.\+m3u8" | head -n 1
https://1420543146.rsc.cdn77.org/Z5XPzYjFisC9SrZHwPgDUg==,1633615580/LS-ATL-...
Why is the command working on one URL but not the other?
Because as Moritz Barsnick <barsnick@gmx.net> Wed, 1 Sep 2021 15:37:21 +0200 already told you: "There's often a lot of magic involved in extracting the video URLs from webpages - parsing JavaScript, downloading and parsing JSON files, re-using cookies and referrers, and so on."
So this is very hard, nearly to impossible to create a tool or receipe that will automatically extract video urls on *any* arbitrary page. Last time I just took a quick look at www.livenewsmag.com and found a simple solution for you, because that page is simple enough. Now let's do the same:
#!/bin/sh #resolution here may be: prem, 720, 480, 240 resolution="480" sdate=`date "+%Y%m%d-%H%M"` filename="rec-msnbc1-${sdate}.mp4" msn1url="https://msnbcdailyshows.com/" ggid=`curl $msn1url | grep -o -e "https://goodgame\.ru/player?[0-9]\+" | grep -o -e "[0-9]\+"` m3u8url="https://hls.goodgame.ru/hls/${ggid}_${resolution}.m3u8" ffmpeg -user_agent "Mozilla" -i $m3u8url -c copy $filename
Also this page contains url of youtube stream of same program, which can be extracted with curl "https://msnbcdailyshows.com/" | grep -o -e "https://youtu.be/[0-9a-zA-Z]\+ <https://youtu.be/%5B0-9a-zA-Z%5D%5C+>" and then fed to youtube-dl
I return again to this subject because the extraction I have used for a long time now has failed again, probably because part of it is a russian site so they changed to something else...
With the war in Ukraine I would very much like to record the MSNBC stream nightly using ffmpeg and it seems impossible now. I have tried my two ffmpeg formats using the page url:
and the m3u8 URL I find if I use F12 in the browser when it plays:
https://cdn.plt.one/cdn/premium327/chunks.m3u8
But my ffmpeg command does not work with these... Example (on one line):
ffmpeg -hide_banner -referer \"https://msnbcdailyshows.com/\" -i https://cdn.plt.one/cdn/premium327/chunks.m3u8 -vf scale=w=-4:h=480 -c:v libx264 -preset fast -crf 26 -c:a copy -t 120 test1.mp4
It fails to actually get any data, so much tghat the timeout does not even trigger...
What can be done to get this working again?
It is premium content for users that pay.
-- Bo Berglund Developer in Sweden
_______________________________________________ ffmpeg-user mailing list ffmpeg-user@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-user
To unsubscribe, visit link above, or email ffmpeg-user-request@ffmpeg.org with subject "unsubscribe".
_______________________________________________ ffmpeg-user mailing list ffmpeg-user@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-user
To unsubscribe, visit link above, or email ffmpeg-user-request@ffmpeg.org with subject "unsubscribe".
On 2022-03-04T18:47:26+0100, Bo Berglund <bo.berglund@gmail.com> wrote:
and the m3u8 URL I find if I use F12 in the browser when it plays:
I don't see that url anywhere. Instead I'm seeing... <iframe width="860" height="460" src="*https://www.youtube.com/embed/9CLF18wQpag*" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> ...in the html-source. And that Youtube-video works just fine for me. -- Reino
On Mon, 7 Mar 2022 00:46:16 +0100, Reino Wijnsma <rwijnsma@xs4all.nl> wrote:
On 2022-03-04T18:47:26+0100, Bo Berglund <bo.berglund@gmail.com> wrote:
and the m3u8 URL I find if I use F12 in the browser when it plays:
I don't see that url anywhere. Instead I'm seeing...
<iframe width="860" height="460" src="*https://www.youtube.com/embed/9CLF18wQpag*" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
...in the html-source. And that Youtube-video works just fine for me.
On the page there is a player that runs MSNBC and a couple of years ago that was a Youtube hosted video. ThenYT started chasing these "publishers" and blocked the stream, whereupon they started another Youtube stream etc. It took up to a week before YT found them. Then they stopped using the Youtube engine and started streaming in other ways and that is also when it no longer was possible to right-click the player and "Copy URL" to get the video URL.... Right now I could not find mention of "youtube" or even "yo" on the "view source" page... And using F12 to show the streaming data gives me the URL: https://hbzpjnctrztwpucm.wzcdn134.net:8443/hls/cshr1ujg6u-32171144940.ts But that is not an m3u8 URL I can use with ffmpeg... I tried using it as an input stream with the.ts ending but no joy... -- Bo Berglund Developer in Sweden
On Fri, 11 Mar 2022 at 17:25, Bo Berglund <bo.berglund@gmail.com> wrote:
And using F12 to show the streaming data gives me the URL:
https://hbzpjnctrztwpucm.wzcdn134.net:8443/hls/cshr1ujg6u-32171144940.ts
But that is not an m3u8 URL I can use with ffmpeg...
I tried using it as an input stream with the.ts ending but no joy...
It gives me a '403 Forbidden'. It might need a valid referer, headers, cookie, IP address, something else, or even a combination. Blithely chucking URLs at ffmpeg as input and hoping to get output is almost certainly not the best way of going about this. Might I again suggest not reinventing the wheel? You could reach out to folks who are like-minded: interested in saving potentially ephemeral video which may be important. You can investigate options, but chatting to the folks behind yt-dlp [1] or the archiveteam [2] might be places to start. Good luck. Cheers, Rob [1]: https://github.com/yt-dlp/yt-dlp - their discord is listed on their GitHub page [2]:https://wiki.archiveteam.org/index.php/Main_Page - worth having a read of their wiki
On 2022-03-11T18:24:46+0100, Bo Berglund <bo.berglund@gmail.com> wrote:
Right now I could not find mention of "youtube" or even "yo" on the "view source" page... And using F12 to show the streaming data gives me the URL:
https://hbzpjnctrztwpucm.wzcdn134.net:8443/hls/cshr1ujg6u-32171144940.ts
But that is not an m3u8 URL I can use with ffmpeg... Now I'm seeing...
<iframe src="https://wigistream.to/embed/cshr1ujg6u" width="888" height="498" scrolling="no" frameborder="0" allowfullscreen="true"></iframe> ...but this "STREAM IS OFFLINE" I'm being shown. Do we see different content depending on our location? I don't see any livestreams that work. -- Reino
On Sun, 13 Mar 2022 15:50:47 +0100, Reino Wijnsma <rwijnsma@xs4all.nl> wrote:
On 2022-03-11T18:24:46+0100, Bo Berglund <bo.berglund@gmail.com> wrote:
Right now I could not find mention of "youtube" or even "yo" on the "view source" page... And using F12 to show the streaming data gives me the URL:
https://hbzpjnctrztwpucm.wzcdn134.net:8443/hls/cshr1ujg6u-32171144940.ts
But that is not an m3u8 URL I can use with ffmpeg... Now I'm seeing...
<iframe src="https://wigistream.to/embed/cshr1ujg6u" width="888" height="498" scrolling="no" frameborder="0" allowfullscreen="true"></iframe>
...but this "STREAM IS OFFLINE" I'm being shown.
Do we see different content depending on our location? I don't see any livestreams that work.
Seems to be changing all the time... Now on https://www.livenewsmag.com/msnbc-news-live-stream/ I get this m3u8: https://1420543146.rsc.cdn77.org/zowLJtLL4UMI57up8Jmbmg==,1647403578/LS-ATL-... But on the original page https://msnbcdailyshows.com/ I get: https://hbzpjnctrztwpucm.wzcdn134.net:8443/hls/cshr1ujg6u.m3u8?s=JaYc9DUQcYOryV3h0eBHsQ&e=1647405745 I do not know what the part after m3u8 means though. F12 on the page in Firefox and then rightclick the line in the Network pane that has the m3u8 and select Copy URL gets me to this... -- Bo Berglund Developer in Sweden
On Wed, Sep 01, 2021 at 08:14:02 +0200, Bo Berglund wrote:
How do I extract the proper m3u8 URL from the webpage in the best/safest way? Some of these pages display a scrolling long list of items when one presses the F12 function key in FireFox and it is not obvious to me what I should try. And there are also URL:s ending with .ts in that scrolling list, but these change all the time so they are probably just short sections of the video, right?
Is there a command line call I can make to extract the m3u8 URL automatically so it can be used in a script only supplied the main page URL?
There's often a lot of magic involved in extracting the video URLs from webpages - parsing JavaScript, downloading and parsing JSON files, re-using cookies and referrers, and so on. Please do use the tool youtube-dl, or the maintained fork yt-dlp, to extract these URL or even to download the actual videos. The tool figures out the formats, the URLs, and often even the de-encryption (not DRM though) of many many web pages. (It can even call ffmpeg to download, and will do so by default in many cases.) To get the (or a) URL: $ youtube-dl -g http://website-of-choice/page If youtube-dl doesn't support a certain page, I load the page in Chrome and look for the original file or m3u8 on the Network tab of the developer console (Ctrl+Shift+C), and then pass that to ffmpeg. Cheers, Moritz
On 2021-09-01T08:14:02+0200, Bo Berglund <bo.berglund@gmail.com> wrote:
Is there a command line call I can make to extract the m3u8 URL automatically so it can be used in a script only supplied the main page URL?
On 2021-08-29T17:41:49+0200, Reino Wijnsma <rwijnsma@xs4all.nl> wrote:
$ ffmpeg -hide_banner -i "$(xidel -s "https://www.livenewsmag.com/msnbc-news-live-stream/" -e 'parse-json(//script/extract(.,"playerInstance.setup\((.+)\)",1,"s"),{"liberal":true()})/file')"
??? Instead of parsing the (pseudo) JSON, you can also try to extract the HLS manifest directly: $ xidel -s "https://www.livenewsmag.com/msnbc-news-live-stream/" -e 'extract(//script,"https.+m3u8")' https://1420543146.rsc.cdn77.org/0YMlFFbIvOHfq9uQgKZexg==,1630548377/LS-ATL-... -- Reino
participants (8)
-
Anatoly -
anatoly＠kazanfieldhockey.ru
-
Bo Berglund -
Dan Schultz -
Moritz Barsnick -
Paul B Mahol -
Reino Wijnsma -
Rob Hallam