Using FFMPEG to encode multiple outputs in 'parallel' (doesn't work?)
Hello Community members, Quoting from the ffmpeg's wiki for creating multiple outputs. https://trac.ffmpeg.org/wiki/Creating%20multiple%20outputs *Parallel encoding*: Outputting and re encoding multiple times in the same FFmpeg process will typically slow down to the "slowest encoder" in your list. Some encoders (like libx264) perform their encoding "threaded and in the background" so they will effectively allow for parallel encodings. Unfortunately, my results are not correlating with what is mentioned above. By that I mean, I am not seeing encoding being performed in parallel on multiple outputs even when using libx264. To simplify the problem and to understand it easily, here's what I am doing for strictly benchmarking purposes. Option1: Run 3 instances of ffmpeg using pipe. ffmpeg -i input.mp4 -c:v libx264 -b:v 12M -f null - | ffmpeg -i input.mp4 -c:v libx264 -b:v 12M -f null - | ffmpeg -i input.mp4 -c:v libx264 -b:v 12M -f null - Option2: run all three encodes in single ffmpeg process ffmpeg -i input.mp4 -c:v libx264 -b:v 12M -f null - -c:v libx264 -b:v 12M -f null - -c:v libx264 -b:v 12M -f null - I am executing this workload on 32 core platform on google cloud. Option1 gives me nearly 100% cpu utilization. In case of Option 2 cpu utilization doesn't go beyond 50% and also fps is about 70% of the option1. Then why don't I just use Option1? Because Option2 decodes the input only once. I need to generate many streams (up to 25 streams) from the same input file which could be 4K or even bigger. So I want to avoid the overhead of decoding the same input so many times in Option1. So basically, I want to make Option2 work but I am not understanding why it's not running all the encodes in parallel to give me better fps and maximum cpu utilization. After taking a quick look at the code, it doesn't seem like ffmpeg is creating multiple threads to run multiple x264 encode sessions in parallel. I know x264 library itself will create multiple threads but that's for a single output. What I am looking for is output level parallelism. Can some one explain if this is possible? Or may be, I can use some different kind of command line to achieve such output level parallelism? Shalin
Hi. Le decadi 10 messidor, an CCXXV, Shalin Mehta a écrit :
Option1: Run 3 instances of ffmpeg using pipe. ffmpeg -i input.mp4 -c:v libx264 -b:v 12M -f null - | ffmpeg -i input.mp4 -c:v libx264 -b:v 12M -f null - | ffmpeg -i input.mp4 -c:v libx264 -b:v 12M -f null -
I think you need to take a little step back and learn what Unix pipes mean. They are meant to connect the output of a process to the input of the next one, but in your examples your commands neither consume their input nor produce output. Regards, -- Nicolas George
Nicolas George wrote
Hi.
Le decadi 10 messidor, an CCXXV, Shalin Mehta a écrit :
Option1: Run 3 instances of ffmpeg using pipe. ffmpeg -i input.mp4 -c:v libx264 -b:v 12M -f null - | ffmpeg -i input.mp4 -c:v libx264 -b:v 12M -f null - | ffmpeg -i input.mp4 -c:v libx264 -b:v 12M -f null -
I think you need to take a little step back and learn what Unix pipes mean. They are meant to connect the output of a process to the input of the next one, but in your examples your commands neither consume their input nor produce output.
-- Nicolas George
signature.asc (849 bytes) <http://www.ffmpeg-archive.org/attachment/4680421/0/signature.asc>;
Piping only shows that I am trying to run all three ffmpeg transcode instances in parallel using single command line. If it makes any clearer, you can read them as three separate commands issued at the same time. With that, I hope we can focus on the concern raised in the original question. -- View this message in context: http://www.ffmpeg-archive.org/Using-FFMPEG-to-encode-multiple-outputs-in-par... Sent from the FFmpeg-users mailing list archive at Nabble.com.
2017-06-28 22:56 GMT+02:00 shalin <shalin186@gmail.com>:
Piping only shows that I am trying to run all three ffmpeg transcode instances in parallel using single command line. If it makes any clearer, you can read them as three separate commands issued at the same time. With that, I hope we can focus on the concern raised in the original question.
Then you don't want pipes, you just want three parallel processes, which you can do using &. More to the point, as far as I know, ffmpeg will not do parallel treatment on either input or output, as you deduced. It certainly is doable, but don't seem trivial to implement in the ffmpeg CLI tool itself since that tool must handle various cases, and introducing (more) special cases would add complexity for not much benefits. You should look in using the various ffmpeg libraries directly; with that you could probably write it to decode once, encode multiple time. But unless you have shockingly similar settings and unlimited processing power, you'll most likely stall some of your encodings while other catch up with them. I don't know how well ffmpeg libraries handle multithreading (I suppose fairly well since most of it's operations are based on context structures) but you'll have to do a fair amount of synchronization by hand.
Cley Faye wrote
More to the point, as far as I know, ffmpeg will not do parallel treatment on either input or output, as you deduced. It certainly is doable, but don't seem trivial to implement in the ffmpeg CLI tool itself since that tool must handle various cases, and introducing (more) special cases would add complexity for not much benefits.
You should look in using the various ffmpeg libraries directly; with that you could probably write it to decode once, encode multiple time. But unless you have shockingly similar settings and unlimited processing power, you'll most likely stall some of your encodings while other catch up with them. I don't know how well ffmpeg libraries handle multithreading (I suppose fairly well since most of it's operations are based on context structures) but you'll have to do a fair amount of synchronization by hand.
Thank you for your answer to my original question. So from this, I am assuming that ffmpeg in its current form doesn't support output level parallelism. Actually I have 'shockingly' similar transcodings to perform on a given input stream (I need to crop different parts of the input video and encode them at same bitrate and also cropped parts are same in size for each stream). With that each encode should roughly run at the same rate and I believe multithreading should be really helpful. I will probably check with ffmpeg developers to know their thoughts on this one. Let me know if anyone has tried some thing like this. Thanks. -- View this message in context: http://www.ffmpeg-archive.org/Using-FFMPEG-to-encode-multiple-outputs-in-par... Sent from the FFmpeg-users mailing list archive at Nabble.com.
On Thursday, June 29, 2017 6:56:51 PM CDT shalin wrote:
Cley Faye wrote
More to the point, as far as I know, ffmpeg will not do parallel treatment on either input or output, as you deduced. It certainly is doable, but don't seem trivial to implement in the ffmpeg CLI tool itself since that tool must handle various cases, and introducing (more) special cases would add complexity for not much benefits.
You should look in using the various ffmpeg libraries directly; with that you could probably write it to decode once, encode multiple time. But unless you have shockingly similar settings and unlimited processing power, you'll most likely stall some of your encodings while other catch up with them. I don't know how well ffmpeg libraries handle multithreading (I suppose fairly well since most of it's operations are based on context structures) but you'll have to do a fair amount of synchronization by hand.
Thank you for your answer to my original question. So from this, I am assuming that ffmpeg in its current form doesn't support output level parallelism. Actually I have 'shockingly' similar transcodings to perform on a given input stream (I need to crop different parts of the input video and encode them at same bitrate and also cropped parts are same in size for each stream). With that each encode should roughly run at the same rate and I believe multithreading should be really helpful.
All this is possible but you need to do a lot of reading. Starting with input mapping. http://www.ffmpeg.org/ffmpeg-all.html#Advanced-options It couldn’t hurt to read up on how anonymous pipes work in a posix shell, as well as other shell related concepts. -Reuben
Reuben Martin wrote
On Thursday, June 29, 2017 6:56:51 PM CDT shalin wrote:
Cley Faye wrote
More to the point, as far as I know, ffmpeg will not do parallel treatment on either input or output, as you deduced. It certainly is doable, but don't seem trivial to implement in the ffmpeg CLI tool itself since that tool must handle various cases, and introducing (more) special cases would add complexity for not much benefits.
You should look in using the various ffmpeg libraries directly; with that you could probably write it to decode once, encode multiple time. But unless you have shockingly similar settings and unlimited processing power, you'll most likely stall some of your encodings while other catch up with them. I don't know how well ffmpeg libraries handle multithreading (I suppose fairly well since most of it's operations are based on context structures) but you'll have to do a fair amount of synchronization by hand.
Thank you for your answer to my original question. So from this, I am assuming that ffmpeg in its current form doesn't support output level parallelism. Actually I have 'shockingly' similar transcodings to perform on a given input stream (I need to crop different parts of the input video and encode them at same bitrate and also cropped parts are same in size for each stream). With that each encode should roughly run at the same rate and I believe multithreading should be really helpful.
All this is possible but you need to do a lot of reading. Starting with input mapping. http://www.ffmpeg.org/ffmpeg-all.html#Advanced-options
It couldn’t hurt to read up on how anonymous pipes work in a posix shell, as well as other shell related concepts.
There's no doubt that I need a lot of reading but I wish you at least read my question properly. It has nothing to do with pipes or shell. It's about running multiple encodes in parallel within single ffmpeg process (option 2 in my question). Simply providing the link doesn't help as it doesn't solve the issue at hand. -- View this message in context: http://www.ffmpeg-archive.org/Using-FFMPEG-to-encode-multiple-outputs-in-par... Sent from the FFmpeg-users mailing list archive at Nabble.com.
Option 2 in your question looks fine to me... I have used ffmpeg to transcode audio and video streams before to achieve adaptive streaming. Here is an example of an audio transcode... input is 256k mp3, output is 128k, 64k, and 24k $ ffmpeg -i http://stream.dubstep.fm -c:a mp3 -b:a 128k -f null /dev/null -c:a mp3 -b:a 64k -f null /dev/null -c:a mp3 -b:a 24k -ar 22050 -f null /dev/null ffmpeg version N-86413-g50be8f2142 Copyright (c) 2000-2017 the FFmpeg developers built with Apple LLVM version 8.1.0 (clang-802.0.42) configuration: --enable-libvidstab --enable-shared --enable-pthreads --cc=/usr/bin/clang --enable-vda --enable-videotoolbox --arch=x86_64 --enable-yasm --enable-gpl --enable-postproc --enable-libx264 --enable-libxvid --enable-filter=coreimage --enable-filter=coreimagesrc --enable-filter=frei0r --enable-filter=frei0r_src --enable-swscale --enable-avfilter --enable-avresample --enable-libmp3lame --enable-libvorbis --enable-libopus --enable-libtheora --enable-libopenjpeg --enable-libmodplug --enable-libvpx --enable-libsoxr --enable-libspeex --enable-libass --enable-libbluray --enable-lzma --enable-gnutls --enable-fontconfig --enable-libfreetype --enable-libfribidi --enable-libx265 --enable-version3 --enable-libwavpack --enable-libssh --enable-librtmp --enable-libfdk-aac --enable-nonfree --enable-libzmq --enable-opencl --enable-ffplay --enable-libcaca --enable-libvidstab libavutil 55. 63.100 / 55. 63.100 libavcodec 57. 96.101 / 57. 96.101 libavformat 57. 72.101 / 57. 72.101 libavdevice 57. 7.100 / 57. 7.100 libavfilter 6. 91.100 / 6. 91.100 libavresample 3. 6. 0 / 3. 6. 0 libswscale 4. 7.101 / 4. 7.101 libswresample 2. 8.100 / 2. 8.100 libpostproc 54. 6.100 / 54. 6.100 [mp3 @ 0x7fa3b2000000] Skipping 88 bytes of junk at 0. Input #0, mp3, from 'http://stream.dubstep.fm': Metadata: icy-genre : Dubstep, Dub, Electronic, Garage icy-name : Dubstep.fm icy-notice1 : <BR>This stream requires <a href="http://www.winamp.com">Winamp</a><BR> icy-notice2 : SHOUTcast DNAS/posix(linux x64) v2.5.1.724<BR> icy-pub : 1 icy-url : https://www.dubstep.fm StreamTitle : ARCHIVE - 2011-01-09 - The Dented McStank With Phaded From Chicago StreamUrl : https://www.dubstep.fm Duration: N/A, start: 0.000000, bitrate: 256 kb/s Stream #0:0: Audio: mp3, 44100 Hz, stereo, s16p, 256 kb/s Stream mapping: Stream #0:0 -> #0:0 (mp3 (native) -> mp3 (libmp3lame)) Stream #0:0 -> #1:0 (mp3 (native) -> mp3 (libmp3lame)) Stream #0:0 -> #2:0 (mp3 (native) -> mp3 (libmp3lame)) Press [q] to stop, [?] for help Output #0, null, to '/dev/null': Metadata: icy-genre : Dubstep, Dub, Electronic, Garage icy-name : Dubstep.fm icy-notice1 : <BR>This stream requires <a href="http://www.winamp.com">Winamp</a><BR> icy-notice2 : SHOUTcast DNAS/posix(linux x64) v2.5.1.724<BR> icy-pub : 1 icy-url : https://www.dubstep.fm StreamTitle : ARCHIVE - 2011-01-09 - The Dented McStank With Phaded From Chicago StreamUrl : https://www.dubstep.fm encoder : Lavf57.72.101 Stream #0:0: Audio: mp3 (libmp3lame), 44100 Hz, stereo, s16p, 128 kb/s Metadata: encoder : Lavc57.96.101 libmp3lame Output #1, null, to '/dev/null': Metadata: icy-genre : Dubstep, Dub, Electronic, Garage icy-name : Dubstep.fm icy-notice1 : <BR>This stream requires <a href="http://www.winamp.com">Winamp</a><BR> icy-notice2 : SHOUTcast DNAS/posix(linux x64) v2.5.1.724<BR> icy-pub : 1 icy-url : https://www.dubstep.fm StreamTitle : ARCHIVE - 2011-01-09 - The Dented McStank With Phaded From Chicago StreamUrl : https://www.dubstep.fm encoder : Lavf57.72.101 Stream #1:0: Audio: mp3 (libmp3lame), 44100 Hz, stereo, s16p, 64 kb/s Metadata: encoder : Lavc57.96.101 libmp3lame Output #2, null, to '/dev/null': Metadata: icy-genre : Dubstep, Dub, Electronic, Garage icy-name : Dubstep.fm icy-notice1 : <BR>This stream requires <a href="http://www.winamp.com">Winamp</a><BR> icy-notice2 : SHOUTcast DNAS/posix(linux x64) v2.5.1.724<BR> icy-pub : 1 icy-url : https://www.dubstep.fm StreamTitle : ARCHIVE - 2011-01-09 - The Dented McStank With Phaded From Chicago StreamUrl : https://www.dubstep.fm encoder : Lavf57.72.101 Stream #2:0: Audio: mp3 (libmp3lame), 22050 Hz, stereo, s16p, 24 kb/s Metadata: encoder : Lavc57.96.101 libmp3lame size=N/A time=00:00:13.14 bitrate=N/A speed=2.69x video:0kB audio:347kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
On Jul 5, 2017, at 2:23 39PM, sh
alin <shalin186@gmail.com> wrote:
Reuben Martin wrote
On Thursday, June 29, 2017 6:56:51 PM CDT shalin wrote:
Cley Faye wrote
More to the point, as far as I know, ffmpeg will not do parallel treatment on either input or output, as you deduced. It certainly is doable, but don't seem trivial to implement in the ffmpeg CLI tool itself since that tool must handle various cases, and introducing (more) special cases would add complexity for not much benefits.
You should look in using the various ffmpeg libraries directly; with that you could probably write it to decode once, encode multiple time. But unless you have shockingly similar settings and unlimited processing power, you'll most likely stall some of your encodings while other catch up with them. I don't know how well ffmpeg libraries handle multithreading (I suppose fairly well since most of it's operations are based on context structures) but you'll have to do a fair amount of synchronization by hand.
Thank you for your answer to my original question. So from this, I am assuming that ffmpeg in its current form doesn't support output level parallelism. Actually I have 'shockingly' similar transcodings to perform on a given input stream (I need to crop different parts of the input video and encode them at same bitrate and also cropped parts are same in size for each stream). With that each encode should roughly run at the same rate and I believe multithreading should be really helpful.
All this is possible but you need to do a lot of reading. Starting with input mapping. http://www.ffmpeg.org/ffmpeg-all.html#Advanced-options
It couldn’t hurt to read up on how anonymous pipes work in a posix shell, as well as other shell related concepts.
There's no doubt that I need a lot of reading but I wish you at least read my question properly. It has nothing to do with pipes or shell. It's about running multiple encodes in parallel within single ffmpeg process (option 2 in my question). Simply providing the link doesn't help as it doesn't solve the issue at hand.
-- View this message in context: http://www.ffmpeg-archive.org/Using-FFMPEG-to-encode-multiple-outputs-in-par... Sent from the FFmpeg-users mailing list archive at Nabble.com. _______________________________________________ ffmpeg-user mailing list ffmpeg-user@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-user
To unsubscribe, visit link above, or email ffmpeg-user-request@ffmpeg.org with subject "unsubscribe".
On Thu, Jul 06, 2017 at 03:44:45 -0700, DopeLabs wrote:
Option 2 in your question looks fine to me...
I have used ffmpeg to transcode audio and video streams before to achieve adaptive streaming.
Here is an example of an audio transcode... input is 256k mp3, output is 128k, 64k, and 24k
$ ffmpeg -i http://stream.dubstep.fm -c:a mp3 -b:a 128k -f null /dev/null -c:a mp3 -b:a 64k -f null /dev/null -c:a mp3 -b:a 24k -ar 22050 -f null /dev/null
Shalin did confirm in the original posting that this does work and is the preferred method. The complaint was that it doesn't optimally make use of parallelism to load all CPUs to the max (and thereby possibly reduce processing time even more). Moritz
On Wed, 28 Jun 2017, shalin wrote:
Nicolas George wrote
I think you need to take a little step back and learn what Unix pipes mean. They are meant to connect the output of a process to the input of the next one, but in your examples your commands neither consume their input nor produce output.
Piping only shows that I am trying to run all three ffmpeg transcode instances in parallel using single command line. If it makes any clearer, you can read them as three separate commands issued at the same time. With that, I hope we can focus on the concern raised in the original question.
As Nicolas already said, you need to take a closer look at "piping" vs "background processing" in the UNIX shell. What you are doing is PIPING: <process 1> | <process 2> | <process 3> This takes the output of process 1 and uses it as the input to process 2, then the output of process 2 and uses it as the input to process 3. It sounds like what you want is BACKGROUND PROCESSING: <process 1> & <process 2> & process 3 & This puts all 3 processes in the background and runs them concurrently. You might want to check out some basics of the UNIX shell, such as this link: https://www.washington.edu/computing/unix/startdoc/shell.html
participants (8)
-
Cley Faye -
DopeLabs -
Hans Carlson -
Moritz Barsnick -
Nicolas George -
Reuben Martin -
shalin -
Shalin Mehta