[FFmpeg-devel] Stream selection algorithm crossing program boundaries

Devin Heitmueller dheitmueller at ltnglobal.com
Fri May 18 18:47:41 EEST 2018

Hello Gyan,

Thanks for your comments.  See inline:

> On May 18, 2018, at 10:24 AM, Gyan Doshi <gyandoshi at gmail.com> wrote:
> If nb_input_files is 1, coupling the stream selection seems to me an improvement. But in the scenario of multiple inputs and no -map options, users usually intend to want a stream from all the sources, so in that case, using one of the streams as a 'key' for the other streams will break expectations. Unless the other inputs are meant for filtering, like a watermark, or background music.

Indeed when I dug into the code it became clear that there is added complexity when multiple input files are involved, if for no other reason than it’s not immediately clear which input stream comes from which file and hence it’s difficult to know which AVFormatContext to be looking at for a given stream.  I need the AVFormatContext to look at the program list, as well as potentially reusing av_find_best_stream() if possible.

I suppose a question worth asking is whether a user would really be passing in multiple files where the streams are already arranged into programs and expect the audio to come from a different input file by default?  My instinct would be that even if he/she passed in multiple files, the default behavior should probably be to pick the audio associated with the video as defined by the program map (and render no audio if none is in the corresponding program).  Of course the user can always use -map to do something different.

I’m tempted to make the heuristic something like the following pseudocode:

If (nb_programs > 0) {
  // Streams arranged into programs, so find first program
  // Pick first video from that program
  // Call av_find_best_stream() to find best audio for that video
} else {
  // Use existing heuristic

With the above approach, we’re taking for granted that if the source input has been arranged into programs, then the author had some preferences in mind with regards to which streams to use by default, and thus by default we should respect those decisions rather than trying to guess which streams to play (even if he/she happened to specify multiple input files).  If the content isn’t arranged into programs then there’s no deterministic way to know which to play by default, so fall back to the current algorithm.  And of course if that’s not what the user wants then he/she can always use -map.

If the above is unacceptable, I’m not against making the new heuristic only if both the file contains one or more programs and there is only a single input file.  In fact the code for that is actually easier to implement (since I only have to ever look at the AVFormatContext for the first input and not have to try to correlate streams back to their corresponding input files).  However I’m not sure if the behavior should really vary based on whether multiple input files are specified.

> Your query has come at an inopportune time, since I intended to rewrite the Stream Selection chapter this weekend, since there are some quirks and nuances not covered or made plain, at present. But if you intend to proceed, of course, I'll wait.

If we can come to a consensus on what the behavior should be (and believe it will be accepted upstream), then I can take a crack at the actual implementation.


More information about the ffmpeg-devel mailing list