[FFmpeg-devel] [PATCH] doc/ffmpeg - rewrite Stream Selection chapter

Gyan Doshi gyandoshi at gmail.com
Thu May 24 16:48:50 EEST 2018

Stream Selection chapter rewritten to be more comprehensive and 
-------------- next part --------------
From a7ff32bdac54a1f257482526a507ab7823cafb41 Mon Sep 17 00:00:00 2001
From: Gyan Doshi <ffmpeg at gyani.pro>
Date: Thu, 24 May 2018 19:11:00 +0530
Subject: [PATCH] doc/ffmpeg - rewrite Stream Selection chapter

Flesh out with details and examples to show quirks and limitations.
 doc/ffmpeg.texi | 170 ++++++++++++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 160 insertions(+), 10 deletions(-)

diff --git a/doc/ffmpeg.texi b/doc/ffmpeg.texi
index 88dbdeb95a..804bcc73ce 100644
--- a/doc/ffmpeg.texi
+++ b/doc/ffmpeg.texi
@@ -216,16 +216,166 @@ filters is obviously also impossible, since filters work on uncompressed data.
 @chapter Stream selection
 @c man begin STREAM SELECTION
-By default, @command{ffmpeg} includes only one stream of each type (video, audio, subtitle)
-present in the input files and adds them to each output file.  It picks the
-"best" of each based upon the following criteria: for video, it is the stream
-with the highest resolution, for audio, it is the stream with the most channels, for
-subtitles, it is the first subtitle stream. In the case where several streams of
-the same type rate equally, the stream with the lowest index is chosen.
-You can disable some of those defaults by using the @code{-vn/-an/-sn/-dn} options. For
-full manual control, use the @code{-map} option, which disables the defaults just
+The @code{-map} option should be used for manual control of stream selection for each
+output file. When used, only user-mapped streams are included in that output file,
+with one possible exception described below.
+In the absence of any map options for a particular output file, ffmpeg inspects the output
+format to check which type of streams can be included in it, viz. video, audio and/or
+subtitles. For each acceptable stream type, ffmpeg will pick one stream, when available,
+from among all the inputs. It will select that stream based upon the following criteria:
+for video, it is the stream with the highest resolution, for audio, it is the stream with
+the most channels, for subtitles, it is the first subtitle stream found but there's a caveat.
+The output format's default subtitle encoder may be text-based or image-based, and only a
+subtitle stream of the same type can be chosen. Data or attachment streams are not
+automatically selected and can only be included using @code{-map}. In the case where several
+streams of the same type rate equally, the stream with the lowest index is chosen.
+The @code{-vn}, @code{-an}, @code{-sn} options can be used to skip automatic stream selection
+for video, audio, and subtitle streams respectively.
+If there are any complex filtergraph output streams with unlabeled pads, they will be added
+to the first output file. This will lead to a fatal error if the stream type is not supported
+by the output format. In the absence of the map option, the inclusion of these streams leads
+to the automatic stream selection of their types being skipped. If map options are present,
+these filtergraph streams are included in addition to the mapped streams.
+Complex filtergraph output streams with labeled pads must be mapped once and exactly once.
+Stream handling is independent of stream selection, with an exception for subtitles described
+below. Stream handling is set via the @code{-codec} option addressed to streams within a
+specific @emph{output} file. In particular, codec options are applied by ffmpeg after the
+stream selection process and thus do not influence the latter. An exception exists for subtitles.
+If a subtitle encoder is specified for an output file, the first subtitle stream found, of type
+text or image, will be included. ffmpeg does not validate if the specified encoder can convert
+the selected stream or if the converted stream is acceptable within the output format. This
+applies generally as well: when the user sets an encoder manually, the stream selection process
+cannot check if the encoded stream can be muxed into the output file. If it cannot, ffmpeg will
+abort and @emph{all} output files will fail to be processed.
+ at section Examples
+The following examples illustrate the behavior, quirks and limitations of ffmpeg's automatic
+stream selection methods.
+They assume the following three input files.
+ at verbatim
+input file 'A.avi'
+      stream 0: video 640x360
+      stream 1: audio 2 channels
+input file 'B.mp4'
+      stream 0: video 1920x1080
+      stream 1: audio 2 channels
+      stream 2: subtitles (text)
+      stream 3: audio 5.1 channels
+      stream 4: subtitles (text)
+input file 'C.mkv'
+      stream 0: video 1280x720
+      stream 1: audio 2 channels
+      stream 2: subtitles (image)
+ at end verbatim
+ at itemize
+ at item
+ at example
+ffmpeg -i A.avi -i B.mp4 out1.mkv out2.wav -map 1:a -c:a copy out3.mov
+ at end example
+There are three output files specified, and for the first two, no @code{-map} options
+are set, so ffmpeg will select streams for these two files automatically.
+ at code{out1.mkv} is a Matroska container file and accepts video, audio and subtitle streams,
+so ffmpeg will try to select one of each type.
+For video, it will select @code{stream 0} from @code{B.mp4}, which has the highest
+resolution among all the input video streams.
+For audio, it will select @code{stream 3} from @code{B.mp4}, since it has the greatest
+number of channels.
+For subtitles, it will select @code{stream 2} from @code{B.mp4}, which is the first subtitle
+stream from among @code{A.avi} and @code{B.mp4}.
+ at code{out2.wav} accepts only audio streams, so only @code{stream 3} from @code{B.mp4} is
+For @code{out3.mov}, since a @code{-map} option is set, no automatic stream selection will
+occur. The @code{-map 1:a} will select all audio streams from the second input @code{B.mp4}.
+No other streams will be included in this output file.
+For the first two outputs, all included streams will be transcoded. The encoders chosen will
+be the default ones registered by each output format, which may not match the codec of the
+selected input streams. For the third output, codec option for audio streams has been set
+to @code{copy}, so no decoding-filtering-encoding operations will occur, or @emph{can} occur.
+Packets of selected streams shall be conveyed from the input file and muxed within the output
+ at item
+ at example
+ffmpeg -i C.mkv out1.mkv -c:s dvdsub -an out2.mkv
+ at end example
+Although @code{out1.mkv} is a Matroska container file which accepts subtitle streams, only a
+video and audio stream shall be selected. The subtitle stream of @code{C.mkv} is image-based
+and the default subtitle encoder of the Matroska muxer is text-based, so a transcode operation
+for the subtitles is expected to fail and hence the stream isn't selected. However, in
+ at code{out2.mkv}, a subtitle encoder is specified in the command and so, the subtitle stream is
+selected, in addition to the video stream. The presence of @code{-an} disables audio stream
+selection for @code{out2.mkv}.
+ at item
+ at example
+ffmpeg -i A.avi -i C.mkv -i B.mp4 -filter_complex "overlay" out1.mp4 out2.srt
+ at end example
+A filtergraph is setup here using the @code{-filter_complex} option and consists of a single
+video filter. The @code{overlay} filter requires exactly two video inputs, but none are
+specified, so the first two available video streams are used, those of @code{A.avi} and
+ at code{C.mkv}. The output pad of the filter has no label and so is sent to the first output file
+ at code{out1.mp4}. Due to this, automatic selection of the video stream is skipped, which would
+have selected the stream in @code{B.mp4}. The audio stream with most channels viz. @code{stream 3}
+in @code{B.mp4}, is chosen automatically. No subtitle stream is chosen however, since the MP4
+format has no default subtitle encoder registered, and the user hasn't specified a subtitle encoder.
+The 2nd output file, @code{out2.srt}, only accepts text-based subtitle streams. So, even though
+the first subtitle stream available belongs to @code{C.mkv}, it is image-based and hence skipped.
+The selected stream, @code{stream 2} in @code{B.mp4}, is the first text-based subtitle stream.
+ at item
+ at example
+ffmpeg -i A.avi -i B.mp4 -i C.mkv -filter_complex "[1:v]hue=s=0[outv];overlay;aresample" -map '[outv]' -an out1.mp4 out2.mkv -map '[outv]' -map 1:a:0 out3.mkv
+ at end example
+This command will fail, as the output pad labelled @code{[outv]} has been mapped twice.
+No output files shall be processed.
+ at example
+ffmpeg -i A.avi -i B.mp4 -i C.mkv -filter_complex "[1:v]hue=s=0[outv];overlay;aresample" -an out1.mp4 out2.mkv -map 1:a:0 out3.mkv
+ at end example
+This command will also fail as the hue filter output is labelled (@code{[outv]}) and hasn't been mapped anywhere.
+The command should be modified as follows,
+ at example
+ffmpeg -i A.avi -i B.mp4 -i C.mkv -filter_complex "[1:v]hue=s=0,split=2[outv1][outv2];overlay;aresample" -map '[outv1]' -an out1.mp4 out2.mkv -map '[outv2]' -map 1:a:0 out3.mkv
+ at end example
+The video stream from @code{B.mp4} is sent to the hue filter, whose output is cloned once,
+and both outputs labelled. Then a copy each is mapped to the first and third output files.
+The overlay filter, requiring two video inputs, uses the first two unused video streams.
+Those are the streams from @code{A.avi} and @code{C.mkv}. The overlay output isn't labelled,
+so it is sent to the first output file @code{out1.mp4}, regardless of the presence of the
+ at code{-map} option. The aresample filter is sent the first unused audio stream, that of
+ at code{A.avi}. Since this filter output is also unlabelled, it too is mapped to the first output
+file. The presence of @code{-an} only suppresses automatic or manual stream selection of audio
+streams, not outputs sent from filtergraphs. Both these mapped streams shall be ordered before
+the mapped stream in @code{out1.mp4}.
+The video, audio and subtitle streams mapped to @code{out2.mkv} are entirely determined by
+automatic stream selection.
+ at code{out3.mkv} consists of the video output from the hue filter and the first audio stream from
+ at code{B.mp4}.
+ at end itemize

More information about the ffmpeg-devel mailing list