[FFmpeg-devel] [PATCH v2] ffmpeg: Add option to force a specific decode format

Mark Thompson sw at jkqxz.net
Wed Nov 14 23:48:26 EET 2018


On 14/11/18 01:35, Fu, Linjie wrote:
>> -----Original Message-----
>> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces at ffmpeg.org] On Behalf
>> Of Mark Thompson
>> Sent: Wednesday, November 14, 2018 09:11
>> To: ffmpeg-devel at ffmpeg.org
>> Subject: Re: [FFmpeg-devel] [PATCH v2] ffmpeg: Add option to force a
>> specific decode format
>>
>> On 14/11/18 00:50, Fu, Linjie wrote:
>>>> -----Original Message-----
>>>> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces at ffmpeg.org] On
>> Behalf
>>>> Of Mark Thompson
>>>> Sent: Wednesday, November 14, 2018 07:44
>>>> To: ffmpeg-devel at ffmpeg.org
>>>> Subject: [FFmpeg-devel] [PATCH v2] ffmpeg: Add option to force a
>> specific
>>>> decode format
>>>>
>>>> Fixes #7519.
>>>> ---
>>>>  doc/ffmpeg.texi      | 13 ++++++++++++
>>>>  fftools/ffmpeg.c     | 10 ++++++++++
>>>>  fftools/ffmpeg.h     |  4 ++++
>>>>  fftools/ffmpeg_opt.c | 47
>>>> ++++++++++++++++++++++++++++++++++++++++++++
>>>>  4 files changed, 74 insertions(+)
>>>>
>>>> diff --git a/doc/ffmpeg.texi b/doc/ffmpeg.texi
>>>> index 3717f22d42..d127bc0f0d 100644
>>>> --- a/doc/ffmpeg.texi
>>>> +++ b/doc/ffmpeg.texi
>>>> @@ -920,6 +920,19 @@ would be more efficient.
>>>>  When doing stream copy, copy also non-key frames found at the
>>>>  beginning.
>>>>
>>>> + at item -decode_format[:@var{stream_specifier}]
>>>> @var{pixfmt}[, at var{pixfmt}...] (@emph{input,per-stream})
>>>> +Set the possible output formats to be used by the decoder for this
>> stream.
>>>> +If the decoder does not natively support any format in the given list for
>>>> +the input stream then decoding will fail rather than continuing with a
>>>> +different format.
>>>> +
>>>> +In general this should not be set - the decoder will select an output
>>>> +format based on the input stream parameters and available components,
>>>> and
>>>> +that will be automatically converted to whatever the output requires.  It
>>>> +may be useful to force a hardware decoder supporting output in
>> multiple
>>>> +different memory types to pick a desired one, or to ensure that a
>> hardware
>>>> +decoder is used when software fallback is also available.
>>>> +
>>>>  @item -init_hw_device
>>>> @var{type}[=@var{name}][:@var{device}[, at var{key=value}...]]
>>>>  Initialise a new hardware device of type @var{type} called @var{name},
>>>> using the
>>>>  given device parameters.
>>>> diff --git a/fftools/ffmpeg.c b/fftools/ffmpeg.c
>>>> index 38c21e944a..c651c8d3a8 100644
>>>> --- a/fftools/ffmpeg.c
>>>> +++ b/fftools/ffmpeg.c
>>>> @@ -598,6 +598,7 @@ static void ffmpeg_cleanup(int ret)
>>>>          avsubtitle_free(&ist->prev_sub.subtitle);
>>>>          av_frame_free(&ist->sub2video.frame);
>>>>          av_freep(&ist->filters);
>>>> +        av_freep(&ist->decode_formats);
>>>>          av_freep(&ist->hwaccel_device);
>>>>          av_freep(&ist->dts_buffer);
>>>>
>>>> @@ -2800,6 +2801,15 @@ static enum AVPixelFormat
>>>> get_format(AVCodecContext *s, const enum AVPixelFormat
>>>>          const AVCodecHWConfig  *config = NULL;
>>>>          int i;
>>>>
>>>> +        if (ist->decode_formats) {
>>>> +            for (i = 0; ist->decode_formats[i] != AV_PIX_FMT_NONE; i++) {
>>>> +                if (ist->decode_formats[i] == *p)
>>>> +                    break;
>>>> +            }
>>>> +            if (ist->decode_formats[i] != *p)
>>>> +                continue;
>>>> +        }
>>>> +
>>>>          if (!(desc->flags & AV_PIX_FMT_FLAG_HWACCEL))
>>>>              break;
>>>>
>>>> diff --git a/fftools/ffmpeg.h b/fftools/ffmpeg.h
>>>> index eb1eaf6363..b06fd18b1c 100644
>>>> --- a/fftools/ffmpeg.h
>>>> +++ b/fftools/ffmpeg.h
>>>> @@ -125,6 +125,8 @@ typedef struct OptionsContext {
>>>>      int        nb_ts_scale;
>>>>      SpecifierOpt *dump_attachment;
>>>>      int        nb_dump_attachment;
>>>> +    SpecifierOpt *decode_formats;
>>>> +    int        nb_decode_formats;
>>>>      SpecifierOpt *hwaccels;
>>>>      int        nb_hwaccels;
>>>>      SpecifierOpt *hwaccel_devices;
>>>> @@ -334,6 +336,8 @@ typedef struct InputStream {
>>>>      int top_field_first;
>>>>      int guess_layout_max;
>>>>
>>>> +    enum AVPixelFormat *decode_formats;
>>>> +
>>>>      int autorotate;
>>>>
>>>>      int fix_sub_duration;
>>>> diff --git a/fftools/ffmpeg_opt.c b/fftools/ffmpeg_opt.c
>>>> index d4851a2cd8..63bb05053b 100644
>>>> --- a/fftools/ffmpeg_opt.c
>>>> +++ b/fftools/ffmpeg_opt.c
>>>> @@ -701,6 +701,7 @@ static void add_input_streams(OptionsContext *o,
>>>> AVFormatContext *ic)
>>>>          AVStream *st = ic->streams[i];
>>>>          AVCodecParameters *par = st->codecpar;
>>>>          InputStream *ist = av_mallocz(sizeof(*ist));
>>>> +        char *decode_formats = NULL;
>>>>          char *framerate = NULL, *hwaccel_device = NULL;
>>>>          const char *hwaccel = NULL;
>>>>          char *hwaccel_output_format = NULL;
>>>> @@ -797,6 +798,49 @@ static void add_input_streams(OptionsContext
>> *o,
>>>> AVFormatContext *ic)
>>>>              ist->top_field_first = -1;
>>>>              MATCH_PER_STREAM_OPT(top_field_first, i, ist->top_field_first, ic,
>>>> st);
>>>>
>>>> +            MATCH_PER_STREAM_OPT(decode_formats, str,
>> decode_formats, ic,
>>>> st);
>>>> +            if (decode_formats) {
>>>> +                const char *p, *q;
>>>> +                int i, nb_formats;
>>>> +                char tmp[32];
>>>> +
>>>> +                nb_formats = 0;
>>>> +                for (p = decode_formats; p; p = strchr(p + 1, ','))
>>>> +                    ++nb_formats;
>>>> +
>>>> +                ist->decode_formats =
>>>> +                    av_malloc_array(nb_formats + 1, sizeof(*ist-
>>> decode_formats));
>>>> +                if (!ist->decode_formats)
>>>> +                    exit_program(1);
>>>> +
>>>> +                p = decode_formats;
>>>> +                for (i = 0; i < nb_formats; i++) {
>>>> +                    q = strchr(p, ',');
>>>> +                    if (!q) {
>>>> +                        ist->decode_formats[i] = av_get_pix_fmt(p);
>>>> +                        if (ist->decode_formats[i] == AV_PIX_FMT_NONE)
>>>> +                            break;
>>>> +                        continue;
>>>> +                    }
>>>> +                    if (q - p > sizeof(tmp) - 1)
>>>> +                        break;
>>>> +                    memcpy(tmp, p, q - p);
>>>> +                    tmp[q - p] = 0;
>>>> +                    ist->decode_formats[i] = av_get_pix_fmt(tmp);
>>>> +                    if (ist->decode_formats[i] == AV_PIX_FMT_NONE)
>>>> +                        break;
>>>> +                    p = q + 1;
>>>> +                }
>>>> +                if (i < nb_formats) {
>>>> +                    av_log(NULL, AV_LOG_FATAL,
>>>> +                           "Unrecognised decode format: %s.\n", p);
>>>> +                    exit_program(1);
>>>> +                }
>>>> +                ist->decode_formats[nb_formats] = AV_PIX_FMT_NONE;
>>>> +            } else {
>>>> +                ist->decode_formats = NULL;
>>>> +            }
>>>> +
>>>>              MATCH_PER_STREAM_OPT(hwaccels, str, hwaccel, ic, st);
>>>>              if (hwaccel) {
>>>>                  // The NVDEC hwaccels use a CUDA device, so remap the name
>> here.
>>>> @@ -3583,6 +3627,9 @@ const OptionDef options[] = {
>>>>          "audio bitrate (please use -b:a)", "bitrate" },
>>>>      { "b",            OPT_VIDEO | HAS_ARG | OPT_PERFILE | OPT_OUTPUT,
>>>> { .func_arg = opt_bitrate },
>>>>          "video bitrate (please use -b:v)", "bitrate" },
>>>> +    { "decode_format",  OPT_VIDEO | OPT_STRING | HAS_ARG |
>>>> OPT_EXPERT |
>>>> +                        OPT_SPEC | OPT_INPUT,                                    { .off =
>>>> OFFSET(decode_formats) },
>>>> +        "set output format(s) to be used by decoder, fail if none of these
>>>> formats are available", "format" },
>>>>      { "hwaccel",          OPT_VIDEO | OPT_STRING | HAS_ARG | OPT_EXPERT
>> |
>>>>                            OPT_SPEC | OPT_INPUT,                                  { .off =
>>>> OFFSET(hwaccels) },
>>>>          "use HW accelerated decoding", "hwaccel name" },
>>>> --
>>>> 2.19.1
>>>>
>>>
>>> Is there any other conditions which will cause the hwaccel failed and
>> fallback to software path?
>>> Like profile was not supported by hardware?
>>
>> If this option is set then get_format will never choose a format not on the list
>> you provide, so it shouldn't be able to use software decode unless a
>> software format is on the list.
>>
>>> I applied the patch and tested some cases, like yuv422  VAAPI decoding
>> (currently not supported), HEVC decoding with unsupported profile.
>>> It reports an error like "hwaccel initialization returned error", and continue
>> to decode and produce less frames than expected. (which I thought fallback
>> still happened).
>>
>> Can you give an example?  With -v debug there will be output saying exactly
>> what happened in ff_get_format().
>>
>> - Mark
> 
> 
> Got information like this:
> 
> [hevc @ 0x55b87457dbc0] Unknown HEVC profile: 0
> [hevc @ 0x55b87457dbc0] Decoding SPS
> [hevc @ 0x55b87457dbc0] Unknown HEVC profile: 0
> [hevc @ 0x55b87457dbc0] Decoding PPS
> [hevc @ 0x55b87457dbc0] Format vaapi_vld chosen by get_format().
> [hevc @ 0x55b87457dbc0] Format vaapi_vld requires hwaccel initialisation.
> [hevc @ 0x55b87457dbc0] Codec hevc profile 0 not supported for hardware decode.
> [hevc @ 0x55b87457dbc0] Failed setup for format vaapi_vld: hwaccel initialisation returned error.
> [hevc @ 0x55b87457dbc0] Format vaapi_vld not usable, retrying get_format() without it.
> [hevc @ 0x55b87457dbc0] Error parsing NAL unit #3.
> cur_dts is invalid (this is harmless if it occurs once at the start per stream)
> [hevc @ 0x55b87458b7c0] nal_unit_type: 1(TRAIL_R), nuh_layer_id: 0, temporal_id: 0
> [hevc @ 0x55b87458b7c0] Could not find ref with POC 0
> 
> See the attached report file  for more details.

This is a decoder bug - get_format() is not correctly called again after it fails, and the decoder then uses the software pixfmt derived from the SPS on subsequent frames.

(I did pretty much all of my testing with H.264, which does not have this problem.)

- Mark


More information about the ffmpeg-devel mailing list