[FFmpeg-devel] [PATCH] lavc/videotoolbox: fix avcc creation for h264 streams missing extradata

[Aman Gupta]

From: Aman Gupta <aman at tmm1.net>

ff_videotoolbox_avcc_extradata_create() was never being called if
avctx->extradata_size==0, even though the function does not need or use
the avctx->extradata.

This manifested itself only on h264 streams in specific containers, and
only on iOS. I guess the macOS version of VideoToolbox is more
forgiving, atleast on my specific combination of OS version and hardware.

I also added an error log message when VTDecompressionSessionCreate()
fails, to help the next poor soul who runs into a bug in this area of the
code. The native OSStatus error codes are much easier to google than
their AVERROR counterparts (especially in this case, with AVERROR_UNKNOWN).
---
 libavcodec/videotoolbox.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/libavcodec/videotoolbox.c b/libavcodec/videotoolbox.c
index 1288aa5..b21eccb 100644
--- a/libavcodec/videotoolbox.c
+++ b/libavcodec/videotoolbox.c
@@ -413,7 +413,7 @@ static CFDictionaryRef videotoolbox_decoder_config_create(CMVideoCodecType codec
                          kVTVideoDecoderSpecification_RequireHardwareAcceleratedVideoDecoder,
                          kCFBooleanTrue);
 
-    if (avctx->extradata_size) {
+    if (avctx->extradata_size || codec_type == kCMVideoCodecType_H264) {
         CFMutableDictionaryRef avc_info;
         CFDataRef data = NULL;
 
@@ -572,13 +572,16 @@ static int videotoolbox_default_init(AVCodecContext *avctx)
     if (buf_attr)
         CFRelease(buf_attr);
 
+    if (status != 0)
+        av_log(avctx, AV_LOG_ERROR, "Error creating videotoolbox decompression session: %d\n", status);
+
     switch (status) {
     case kVTVideoDecoderNotAvailableNowErr:
     case kVTVideoDecoderUnsupportedDataFormatErr:
         return AVERROR(ENOSYS);
     case kVTVideoDecoderMalfunctionErr:
         return AVERROR(EINVAL);
-    case kVTVideoDecoderBadDataErr :
+    case kVTVideoDecoderBadDataErr:
         return AVERROR_INVALIDDATA;
     case 0:
         return 0;
-- 
2.8.1