[FFmpeg-devel] [PATCH 3/9] avcodec/videotoolbox: h264 decoder on iOS is unable to decode interlaced video

Aman Gupta ffmpeg at tmm1.net
Tue Sep 26 03:36:25 EEST 2017


From: Aman Gupta <aman at tmm1.net>

VideoToolbox's support for interlaced H264 is quite poor.

On macOS, VTSessionCopySupportedPropertyDictionary() will show that
kVTDecompressionPropertyKey_FieldMode is supported. Possible values for
this option include DeinterlaceFields and BothFields.

However, files that use MBAFF interlacing are not deinterlaced on macOS
even if the FieldMode=DeinterlacedFields option is specified. Although
it doesn't always deinterlace, the macOS version of VideoToolbox will
always decode and return frame data even when the H264 source is
interlaced.

On iOS, FieldMode is not a valid option and interlaced H264 is not
supported at all. You can create a valid decompression session, but no
frames are returned and almost every DecodeFrame() call returns a
malfunction error.

I opened rdar://30669495 about this, and Apple's response was:

> Correct and intentional. Please stop using interlaced video.

So this commit forces the VideoToolbox hwaccel to fail early when
interlaced H264 is encountered. Thus the API user can easily detect the
failure and use a different decoder.

Here are some sample files I tested on iOS 10 and 11:

https://s3.amazonaws.com/tmm1/videotoolbox/interlaced.ts
https://s3.amazonaws.com/tmm1/videotoolbox/interlaced2.ts
https://s3.amazonaws.com/tmm1/videotoolbox/interlaced3.ts

Decoding interlaced2.ts with VideoToolbox never produces any frames.
Decoding interlaced3.ts ocassional produces frames, but most slices
cause errors.

Decoding interlaced.ts actually works as expected, even though its SPS
matches the other two samples which fail. This means my test is not
comprehensive, and it is not possible to fully detect if a file is
compatible with VideoToolbox using its SPS alone. Still, though this
method produces false positives, it does not produce false negatives.
---
 libavcodec/videotoolbox.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/libavcodec/videotoolbox.c b/libavcodec/videotoolbox.c
index 7dfcf14c00..8ed56392cc 100644
--- a/libavcodec/videotoolbox.c
+++ b/libavcodec/videotoolbox.c
@@ -496,11 +496,18 @@ static CFDictionaryRef videotoolbox_decoder_config_create(CMVideoCodecType codec
             if (data)
                 CFDictionarySetValue(avc_info, CFSTR("esds"), data);
             break;
-        case kCMVideoCodecType_H264 :
+        case kCMVideoCodecType_H264 : {
+            H264Context *h = avctx->priv_data;
+            if (TARGET_OS_IPHONE && h->ps.sps->frame_mbs_only_flag == 0) {
+                av_log(avctx, AV_LOG_ERROR, "VideoToolbox cannot decode interlaced fields on iOS\n");
+                CFRelease(avc_info);
+                goto fail;
+            }
             data = ff_videotoolbox_avcc_extradata_create(avctx);
             if (data)
                 CFDictionarySetValue(avc_info, CFSTR("avcC"), data);
             break;
+        }
         default:
             break;
         }
@@ -515,6 +522,10 @@ static CFDictionaryRef videotoolbox_decoder_config_create(CMVideoCodecType codec
         CFRelease(avc_info);
     }
     return config_info;
+
+fail:
+    CFRelease(config_info);
+    return NULL;
 }
 
 static CFDictionaryRef videotoolbox_buffer_attributes_create(int width,
-- 
2.13.5 (Apple Git-94)



More information about the ffmpeg-devel mailing list