[FFmpeg-trac] #5593(avcodec:new): OS X h264_videotoolbox encoder and codecpar API race condition with extradite when using MP4 / mov

[FFmpeg]

#5593: OS X h264_videotoolbox encoder and codecpar API race condition with
extradite when using MP4 / mov
-------------------------------------+-------------------------------------
             Reporter:  vade         |                     Type:  defect
               Status:  new          |                 Priority:  normal
            Component:  avcodec      |                  Version:  git-
             Keywords:               |  master
  h264_videotoolbox, extradata,      |               Blocked By:
  codecpar                           |  Reproduced by developer:  0
             Blocking:               |
Analyzed by developer:  0            |
-------------------------------------+-------------------------------------
 Summary of the bug:

 On OS X, using the new send frame / receive packet / codec parameter API,
 it appears that there is a race condition or a lack of setting of the
 extra data / extra data size when specifying h264_videotoolbox encoder.

 We create do our normal output context setup, make a stream, and using the
 new codecparam API we:

 * avcodec_alloc_context3 a new AVCodecContext and set it up as per our
 streams setting.
 * avcodec_open2 the codec
 * avcodec_parameters_from_context to our target stream

 We then avcodec_send_frame / avcodec_receive_packet as expected, with no
 errors, and mux our resulting packets using av_interleaved_write_frame.

 Our resulting MP4 has invalid video samples as per Apples Quicktime
 Player, resulting in errors in the console :

 5/27/16 12:40:50.282 PM VTDecoderXPCService[26681]: [12:40:50.282]
 H264VideoDecoder_DecodeFrame signalled err=-12910 (err)
 (createJVTLibInstance failed) at
 /Library/Caches/com.apple.xbs/Sources/CoreMedia_frameworks/CoreMedia-1731.15.202/Sources/VideoCodecs/H264/H264VideoDecoder.c
 line 2477

 Quicktime Player 7, Quicklook, and other video programs also have errors
 opening the file.

 if we switch our encoder to libx264, and everything works as expected with
 no errors in sight.

 One can fix the above issue if on the first output packet received from
 avcodec_receive_packet, we set our streams codecpar extra data / extra
 data size from our codec context above.

 This is the race condition. Pseudocode :


 {{{
  ret = avcodec_send_frame(mOutputVideoEncoderContext, inputFrame);
             if( ret >= 0 ) {
                 ret = avcodec_receive_packet(mOutputVideoEncoderContext,
 outputPacket);
                 if( ret >= 0 ) {
                     encodedSuccess = 1;
                     outputPacket->stream_index = videoStream->index;
                     outputPacket->duration = mAvgVideoPacketDuration;

                     // DEBUG FOR VIDEO TOOLBOX BUG
                     if( videoStream->codecpar->extradata == NULL ) {
                         videoStream->codecpar->extradata =
 mOutputVideoEncoderContext->extradata;
                         videoStream->codecpar->extradata_size =
 mOutputVideoEncoderContext->extradata_size;
                     }
                 }
                 else {
                     std::cout << "avcodec_receive_packet got " <<
 av_err2str(ret) << std::endl;
                 }
             }
             else {
                 std::cout << "avcodec_send_frame got " << av_err2str(ret)
 << std::endl;
             }

 }}}


 This bug was verified on IRC by rkern, and I was requested to make a trac
 issue for official reporting of the bug.

 Please note this issue is relevant to git master as of commit
 d93495c95411b95cfc5c050dbabfcdff1c10890a

 Thank you!

--
Ticket URL: <https://trac.ffmpeg.org/ticket/5593>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker