[FFmpeg-trac] #3900(avformat:new): ff_check_h264_startcode does not support 3-byte startcodes

[FFmpeg]

#3900: ff_check_h264_startcode does not support 3-byte startcodes
----------------------------------+----------------------------------
             Reporter:  Lastique  |                     Type:  defect
               Status:  new       |                 Priority:  normal
            Component:  avformat  |                  Version:  2.3
             Keywords:            |               Blocked By:
             Blocking:            |  Reproduced by developer:  0
Analyzed by developer:  0         |
----------------------------------+----------------------------------
 This is a regression from ffmpeg 2.2 series. The problem was reproduced
 with ffmpeg 2.3.3.

 ff_check_h264_startcode (in libavformat/mpegtsenc.c) fails to recognize
 3-byte h264 startcodes (i.e. 00 00 01), which results in errors while
 saving h264 content to some containers (avi in my case) with message
 "H.264 bitstream malformed, no startcode found, use the h264_mp4toannexb
 bitstream filter (-bsf h264_mp4toannexb)". The attached patch fixes the
 problem.

 The 3-byte startcode is produced by libx264 for SEI NAL. Our application
 uses libx264 and ffmpeg through C API and at some point passes an AVPacket
 that starts with SEI to av_interleaved_write_frame, which fails.

--
Ticket URL: <https://trac.ffmpeg.org/ticket/3900>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker