[FFmpeg-trac] #4056(avcodec:new): mplayer crashed with vda h264 decoder

[FFmpeg]

#4056: mplayer crashed with vda h264 decoder
------------------------------------+-----------------------------------
             Reporter:  niltsh      |                    Owner:
                 Type:  defect      |                   Status:  new
             Priority:  important   |                Component:  avcodec
              Version:  git-master  |               Resolution:
             Keywords:              |               Blocked By:
             Blocking:              |  Reproduced by developer:  0
Analyzed by developer:  0           |
------------------------------------+-----------------------------------

Comment (by niltsh):

 Hi Rimar

 Thank you for the reply.

 Let me share the call stack, I also tried as far as I could, but clearly
 this part is beyond my knowledge.

 1. the original crash callstack is following,
 ============================================
     frame #0: 0x00007fff92da2282 libsystem_kernel.dylib`__pthread_kill +
 10
     frame #1: 0x00007fff92d874c3 libsystem_pthread.dylib`pthread_kill + 90
     frame #2: 0x00007fff8a870b73 libsystem_c.dylib`abort + 129
     frame #3: 0x00000001006ac519
 mplayer`ff_get_format(avctx=<unavailable>, fmt=<unavailable>) + 601 at
 utils.c:1232
     frame #4: 0x00000001003b18c9 mplayer`get_pixel_format(h=<unavailable>,
 force_callback=<unavailable>) + 761 at h264_slice.c:1112
     frame #5: 0x00000001003af679
 mplayer`ff_h264_decode_slice_header(h=0x0000000103201000,
 h0=0x0000000103201000) + 1801 at h264_slice.c:1415
     frame #6: 0x000000010036cf24
 mplayer`decode_nal_units(h=0x0000000103201000, buf=0x000000010281d000,
 buf_size=53775, parse_extradata=0) + 3076 at h264.c:1592
     frame #7: 0x000000010036eb3c
 mplayer`h264_decode_frame(avctx=0x00000001028a4400,
 data=0x0000000101d3f740, got_frame=0x00007fff5fbfe6cc,
 avpkt=<unavailable>) + 540 at h264.c:1908
     frame #8: 0x00000001006e6f58
 mplayer`vdadec_decode(avctx=0x00000001028a4400, data=0x0000000101d3f740,
 got_frame=0x00007fff5fbfe6cc, avpkt=<unavailable>) + 152 at
 vda_h264_dec.c:136
     frame #9: 0x00000001006af3d8
 mplayer`avcodec_decode_video2(avctx=0x00000001028a4400,
 picture=0x0000000101d3f740, got_picture_ptr=0x00007fff5fbfe6cc,
 avpkt=0x00007fff5fbfe668) + 392 at utils.c:2349
     frame #10: 0x0000000100128759 mplayer`decode(sh=0x0000000101c4d6c0,
 data=<unavailable>, len=53775, flags=<unavailable>) + 665 at
 vd_ffmpeg.c:944
     frame #11: 0x000000010004aa04
 mplayer`decode_video(sh_video=0x0000000101c4d6c0, start=<unavailable>,
 in_size=<unavailable>, drop_frame=0, pts=0, full_frame=0x0000000000000000)
 + 68 at dec_video.c:398
     frame #12: 0x0000000100011f88 mplayer`update_video [inlined]
 generate_video_frame(sh_video=0x0000000101c4d6c0,
 d_video=0x0000000101d00160) + 431 at mplayer.c:1815
     frame #13: 0x0000000100011dd9
 mplayer`update_video(blit_frame=0x00007fff5fbfe944) + 57 at mplayer.c:2527
 ============================================

 And about the assertion, I think it is because that via_h264_dec.c, LINE
 73, the decoder returns AV_PIX_FMT_VDA_VLD in get_format.
 Then I tried to change the return value to AV_PIX_FMT_VDA, the assertion
 passed, but the crash call stack changed to the following

 =======================================
 thread #1: tid = 0xddd45, 0x00000001006e6fc7
 mplayer`vdadec_decode(avctx=0x0000000103017800, data=0x0000000101e162e0,
 got_frame=<unavailable>, avpkt=<unavailable>) + 263 at vda_h264_dec.c:145,
 queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1,
 address=0x0)
     frame #0: 0x00000001006e6fc7
 mplayer`vdadec_decode(avctx=0x0000000103017800, data=0x0000000101e162e0,
 got_frame=<unavailable>, avpkt=<unavailable>) + 263 at vda_h264_dec.c:145
    142
    143          CVPixelBufferRetain(cv_buffer);
    144          CVPixelBufferLockBaseAddress(cv_buffer, 0);
 -> 145          context->cv_buffer = cv_buffer;
    146          pic->format = ctx->pix_fmt;
    147          if (CVPixelBufferIsPlanar(cv_buffer)) {
    148              int i, count = CVPixelBufferGetPlaneCount(cv_buffer);
 (lldb) bt
 thread #1: tid = 0xddd45, 0x00000001006e6fc7
 mplayer`vdadec_decode(avctx=0x0000000103017800, data=0x0000000101e162e0,
 got_frame=<unavailable>, avpkt=<unavailable>) + 263 at vda_h264_dec.c:145,
 queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1,
 address=0x0)
     frame #0: 0x00000001006e6fc7
 mplayer`vdadec_decode(avctx=0x0000000103017800, data=0x0000000101e162e0,
 got_frame=<unavailable>, avpkt=<unavailable>) + 263 at vda_h264_dec.c:145
     frame #1: 0x00000001006af3d8
 mplayer`avcodec_decode_video2(avctx=0x0000000103017800,
 picture=0x0000000101e162e0, got_picture_ptr=0x00007fff5fbfe6cc,
 avpkt=0x00007fff5fbfe668) + 392 at utils.c:2349
     frame #2: 0x0000000100128759 mplayer`decode(sh=0x0000000101c2a820,
 data=<unavailable>, len=144, flags=<unavailable>) + 665 at vd_ffmpeg.c:944
     frame #3: 0x000000010004aa04
 mplayer`decode_video(sh_video=0x0000000101c2a820, start=<unavailable>,
 in_size=<unavailable>, drop_frame=0, pts=0.056677777777777778,
 full_frame=0x0000000000000000) + 68 at dec_video.c:398
     frame #4: 0x0000000100011f88 mplayer`update_video [inlined]
 generate_video_frame(sh_video=0x0000000101c2a820,
 d_video=0x0000000101e00160) + 431 at mplayer.c:1815
     frame #5: 0x0000000100011dd9
 mplayer`update_video(blit_frame=0x00007fff5fbfe944) + 57 at mplayer.c:2527
 =======================================

--
Ticket URL: <https://trac.ffmpeg.org/ticket/4056#comment:4>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker