[FFmpeg-trac] #8781(avcodec:new): H.264 Decoding Fails on Raw Annex B Streams from VideoToolbox

[FFmpeg]

#8781: H.264 Decoding Fails on Raw Annex B Streams from VideoToolbox
---------------------------------+--------------------------------------
             Reporter:  Sox      |                     Type:  defect
               Status:  new      |                 Priority:  normal
            Component:  avcodec  |                  Version:  git-master
             Keywords:  d3d11va  |               Blocked By:
             Blocking:           |  Reproduced by developer:  0
Analyzed by developer:  0        |
---------------------------------+--------------------------------------
 '''Summary of the bug:'''
 ffmpeg can successfully decode my stream of H264 Annex B NALs without a
 container (generated with VideoToolbox on an iOS device) using the x264
 decoder, or even "-hwaccel cuda" on my Nvidia GPU, but fails when using
 d3d11va.

 The bug seems to be in the d3d11va_validate_output() function in dxva2.c.
 I haven't been able to trace it down completely and create a proper fix
 yet, but I was able get decoding to work for my own use cases with a hacky
 workaround here:
 https://github.com/LIV/FFmpeg/commit/35e8d409085d225d7cbd4553b59bd40a8a6b9716.
 I'm forcing ffmpeg to use DXVA2_ModeH264_E regardless of the input stream,
 so this obviously isn't a proper solution.

 '''How to reproduce:'''
 {{{
 % ffmpeg -hwaccel d3d11va -i h264_annexb_1440x1280.h264 output.mp4
 '''ffmpeg version:'''
 4.2.2 and 20200628-4cfcfb3
 '''built on:'''
 Downloaded from ffmpeg.zeranoe.com/builds, and was able to reproduce with
 self-built 4.2.2 on Windows 10 1909.
 }}}

 Note that the white bottom half of my sample video is normal (it's an
 alpha channel).

--
Ticket URL: <https://trac.ffmpeg.org/ticket/8781>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker