[FFmpeg-trac] #6649(avcodec:new): DNxHR 444 encoding crashes because of wrong macroblock size

[FFmpeg]

#6649: DNxHR 444 encoding crashes because of wrong macroblock size
-------------------------------------+-----------------------------------
             Reporter:  devernay     |                    Owner:
                 Type:  defect       |                   Status:  new
             Priority:  normal       |                Component:  avcodec
              Version:  git-master   |               Resolution:
             Keywords:  DNxHD DNxHR  |               Blocked By:
             Blocking:               |  Reproduced by developer:  0
Analyzed by developer:  0            |
-------------------------------------+-----------------------------------

Comment (by devernay):

 ok I got a full fix.
 That part of the code had several bugs, because it was basically dead
 code, since the condition was wrong.
 Overall, this code looks very fragile:
 - hardcoded numbers such as 16 (shouldn't the variable bs be used instead)
 and 1080 (why?) at several places
 - the test ctx->bit_depth != 10 in dnxhd_get_blocks should really be
 ctx->bit_depth == 8
 - the test ctx->bit_depth == 10 in dnxhd_get_blocks should really be
 ctx->bit_depth > 8 (remember that DNxHR 444 should really be 12-bit deep,
 so I hope we will get that into ffmpeg someday)
 - hardcoded edge macroblock buffer sizes of 256 bytes (edge_buf_y and
 edge_buf_uv in dnxhdenc.h)

 but at least I got DNxHR 444 working using the API!

 I think the ffmpeg command-line tool worked by chance, because there is
 accessible memory after the frame to be encoded, so reading this area
 didn't make ffmpeg crash.

--
Ticket URL: <https://trac.ffmpeg.org/ticket/6649#comment:4>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker