[FFmpeg-trac] #9063(avcodec:new): Segmentation fault when encoding with dnxhd with yuv444p10le

[FFmpeg]

#9063: Segmentation fault when encoding with dnxhd with yuv444p10le
-------------------------------------+-------------------------------------
             Reporter:  difs         |                    Owner:
                 Type:  defect       |                   Status:  new
             Priority:  important    |                Component:  avcodec
              Version:  git-master   |               Resolution:
             Keywords:  dnxhd crash  |               Blocked By:
  SIGSEGV                            |
             Blocking:               |  Reproduced by developer:  1
Analyzed by developer:  0            |
-------------------------------------+-------------------------------------
Changes (by cehoyos):

 * keywords:   => dnxhd crash SIGSEGV
 * priority:  normal => important
 * component:  undetermined => avcodec
 * reproduced:  0 => 1


Comment:

 This command line also allows to reproduce but needs less memory (I was
 unable to reproduce with 4k input):
 {{{
 $ ffmpeg -f lavfi -i nullsrc=size=5760x3240:rate=60 -vf
 "geq=random(1)*255:128:128" -threads 1 -pix_fmt yuv444p10le -c:v dnxhd
 -profile:v 5 -f null -
 }}}
 The following patch illustrates the issue, sum of slize_size[] is bigger
 than coding_unit_size:
 {{{
 diff --git a/libavcodec/dnxhdenc.c b/libavcodec/dnxhdenc.c
 index 2461c51727..2b3170c56a 100644
 --- a/libavcodec/dnxhdenc.c
 +++ b/libavcodec/dnxhdenc.c
 @@ -1304,7 +1304,7 @@ encode_coding_unit:

      avctx->execute2(avctx, dnxhd_encode_thread, buf, NULL,
 ctx->m.mb_height);

 -    av_assert1(ctx->data_offset + offset + 4 <= ctx->coding_unit_size);
 +    av_assert0(ctx->data_offset + offset + 4 <= ctx->coding_unit_size);
      memset(buf + ctx->data_offset + offset, 0,
             ctx->coding_unit_size - 4 - offset - ctx->data_offset);

 }}}

--
Ticket URL: <https://trac.ffmpeg.org/ticket/9063#comment:1>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker