[FFmpeg-devel] [FFmpeg-cvslog] avcodec/dnxhdenc: fix possible out of bound writes for big w/h

[Paul B Mahol]

On 3/5/22, Martin Storsjö <martin at martin.st> wrote:
> On Fri, 4 Mar 2022, Paul B Mahol wrote:
>
>> ffmpeg | branch: master | Paul B Mahol <onemda at gmail.com> | Fri Mar  4
>> 23:25:48 2022 +0100| [37480b1b85b0405563962b581dc2899b1b4bec59] |
>> committer: Paul B Mahol
>>
>> avcodec/dnxhdenc: fix possible out of bound writes for big w/h
>>
>> It was caused by integer overflows.
>>
>>> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=37480b1b85b0405563962b581dc2899b1b4bec59
>> ---
>
> This caused changes to the hashes in the fate-lavf-mxf_opatom test - can
> you have a look?
>

It caused yet another integer overflow, and max_bits was still bigger
than allowed by few bits.

I changed code to keep increasing qscale until max_bits is lowered.
max_bits is just worst case scenario for bit allocations.

> // Martin
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
>