[FFmpeg-devel] [PATCH] avcodec/agm: add support for higher compression

[Paul B Mahol]

On 4/8/19, James Almer <jamrial at gmail.com> wrote:
> On 4/8/2019 12:42 PM, Paul B Mahol wrote:
>>>> +static int decode_huffman2(AVCodecContext *avctx, int header, int size)
>>>> +{
>>>> +    AGMContext *s = avctx->priv_data;
>>>> +    GetBitContext *gb = &s->gb;
>>>> +    uint8_t lens[256];
>>>> +    uint32_t output_size;
>>>> +    int ret, x, len;
>>>> +
>>>> +    if ((ret = init_get_bits8(gb, s->gbyte.buffer,
>>>> +                              bytestream2_get_bytes_left(&s->gbyte))) <
>>>> 0)
>>>> +        return ret;
>>>> +
>>>> +    output_size = get_bits_long(gb, 32);
>>>> +
>>>> +    av_fast_padded_malloc(&s->output, &s->output_size,
>>>> +                          output_size * sizeof(*s->output));
>>> Several chances for overflow here.
>> Yes, changed output_size to int.
>
> No, it needs to be unsigned for av_fast_padded_malloc(). What you need
> to also make unsigned is s->output_size instead.
>
> Also, that sizeof(*s->output) seems superfluous.

Done locally.