[FFmpeg-devel] A patch to fix buffer overflow when decoding h264

[Antti Nietosvaara]

Michael Niedermayer wrote:
> On Wed, May 26, 2010 at 03:34:38PM +0300, Antti Nietosvaara wrote:
>   
>> I was experiencing crashes when decoding certain h264 videos (unfortunately 
>> it is quite hard to extract the problematic stream for replication, since 
>> its in proprietary DVR format).
>> It seems that s->mb_height can change in decode_slice_header after 
>> alloc_tables has been called for the current context, which causes 
>> overflows later. Hopefully this behaviour can be confirmed without a sample 
>> stream.
>> I have attached a patch that reallocates the tables if mb_width or 
>> mb_height change.
>>     
>
> what is changing mb_height without changing height?
>
>
> [...]
>   
> ------------------------------------------------------------------------
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at mplayerhq.hu
> https://lists.mplayerhq.hu/mailman/listinfo/ffmpeg-devel
I dug a little deeper and I may have found a reason for the crash on our 
software. Before decompressing the frame I set AVCodecContext's width 
and height to values that the frame should have been compressed to. This 
seems to end up crashing the program later on.
I suppose altering AVCodecContext::width and height outside libavcodec 
is not using the library as intended, and as such, this patch is 
probably useless.
If you are interested in replicating the crash anyway, I could slap 
together a small C program that does just that.

-- 
Antti Nietosvaara
Turun Turvatekniikka Oy