[FFmpeg-devel] avpicture_layout() writes passed the buffer length for pseudo-paletted formats (bug)

Stefano Sabatini stefano.sabatini-lala at poste.it
Thu Aug 25 16:21:40 CEST 2011

On date Sunday 2011-08-21 14:09:12 -0400, Matthew Einhorn encoded:
> Hi,
> The problem is that in the docs avpicture_layout() says it writes a
> picture with size avpicture_get_size(). So one would make the buffer
> input to avpicture_layout() of that size. However, for pseudo-paletted
> formats (as listed in avpicture_get_size()) the amount written to
> buffer is avpicture_get_size()+1024 (which is also the
> avpicture_fill() size).
> If the buffer size is less than avpicture_get_size() (as indicated by
> dest_size in avpicture_layout()) an error is returned by
> avpicture_layout(). However, for these pseudo-paletted formats no
> error is returned if the buffer is less than avpicture_get_size()+1024
> and the function simply writes passed the actual buffer length for a
> total of avpicture_get_size()+1024.
> The reason for this it seems is that avpicture_layout() wasn't
> supposed to write the palette for these pseudo-paletted formats, but
> the function still writes it resulting in the problem.
> The solution is to:
> *Either remove the exception for pseudo-paletted formats so that
> avpicture_get_size() now completely relies on avpicture_fill() (will
> probably break backwards compatibility) and both return the same size.
> *Make an exception in avpicture_layout() as well for the
> pseudo-paletted formats so that the palette is not written for them.
> *If this "is" intended behavior update avpicture_layout() docs to
> indicate that for these formats the buffer needs to be larger than
> avpicture_get_size() by 1024.

> *Or maybe there's some other deeper issue here (other then to prevent
> copying of a standard palette) as to why these pseudo-paletted formats
> are treated specially?

Best solution is to get rid of these pseudo-palette hacks altogheter,
please wait since I have some cached patches for that.
FFmpeg = Frightening & Formidable Mystic Pacific Eretic Gymnast

More information about the ffmpeg-devel mailing list