[FFmpeg-devel] [PATCH 5/6] avcodec/rv34, mpegvideo: Fix segfault upon frame size change error

[Michael Niedermayer]

On Mon, Apr 05, 2021 at 03:44:33AM +0200, Andreas Rheinhardt wrote:
> The RealVideo 3.0 and 4.0 decoders call ff_mpv_common_init() only during
> their init function and not during decode_frame(); when the size of the
> frame changes, they call ff_mpv_common_frame_size_change(). Yet upon
> error, said function calls ff_mpv_common_end() which frees the whole
> MpegEncContext and not only those parts that
> ff_mpv_common_frame_size_change() reinits. As a result, the context will
> never be usable again; worse, because decode_frame() contains no check
> for whether the context is initialized or not, it is presumed that it is
> initialized, leading to segfaults. Basically the same happens if
> rv34_decoder_realloc() fails.
> 
> This commit fixes this by only resetting the parts that
> ff_mpv_common_frame_size_change() changes upon error and by actually
> checking whether the context is in need of reinitialization in
> ff_rv34_decode_frame().
> 
> Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt at outlook.com>

> ---
> I actually don't like that we have two flags that indicate whether
> a MpegEncContext is usable or not; how about we always call
> ff_mpv_common_init() during init (and never lateron) and make it
> unconditionally allocate the stuff that does not depend upon resolution
> etc. and add a parameter to said function to also allocate the latter.
> The decode_frame functions would then be modified to always use
> ff_mpv_common_frame_size_change().

sure if that ends up being cleaner


[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

I know you won't believe me, but the highest form of Human Excellence is
to question oneself and others. -- Socrates
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20210407/ed0a7c4b/attachment.sig>