[FFmpeg-devel] [PATCH] avcodec/decode: guard against NULL hw_frames_ctx

[Hendrik Leppkes]

On Tue, Nov 14, 2023 at 7:23 PM Dmitry Rogozhkin
<dmitry.v.rogozhkin-at-intel.com at ffmpeg.org> wrote:
>
> Gurd against segfault running VLC decoding under msys2 [1]:
>
> Thread 33 received signal SIGSEGV, Segmentation fault.
> [Switching to Thread 37728.0xadd0]
> ff_hwaccel_frame_priv_alloc (avctx=0x6447b00, hwaccel_picture_private=0x65dfd00)
>     at libavcodec/decode.c:1848
> 1848        frames_ctx = (AVHWFramesContext *)avctx->hw_frames_ctx->data;
> (gdb) bt
>     at libavcodec/decode.c:1848
>     at libavcodec/h264_slice.c:208
>     first_slice=1) at libavcodec/h264_slice.c:1599
>     at libavcodec/h264_slice.c:2130
>     at libavcodec/h264dec.c:652
>     got_frame=0x646e4b0, avpkt=0x64522c0) at libavcodec/h264dec.c:1048
>
> (gdb) p avctx
> $1 = (AVCodecContext *) 0x6447b00
> (gdb) p avctx->hw_frames_ctx
> $2 = (AVBufferRef *) 0x0
>
> See[1]: https://github.com/msys2/MINGW-packages/pull/19050
> Fixes: be07145109 ("avcodec: add AVHWAccel.free_frame_priv callback")
> CC: Lynne <dev at lynne.ee>
> CC: Christoph Reiter <reiter.christoph at gmail.com>
> Signed-off-by: Dmitry Rogozhkin <dmitry.v.rogozhkin at intel.com>
> ---
>  libavcodec/decode.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/libavcodec/decode.c b/libavcodec/decode.c
> index ad39021..3caaeec 100644
> --- a/libavcodec/decode.c
> +++ b/libavcodec/decode.c
> @@ -1845,9 +1845,9 @@ int ff_hwaccel_frame_priv_alloc(AVCodecContext *avctx, void **hwaccel_picture_pr
>
>      av_assert0(!*hwaccel_picture_private);
>
> -    frames_ctx = (AVHWFramesContext *)avctx->hw_frames_ctx->data;
> +    frames_ctx = (AVHWFramesContext *) avctx->hw_frames_ctx ? avctx->hw_frames_ctx->data : NULL;
>      *hwaccel_picture_private = ff_refstruct_alloc_ext(hwaccel->frame_priv_data_size, 0,
> -                                                      frames_ctx->device_ctx,
> +                                                      frames_ctx ? frames_ctx->device_ctx : NULL,
>                                                        hwaccel->free_frame_priv);

The free_frame_priv callback is not usable when no context is
available. The code should be updated to check if the hwaccel has a
free_frame_priv callback and then require a frames_ctx to be set and
otherwise error out (instead of crashing), and if free_frame_priv is
not set, then it can allocate a buffer without these requirements.

- Hendrik