[FFmpeg-trac] #9537(avcodec:new): avcodec_open2 segfault for h264_vaapi

29 Nov 2021

      #9537: avcodec_open2 segfault for h264_vaapi
-------------------------------------+-------------------------------------
             Reporter:  kurufu       |                     Type:  defect
               Status:  new          |                 Priority:  important
            Component:  avcodec      |                  Version:
             Keywords:  crash,       |  unspecified
  VAAPI                              |               Blocked By:
             Blocking:               |  Reproduced by developer:  0
Analyzed by developer:  0            |
-------------------------------------+-------------------------------------
 Summary of the bug:
 avcodec_open2 segfaults if you try to open vaapi with well formed but
 incorrect options.

 How to reproduce:
 {{{
 #include <libavcodec/avcodec.h>

 int main(int argc, char *argv[]) {
   AVDictionary *opts = NULL;
   av_dict_set(&opts, "b", "2.5M", 0);
   AVCodec *codec = avcodec_find_encoder_by_name("h264_vaapi");
   if (!codec)
     exit(1);

   AVCodecContext *context = avcodec_alloc_context3(codec);
   context->time_base = (AVRational){1, 60};
   context->pix_fmt = AV_PIX_FMT_VAAPI;
   context->width = 640;
   context->height = 480;

   if (avcodec_open2(context, codec, &opts) < 0)
     exit(1);
   exit(0);
 }
 }}}

 Build with `gcc -o example main.c -lavcodec -lavutil` on ffmpeg 4.4.1.
 Though the given options are not enough to correctly open the codec, (the
 error `[h264_vaapi @ 0x55fc49a13c00] A hardware frames reference is
 required to associate the encoding device.` is reported). Libavcodec then
 proceeds to segfault.

 Preferably ffmpeg would not segfault when provided well formed input.
 Marking important as its a crash, feel free to re-prioritize if thats not
 the correct choice. Also preferably a fix would be backported if possible.
-- 
Ticket URL: <https://trac.ffmpeg.org/ticket/9537>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker