[Libav-user] Assertion failure with AV_PIX_FMT_VIDEOTOOLBOX

[Nikolai Lineckiy]

Hi. I came across a problem when using h264_videotoolbox encoder with AV_PIX_FMT_VIDEOTOOLBOX. It crashes with assertion failure on avcodec_open2:

Assertion *cv_img failed at src/libavcodec/videotoolboxenc.c:2049

The issue seems to be in following code in create_cv_pixel_buffer:

if (avctx->pix_fmt == AV_PIX_FMT_VIDEOTOOLBOX) {
    av_assert0(frame->format == AV_PIX_FMT_VIDEOTOOLBOX);

    *cv_img = (CVPixelBufferRef)frame->data[3];
    av_assert0(*cv_img);

    CFRetain(*cv_img);
    return 0;
}

The calling sequence seems to be vtenc_init -> vtenc_populate_extradata -> vtenc_send_frame -> create_cv_pixel_buffer. It checks frame->data[3] for nullability, and always failes becuse during the construction of internal frame in vtenc_populate_extradata there is no initialization code for data[3].

Or maybe I'm missing something?