[FFmpeg-devel] [PATCH] lavc/vaapi_encode: Don't pass VAConfigAttribEncPackedHeaders with value set to 0

[Haihao Xiang]

Recent Intel i965 driver commit strictly disallows application to set
unsupported attribute values, VA_ENC_PACKED_HEADER_NONE (0) is not used
in Intel i965 driver, so application shouldn't pass this value to the
driver. On the other hand, VA_ENC_PACKED_HEADER_NONE (0) means the
driver doesn't support any packed header mode, so application also
shouldn't pass packed header to driver if a driver returns
VA_ENC_PACKED_HEADER_NONE (0), the driver should work without
VAConfigAttribEncPackedHeaders set for this case.

In addition, VA_ATTRIB_NOT_SUPPORTED and VA_ENC_PACKED_HEADER_NONE make
thing messy, we will deprecate VA_ENC_PACKED_HEADER_NONE in the
future. See https://github.com/intel/libva/issues/178 for more information.

This fixes broken vp9 encoder on Kably Lake with Intel I965 driver.

Signed-off-by: Haihao Xiang <haihao.xiang at intel.com>
---
 libavcodec/vaapi_encode.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/libavcodec/vaapi_encode.c b/libavcodec/vaapi_encode.c
index e371f5761ee..1d30aabed40 100644
--- a/libavcodec/vaapi_encode.c
+++ b/libavcodec/vaapi_encode.c
@@ -1111,6 +1111,10 @@ static av_cold int vaapi_encode_config_attributes(AVCodecContext *avctx)
                        ctx->va_packed_headers, attr[i].value);
                 ctx->va_packed_headers &= attr[i].value;
             }
+
+            if (!ctx->va_packed_headers)
+                continue;
+
             ctx->config_attributes[ctx->nb_config_attributes++] =
                 (VAConfigAttrib) {
                 .type  = VAConfigAttribEncPackedHeaders,
-- 
2.14.1