[FFmpeg-trac] #8902(avcodec:new): VAAPI pixel format support regression between FFmpeg 3.x and 4.x

[FFmpeg]

#8902: VAAPI pixel format support regression between FFmpeg 3.x and 4.x
-------------------------------------+-------------------------------------
             Reporter:  bmegli       |                    Owner:
                 Type:  defect       |                   Status:  new
             Priority:  normal       |                Component:  avcodec
              Version:  git-master   |               Resolution:
             Keywords:  vaapi        |               Blocked By:
  regression                         |
             Blocking:               |  Reproduced by developer:  0
Analyzed by developer:  0            |
-------------------------------------+-------------------------------------

Comment (by bmegli):

 Tested with:
 - Ubuntu 18.04
 - FFmpeg version 7:3.4.8-0ubuntu0.2 (from system package)
 - Kaby Lake platfrom
 - i965 driver
 - [https://github.com/bmegli/hardware-video-encoder/tree/bgr0 own bgr0
 test], [https://github.com/bmegli/hardware-video-encoder/tree/yuyv422 own
 yuyv422 test]
 - profile is set to FF_PROFILE_H264_HIGH (similar effect with other)

 When enabled with trace debugging or queried with
 av_hwframe_transfer_get_formats following formats are reported:

 {{{
 [AVHWDeviceContext @ 0x55dcc90cf7e0] Format 0x32315659 -> yuv420p.
 [AVHWDeviceContext @ 0x55dcc90cf7e0] Format 0x30323449 -> unknown.
 [AVHWDeviceContext @ 0x55dcc90cf7e0] Format 0x3231564e -> nv12.
 [AVHWDeviceContext @ 0x55dcc90cf7e0] Format 0x32595559 -> yuyv422.
 [AVHWDeviceContext @ 0x55dcc90cf7e0] Format 0x59565955 -> uyvy422.
 [AVHWDeviceContext @ 0x55dcc90cf7e0] Format 0x48323234 -> yuv422p.
 [AVHWDeviceContext @ 0x55dcc90cf7e0] Format 0x58424752 -> rgb0.
 [AVHWDeviceContext @ 0x55dcc90cf7e0] Format 0x58524742 -> bgr0.
 [AVHWDeviceContext @ 0x55dcc90cf7e0] Format 0x30313050 -> p010le.

 }}}

 I have tried bgr0, rgb0 and yuyv422 (all of which I or my users use
 directly).

 The raw avcodec output dumped to file, queried with ffprobe returns:

 {{{
 Stream #0:0: Video: h264 (High), yuv420p(progressive), 1280x720 [SAR 1:1
 DAR 16:9], 30 fps, 30 tbr, 1200k tbn, 60 tbc
 }}}

 So looks like YUV420P regardless of sw pixel format.

 Doing the same with doc/examples/vaapi_encode.c would require at least
 some changes when
 [https://github.com/FFmpeg/FFmpeg/blob/f30a41a6086eb8c10f66090739a2a4f8491c3c7a/doc/examples/vaapi_encode.c#L177
 loading data/setting linesize] (bgr0 is 32 bpp single plane, yuyv422 is
 16bpp single plane).

 All three formats don't work in 4.2 (Ubuntu 20.04) with both i965 and iHD
 with same effect and message ("No usable encoding profile found")

--
Ticket URL: <https://trac.ffmpeg.org/ticket/8902#comment:2>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker