[Libav-user] Using VAAPI to decode to RGB colors and to DRM PRIME fd

[Manuel Stoeckl]

Problem solved; it turns out that VAAPI has a postprocessing module
which can do (among other things) format conversions. The FFMPEG
interface to this is the scale_vaapi filter in libavfilter. As
integrating the scale_vaapi filter would have been tricky for my
specific use case, I instead used the VAAPI interface directly to 
set up a pipeline without any filters that copies the video output
surface onto a provided target DMABUF.

In case anyone is interested in the code, see [0].

[0]
https://gitlab.freedesktop.org/mstoeckl/waypipe/blob/77c92bb1a40b4ac779414d9ec2ace9841b283993/video.c#L207-363

On Tue, 9 Jul 2019 19:14:34 -0400
"M. Stoeckl" <libavuser at mstoeckl.com> wrote:

> Hello,
> 
> I have the following problem: there is a program outside of my control
> which uses and produces DRM PRIME fds only with a specific format (for
> example, DRM_FORMAT_XRGB8888/AV_PIX_FMT_BGR0), and I'd like to decode
> an input video format (such as H264, or VP9) in YUV colorspace and
> have its frames be written to the DRM PRIME fds, without copying the
> entire frame back to the CPU to do a color format conversion.
> 
> With the h264_vaapi encoder, I can use av_hwframe_map with
> AV_PIX_FMT_DRM_PRIME frames (with format DRM_FORMAT_XRGB8888) to
> produce AV_PIX_FMT_VAAPI frames with the software format
> AV_PIX_FMT_NV12, that the encoder then uses successfully. [Unless I
> am mistaken, this calls vaCreateSurfaces using the DRM frame data,
> and the VAAPI driver does the format conversion work.]
> 
> However, I'm not sure how best to decode with VAAPI; a similar
> approach trying av_hwframe_map before the decoder started produced
> AVERROR(ENOSYS). When I configure AVVAAPIFramesContext to have the
> frames be produced from an existing DRM PRIME file descriptor, the 
> decoder runs, but the original file descriptor contents aren't
> updated. Alternatively, converting the frames produced by the
> decoder, with av_hwframe_map, to DRM PRIME frames gives me data in
> NV12 format, which isn't what I want.
> 
> Has anyone managed to get something similar working, or even any
> GPU-only decoding pipeline that can decode YUV video into RGB images,
> handling any conversions on-GPU?
> 
> Any additional advice (or pointers to relevant example code not
> already in FFMPEG) would be appreciated.