[FFmpeg-devel] [PATCH 1/2] docs/filters: add documentation to all existing OpenCL filters

[Gyan Doshi]

Hi Danil,

Except overlay and the topic of format requirements, rest LGTM.

On 15-08-2018 05:30 PM, Danil Iashchenko wrote:
> docs/filters: add documentation to all existing OpenCL filters

> Added general instruction for format conversion in the start of overall section and separatly for overlay_opencl.
> 
> As far as I understand, different format conversion depends on which formats are supported by hwupload/hwdownload.
> So maybe it is better to clarify what formats are supported by them in hwupload/hwdownload section, since it is used not only by OpenCL and show set of possible input formats which do not need the conversion. All other unexpected formats should be converted by format filter.


> +Format conversion depends on format of the INPUT, on which logo is overlaying on.
> +
> +These are examples of different format conversions:
> + at example
> +if the formats of INPUT and LOGO are the same, no conversion is required. For example yuv420p is overlaying on yuv420p.
> + at end example
> + at example
> +if initial planes match, but the overlay has additional alpha plane, this case requires conversion to matching format. For example if the INPUT is yuv420p, LOGO should be yuva420p.
> + at end example
> + at example
> +if INPUT and LOGO has same-plane alpha, same layout and different but mathing format. For example RGBA is overlaying on RGB0.
> + at end example

> +Insert a JPG logo in the top-left corner of the mp4 INPUT. No conversion needed.
> + at example
> +-i INPUT -i LOGO -filter_complex "[0:v]hwupload[a], [1:v]hwupload[b], [a][b]overlay_opencl, hwdownload" OUTPUT

I tried to overlay a 4:4:4 JPG on a 4:2:0 video, using

     ffmpeg -i video.mp4 -i image.jpg
            -init_hw_device opencl=gpu:0.0 -filter_hw_device gpu
            -filter_complex "[0]hwupload at m[m];[1]hwupload at o[o];
                             [m][o]overlay_opencl=100:100,hwdownload"
            -t 1 -an -y test.mp4

This "succeeded", but the colors are, obviously, wrong. It works 
correctly with format=yuv420p before hwupload for the JPG. It also 
"works" with format=rgba, but the colors are again wrong.

Shouldn't there be validation of the matching of pixel formats within 
the filter?

> +Insert a PNG logo in the top-left corner of the mp4 INPUT. Explicit format conversion is a must.
> + at example
> +-i INPUT -i LOGO -filter_complex "[0:v]hwupload[a], [1:v]format=yuva420p, hwupload[b], [a][b]overlay_opencl, hwdownload" OUTPUT

I tried to overlay a RGBA input on a 4:2:0 video, using

     ffmpeg -i video.mp4 -i image.png
            -init_hw_device opencl=gpu:0.0 -filter_hw_device gpu
            -filter_complex "[0]hwupload at m[m];
                             [1]format=yuva420p,hwupload at o[o];
                             [m][o]overlay_opencl=100:100,hwdownload"
            -t 1 -an -y test.mp4

This failed with

     [hwupload @ 0000000006ee5e80] Failed to allocate frame to upload to.
     Error while filtering: Cannot allocate memory
     Failed to inject frame into filter network: Cannot allocate memory
     Error while processing the decoded data for stream #1:0

If I skip format, I get

     [AVHWDeviceContext @ 0000000002983700] OpenCL error: Invalid memory 
object.
     [Parsed_overlay_opencl_3 @ 0000000007125f00] Failed to set kernel 
argument 3: error -38.

If I use instead format=rgb24, it "works" but with a weird result. 
Again, validation?

If I insert format with arg yuva420p before both video and image, I get

     [hwdownload @ 0000000006ce14c0] Invalid output format yuv420p for 
hwframe download.
     [Parsed_hwdownload_5 @ 0000000006efb040] Failed to configure output 
pad on Parsed_hwdownload_5


Adding format=yuv420p after hwdownload, I get

[hwdownload @ 0000000006f2b440] Invalid output format yuv420p for 
hwframe download.
[Parsed_hwdownload_5 @ 00000000070148c0] Failed to configure output pad 
on Parsed_hwdownload_5


Adding instead format=yuva420p after hwdownload, I get

[hwupload @ 0000000006aaf2c0] Failed to allocate frame to upload to.


If I change both input formats to rgba *and* output format to rgba, it 
works but the image's alpha is ignored, with the result being a straight 
overlay of the image RGB components.

In overlay_opencl, I see

     if (main_planes == overlay_planes) {
         if (main_desc->nb_components == overlay_desc->nb_components)
             kernel = "overlay_no_alpha";
         else
             kernel = "overlay_internal_alpha";
         ctx->alpha_separate = 0;
     } else {
         kernel = "overlay_external_alpha";
         ctx->alpha_separate = 1;
     }

This looks wrong to me. For the purposes of overlay, only the presence 
of alpha on the 2nd input matters and should be handled when present.

All tests performed with on-die Intel HD / OpenCL runtime 1.2 on Win 7 
x64. My driver looks to be 3 years old.

Regards,
Gyan