[FFmpeg-devel] [PATCH V2] libavfilter/vaapi: enable vaapi rotation feature via call Intel iHD driver

Zhou, Zachary zachary.zhou at intel.com
Fri Oct 26 18:13:21 EEST 2018



> -----Original Message-----
> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces at ffmpeg.org] On Behalf Of
> Rostislav Pehlivanov
> Sent: Friday, October 26, 2018 9:02 PM
> To: FFmpeg development discussions and patches <ffmpeg-devel at ffmpeg.org>
> Subject: Re: [FFmpeg-devel] [PATCH V2] libavfilter/vaapi: enable vaapi
> rotation feature via call Intel iHD driver
> 
> On Fri, 26 Oct 2018 at 04:10, Zhou, Zachary <zachary.zhou at intel.com> wrote:
> 
> >
> >
> > > -----Original Message-----
> > > From: ffmpeg-devel [mailto:ffmpeg-devel-bounces at ffmpeg.org] On
> > > Behalf Of sean darcy
> > > Sent: Thursday, October 25, 2018 11:51 PM
> > > To: ffmpeg-devel at ffmpeg.org
> > > Subject: Re: [FFmpeg-devel] [PATCH V2] libavfilter/vaapi: enable
> > > vaapi rotation feature via call Intel iHD driver
> > >
> > > On 10/25/18 2:52 AM, Zachary Zhou wrote:
> > > > It supports clockwise rotation by 0/90/180/270 degrees defined in
> > > > va/va_vpp.h, tested following command line on SKL platform
> > > >
> > > > dir=0 for 0   degree
> > > > dir=1 for 90  degree
> > > > dir=2 for 180 degree
> > > > dir=3 for 270 degree
> > > >
> > > > ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128
> > > > -hwaccel_output_format vaapi -i input.264 -vf "rotation_vaapi=dir=1"
> > > > -c:v h264_vaapi output.h264
> > > >
> > > > Signed-off-by: Zachary Zhou <zachary.zhou at intel.com>
> > > > ---
> > > >   configure                     |   3 +
> > > >   libavfilter/Makefile          |   1 +
> > > >   libavfilter/allfilters.c      |   1 +
> > > >   libavfilter/vaapi_vpp.h       |   1 +
> > > >   libavfilter/vf_rotate_vaapi.c | 252
> > ++++++++++++++++++++++++++++++++++
> > > >   5 files changed, 258 insertions(+)
> > > >   create mode 100644 libavfilter/vf_rotate_vaapi.c
> > > >
> > > > diff --git a/configure b/configure index 85d5dd5962..33aced3d78
> > > > 100755
> > > > --- a/configure
> > > > +++ b/configure
> > > > @@ -6390,6 +6390,9 @@ if enabled vaapi; then
> > > >       fi
> > > >
> > > >       check_cpp_condition vaapi_1 "va/va.h" "VA_CHECK_VERSION(1, 0,
> 0)"
> > > > +    if ! check_struct "va/va.h" "struct _VAProcPipelineCaps"
> > rotation_flags;
> > > then
> > > > +        disable rotation_vaapi_filter
> > > > +    fi
> > > >   fi
> > > >
> > > >   if enabled_all opencl libdrm ; then diff --git
> > > > a/libavfilter/Makefile b/libavfilter/Makefile index
> > > > 108a2f87d7..534650364a 100644
> > > > --- a/libavfilter/Makefile
> > > > +++ b/libavfilter/Makefile
> > > > @@ -349,6 +349,7 @@ OBJS-$(CONFIG_SETRANGE_FILTER)               +=
> > > vf_setparams.o
> > > >   OBJS-$(CONFIG_SETSAR_FILTER)                 += vf_aspect.o
> > > >   OBJS-$(CONFIG_SETTB_FILTER)                  += settb.o
> > > >   OBJS-$(CONFIG_SHARPNESS_VAAPI_FILTER)        += vf_misc_vaapi.o
> > > vaapi_vpp.o
> > > > +OBJS-$(CONFIG_ROTATION_VAAPI_FILTER)         += vf_rotate_vaapi.o
> > > vaapi_vpp.o
> > > >   OBJS-$(CONFIG_SHOWINFO_FILTER)               += vf_showinfo.o
> > > >   OBJS-$(CONFIG_SHOWPALETTE_FILTER)            += vf_showpalette.o
> > > >   OBJS-$(CONFIG_SHUFFLEFRAMES_FILTER)          += vf_shuffleframes.o
> > > > diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c
> > > > index
> > > > 557590850b..4b90a7f440 100644
> > > > --- a/libavfilter/allfilters.c
> > > > +++ b/libavfilter/allfilters.c
> > > > @@ -333,6 +333,7 @@ extern AVFilter ff_vf_setrange;
> > > >   extern AVFilter ff_vf_setsar;
> > > >   extern AVFilter ff_vf_settb;
> > > >   extern AVFilter ff_vf_sharpness_vaapi;
> > > > +extern AVFilter ff_vf_rotation_vaapi;
> > > >   extern AVFilter ff_vf_showinfo;
> > > >   extern AVFilter ff_vf_showpalette;
> > > >   extern AVFilter ff_vf_shuffleframes; diff --git
> > > > a/libavfilter/vaapi_vpp.h b/libavfilter/vaapi_vpp.h index
> > > > 0bc31018d4..cfe19b689f 100644
> > > > --- a/libavfilter/vaapi_vpp.h
> > > > +++ b/libavfilter/vaapi_vpp.h
> > > > @@ -44,6 +44,7 @@ typedef struct VAAPIVPPContext {
> > > >       int output_width;   // computed width
> > > >       int output_height;  // computed height
> > > >
> > > > +    int rotation_state;
> > > >       VABufferID         filter_buffers[VAProcFilterCount];
> > > >       int                nb_filter_buffers;
> > > >
> > > > diff --git a/libavfilter/vf_rotate_vaapi.c
> > > > b/libavfilter/vf_rotate_vaapi.c new file mode 100644 index
> > > > 0000000000..34c270e9c7
> > > > --- /dev/null
> > > > +++ b/libavfilter/vf_rotate_vaapi.c
> > > > @@ -0,0 +1,252 @@
> > > > +/*
> > > > + * This file is part of FFmpeg.
> > > > + *
> > > > + * FFmpeg is free software; you can redistribute it and/or
> > > > + * modify it under the terms of the GNU Lesser General Public
> > > > + * License as published by the Free Software Foundation; either
> > > > + * version 2.1 of the License, or (at your option) any later version.
> > > > + *
> > > > + * FFmpeg is distributed in the hope that it will be useful,
> > > > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > > > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> > > GNU
> > > > + * Lesser General Public License for more details.
> > > > + *
> > > > + * You should have received a copy of the GNU Lesser General
> > > > +Public
> > > > + * License along with FFmpeg; if not, write to the Free Software
> > > > + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
> > > > +02110-1301 USA  */ #include <string.h>
> > > > +
> > > > +#include "libavutil/avassert.h"
> > > > +#include "libavutil/mem.h"
> > > > +#include "libavutil/opt.h"
> > > > +#include "libavutil/pixdesc.h"
> > > > +
> > > > +#include "avfilter.h"
> > > > +#include "formats.h"
> > > > +#include "internal.h"
> > > > +#include "vaapi_vpp.h"
> > > > +
> > > > +// Rotation angle values
> > > > +enum RotationAngle {
> > > > +    ROTATION_0   = 0,
> > > > +    ROTATION_90  = 1,
> > > > +    ROTATION_180 = 2,
> > > > +    ROTATION_270 = 3,
> > > > +
> > > > +    ROTATION_MIN     = ROTATION_0,
> > > > +    ROTATION_MAX     = ROTATION_270,
> > > > +    ROTATION_DEFAULT = ROTATION_0, };
> > > > +
> > > > +typedef struct RotationVAAPIContext {
> > > > +    VAAPIVPPContext vpp_ctx; // must be the first field
> > > > +
> > > > +    int rotation;        // enable rotation.
> > > > +} RotationVAAPIContext;
> > > > +
> > > > +static int rotation_vaapi_build_filter_params(AVFilterContext
> > > > +*avctx) {
> > > > +    VAAPIVPPContext *vpp_ctx  = avctx->priv;
> > > > +    RotationVAAPIContext *ctx = avctx->priv;
> > > > +
> > > > +    VAStatus vas;
> > > > +    int support_flag;
> > > > +
> > > > +    VAProcPipelineCaps pipeline_caps;
> > > > +
> > > > +    memset(&pipeline_caps, 0, sizeof(pipeline_caps));
> > > > +    vas = vaQueryVideoProcPipelineCaps(vpp_ctx->hwctx->display,
> > > > +                                       vpp_ctx->va_context,
> > > > +                                       NULL, 0,
> > > > +                                       &pipeline_caps);
> > > > +    if (vas != VA_STATUS_SUCCESS) {
> > > > +        av_log(avctx, AV_LOG_ERROR, "Failed to query pipeline "
> > > > +               "caps: %d (%s).\n", vas, vaErrorStr(vas));
> > > > +        return AVERROR(EIO);
> > > > +    }
> > > > +
> > > > +    if (!pipeline_caps.rotation_flags) {
> > > > +        av_log(avctx, AV_LOG_ERROR, "VAAPI driver doesn't support
> > > rotation\n");
> > > > +        return AVERROR(EINVAL);
> > > > +    }
> > > > +
> > > > +    switch (ctx->rotation) {
> > > > +    case ROTATION_0:
> > > > +        vpp_ctx->rotation_state = VA_ROTATION_NONE;
> > > > +        break;
> > > > +    case ROTATION_90:
> > > > +        vpp_ctx->rotation_state = VA_ROTATION_90;
> > > > +        break;
> > > > +    case ROTATION_180:
> > > > +        vpp_ctx->rotation_state = VA_ROTATION_180;
> > > > +        break;
> > > > +    case ROTATION_270:
> > > > +        vpp_ctx->rotation_state = VA_ROTATION_270;
> > > > +        break;
> > > > +    default:
> > > > +        av_log(avctx, AV_LOG_ERROR, "Failed to set rotation_state
> > > > + to
> > %d. "
> > > > +               "Clockwise %d, %d, %d and %d are only supported\n",
> > > > +               ctx->rotation,
> > > > +               ROTATION_0, ROTATION_90, ROTATION_180, ROTATION_270);
> > > > +        return AVERROR(EINVAL);
> > > > +    }
> > > > +
> > > > +    support_flag = pipeline_caps.rotation_flags &
> > > > +                   (1 << vpp_ctx->rotation_state);
> > > > +    if (!support_flag) {
> > > > +        av_log(avctx, AV_LOG_ERROR, "VAAPI driver doesn't support
> > %d\n",
> > > > +               ctx->rotation);
> > > > +        return AVERROR(EINVAL);
> > > > +    }
> > > > +
> > > > +    return 0;
> > > > +}
> > > > +
> > > > +static int rotation_vaapi_filter_frame(AVFilterLink *inlink,
> > > > +AVFrame
> > > > +*input_frame) {
> > > > +    AVFilterContext *avctx   = inlink->dst;
> > > > +    AVFilterLink *outlink    = avctx->outputs[0];
> > > > +    VAAPIVPPContext *vpp_ctx = avctx->priv;
> > > > +    AVFrame *output_frame    = NULL;
> > > > +    VASurfaceID input_surface, output_surface;
> > > > +    VARectangle input_region;
> > > > +
> > > > +    VAProcPipelineParameterBuffer params;
> > > > +    int err;
> > > > +
> > > > +    av_log(avctx, AV_LOG_DEBUG, "Filter input: %s, %ux%u
> > (%"PRId64").\n",
> > > > +           av_get_pix_fmt_name(input_frame->format),
> > > > +           input_frame->width, input_frame->height,
> > > > + input_frame->pts);
> > > > +
> > > > +    if (vpp_ctx->va_context == VA_INVALID_ID)
> > > > +        return AVERROR(EINVAL);
> > > > +
> > > > +    input_surface = (VASurfaceID)(uintptr_t)input_frame->data[3];
> > > > +    av_log(avctx, AV_LOG_DEBUG, "Using surface %#x for misc vpp
> > input.\n",
> > > > +           input_surface);
> > > > +
> > > > +    output_frame = ff_get_video_buffer(outlink, vpp_ctx->output_width,
> > > > +                                       vpp_ctx->output_height);
> > > > +    if (!output_frame) {
> > > > +        err = AVERROR(ENOMEM);
> > > > +        goto fail;
> > > > +    }
> > > > +
> > > > +    output_surface = (VASurfaceID)(uintptr_t)output_frame->data[3];
> > > > +    av_log(avctx, AV_LOG_DEBUG, "Using surface %#x for misc vpp
> > > output.\n",
> > > > +           output_surface);
> > > > +    memset(&params, 0, sizeof(params));
> > > > +    input_region = (VARectangle) {
> > > > +        .x      = 0,
> > > > +        .y      = 0,
> > > > +        .width  = input_frame->width,
> > > > +        .height = input_frame->height,
> > > > +    };
> > > > +
> > > > +    switch (vpp_ctx->rotation_state) {
> > > > +    case VA_ROTATION_NONE:
> > > > +    case VA_ROTATION_90:
> > > > +    case VA_ROTATION_180:
> > > > +    case VA_ROTATION_270:
> > > > +        params.rotation_state = vpp_ctx->rotation_state;
> > > > +        break;
> > > > +    default:
> > > > +        av_log(avctx, AV_LOG_ERROR, "VAAPI doesn't support %d\n",
> > > > +               vpp_ctx->rotation_state);
> > > > +        goto fail;
> > > > +    }
> > > > +
> > > > +    if (vpp_ctx->nb_filter_buffers) {
> > > > +        params.filters     = &vpp_ctx->filter_buffers[0];
> > > > +        params.num_filters = vpp_ctx->nb_filter_buffers;
> > > > +    }
> > > > +    params.surface = input_surface;
> > > > +    params.surface_region = &input_region;
> > > > +    params.surface_color_standard =
> > > > +        ff_vaapi_vpp_colour_standard(input_frame->colorspace);
> > > > +
> > > > +    params.output_region = NULL;
> > > > +    params.output_background_color = 0xff000000;
> > > > +    params.output_color_standard = params.surface_color_standard;
> > > > +
> > > > +    params.pipeline_flags = 0;
> > > > +    params.filter_flags = VA_FRAME_PICTURE;
> > > > +
> > > > +    err = ff_vaapi_vpp_render_picture(avctx, &params, output_surface);
> > > > +    if (err < 0)
> > > > +        goto fail;
> > > > +
> > > > +    err = av_frame_copy_props(output_frame, input_frame);
> > > > +    if (err < 0)
> > > > +        goto fail;
> > > > +    av_frame_free(&input_frame);
> > > > +
> > > > +    av_log(avctx, AV_LOG_DEBUG, "Filter output: %s, %ux%u
> > > (%"PRId64").\n",
> > > > +           av_get_pix_fmt_name(output_frame->format),
> > > > +           output_frame->width, output_frame->height,
> > > > + output_frame->pts);
> > > > +
> > > > +    return ff_filter_frame(outlink, output_frame);
> > > > +
> > > > +fail:
> > > > +    av_frame_free(&input_frame);
> > > > +    av_frame_free(&output_frame);
> > > > +    return err;
> > > > +}
> > > > +
> > > > +static av_cold int rotation_vaapi_init(AVFilterContext *avctx) {
> > > > +    VAAPIVPPContext *vpp_ctx = avctx->priv;
> > > > +
> > > > +    ff_vaapi_vpp_ctx_init(avctx);
> > > > +    vpp_ctx->pipeline_uninit     = ff_vaapi_vpp_pipeline_uninit;
> > > > +    vpp_ctx->build_filter_params = rotation_vaapi_build_filter_params;
> > > > +    vpp_ctx->output_format       = AV_PIX_FMT_NONE;
> > > > +
> > > > +    return 0;
> > > > +}
> > > > +
> > > > +#define ROFFSET(x) offsetof(RotationVAAPIContext, x) #define
> > > > +FLAGS (AV_OPT_FLAG_VIDEO_PARAM |
> AV_OPT_FLAG_FILTERING_PARAM)
> > > > +static
> > > const
> > > > +AVOption rotation_vaapi_options[] = {
> > > > +    { "dir", "set roation direction", ROFFSET(rotation),
> > AV_OPT_TYPE_INT,
> > > { .i64 = ROTATION_MIN }, ROTATION_MIN, ROTATION_MAX, FLAGS, "dir" },
> > > > +        { "clock_0",   "rotate clockwise 0 degrees",   0,
> > AV_OPT_TYPE_CONST,
> > > { .i64 = ROTATION_0 },   .flags=FLAGS, .unit = "dir" },
> > > > +        { "clock_90",  "rotate clockwise 90 degrees",  0,
> > AV_OPT_TYPE_CONST,
> > > { .i64 = ROTATION_90},   .flags=FLAGS, .unit = "dir" },
> > > > +        { "clock_180", "rotate clockwise 180 degrees", 0,
> > AV_OPT_TYPE_CONST,
> > > { .i64 = ROTATION_180 }, .flags=FLAGS, .unit = "dir" },
> > > > +        { "clock_270", "rotate clockwise 270 degrees", 0,
> > AV_OPT_TYPE_CONST,
> > > { .i64 = ROTATION_270 }, .flags=FLAGS, .unit = "dir" },
> > > > +    { NULL },
> > > > +};
> > > > +
> > > > +AVFILTER_DEFINE_CLASS(rotation_vaapi);
> > > > +
> > > > +static const AVFilterPad rotation_vaapi_inputs[] = {
> > > > +    {
> > > > +        .name         = "default",
> > > > +        .type         = AVMEDIA_TYPE_VIDEO,
> > > > +        .filter_frame = &rotation_vaapi_filter_frame,
> > > > +        .config_props = &ff_vaapi_vpp_config_input,
> > > > +    },
> > > > +    { NULL }
> > > > +};
> > > > +
> > > > +static const AVFilterPad rotation_vaapi_outputs[] = {
> > > > +    {
> > > > +        .name = "default",
> > > > +        .type = AVMEDIA_TYPE_VIDEO,
> > > > +        .config_props = &ff_vaapi_vpp_config_output,
> > > > +    },
> > > > +    { NULL }
> > > > +};
> > > > +
> > > > +AVFilter ff_vf_rotation_vaapi = {
> > > > +    .name          = "rotation_vaapi",
> > > > +    .description   = NULL_IF_CONFIG_SMALL("VAAPI VPP for rotation"),
> > > > +    .priv_size     = sizeof(RotationVAAPIContext),
> > > > +    .init          = &rotation_vaapi_init,
> > > > +    .uninit        = &ff_vaapi_vpp_ctx_uninit,
> > > > +    .query_formats = &ff_vaapi_vpp_query_formats,
> > > > +    .inputs        = rotation_vaapi_inputs,
> > > > +    .outputs       = rotation_vaapi_outputs,
> > > > +    .priv_class    = &rotation_vaapi_class,
> > > > +    .flags_internal = FF_FILTER_FLAG_HWFRAME_AWARE, };
> > > >
> > > How about:
> > >
> > > deg=0 for 0   degree
> > > deg=90 for 90  degree
> > > deg=180 for 180 degree
> > > deg=270 for 270 degree
> > >
> > > ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128
> > >   -hwaccel_output_format vaapi -i input.264 -vf "rotation_vaapi=deg=90"
> > >   -c:v h264_vaapi output.h264
> > >
> > > The rotation is in degrees , not direction. And using the actual
> > > degrees
> > instead
> > > of magic numbers makes it more obvious for the user.
> >
> >
> > My first version of the
> > patch(https://patchwork.ffmpeg.org/patch/10754/)
> > just like this.
> >
> > ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128
> > -hwaccel_output_format vaapi -i input.264 -vf "rotation_vaapi=angle=90"
> > -c:v h264_vaapi output.h264
> >
> > And I got comments to add sub-options. the sub-options are clock_0,
> > clock_90, clock_180, clock_270, the command line likes this.
> >
> > ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128
> > -hwaccel_output_format vaapi -i input.264 -vf "rotation_vaapi= clock_270"
> > -c:v h264_vaapi output.h264
> >
> >
> > >
> > > _______________________________________________
> > > ffmpeg-devel mailing list
> > > ffmpeg-devel at ffmpeg.org
> > > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> > _______________________________________________
> > ffmpeg-devel mailing list
> > ffmpeg-devel at ffmpeg.org
> > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> 
> Please, call it transpose_vaapi, and make it accept the same options as the
> transpose filter we have.
> transpose != rotation.

Actually, I want to change back to "rotation_vaapi=angle=90", I agree  "sean darcy <seandarcy2 at gmail.com>" comments, and I think this filter is different with transpose.

1. This filter no vflip/hflip support, but transpose has vflip/hflip, in VAAPI another feature Mirroring, hflip/vflip are supported by using VAProcPipelineParameterBuffer::mirror_state. I was planning enable VAAPI mirroring in another patch.
2. Currently VAAPI rotation only support 0/90/180/270 degrees, not sure VAAPI will add more degrees in future, so I want to keep the name rotation_vaapi to align VAAPI. 
3. I don't want to put VAAPI rotation and VAAPI mirroring together to enable a similar but not same with transpose currently, Because the same options and the different functions may call people confuse.

My first version patch has already included  the discussion regarding the filter name and options. https://patchwork.ffmpeg.org/patch/10754/


> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel


More information about the ffmpeg-devel mailing list