[FFmpeg-devel] [PATCH 1/2] avfilter/vf_colorkey: Add colorkey video filter

Michael Niedermayer michaelni at gmx.at
Sat Jun 13 04:40:12 CEST 2015


On Thu, Jun 11, 2015 at 09:06:19PM +0200, Timo Rothenpieler wrote:
> ---
>  Changelog                 |   1 +
>  MAINTAINERS               |   1 +
>  doc/filters.texi          |  39 ++++++++
>  libavfilter/Makefile      |   1 +
>  libavfilter/allfilters.c  |   1 +
>  libavfilter/vf_colorkey.c | 221 ++++++++++++++++++++++++++++++++++++++++++++++
>  6 files changed, 264 insertions(+)
>  create mode 100644 libavfilter/vf_colorkey.c
> 
> diff --git a/Changelog b/Changelog
> index aa5753e..9a53286 100644
> --- a/Changelog
> +++ b/Changelog
> @@ -2,6 +2,7 @@ Entries are sorted chronologically from oldest to youngest within each release,
>  releases are sorted from youngest to oldest.
>  
>  version <next>:
> +- colorkey video filter
>  
>  
>  version 2.7:
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 795e904..7b239a1 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -352,6 +352,7 @@ Filters:
>    avf_showcqt.c                         Muhammad Faiz
>    vf_blend.c                            Paul B Mahol
>    vf_colorbalance.c                     Paul B Mahol
> +  vf_colorkey.c                         Timo Rothenpieler
>    vf_dejudder.c                         Nicholas Robbins
>    vf_delogo.c                           Jean Delvare (CC <khali at linux-fr.org>)
>    vf_drawbox.c/drawgrid                 Andrey Utkin
> diff --git a/doc/filters.texi b/doc/filters.texi
> index 4b5c3c1..5696362 100644
> --- a/doc/filters.texi
> +++ b/doc/filters.texi
> @@ -3029,6 +3029,45 @@ colorbalance=rs=.3
>  @end example
>  @end itemize
>  
> + at section colorkey
> +RGB colorspace color keying.
> +
> +The filter accepts the following options:
> +
> + at table @option
> + at item color
> +The color which will be replaced with transparency.
> +
> + at item similarity
> +Similarity percentage with the key color.
> +
> +0.01 matches only the exact key color, while 1.0 matches everything.
> +
> + at item blend
> +Blend percentage.
> +
> +0.0 makes pixels either fully transparent, or not transparent at all.
> +
> +Higher values result in semi-transparent pixels, with a higher transparency
> +the more similar the pixels color is to the key color.
> + at end table
> +
> + at subsection Examples
> +
> + at itemize
> + at item
> +Make every green pixel in the input image transparent:
> + at example
> +ffmpeg -i input.png -vf colorkey=green out.png
> + at end example
> +
> + at item
> +Overlay a greenscreen-video on top of a static background image.
> + at example
> +ffmpeg -i background.png -i video.mp4 -filter_complex "[1:v]colorkey=0x3BBD1E:0.3:0.2[ckout];[0:v][ckout]overlay[out]" -map "[out]" output.flv
> + at end example
> + at end itemize
> +
>  @section colorlevels
>  
>  Adjust video input frames using levels.
> diff --git a/libavfilter/Makefile b/libavfilter/Makefile
> index bf5a549..fc9f455 100644
> --- a/libavfilter/Makefile
> +++ b/libavfilter/Makefile
> @@ -102,6 +102,7 @@ OBJS-$(CONFIG_BOXBLUR_FILTER)                += vf_boxblur.o
>  OBJS-$(CONFIG_CODECVIEW_FILTER)              += vf_codecview.o
>  OBJS-$(CONFIG_COLORBALANCE_FILTER)           += vf_colorbalance.o
>  OBJS-$(CONFIG_COLORCHANNELMIXER_FILTER)      += vf_colorchannelmixer.o
> +OBJS-$(CONFIG_COLORKEY_FILTER)               += vf_colorkey.o
>  OBJS-$(CONFIG_COLORLEVELS_FILTER)            += vf_colorlevels.o
>  OBJS-$(CONFIG_COLORMATRIX_FILTER)            += vf_colormatrix.o
>  OBJS-$(CONFIG_COPY_FILTER)                   += vf_copy.o
> diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c
> index 0244585..415ed1b 100644
> --- a/libavfilter/allfilters.c
> +++ b/libavfilter/allfilters.c
> @@ -118,6 +118,7 @@ void avfilter_register_all(void)
>      REGISTER_FILTER(CODECVIEW,      codecview,      vf);
>      REGISTER_FILTER(COLORBALANCE,   colorbalance,   vf);
>      REGISTER_FILTER(COLORCHANNELMIXER, colorchannelmixer, vf);
> +    REGISTER_FILTER(COLORKEY,       colorkey,       vf);
>      REGISTER_FILTER(COLORLEVELS,    colorlevels,    vf);
>      REGISTER_FILTER(COLORMATRIX,    colormatrix,    vf);
>      REGISTER_FILTER(COPY,           copy,           vf);
> diff --git a/libavfilter/vf_colorkey.c b/libavfilter/vf_colorkey.c
> new file mode 100644
> index 0000000..5db30e8
> --- /dev/null
> +++ b/libavfilter/vf_colorkey.c
> @@ -0,0 +1,221 @@
> +/*
> + * Copyright (c) 2015 Timo Rothenpieler <timo at rothenpieler.org>
> + *
> + * This file is part of FFmpeg.
> + *
> + * FFmpeg is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU Lesser General Public
> + * License as published by the Free Software Foundation; either
> + * version 2.1 of the License, or (at your option) any later version.
> + *
> + * FFmpeg is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> + * Lesser General Public License for more details.
> + *
> + * You should have received a copy of the GNU Lesser General Public
> + * License along with FFmpeg; if not, write to the Free Software
> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
> + */
> +
> +#include "libavutil/opt.h"
> +#include "libavutil/imgutils.h"
> +#include "avfilter.h"
> +#include "formats.h"
> +#include "internal.h"
> +#include "video.h"
> +
> +typedef struct ColorkeyContext {
> +    const AVClass *class;
> +
> +    /* color offsets rgba */
> +    int co[4];
> +
> +    uint8_t colorkey_rgba[4];
> +    float similarity;
> +    float blend;
> +} ColorkeyContext;
> +
> +static int convert_format(int fmt)
> +{
> +    switch(fmt) {
> +    case AV_PIX_FMT_0RGB:
> +        return AV_PIX_FMT_ARGB;
> +    case AV_PIX_FMT_0BGR:
> +        return AV_PIX_FMT_ABGR;
> +    case AV_PIX_FMT_BGR0:
> +        return AV_PIX_FMT_BGRA;
> +    case AV_PIX_FMT_RGB0:
> +        return AV_PIX_FMT_RGBA;
> +    default:
> +        return fmt;
> +    }
> +}
> +
> +static uint8_t do_colorkey_pixel(ColorkeyContext *ctx, uint8_t r, uint8_t g, uint8_t b)
> +{
> +    int dr = (int)r - ctx->colorkey_rgba[0];
> +    int dg = (int)g - ctx->colorkey_rgba[1];
> +    int db = (int)b - ctx->colorkey_rgba[2];
> +
> +    double diff = sqrt((dr * dr + dg * dg + db * db) / (255.0 * 255.0));
> +
> +    if (ctx->blend > 0.0001) {

> +        return av_clipd(FFMAX(diff - ctx->similarity, 0.0) / ctx->blend, 0.0, 1.0) * 255.0;

isnt the FFMAX redundant ?


> +    } else {
> +        return ((diff - ctx->similarity) > 0.0) ? 255 : 0;

diff > ctx->similarity ? ...


> +    }
> +}
> +
> +static int do_colorkey_slice(AVFilterContext *avctx, void *arg, int jobnr, int nb_jobs)
> +{
> +    AVFrame *frame = arg;
> +
> +    const int slice_start = (frame->height * jobnr) / nb_jobs;
> +    const int slice_end = (frame->height * (jobnr + 1)) / nb_jobs;
> +
> +    ColorkeyContext *ctx = avctx->priv;
> +
> +    int o, x, y;
> +
> +    for (y = slice_start; y < slice_end; ++y) {
> +        for (x = 0; x < frame->width; ++x) {
> +            o = frame->linesize[0] * y + x * 4;
> +
> +            frame->data[0][o + ctx->co[3]] =
> +                do_colorkey_pixel(ctx,
> +                                  frame->data[0][o + ctx->co[0]],
> +                                  frame->data[0][o + ctx->co[1]],
> +                                  frame->data[0][o + ctx->co[2]]);
> +        }
> +    }
> +
> +    return 0;
> +}
> +
> +static int filter_frame(AVFilterLink *link, AVFrame *frame)
> +{
> +    AVFilterContext *avctx = link->dst;
> +    int res;
> +
> +    if (res = av_frame_make_writable(frame))
> +        return res;
> +
> +    frame->format = convert_format(frame->format);
> +
> +    if (res = avctx->internal->execute(avctx, do_colorkey_slice, frame, NULL, FFMIN(frame->height, avctx->graph->nb_threads)))
> +        return res;
> +
> +    return ff_filter_frame(avctx->outputs[0], frame);
> +}
> +
> +static av_cold int config_output(AVFilterLink *outlink)
> +{
> +    AVFilterContext *avctx = outlink->src;
> +    ColorkeyContext *ctx = avctx->priv;
> +    const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(outlink->format);
> +    int i;
> +
> +    outlink->w = avctx->inputs[0]->w;
> +    outlink->h = avctx->inputs[0]->h;
> +    outlink->time_base = avctx->inputs[0]->time_base;
> +
> +    for (i = 0; i < 4; ++i)
> +        ctx->co[i] = desc->comp[i].offset_plus1 - 1;
> +
> +    return 0;
> +}
> +
> +static av_cold int query_formats(AVFilterContext *avctx)
> +{
> +    static const enum AVPixelFormat pixel_fmts_in[] = {
> +        AV_PIX_FMT_0RGB,
> +        AV_PIX_FMT_ARGB,
> +        AV_PIX_FMT_RGB0,
> +        AV_PIX_FMT_RGBA,
> +        AV_PIX_FMT_0BGR,
> +        AV_PIX_FMT_ABGR,
> +        AV_PIX_FMT_BGR0,
> +        AV_PIX_FMT_BGRA,
> +        AV_PIX_FMT_NONE
> +    };
> +
> +    static const enum AVPixelFormat pixel_fmts_out[] = {
> +        AV_PIX_FMT_ARGB,
> +        AV_PIX_FMT_RGBA,
> +        AV_PIX_FMT_ABGR,
> +        AV_PIX_FMT_BGRA,
> +        AV_PIX_FMT_NONE
> +    };
> +
> +    AVFilterFormats *input_formats;
> +    AVFilterFormats *output_formats;
> +
> +    output_formats = ff_make_format_list(pixel_fmts_out);
> +    if (!output_formats)
> +        return AVERROR(ENOMEM);
> +
> +    if (!avctx->inputs[0]->out_formats) {
> +        input_formats = ff_make_format_list(pixel_fmts_in);
> +
> +        if (!input_formats)
> +            return AVERROR(ENOMEM);
> +
> +        ff_formats_ref(input_formats, &avctx->inputs[0]->out_formats);
> +    }
> +
> +    if (!avctx->outputs[0]->in_formats) {
> +        output_formats = ff_make_format_list(pixel_fmts_out);
> +
> +        if (!output_formats)
> +            return AVERROR(ENOMEM);
> +
> +        ff_formats_ref(output_formats, &avctx->outputs[0]->in_formats);
> +    }
> +

> +    if (avctx->outputs[0]->format != convert_format(avctx->inputs[0]->format))
> +        return AVERROR(EAGAIN);

maybe iam missing something but
what gurantees that these will ever be equal ?

I suggest you only support RGBA and permutated formats as input

RGB0->RGBA convertion can be improved to not copy if it does

alternatively you can support any input and output format and do the
permutation if needed if they happen not to match up, this may in
case the previous filter and the next filter have no common format
even avoid a swcale sample permutation pass

I suggets against the use of EAGAIN, its not trivial to use
one cannot just return EAGAIN arbitrarily and hope that somehow
whatever lead to the situation triggering EAGAIN will not be there
on every future call.
Also if  one filter waits on the next filter to provide something
and the next filter waits on the previous, each returning EAGAIN
you have a deadlock
or if every filter waits on  the previous and the filters form a
loop/cycle there too will be a problem

The way the system works is that each filter exports for each of
its in and outputs a list of formats that it supports and which of
these links need to match format wise.
The avfilter code then uses this information for a filter graph which
can contain arbitrary many filters branches and loops/cycles and
it will find a solution with few converts in polynomial time


[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Why not whip the teacher when the pupil misbehaves? -- Diogenes of Sinope
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: Digital signature
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20150613/dbd4a2a0/attachment.asc>


More information about the ffmpeg-devel mailing list