[PATCH] lavfi: add realtime filter.
Similar to the -re option in ffmpeg that only works for input files. Can be used at any place in the filter graph. Signed-off-by: Nicolas George <george@nsup.org> --- Changelog | 1 + doc/filters.texi | 16 ++++++ libavfilter/Makefile | 2 + libavfilter/allfilters.c | 2 + libavfilter/f_realtime.c | 130 +++++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 151 insertions(+) create mode 100644 libavfilter/f_realtime.c diff --git a/Changelog b/Changelog index abaac19..1c8fe37 100644 --- a/Changelog +++ b/Changelog @@ -25,6 +25,7 @@ version <next>: - wve demuxer - zero-copy Intel QSV transcoding in ffmpeg - shuffleframes filter +- realtime filter version 2.8: diff --git a/doc/filters.texi b/doc/filters.texi index 5a35bde..141394c 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -13052,6 +13052,22 @@ following one, the permission might not be received as expected in that following filter. Inserting a @ref{format} or @ref{aformat} filter before the perms/aperms filter can avoid this problem. +@section realtime, arealtime + +Slow down filtering to match real time approximatively. + +These filters will pause the filtering for a variable amount of time to +match the output rate with the input timestamps. +They are similar to the @option{re} option to @code{ffmpeg}. + +They accept the following options: + +@table @option +@item limit +Time limit for the pauses. Any pause longer than that will be considered +a timestamp discontinuity and reset the timer. Default is 2 seconds. +@end table + @section select, aselect Select frames to pass in output. diff --git a/libavfilter/Makefile b/libavfilter/Makefile index dd4f547..9c07d98 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -39,6 +39,7 @@ OBJS-$(CONFIG_ANULL_FILTER) += af_anull.o OBJS-$(CONFIG_APAD_FILTER) += af_apad.o OBJS-$(CONFIG_APERMS_FILTER) += f_perms.o OBJS-$(CONFIG_APHASER_FILTER) += af_aphaser.o generate_wave_table.o +OBJS-$(CONFIG_AREALTIME_FILTER) += f_realtime.o OBJS-$(CONFIG_ARESAMPLE_FILTER) += af_aresample.o OBJS-$(CONFIG_AREVERSE_FILTER) += f_reverse.o OBJS-$(CONFIG_ASELECT_FILTER) += f_select.o @@ -196,6 +197,7 @@ OBJS-$(CONFIG_PSNR_FILTER) += vf_psnr.o dualinput.o framesync. OBJS-$(CONFIG_PULLUP_FILTER) += vf_pullup.o OBJS-$(CONFIG_QP_FILTER) += vf_qp.o OBJS-$(CONFIG_RANDOM_FILTER) += vf_random.o +OBJS-$(CONFIG_REALTIME_FILTER) += f_realtime.o OBJS-$(CONFIG_REMOVEGRAIN_FILTER) += vf_removegrain.o OBJS-$(CONFIG_REMOVELOGO_FILTER) += bbox.o lswsutils.o lavfutils.o vf_removelogo.o OBJS-$(CONFIG_REPEATFIELDS_FILTER) += vf_repeatfields.o diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 8e363a0..239fbb5 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -61,6 +61,7 @@ void avfilter_register_all(void) REGISTER_FILTER(APAD, apad, af); REGISTER_FILTER(APERMS, aperms, af); REGISTER_FILTER(APHASER, aphaser, af); + REGISTER_FILTER(AREALTIME, arealtime, af); REGISTER_FILTER(ARESAMPLE, aresample, af); REGISTER_FILTER(AREVERSE, areverse, af); REGISTER_FILTER(ASELECT, aselect, af); @@ -217,6 +218,7 @@ void avfilter_register_all(void) REGISTER_FILTER(PULLUP, pullup, vf); REGISTER_FILTER(QP, qp, vf); REGISTER_FILTER(RANDOM, random, vf); + REGISTER_FILTER(REALTIME, realtime, vf); REGISTER_FILTER(REMOVEGRAIN, removegrain, vf); REGISTER_FILTER(REMOVELOGO, removelogo, vf); REGISTER_FILTER(REPEATFIELDS, repeatfields, vf); diff --git a/libavfilter/f_realtime.c b/libavfilter/f_realtime.c new file mode 100644 index 0000000..a50141c --- /dev/null +++ b/libavfilter/f_realtime.c @@ -0,0 +1,130 @@ +/* + * Copyright (c) 2015 Nicolas George + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public License + * as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public License + * along with FFmpeg; if not, write to the Free Software Foundation, Inc., + * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "libavutil/opt.h" +#include "libavutil/time.h" +#include "avfilter.h" +#include "internal.h" + +typedef struct RealtimeContext { + const AVClass *class; + int64_t delta; + int64_t limit; + unsigned inited; +} RealtimeContext; + +static int filter_frame(AVFilterLink *inlink, AVFrame *frame) +{ + AVFilterContext *ctx = inlink->dst; + RealtimeContext *s = ctx->priv; + + if (frame->pts != AV_NOPTS_VALUE) { + int64_t pts = av_rescale_q(frame->pts, inlink->time_base, AV_TIME_BASE_Q); + int64_t now = av_gettime_relative(); + int64_t sleep = pts - now + s->delta; + if (!s->inited) { + s->inited = 1; + sleep = 0; + s->delta = now - pts; + } + if (sleep > s->limit || sleep < -s->limit) { + av_log(ctx, AV_LOG_WARNING, + "time discontinuity detected: %"PRIi64" us, resetting\n", + sleep); + sleep = 0; + s->delta = now - pts; + } + if (sleep > 0) { + av_log(ctx, AV_LOG_DEBUG, "sleeping %"PRIi64" us\n", sleep); + av_usleep(sleep); + } + } + return ff_filter_frame(inlink->dst->outputs[0], frame); +} + +#define OFFSET(x) offsetof(RealtimeContext, x) +#define FLAGS AV_OPT_FLAG_VIDEO_PARAM | AV_OPT_FLAG_AUDIO_PARAM | AV_OPT_FLAG_FILTERING_PARAM +static const AVOption options[] = { + { "limit", "sleep time limit", OFFSET(limit), AV_OPT_TYPE_DURATION, { .i64 = 2000000 }, 0, INT64_MAX, FLAGS }, + { NULL } +}; + +#if CONFIG_REALTIME_FILTER +#define realtime_options options +AVFILTER_DEFINE_CLASS(realtime); + +static const AVFilterPad avfilter_vf_realtime_inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .filter_frame = filter_frame, + }, + { NULL } +}; + +static const AVFilterPad avfilter_vf_realtime_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + }, + { NULL } +}; + +AVFilter ff_vf_realtime = { + .name = "realtime", + .description = NULL_IF_CONFIG_SMALL("Slow down filtering to match realtime"), + .priv_size = sizeof(RealtimeContext), + .priv_class = &realtime_class, + .inputs = avfilter_vf_realtime_inputs, + .outputs = avfilter_vf_realtime_outputs, +}; +#endif /* CONFIG_REALTIME_FILTER */ + +#if CONFIG_AREALTIME_FILTER + +#define arealtime_options options +AVFILTER_DEFINE_CLASS(arealtime); + +static const AVFilterPad arealtime_inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_AUDIO, + .filter_frame = filter_frame, + }, + { NULL } +}; + +static const AVFilterPad arealtime_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_AUDIO, + }, + { NULL } +}; + +AVFilter ff_af_arealtime = { + .name = "arealtime", + .description = NULL_IF_CONFIG_SMALL("Slow down filtering to match realtime"), + .priv_size = sizeof(RealtimeContext), + .priv_class = &arealtime_class, + .inputs = arealtime_inputs, + .outputs = arealtime_outputs, +}; +#endif /* CONFIG_AREALTIME_FILTER */ -- 2.6.1
+static int filter_frame(AVFilterLink *inlink, AVFrame *frame) +{ + AVFilterContext *ctx = inlink->dst; + RealtimeContext *s = ctx->priv; + + if (frame->pts != AV_NOPTS_VALUE) { + int64_t pts = av_rescale_q(frame->pts, inlink->time_base, AV_TIME_BASE_Q); + int64_t now = av_gettime_relative(); + int64_t sleep = pts - now + s->delta; + if (!s->inited) { + s->inited = 1; + sleep = 0; + s->delta = now - pts; + } + if (sleep > s->limit || sleep < -s->limit) { + av_log(ctx, AV_LOG_WARNING, + "time discontinuity detected: %"PRIi64" us, resetting\n", + sleep);
Won't this also be shown when there is no discontinuity but it isn't able to keep up with realtime (e.g. due to a very high frame rate)? The message is misleading in that situation.
+ sleep = 0; + s->delta = now - pts; + } + if (sleep > 0) { + av_log(ctx, AV_LOG_DEBUG, "sleeping %"PRIi64" us\n", sleep); + av_usleep(sleep); + } + } + return ff_filter_frame(inlink->dst->outputs[0], frame); +} + +#define OFFSET(x) offsetof(RealtimeContext, x) +#define FLAGS AV_OPT_FLAG_VIDEO_PARAM | AV_OPT_FLAG_AUDIO_PARAM | AV_OPT_FLAG_FILTERING_PARAM +static const AVOption options[] = { + { "limit", "sleep time limit", OFFSET(limit), AV_OPT_TYPE_DURATION, { .i64 = 2000000 }, 0, INT64_MAX, FLAGS }, + { NULL } +};
The argument to av_usleep() is an unsigned int. Should the maximum limit be UINT_MAX rather than INT64_MAX? Alternatively it could call av_usleep() in a loop if the value is too large for one call. - Mark
Le quartidi 4 brumaire, an CCXXIV, Mark Harris a écrit :
+ if (sleep > s->limit || sleep < -s->limit) { + av_log(ctx, AV_LOG_WARNING, + "time discontinuity detected: %"PRIi64" us, resetting\n", + sleep); Won't this also be shown when there is no discontinuity but it isn't able to keep up with realtime (e.g. due to a very high frame rate)? The message is misleading in that situation.
This is true, but I consider it an acceptable approximation for now. I am sure any user would guess what it means if the CPU is permanently at 100% and this warning is printed repeatedly. An heuristic could be added to detect when processing is slower than realtime and print a warning accordingly, but tuning it would require more time than I want to spend on this right now.
The argument to av_usleep() is an unsigned int. Should the maximum limit be UINT_MAX rather than INT64_MAX? Alternatively it could call av_usleep() in a loop if the value is too large for one call.
Good catch, although it would probably not matter in practice: that would involve a >1 hour gap between frames, the user actually wanting to sleep for that time and having set the limit accordingly. But all the same, I added this locally: + for (; sleep > 600000000; sleep -= 600000000) + av_usleep(600000000); (I used 600000000 instead of UINT_MAX because I do not trust values too near the limit value for that kind of system call; a spurious wakeup every 10 minutes is acceptable. Also, it will probably be easier to guess what it means when reading strace output.) Thanks for the review. Regards, -- Nicolas George
On Sun, Oct 25, 2015 at 17:35:32 +0100, Nicolas George wrote:
+@table @option +@item limit +Time limit for the pauses. Any pause longer than that will be considered +a timestamp discontinuity and reset the timer. Default is 2 seconds. +@end table [...] +static const AVOption options[] = { + { "limit", "sleep time limit", OFFSET(limit), AV_OPT_TYPE_DURATION, { .i64 = 2000000 }, 0, INT64_MAX, FLAGS }, + { NULL } +};
$ ffmpeg -h filter=realtime [...] realtime AVOptions: limit <duration> ..FVA... sleep time limit (default 2e+06) It's misleading to mention "seconds" and to expect microseconds. Or to mention "2" in one place and "2000000" in the other. -> Time limit for the pauses in microseconds. [...] Default is 2000000 (2 seconds). [...] { "limit", "sleep time limit (in microseconds)", OFFSET(limit), AV_OPT_TYPE_DURATION, { .i64 = 2000000 }, 0, INT64_MAX, FLAGS }, Moritz
Le quintidi 5 brumaire, an CCXXIV, Moritz Barsnick a écrit :
$ ffmpeg -h filter=realtime [...] realtime AVOptions: limit <duration> ..FVA... sleep time limit (default 2e+06)
It's misleading to mention "seconds" and to expect microseconds. Or to mention "2" in one place and "2000000" in the other.
-> Time limit for the pauses in microseconds. [...] Default is 2000000 (2 seconds). [...] { "limit", "sleep time limit (in microseconds)", OFFSET(limit), AV_OPT_TYPE_DURATION, { .i64 = 2000000 }, 0, INT64_MAX, FLAGS },
This is how it works with fields of type AV_OPT_TYPE_DURATION: the value is internally in microseconds (AV_TIME_BASE), but the user interface converts the strings in seconds, multiplying them by 1000000, or even in H:MM:SS.DDDDDD. The display of the default value is inconsistent, though. I just sent a patch about it. Regards, -- Nicolas George
This is how it works with fields of type AV_OPT_TYPE_DURATION: the value is internally in microseconds (AV_TIME_BASE), but the user interface converts the strings in seconds, multiplying them by 1000000, or even in H:MM:SS.DDDDDD.
Ah, I missed that fact. I just scanned "ffmpeg -h full" for other options using microseconds and how they document them - not their actual source and the type.
The display of the default value is inconsistent, though. I just sent a patch about it.
Good work. Moritz
Le quartidi 4 brumaire, an CCXXIV, Nicolas George a écrit :
Similar to the -re option in ffmpeg that only works for input files. Can be used at any place in the filter graph.
Signed-off-by: Nicolas George <george@nsup.org> --- Changelog | 1 + doc/filters.texi | 16 ++++++ libavfilter/Makefile | 2 + libavfilter/allfilters.c | 2 + libavfilter/f_realtime.c | 130 +++++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 151 insertions(+) create mode 100644 libavfilter/f_realtime.c
Pushed. Regards, -- Nicolas George
Hi, On 11/7/15, Nicolas George <george@nsup.org> wrote:
Le quartidi 4 brumaire, an CCXXIV, Nicolas George a écrit :
Similar to the -re option in ffmpeg that only works for input files. Can be used at any place in the filter graph.
Signed-off-by: Nicolas George <george@nsup.org> --- Changelog | 1 + doc/filters.texi | 16 ++++++ libavfilter/Makefile | 2 + libavfilter/allfilters.c | 2 + libavfilter/f_realtime.c | 130 +++++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 151 insertions(+) create mode 100644 libavfilter/f_realtime.c
Pushed.
Regards,
-- Nicolas George
What about making target speed configurable? 1x .. 1.5x .. 2x and so on.
Paul B Mahol (2018-10-25):
What about making target speed configurable? 1x .. 1.5x .. 2x and so on.
That would certainly be a useful feature. Regards, -- Nicolas George
participants (4)
-
Mark Harris -
Moritz Barsnick -
Nicolas George -
Paul B Mahol