[WIP] lavfi: add showaudio filter
Hi, in attachment an incomplete (but basically working) trasmedia A->V filter. Merry XMas. -- FFmpeg = Faithless & Frenzy Merry Perennial Extroverse Gangster
On Mon, Dec 26, 2011 at 12:14:39AM +0100, Stefano Sabatini wrote:
Hi,
in attachment an incomplete (but basically working) trasmedia A->V filter.
Merry XMas.
Yay ♥.♥ [...] -- Clément B.
On Mon, Dec 26, 2011 at 12:14:39AM +0100, Stefano Sabatini wrote:
Hi,
in attachment an incomplete (but basically working) trasmedia A->V filter.
\o/
Merry XMas.
merry xmas too [...] -- Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB While the State exists there can be no freedom; when there is freedom there will be no State. -- Vladimir Lenin
On date Monday 2011-12-26 00:14:39 +0100, Stefano Sabatini encoded:
Hi,
in attachment an incomplete (but basically working) trasmedia A->V filter.
Merry XMas. -- FFmpeg = Faithless & Frenzy Merry Perennial Extroverse Gangster
From 6eaf17db1b6ad105e0f273434bdbd3dc59026e81 Mon Sep 17 00:00:00 2001 From: Stefano Sabatini <stefasab@gmail.com> Date: Sun, 25 Dec 2011 17:37:53 +0100 Subject: [PATCH] lavfi: add showaudio filter
--- libavfilter/Makefile | 3 +- libavfilter/allfilters.c | 3 + libavfilter/avf_showaudio.c | 174 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 179 insertions(+), 1 deletions(-) create mode 100644 libavfilter/avf_showaudio.c
Main problem is now to understand how to visualize N samples in a packet using only w columns. One possibility: w defines the number of samples to visualize, this means that we have to visualize samplerate/w frames, i.e.: rate = samplerate/w For example with w=640, samplerate=22100: rate = 22100/640 ~= 35 Alternatively I could fix the rate, and scale the image accordingly by imposing: samples_to_show = samplerate / rate For example with rate = 10, samplerate = 22100 samples_to_show = 22100 / 10 = 2210 and I could use lsws to scale such an image to make it suit w. RDFT should be easier to add (same for some power/volume visualization). For more complex audio visualizations we may consider to write a MilkDrop/ProjectM wrapper. -- FFmpeg = Formidable & Fierce Mastodontic Pitiful Enlightened Gem
On date Tuesday 2011-12-27 14:58:03 +0100, Stefano Sabatini encoded:
On date Monday 2011-12-26 00:14:39 +0100, Stefano Sabatini encoded:
Subject: [PATCH] lavfi: add showaudio filter
--- libavfilter/Makefile | 3 +- libavfilter/allfilters.c | 3 + libavfilter/avf_showaudio.c | 174 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 179 insertions(+), 1 deletions(-) create mode 100644 libavfilter/avf_showaudio.
Main problem is now to understand how to visualize N samples in a packet using only w columns.
One possibility: w defines the number of samples to visualize, this means that we have to visualize samplerate/w frames, i.e.: rate = samplerate/w
For example with w=640, samplerate=22100: rate = 22100/640 ~= 35
Alternatively I could fix the rate, and scale the image accordingly by imposing: samples_to_show = samplerate / rate
For example with rate = 10, samplerate = 22100 samples_to_show = 22100 / 10 = 2210
and I could use lsws to scale such an image to make it suit w.
Updated implementing the first option. Still missing: * multiple channels support, every channel may be displayed on a separate area a-la ffplay or on the same area, possibly letting the user choose the color for each channel wave.
RDFT should be easier to add (same for some power/volume visualization).
Due to different caching logic this could be a separate filter (e.g. showaudio -> showwaves, showrdft?).
For more complex audio visualizations we may consider to write a MilkDrop/ProjectM wrapper.
Started to look at this, it's quite complex due to the use of C++ and to the opaque API (I can't still guess how to just get the video data out of the SDL/GL mess). Hints are welcome if you have experience with it (IIRC at least VLC has/planned to have a projectm wrapper, don't know how it ended). -- FFmpeg = Faithless & Faithful Most Plastic Egregious Gargoyle
On Tue, Dec 27, 2011 at 07:20:04PM +0100, Stefano Sabatini wrote:
On date Tuesday 2011-12-27 14:58:03 +0100, Stefano Sabatini encoded:
On date Monday 2011-12-26 00:14:39 +0100, Stefano Sabatini encoded:
Subject: [PATCH] lavfi: add showaudio filter
--- libavfilter/Makefile | 3 +- libavfilter/allfilters.c | 3 + libavfilter/avf_showaudio.c | 174 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 179 insertions(+), 1 deletions(-) create mode 100644 libavfilter/avf_showaudio.
Main problem is now to understand how to visualize N samples in a packet using only w columns.
One possibility: w defines the number of samples to visualize, this means that we have to visualize samplerate/w frames, i.e.: rate = samplerate/w
For example with w=640, samplerate=22100: rate = 22100/640 ~= 35
Alternatively I could fix the rate, and scale the image accordingly by imposing: samples_to_show = samplerate / rate
For example with rate = 10, samplerate = 22100 samples_to_show = 22100 / 10 = 2210
and I could use lsws to scale such an image to make it suit w.
Updated implementing the first option. Still missing: * multiple channels support, every channel may be displayed on a separate area a-la ffplay or on the same area, possibly letting the user choose the color for each channel wave.
RDFT should be easier to add (same for some power/volume visualization).
Due to different caching logic this could be a separate filter (e.g. showaudio -> showwaves, showrdft?).
Sounds like a good idea; if you want to continue on showaudio, I'd like to do the showrdft/spectrum one, so we could replace the ffplay builtin with this filter.
For more complex audio visualizations we may consider to write a MilkDrop/ProjectM wrapper.
Started to look at this, it's quite complex due to the use of C++ and to the opaque API (I can't still guess how to just get the video data out of the SDL/GL mess). Hints are welcome if you have experience with it (IIRC at least VLC has/planned to have a projectm wrapper, don't know how it ended).
The get_screenshot function in mplayer2 might give you some hints: http://git.mplayer2.org/mplayer2/tree/libvo/vo_gl.c#n1127 Also, maybe the other plugins such as XMMS one should give an overview of the API usage. Anyway, I think an example like "amovie=a.mp3,asplit[a][out1]; [a]showaudio[out0]" in the documentation will be welcome when adding this first filter. [...] -- Clément B.
On date Tuesday 2011-12-27 20:15:34 +0100, Clément Bœsch encoded:
On Tue, Dec 27, 2011 at 07:20:04PM +0100, Stefano Sabatini wrote: [...]
Still missing: * multiple channels support, every channel may be displayed on a separate area a-la ffplay or on the same area, possibly letting the user choose the color for each channel wave.
RDFT should be easier to add (same for some power/volume visualization).
Due to different caching logic this could be a separate filter (e.g. showaudio -> showwaves, showrdft?).
Sounds like a good idea; if you want to continue on showaudio, I'd like to do the showrdft/spectrum one, so we could replace the ffplay builtin with this filter. [...]
Anyway, I think an example like "amovie=a.mp3,asplit[a][out1]; [a]showaudio[out0]" in the documentation will be welcome when adding this first filter.
Added. Updated against the new API, and renamed showwaves. The functionality is still quite limited. For example: we could split the channel visualization a la ffplay, or create a color map for showing each channel "samples" in a different color, but I have to think at a sensible syntax for specifying the map. This can be added later though with no harm. -- FFmpeg = Furious Free Monstrous Pitiful Extroverse Geisha
Le primidi 1er prairial, an CCXX, Stefano Sabatini a écrit :
Updated against the new API, and renamed showwaves. The functionality is still quite limited. For example: we could split the channel visualization a la ffplay, or create a color map for showing each channel "samples" in a different color, but I have to think at a sensible syntax for specifying the map.
Nice work, although I did not read the code yet. Should we consider implementing some kind of mechanism to flag this kind of "visualization" filter, so that applications like ffplay could pick them automatically? Regards, -- Nicolas George
On Sun, May 20, 2012 at 12:21:24PM +0200, Nicolas George wrote:
Le primidi 1er prairial, an CCXX, Stefano Sabatini a écrit :
Updated against the new API, and renamed showwaves. The functionality is still quite limited. For example: we could split the channel visualization a la ffplay, or create a color map for showing each channel "samples" in a different color, but I have to think at a sensible syntax for specifying the map.
Nice work, although I did not read the code yet.
Should we consider implementing some kind of mechanism to flag this kind of "visualization" filter, so that applications like ffplay could pick them automatically?
i think that would be a good idea [...] -- Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB Concerning the gods, I have no means of knowing whether they exist or not or of what sort they may be, because of the obscurity of the subject, and the brevity of human life -- Protagoras
Le primidi 1er prairial, an CCXX, Stefano Sabatini a écrit :
From 1a9c356febb48a6f5b4e028728d638b199a40a51 Mon Sep 17 00:00:00 2001 From: Stefano Sabatini <stefasab@gmail.com> Date: Sun, 25 Dec 2011 17:37:53 +0100 Subject: [PATCH] lavfi: add showwaves filter
--- doc/filters.texi | 23 +++++ libavfilter/Makefile | 2 + libavfilter/allfilters.c | 3 + libavfilter/avf_showwaves.c | 195 +++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 223 insertions(+), 0 deletions(-) create mode 100644 libavfilter/avf_showwaves.c
diff --git a/doc/filters.texi b/doc/filters.texi index 467fb4a..9c4575d 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -3678,3 +3678,26 @@ mainly useful as a template and to be employed in analysis / debugging tools.
@c man end VIDEO SINKS + +@chapter Transmedia Filters +@c man begin TRANSMEDIA FILTERS + +Below is a description of the currently available transmedia filters. + +@section showwaves + +Convert input audio to a video output, representing the samples waves. + +The filter accepts the following named parameters: +@table @option +@item size, s +Specify the video size for the output. Default value is "320x240". +@end table + +The following example shows how to output the input file audio and the +corresponding video representation at the same time: +@example +amovie=a.mp3,asplit[out0],showwaves[out1] +@end example + +@c man end TRANSMEDIA FILTERS diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 91f2527..7250e29 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -184,6 +184,8 @@ OBJS-$(CONFIG_MP_FILTER) += libmpcodecs/vf_yuvcsp.o OBJS-$(CONFIG_MP_FILTER) += libmpcodecs/vf_yvu9.o OBJS-$(CONFIG_MP_FILTER) += libmpcodecs/pullup.o
+# transmedia filters +OBJS-$(CONFIG_SHOWWAVES_FILTER) += avf_showwaves.o
TESTPROGS = drawutils formats
diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 29cd3e3..275744b 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -123,6 +123,9 @@ void avfilter_register_all(void) REGISTER_FILTER (BUFFERSINK, buffersink, vsink); REGISTER_FILTER (NULLSINK, nullsink, vsink);
+ /* transmedia filters */ + REGISTER_FILTER (SHOWWAVES, showwaves, avf); + /* those filters are part of public or internal API => registered * unconditionally */ { diff --git a/libavfilter/avf_showwaves.c b/libavfilter/avf_showwaves.c new file mode 100644 index 0000000..fdaa718 --- /dev/null +++ b/libavfilter/avf_showwaves.c @@ -0,0 +1,195 @@ +/* + * Copyright (c) 2011 Stefano Sabatini + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * audio to video transmedia filter + */ + +#include "libavutil/audioconvert.h" +#include "libavutil/opt.h" +#include "libavutil/parseutils.h" +#include "avfilter.h" +#include "formats.h" + +typedef struct { + const AVClass *class; + int w, h; + char *size_str; + int buf_idx; + AVFilterBufferRef *outpicref; +} ShowWavesContext; + +#define OFFSET(x) offsetof(ShowWavesContext, x) + +static const AVOption showwaves_options[] = { + { "size", "set video size", OFFSET(size_str), AV_OPT_TYPE_STRING, {.str = "320x240"}, 0, 0 }, + { "s", "set video size", OFFSET(size_str), AV_OPT_TYPE_STRING, {.str = "320x240"}, 0, 0 }, + { NULL }, +}; + +static const char *showwaves_get_name(void *ctx) +{ + return "showwaves"; +} + +static const AVClass showwaves_class = { + "ShowWavesContext", + showwaves_get_name, + showwaves_options +}; + +static av_cold int init(AVFilterContext *ctx, const char *args, void *opaque) +{ + ShowWavesContext *showwaves = ctx->priv; + int err; + + showwaves->class = &showwaves_class; + av_opt_set_defaults(showwaves); + + if ((err = (av_set_options_string(showwaves, args, "=", ":"))) < 0) { + av_log(ctx, AV_LOG_ERROR, "Error parsing options string: '%s'\n", args); + return err; + } + + if (av_parse_video_size(&showwaves->w, &showwaves->h, showwaves->size_str) < 0) { + av_log(ctx, AV_LOG_ERROR, "Invalid frame size: %s\n", showwaves->size_str); + return AVERROR(EINVAL); + }
Can be simplified now.
+ + return 0; +} + +static int query_formats(AVFilterContext *ctx) +{ + AVFilterFormats *formats = NULL; + AVFilterChannelLayouts *layouts = NULL; + AVFilterLink *inlink = ctx->inputs[0]; + AVFilterLink *outlink = ctx->outputs[0]; + static const enum AVSampleFormat sample_fmts[] = { AV_SAMPLE_FMT_DBL, -1 };
Why double? A reasonable video size will never go beyond the precision of S16.
+ static const enum PixelFormat pix_fmts[] = { PIX_FMT_GRAY8, -1 }; + + /* set input audio formats */ + formats = avfilter_make_format_list(sample_fmts); + if (!formats) + return AVERROR(ENOMEM); + avfilter_formats_ref(formats, &inlink->out_formats); + + layouts = ff_all_channel_layouts(); + if (!layouts) + return AVERROR(ENOMEM); + ff_channel_layouts_ref(layouts, &inlink->out_channel_layouts); + + formats = ff_all_samplerates(); + if (!formats) + return AVERROR(ENOMEM); + avfilter_formats_ref(formats, &inlink->out_samplerates); + + /* set output video format */ + formats = avfilter_make_format_list(pix_fmts); + if (!formats) + return AVERROR(ENOMEM); + avfilter_formats_ref(formats, &outlink->in_formats); + + return 0; +} + +static int config_output(AVFilterLink *outlink) +{ + AVFilterContext *ctx = outlink->src; + ShowWavesContext *showwaves = ctx->priv; + + outlink->w = showwaves->w; + outlink->h = showwaves->h; + + av_log(ctx, AV_LOG_INFO, "s:%dx%d\n", showwaves->w, showwaves->h); + return 0; +} + +static void filter_samples(AVFilterLink *inlink, AVFilterBufferRef *insamples) +{ + AVFilterContext *ctx = inlink->dst; + AVFilterLink *outlink = ctx->outputs[0]; + ShowWavesContext *showwaves = ctx->priv; + const int nb_samples = insamples->audio->nb_samples; + AVFilterBufferRef *outpicref = showwaves->outpicref; + int linesize = outpicref ? outpicref->linesize[0] : 0; + double *p = (double *)insamples->data[0]; + int nb_channels = av_get_channel_layout_nb_channels(insamples->audio->channel_layout); + int i, j, h; + + /* draw data in the buffer */ + for (i = 0; i < nb_samples; i++) { + if (showwaves->buf_idx == 0) { + showwaves->outpicref = outpicref = + avfilter_get_video_buffer(outlink, AV_PERM_WRITE|AV_PERM_ALIGN, + outlink->w, outlink->h); + outpicref->video->w = outlink->w; + outpicref->video->h = outlink->h; + outpicref->pts = insamples->pts + i * inlink->time_base.num / (inlink->time_base.den * inlink->sample_rate);
av_rescale?
+ outlink->out_buf = outpicref; + linesize = outpicref->linesize[0]; + memset(outpicref->data[0], 0, showwaves->h*linesize); + } + for (j = 0; j < nb_channels; j++) { + h = showwaves->h/2 - (*p++ * showwaves->h/2);
av_clip(av_rescale())?
+ if (h < outlink->h) + *(outpicref->data[0] + showwaves->buf_idx + h * linesize) = 255; + } + showwaves->buf_idx++;
With the default frame size, at 44100Hz it outputs ~138 frames per second. That would be barely readable. Dividing buf_idx by some constant would solve the problem at almost no cost.
+ + if (showwaves->buf_idx == showwaves->w-1) { + avfilter_start_frame(outlink, outpicref); + avfilter_draw_slice(outlink, 0, outlink->h, 1); + avfilter_end_frame(outlink); + showwaves->outpicref = NULL; + showwaves->buf_idx = 0; + } + }
No EOF flushing, it seems. Fine by me. OTOH, it looks like the last buffer is leaked.
+ + avfilter_unref_buffer(insamples); +} + +AVFilter avfilter_avf_showwaves = { + .name = "showwaves", + .description = NULL_IF_CONFIG_SMALL("Convert input audio to a video output."), + .init = init, + .query_formats = query_formats, + .priv_size = sizeof(ShowWavesContext), + + .inputs = (const AVFilterPad[]) { + { + .name = "default", + .type = AVMEDIA_TYPE_AUDIO, + .filter_samples = filter_samples, + .min_perms = AV_PERM_READ, + }, + { .name = NULL } + }, + + .outputs = (const AVFilterPad[]) { + { + .name = "default", + .config_props = config_output, + .type = AVMEDIA_TYPE_VIDEO, + }, + { .name = NULL } + }, +};
Regards, -- Nicolas George
On date Monday 2012-05-21 00:07:46 +0200, Nicolas George encoded:
Le primidi 1er prairial, an CCXX, Stefano Sabatini a écrit :
From 1a9c356febb48a6f5b4e028728d638b199a40a51 Mon Sep 17 00:00:00 2001 From: Stefano Sabatini <stefasab@gmail.com> Date: Sun, 25 Dec 2011 17:37:53 +0100 Subject: [PATCH] lavfi: add showwaves filter
--- doc/filters.texi | 23 +++++ libavfilter/Makefile | 2 + libavfilter/allfilters.c | 3 + libavfilter/avf_showwaves.c | 195 +++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 223 insertions(+), 0 deletions(-) create mode 100644 libavfilter/avf_showwaves.c [...]
Updated work in progress, with some of the issues addressed. -- FFmpeg = Fierce Fierce Monstrous Portable Ecumenical Gorilla
On Sat, Jun 16, 2012 at 01:49:40AM +0200, Stefano Sabatini wrote:
On date Monday 2012-05-21 00:07:46 +0200, Nicolas George encoded:
Le primidi 1er prairial, an CCXX, Stefano Sabatini a écrit :
From 1a9c356febb48a6f5b4e028728d638b199a40a51 Mon Sep 17 00:00:00 2001 From: Stefano Sabatini <stefasab@gmail.com> Date: Sun, 25 Dec 2011 17:37:53 +0100 Subject: [PATCH] lavfi: add showwaves filter
--- doc/filters.texi | 23 +++++ libavfilter/Makefile | 2 + libavfilter/allfilters.c | 3 + libavfilter/avf_showwaves.c | 195 +++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 223 insertions(+), 0 deletions(-) create mode 100644 libavfilter/avf_showwaves.c [...]
Updated work in progress, with some of the issues addressed. -- FFmpeg = Fierce Fierce Monstrous Portable Ecumenical Gorilla
From 40c0f12fb54449efdddf8d2c0412228464781fd5 Mon Sep 17 00:00:00 2001 From: Stefano Sabatini <stefasab@gmail.com> Date: Sun, 25 Dec 2011 17:37:53 +0100 Subject: [PATCH] lavfi: add showwaves filter
--- doc/filters.texi | 23 ++++ libavfilter/Makefile | 2 + libavfilter/allfilters.c | 3 + libavfilter/avf_showwaves.c | 235 +++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 263 insertions(+), 0 deletions(-) create mode 100644 libavfilter/avf_showwaves.c
[...]
+static int request_frame(AVFilterLink *outlink) +{ + ShowWavesContext *showwaves = outlink->src->priv; + AVFilterLink *inlink = outlink->src->inputs[0]; + AVFilterBufferRef *outpicref = showwaves->outpicref; + int ret; + + showwaves->req_fullfilled = 0; + do { + ret = avfilter_request_frame(inlink); + } while (!showwaves->req_fullfilled && ret >= 0); + + if (ret == AVERROR_EOF && showwaves->outpicref) { + int j, linesize = showwaves->outpicref->linesize[0]; + + /* fill last frame with zeroes */ + while (showwaves->buf_idx < showwaves->w) { + for (j = 0; j < showwaves->nb_channels; j++) + *(showwaves->outpicref->data[0] + + showwaves->buf_idx + showwaves->h/2 * linesize) = 255; + showwaves->buf_idx++; + }
The comment looks clumsy; it looks more correct to say "fill gap in the last frame with a zero signal" or something like that. You clearly are not creating silence/zero which is then drawn here. Maybe you can create a function for this so it can be called with either 0 or *p++ (in filter_samples()).
+ + /* push frame */ + ff_start_frame(outlink, outpicref); + ff_draw_slice(outlink, 0, outlink->h, 1); + ff_end_frame(outlink); + showwaves->req_fullfilled = 1; + showwaves->outpicref = NULL; + showwaves->buf_idx = 0;
You have the exact same chunk in filter_samples(), can't it be factorized?
+ } + + return ret; +} + +static void filter_samples(AVFilterLink *inlink, AVFilterBufferRef *insamples) +{ + AVFilterContext *ctx = inlink->dst; + AVFilterLink *outlink = ctx->outputs[0]; + ShowWavesContext *showwaves = ctx->priv; + const int nb_samples = insamples->audio->nb_samples; + AVFilterBufferRef *outpicref = showwaves->outpicref; + int linesize = outpicref ? outpicref->linesize[0] : 0; + double *p = (double *)insamples->data[0]; + int i, j, h; + + showwaves->nb_channels = av_get_channel_layout_nb_channels(insamples->audio->channel_layout);
Can't this be in a config_props() instead? Or just call a second time av_get_channel_layout_nb_channels() in request_frame()?
+ + /* draw data in the buffer */ + for (i = 0; i < nb_samples; i++) { + if (showwaves->buf_idx == 0) { + showwaves->outpicref = outpicref = + ff_get_video_buffer(outlink, AV_PERM_WRITE|AV_PERM_ALIGN, + outlink->w, outlink->h); + outpicref->video->w = outlink->w; + outpicref->video->h = outlink->h; + outpicref->pts = insamples->pts; + outlink->out_buf = outpicref; + linesize = outpicref->linesize[0]; + memset(outpicref->data[0], 0, showwaves->h*linesize); + } + for (j = 0; j < showwaves->nb_channels; j++) { + h = showwaves->h/2 - (*p++ * showwaves->h/2); + if (h < outlink->h) + *(outpicref->data[0] + showwaves->buf_idx + h * linesize) = 255; + } + showwaves->buf_idx++; + + if (showwaves->buf_idx == showwaves->w) { + ff_start_frame(outlink, outpicref); + ff_draw_slice(outlink, 0, outlink->h, 1); + ff_end_frame(outlink); + showwaves->req_fullfilled = 1; + showwaves->outpicref = NULL; + showwaves->buf_idx = 0; + } + } + + avfilter_unref_buffer(insamples); +} +
[...] -- Clément B.
On date Saturday 2012-06-16 10:19:22 +0200, Clément Bœsch encoded:
On Sat, Jun 16, 2012 at 01:49:40AM +0200, Stefano Sabatini wrote: [...]
+static int request_frame(AVFilterLink *outlink) +{ + ShowWavesContext *showwaves = outlink->src->priv; + AVFilterLink *inlink = outlink->src->inputs[0]; + AVFilterBufferRef *outpicref = showwaves->outpicref; + int ret; + + showwaves->req_fullfilled = 0; + do { + ret = avfilter_request_frame(inlink); + } while (!showwaves->req_fullfilled && ret >= 0); + + if (ret == AVERROR_EOF && showwaves->outpicref) { + int j, linesize = showwaves->outpicref->linesize[0]; + + /* fill last frame with zeroes */ + while (showwaves->buf_idx < showwaves->w) { + for (j = 0; j < showwaves->nb_channels; j++) + *(showwaves->outpicref->data[0] + + showwaves->buf_idx + showwaves->h/2 * linesize) = 255; + showwaves->buf_idx++; + }
The comment looks clumsy; it looks more correct to say "fill gap in the last frame with a zero signal" or something like that. You clearly are not creating silence/zero which is then drawn here.
Maybe you can create a function for this so it can be called with either 0 or *p++ (in filter_samples()).
Removed, after all the samples are missing are not set to zero.
+ + /* push frame */ + ff_start_frame(outlink, outpicref); + ff_draw_slice(outlink, 0, outlink->h, 1); + ff_end_frame(outlink); + showwaves->req_fullfilled = 1; + showwaves->outpicref = NULL; + showwaves->buf_idx = 0;
You have the exact same chunk in filter_samples(), can't it be factorized?
Factorized.
+ } + + return ret; +} + +static void filter_samples(AVFilterLink *inlink, AVFilterBufferRef *insamples) +{ + AVFilterContext *ctx = inlink->dst; + AVFilterLink *outlink = ctx->outputs[0]; + ShowWavesContext *showwaves = ctx->priv; + const int nb_samples = insamples->audio->nb_samples; + AVFilterBufferRef *outpicref = showwaves->outpicref; + int linesize = outpicref ? outpicref->linesize[0] : 0; + double *p = (double *)insamples->data[0]; + int i, j, h; + + showwaves->nb_channels = av_get_channel_layout_nb_channels(insamples->audio->channel_layout);
Can't this be in a config_props() instead? Or just call a second time av_get_channel_layout_nb_channels() in request_frame()?
Removed, no longer required. -- FFmpeg = Fancy and Fierce Magic Puritan Evangelical Gorilla
(Maybe I already posted some of these remarks, I am not sure.) Le decadi 30 prairial, an CCXX, Stefano Sabatini a écrit :
From: Stefano Sabatini <stefasab@gmail.com> Date: Sun, 25 Dec 2011 17:37:53 +0100 Subject: [PATCH] lavfi: add showwaves filter
--- doc/filters.texi | 23 +++++ libavfilter/Makefile | 2 + libavfilter/allfilters.c | 3 + libavfilter/avf_showwaves.c | 220 +++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 248 insertions(+), 0 deletions(-) create mode 100644 libavfilter/avf_showwaves.c
diff --git a/doc/filters.texi b/doc/filters.texi index a354e97..cb61ed7 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -3815,3 +3815,26 @@ mainly useful as a template and to be employed in analysis / debugging tools.
@c man end VIDEO SINKS + +@chapter Transmedia Filters +@c man begin TRANSMEDIA FILTERS + +Below is a description of the currently available transmedia filters. + +@section showwaves + +Convert input audio to a video output, representing the samples waves. + +The filter accepts the following named parameters: +@table @option +@item size, s +Specify the video size for the output. Default value is "320x240". +@end table + +The following example shows how to output the input file audio and the +corresponding video representation at the same time: +@example +amovie=a.mp3,asplit[out0],showwaves[out1] +@end example + +@c man end TRANSMEDIA FILTERS diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 72bb640..6de6ef4 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -193,6 +193,8 @@ OBJS-$(CONFIG_MP_FILTER) += libmpcodecs/vf_yuvcsp.o OBJS-$(CONFIG_MP_FILTER) += libmpcodecs/vf_yvu9.o OBJS-$(CONFIG_MP_FILTER) += libmpcodecs/pullup.o
+# transmedia filters +OBJS-$(CONFIG_SHOWWAVES_FILTER) += avf_showwaves.o
TESTPROGS = drawutils formats
diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 35065d5..6d2d6cf 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -129,6 +129,9 @@ void avfilter_register_all(void) REGISTER_FILTER (BUFFERSINK, buffersink, vsink); REGISTER_FILTER (NULLSINK, nullsink, vsink);
+ /* transmedia filters */ + REGISTER_FILTER (SHOWWAVES, showwaves, avf); + /* those filters are part of public or internal API => registered * unconditionally */ { diff --git a/libavfilter/avf_showwaves.c b/libavfilter/avf_showwaves.c new file mode 100644 index 0000000..bcb6545 --- /dev/null +++ b/libavfilter/avf_showwaves.c @@ -0,0 +1,220 @@ +/* + * Copyright (c) 2011 Stefano Sabatini + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * audio to video transmedia filter + */ + +#include "libavutil/audioconvert.h" +#include "libavutil/opt.h" +#include "libavutil/parseutils.h" +#include "avfilter.h" +#include "formats.h" +#include "audio.h" +#include "video.h" + +typedef struct { + const AVClass *class; + int w, h; + int buf_idx; + AVFilterBufferRef *outpicref; + int req_fullfilled; +} ShowWavesContext; + +#define OFFSET(x) offsetof(ShowWavesContext, x) + +static const AVOption showwaves_options[] = { + { "size", "set video size", OFFSET(w), AV_OPT_TYPE_IMAGE_SIZE, {.str = "320x240"}, 0, 0 }, + { "s", "set video size", OFFSET(w), AV_OPT_TYPE_IMAGE_SIZE, {.str = "320x240"}, 0, 0 }, + { NULL }, +}; + +static const AVClass showwaves_class = { + "showwaves", + av_default_item_name, + showwaves_options +}; + +static av_cold int init(AVFilterContext *ctx, const char *args, void *opaque) +{ + ShowWavesContext *showwaves = ctx->priv; + int err; + + showwaves->class = &showwaves_class; + av_opt_set_defaults(showwaves); + showwaves->buf_idx = 0; + + if ((err = av_set_options_string(showwaves, args, "=", ":")) < 0) { + av_log(ctx, AV_LOG_ERROR, "Error parsing options string: '%s'\n", args); + return err; + } + + return 0; +} + +static av_cold void uninit(AVFilterContext *ctx) +{ + ShowWavesContext *showwaves = ctx->priv; + + avfilter_unref_bufferp(&showwaves->outpicref); +} + +static int query_formats(AVFilterContext *ctx) +{ + AVFilterFormats *formats = NULL; + AVFilterChannelLayouts *layouts = NULL; + AVFilterLink *inlink = ctx->inputs[0]; + AVFilterLink *outlink = ctx->outputs[0]; + static const enum AVSampleFormat sample_fmts[] = { AV_SAMPLE_FMT_DBL, -1 };
Using S16 would make the filter require less conversions and be bit-exact on various architectures. You can use av_rescale to compute the height of the bars.
+ static const enum PixelFormat pix_fmts[] = { PIX_FMT_GRAY8, -1 }; + + /* set input audio formats */ + formats = ff_make_format_list(sample_fmts); + if (!formats) + return AVERROR(ENOMEM); + ff_formats_ref(formats, &inlink->out_formats); + + layouts = ff_all_channel_layouts(); + if (!layouts) + return AVERROR(ENOMEM); + ff_channel_layouts_ref(layouts, &inlink->out_channel_layouts); + + formats = ff_all_samplerates(); + if (!formats) + return AVERROR(ENOMEM); + ff_formats_ref(formats, &inlink->out_samplerates); + + /* set output video format */ + formats = ff_make_format_list(pix_fmts); + if (!formats) + return AVERROR(ENOMEM); + ff_formats_ref(formats, &outlink->in_formats); + + return 0; +} + +static int config_output(AVFilterLink *outlink) +{ + AVFilterContext *ctx = outlink->src; + ShowWavesContext *showwaves = ctx->priv; + + outlink->w = showwaves->w; + outlink->h = showwaves->h;
outlink->frame_rate = av_div_q((AVRational){ inlink->sample_rate, 1}, (AVRational){ showwaves->w, 1 }); ... or something like that should allow it to work with -filter_complex too.
+ + av_log(ctx, AV_LOG_INFO, "s:%dx%d\n", showwaves->w, showwaves->h); + return 0; +} + +inline static void push_frame(AVFilterLink *outlink) +{ + ShowWavesContext *showwaves = outlink->src->priv; + + ff_start_frame(outlink, showwaves->outpicref); + ff_draw_slice(outlink, 0, outlink->h, 1); + ff_end_frame(outlink); + showwaves->req_fullfilled = 1; + showwaves->outpicref = NULL; + showwaves->buf_idx = 0; +} + +static int request_frame(AVFilterLink *outlink) +{ + ShowWavesContext *showwaves = outlink->src->priv; + AVFilterLink *inlink = outlink->src->inputs[0]; + int ret; + + showwaves->req_fullfilled = 0; + do { + ret = avfilter_request_frame(inlink); + } while (!showwaves->req_fullfilled && ret >= 0); + + if (ret == AVERROR_EOF && showwaves->outpicref) + push_frame(outlink); + return ret; +} + +static void filter_samples(AVFilterLink *inlink, AVFilterBufferRef *insamples) +{ + AVFilterContext *ctx = inlink->dst; + AVFilterLink *outlink = ctx->outputs[0]; + ShowWavesContext *showwaves = ctx->priv; + const int nb_samples = insamples->audio->nb_samples; + AVFilterBufferRef *outpicref = showwaves->outpicref; + int linesize = outpicref ? outpicref->linesize[0] : 0; + double *p = (double *)insamples->data[0]; + int nb_channels = av_get_channel_layout_nb_channels(insamples->audio->channel_layout); + int i, j, h; + + /* draw data in the buffer */ + for (i = 0; i < nb_samples; i++) { + if (showwaves->buf_idx == 0) { + showwaves->outpicref = outpicref = + ff_get_video_buffer(outlink, AV_PERM_WRITE|AV_PERM_ALIGN, + outlink->w, outlink->h); + outpicref->video->w = outlink->w; + outpicref->video->h = outlink->h; + outpicref->pts = insamples->pts; + outlink->out_buf = outpicref; + linesize = outpicref->linesize[0]; + memset(outpicref->data[0], 0, showwaves->h*linesize); + } + for (j = 0; j < nb_channels; j++) { + h = showwaves->h/2 - (*p++ * showwaves->h/2); + if (h < outlink->h) + *(outpicref->data[0] + showwaves->buf_idx + h * linesize) = 255; + } + showwaves->buf_idx++; + + if (showwaves->buf_idx == showwaves->w) + push_frame(outlink);
With sample_rate = 44100 and w = 320, that makes 138 frames per second, it is probably too much. What about incrementing buf_idx only once in n times, n being an option defaulting to maybe 4. And in that case, "= 255" may be replaced by "+= x", where x would be 255 / (nb_channels * n)
+ } + + avfilter_unref_buffer(insamples); +} + +AVFilter avfilter_avf_showwaves = { + .name = "showwaves", + .description = NULL_IF_CONFIG_SMALL("Convert input audio to a video output."), + .init = init, + .uninit = uninit, + .query_formats = query_formats, + .priv_size = sizeof(ShowWavesContext), + + .inputs = (const AVFilterPad[]) { + { + .name = "default", + .type = AVMEDIA_TYPE_AUDIO, + .filter_samples = filter_samples, + .min_perms = AV_PERM_READ, + }, + { .name = NULL } + }, + + .outputs = (const AVFilterPad[]) { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = config_output, + .request_frame = request_frame, + }, + { .name = NULL } + }, +};
Regards, -- Nicolas George
On date Sunday 2012-06-17 14:56:35 +0200, Nicolas George encoded:
(Maybe I already posted some of these remarks, I am not sure.) [...]
Yep sure, I didn't forget that, consider the patch still a WIP. -- FFmpeg = Fierce Fierce Majestic Peaceless Exciting Gymnast
On date Sunday 2012-06-17 14:56:35 +0200, Nicolas George encoded: [...]
+static int query_formats(AVFilterContext *ctx) +{ + AVFilterFormats *formats = NULL; + AVFilterChannelLayouts *layouts = NULL; + AVFilterLink *inlink = ctx->inputs[0]; + AVFilterLink *outlink = ctx->outputs[0]; + static const enum AVSampleFormat sample_fmts[] = { AV_SAMPLE_FMT_DBL, -1 };
Using S16 would make the filter require less conversions and be bit-exact on various architectures. You can use av_rescale to compute the height of the bars.
Changed. [...]
+static int config_output(AVFilterLink *outlink) +{ + AVFilterContext *ctx = outlink->src; + ShowWavesContext *showwaves = ctx->priv; + + outlink->w = showwaves->w; + outlink->h = showwaves->h;
outlink->frame_rate = av_div_q((AVRational){ inlink->sample_rate, 1}, (AVRational){ showwaves->w, 1 });
... or something like that should allow it to work with -filter_complex too.>
Fixed. [...]
+static void filter_samples(AVFilterLink *inlink, AVFilterBufferRef *insamples) +{ + AVFilterContext *ctx = inlink->dst; + AVFilterLink *outlink = ctx->outputs[0]; + ShowWavesContext *showwaves = ctx->priv; + const int nb_samples = insamples->audio->nb_samples; + AVFilterBufferRef *outpicref = showwaves->outpicref; + int linesize = outpicref ? outpicref->linesize[0] : 0; + double *p = (double *)insamples->data[0]; + int nb_channels = av_get_channel_layout_nb_channels(insamples->audio->channel_layout); + int i, j, h; + + /* draw data in the buffer */ + for (i = 0; i < nb_samples; i++) { + if (showwaves->buf_idx == 0) { + showwaves->outpicref = outpicref = + ff_get_video_buffer(outlink, AV_PERM_WRITE|AV_PERM_ALIGN, + outlink->w, outlink->h); + outpicref->video->w = outlink->w; + outpicref->video->h = outlink->h; + outpicref->pts = insamples->pts; + outlink->out_buf = outpicref; + linesize = outpicref->linesize[0]; + memset(outpicref->data[0], 0, showwaves->h*linesize); + } + for (j = 0; j < nb_channels; j++) { + h = showwaves->h/2 - (*p++ * showwaves->h/2); + if (h < outlink->h) + *(outpicref->data[0] + showwaves->buf_idx + h * linesize) = 255; + } + showwaves->buf_idx++; + + if (showwaves->buf_idx == showwaves->w) + push_frame(outlink);
With sample_rate = 44100 and w = 320, that makes 138 frames per second, it is probably too much. What about incrementing buf_idx only once in n times, n being an option defaulting to maybe 4.
And in that case, "= 255" may be replaced by "+= x", where x would be 255 / (nb_channels * n)
Very good idea, I changed this and added a new rate parameters, which changes the n parameter to honor the specified framerate. I also changed the default size from 320x240 to 800x200, which seems more useful for visualizing audio data. [...] -- FFmpeg = Formidable and Forgiving Mystic Portentous Epic God
On date Monday 2012-06-18 18:19:26 +0200, Stefano Sabatini encoded:
On date Sunday 2012-06-17 14:56:35 +0200, Nicolas George encoded: [...] From c0b07ef2d6bee8db57e41036af2ea6454d6d98ec Mon Sep 17 00:00:00 2001 From: Stefano Sabatini <stefasab@gmail.com> Date: Sun, 25 Dec 2011 17:37:53 +0100 Subject: [PATCH] lavfi: add showwaves filter
Ping Nicolas, the filter is still missing a few things (e.g. background/channel colors, and maybe a more flexible system for setting the framerate), but I believe it is already in a committable state if you don't have more suggestions/remarks. -- FFmpeg = Fundamentalist and Fierce Most Philosofic Enhancing Guru
Le tridi 3 messidor, an CCXX, Stefano Sabatini a écrit :
Ping Nicolas,
the filter is still missing a few things (e.g. background/channel colors, and maybe a more flexible system for setting the framerate), but I believe it is already in a committable state if you don't have more suggestions/remarks.
I gave it another look, and it looks fine. If it works reasonably well in both -f lavfi and -filter_complex cases, I see no reason not to go forward. I hope users will realize that the exact output may change without notice as new features are added. Regards, -- Nicolas George
On date Wednesday 2012-06-20 10:10:25 +0200, Nicolas George encoded:
Le tridi 3 messidor, an CCXX, Stefano Sabatini a écrit :
Ping Nicolas,
the filter is still missing a few things (e.g. background/channel colors, and maybe a more flexible system for setting the framerate), but I believe it is already in a committable state if you don't have more suggestions/remarks.
I gave it another look, and it looks fine. If it works reasonably well in both -f lavfi and -filter_complex cases, I see no reason not to go forward.
I hope users will realize that the exact output may change without notice as new features are added.
Regards,
Applied. -- FFmpeg = Frightening & Faithful Mystic Picky Epic Gangster
participants (4)
-
Clément Bœsch -
Michael Niedermayer -
Nicolas George -
Stefano Sabatini