[PATCH] af_mix: mix two audio streams
Hi there, Here is a first attempt to write an audio filter which merge two audio streams. Usage is: ffplay -f lavfi "amovie=test1.mp3[a];amovie=test2.mp3[b];[a][b]mix=0.7:0.3[out]" The volume of each streams can be configured and default to 0.5:0.5 is not specified. It only support two s16 audio streams with same sample rate. Feel free to comment. Matthieu
Le tridi 23 frimaire, an CCXX, Matthieu Bouron a écrit :
Hi there,
Here is a first attempt to write an audio filter which merge two audio streams. Usage is: ffplay -f lavfi "amovie=test1.mp3[a];amovie=test2.mp3[b];[a][b]mix=0.7:0.3[out]"
The volume of each streams can be configured and default to 0.5:0.5 is not specified. It only support two s16 audio streams with same sample rate. Feel free to comment.
This is a much asked for feature, but I do not like the design you adopted. I think your version is confusing two different tasks: merging the streams, and mixing the channels with configurable coefficients. Thus, it only satisfies the most basic needs. I believe that it is better to separate both tasks: first merging a n-channels streams and a p-channels streams into a stream with n+p channels, and then mixing the channels together if necessary. For the first task, I started some time ago a "amerge" filter. The work as I left it is there: http://ffmpeg.org/pipermail/ffmpeg-devel/2011-November/116523.html I intend to start working on it again soon. For the second task, the pan filter can do almost anything we may need from it. Regards, -- Nicolas George
On Tue, Dec 13, 2011 at 05:08:26PM +0100, Nicolas George wrote:
Le tridi 23 frimaire, an CCXX, Matthieu Bouron a écrit :
Hi there,
Here is a first attempt to write an audio filter which merge two audio streams. Usage is: ffplay -f lavfi "amovie=test1.mp3[a];amovie=test2.mp3[b];[a][b]mix=0.7:0.3[out]"
The volume of each streams can be configured and default to 0.5:0.5 is not specified. It only support two s16 audio streams with same sample rate. Feel free to comment.
This is a much asked for feature, but I do not like the design you adopted. I think your version is confusing two different tasks: merging the streams, and mixing the channels with configurable coefficients. Thus, it only satisfies the most basic needs.
I believe that it is better to separate both tasks: first merging a n-channels streams and a p-channels streams into a stream with n+p channels, and then mixing the channels together if necessary.
I think this might get complicated with amerge+pan for the user... This filter is IMO a really user friendly way to do a pretty common operation: add a background music to a video. Just like we can use af pan to downmix from 5.1 to stereo, having a "-ac 2" option is a nice have.
For the first task, I started some time ago a "amerge" filter. The work as I left it is there: http://ffmpeg.org/pipermail/ffmpeg-devel/2011-November/116523.html I intend to start working on it again soon.
Heh, I was just testing it again today :) Unfortunately I got issues, I'm looking forward the next version. So to sum up, I think this filter is pretty useful and might just be the "easy way" for basic and common mixing usage. -- Clément B.
On Tue, Dec 13, 2011 at 04:57:15PM +0100, Matthieu Bouron wrote:
Hi there,
Here is a first attempt to write an audio filter which merge two audio streams. Usage is: ffplay -f lavfi "amovie=test1.mp3[a];amovie=test2.mp3[b];[a][b]mix=0.7:0.3[out]"
The volume of each streams can be configured and default to 0.5:0.5 is not specified. It only support two s16 audio streams with same sample rate. Feel free to comment.
Matthieu
Makefile | 1 af_mix.c | 223 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ allfilters.c | 1 3 files changed, 225 insertions(+) 9d3c49c3da24f0abf3df45166fec82c2245febd5 0001-af_mix-mix-two-audio-streams.patch From 78537e1c874d62b42b58de8da210dde28eb17859 Mon Sep 17 00:00:00 2001 From: Matthieu Bouron <matthieu.bouron@gmail.com> Date: Sat, 10 Dec 2011 12:31:32 +0100 Subject: [PATCH] af_mix: mix two audio streams.
--- libavfilter/Makefile | 1 + libavfilter/af_mix.c | 223 ++++++++++++++++++++++++++++++++++++++++++++++ libavfilter/allfilters.c | 1 + 3 files changed, 225 insertions(+), 0 deletions(-) create mode 100644 libavfilter/af_mix.c
diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 4351d5e..848a033 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -31,6 +31,7 @@ OBJS-$(CONFIG_ASHOWINFO_FILTER) += af_ashowinfo.o OBJS-$(CONFIG_EARWAX_FILTER) += af_earwax.o OBJS-$(CONFIG_PAN_FILTER) += af_pan.o OBJS-$(CONFIG_VOLUME_FILTER) += af_volume.o +OBJS-$(CONFIG_MIX_FILTER) += af_mix.o
OBJS-$(CONFIG_ABUFFER_FILTER) += asrc_abuffer.o OBJS-$(CONFIG_AEVALSRC_FILTER) += asrc_aevalsrc.o diff --git a/libavfilter/af_mix.c b/libavfilter/af_mix.c new file mode 100644 index 0000000..70a168d --- /dev/null +++ b/libavfilter/af_mix.c @@ -0,0 +1,223 @@ +/* + * Copyright (c) 2011 Matthieu Bouron + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * audio mix filter + */ + +#include "libavutil/audioconvert.h" +#include "libavutil/eval.h" +#include "avfilter.h" +#include "internal.h" + +typedef struct { + double vol[2]; + int vol_i[2]; + int ended[2]; + int nb_channels; + AVFilterBufferRef *buf[2]; + int buf_pos[2]; +} MixContext; + +static av_cold int init(AVFilterContext *ctx, const char *args, void *opaque) +{ + int i, n; + double vol1, vol2; + MixContext *mix = ctx->priv; + + if (args) { + n = sscanf(args, "%lf:%lf", &vol1, &vol2); + switch (n) { + case 0: + mix->vol[0] = 0.5; + mix->vol[1] = 0.5; + break; + case 1: + mix->vol[0] = vol1; + mix->vol[1] = 0.5;
{vol1, 1.0-vol1} might be more usefull than {vol1, 0.5} also it would be cool if it was set by using the expression evaluator so it could change over time in a user specified way. But thats probably better in a seperate patch later ... [...]
+static void filter_samples(AVFilterLink *inlink, AVFilterBufferRef *insamples) +{ + AVFilterContext *ctx = inlink->dst; + MixContext *mix = inlink->dst->priv; + AVFilterLink *outlink = inlink->dst->outputs[0]; + AVFilterBufferRef *outsamples; + int i, pos; + int nb_samples; + int nb_samples_per_channel; + int nb_samples_input1 = INT_MAX; + int nb_samples_input2 = INT_MAX; + + pos = inlink == ctx->inputs[1]; + mix->buf[pos] = avfilter_ref_buffer(insamples, AV_PERM_READ); + mix->buf_pos[pos] = 0;
does this work if one input receives several frames before the other receives any ?
+ + // If opposite buffer is empty, wait for it. + if (!mix->buf[!pos] && !mix->ended[!pos]) + return; + + if (!mix->ended[0]) + nb_samples_input1 = mix->buf[0]->audio->nb_samples - mix->buf_pos[0]; + if (!mix->ended[1]) + nb_samples_input2 = mix->buf[1]->audio->nb_samples - mix->buf_pos[1]; + + nb_samples_per_channel = FFMIN(nb_samples_input1, nb_samples_input2); + nb_samples = nb_samples_per_channel * mix->nb_channels; + + outsamples = avfilter_get_audio_buffer(outlink, AV_PERM_WRITE, nb_samples_per_channel); + avfilter_copy_buffer_ref_props(outsamples, insamples); + + if (insamples->format == AV_SAMPLE_FMT_S16) {
+ int16_t *in1 = NULL; + int16_t *in2 = NULL;
const int16_t *
+ int16_t *out = NULL; + + if (!mix->ended[0]) + in1 = (int16_t*)mix->buf[0]->data[0]; + if (!mix->ended[1]) + in2 = (int16_t*)mix->buf[1]->data[0]; + out = (int16_t*)outsamples->data[0]; + + for (i = 0; i < nb_samples; i++) { + int64_t v1 = 0; + int64_t v2 = 0; + if (!mix->ended[0]) + v1 = (((int64_t)*in1++ * mix->vol_i[0] + 128) >> 8); + if (!mix->ended[1]) + v2 = (((int64_t)*in2++ * mix->vol_i[1] + 128) >> 8); + *out++ = av_clip_int16((v1 + v2) / 2); + } + + for (i = 0; i < 2; i++) { + if (!mix->ended[i]) { + mix->buf_pos[i] += nb_samples; + if (mix->buf_pos[i] == nb_samples) { + avfilter_unref_buffer(mix->buf[i]); + mix->buf[i] = 0; + } + } + } + } + + avfilter_filter_samples(outlink, outsamples); +} +
+static int request_frame(AVFilterLink *outlink) +{ + AVFilterContext *ctx = outlink->src; + MixContext *mix = ctx->priv; + int i; + + for (i = 0; i < 2; i++) + if (avfilter_request_frame(ctx->inputs[i]) != 0) {
this should check if data is in the internal buffer before requesting more [...] -- Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB No human being will ever know the Truth, for even if they happen to say it by chance, they would not even known they had done so. -- Xenophanes
On Tue, Dec 13, 2011 at 04:57:15PM +0100, Matthieu Bouron wrote:
Hi there,
Here is a first attempt to write an audio filter which merge two audio streams. Usage is: ffplay -f lavfi "amovie=test1.mp3[a];amovie=test2.mp3[b];[a][b]mix=0.7:0.3[out]"
The volume of each streams can be configured and default to 0.5:0.5 is not specified. It only support two s16 audio streams with same sample rate. Feel free to comment.
Matthieu
From 78537e1c874d62b42b58de8da210dde28eb17859 Mon Sep 17 00:00:00 2001 From: Matthieu Bouron <matthieu.bouron@gmail.com> Date: Sat, 10 Dec 2011 12:31:32 +0100 Subject: [PATCH] af_mix: mix two audio streams.
--- libavfilter/Makefile | 1 + libavfilter/af_mix.c | 223 ++++++++++++++++++++++++++++++++++++++++++++++ libavfilter/allfilters.c | 1 + 3 files changed, 225 insertions(+), 0 deletions(-) create mode 100644 libavfilter/af_mix.c
You will need to bump lavfi minor version, and ideally add a Changelog entry and document it in doc/filters.texi. [...]
+static av_cold int init(AVFilterContext *ctx, const char *args, void *opaque) +{ + int i, n; + double vol1, vol2; + MixContext *mix = ctx->priv; + + if (args) { + n = sscanf(args, "%lf:%lf", &vol1, &vol2); + switch (n) { + case 0: + mix->vol[0] = 0.5; + mix->vol[1] = 0.5; + break; + case 1: + mix->vol[0] = vol1; + mix->vol[1] = 0.5; + break; + case 2: + mix->vol[0] = vol1; + mix->vol[1] = vol2; + break; + default: + av_log(ctx, AV_LOG_ERROR, "sscanf error"); + return 1;
nit: you should remove one level of indent in the switch. Also, the error isn't very explicit...
+ } + for (i = 0; i < 2; i++) { + mix->vol_i[i] = (int)(mix->vol[i] * 256 + 0.5); + } + } + + return 0; +} + +static int query_formats(AVFilterContext *ctx) +{ + int i; + MixContext *mix = ctx->priv; + AVFilterFormats *formats = NULL; + enum AVSampleFormat sample_fmts[] = { + AV_SAMPLE_FMT_S16, + AV_SAMPLE_FMT_NONE + }; + int packing_fmts[] = { AVFILTER_PACKED, -1 };
sample_fmts and packing_fmts could be const.
+ int layout_fmts[] = {0, -1}; + [...] + if (insamples->format == AV_SAMPLE_FMT_S16) { + int16_t *in1 = NULL; + int16_t *in2 = NULL; + int16_t *out = NULL; + + if (!mix->ended[0]) + in1 = (int16_t*)mix->buf[0]->data[0]; + if (!mix->ended[1]) + in2 = (int16_t*)mix->buf[1]->data[0]; + out = (int16_t*)outsamples->data[0]; + + for (i = 0; i < nb_samples; i++) { + int64_t v1 = 0; + int64_t v2 = 0; + if (!mix->ended[0]) + v1 = (((int64_t)*in1++ * mix->vol_i[0] + 128) >> 8); + if (!mix->ended[1]) + v2 = (((int64_t)*in2++ * mix->vol_i[1] + 128) >> 8); + *out++ = av_clip_int16((v1 + v2) / 2);
Since your volumes are by default 0.5, is the /2 really needed here?
+ } + + for (i = 0; i < 2; i++) { + if (!mix->ended[i]) { + mix->buf_pos[i] += nb_samples; + if (mix->buf_pos[i] == nb_samples) { + avfilter_unref_buffer(mix->buf[i]); + mix->buf[i] = 0; + } + } + } + } + + avfilter_filter_samples(outlink, outsamples); +} + +static int request_frame(AVFilterLink *outlink) +{ + AVFilterContext *ctx = outlink->src; + MixContext *mix = ctx->priv; + int i; + + for (i = 0; i < 2; i++) + if (avfilter_request_frame(ctx->inputs[i]) != 0) { + mix->ended[i] = 1; + }
nit: the for without { } looks weird here, but it's at your own discretion.
+ return 0; +} + +AVFilter avfilter_af_mix = { + .name = "mix", + .description = NULL_IF_CONFIG_SMALL("Mix two audio streams."),
Superfluous period -- Clément B.
Patch updated.
Patch rebased and updated.
Le duodi 12 germinal, an CCXX, Matthieu Bouron a écrit :
Patch rebased and updated.
Hi. Before looking at the code, I would like to know: what is the benefit of these 200+ lines of code compared to amerge+pan? Regards, -- Nicolas George
2012/3/31 Nicolas George <nicolas.george@normalesup.org>:
Le duodi 12 germinal, an CCXX, Matthieu Bouron a écrit :
Patch rebased and updated.
Hi. Before looking at the code, I would like to know: what is the benefit of these 200+ lines of code compared to amerge+pan?
Hi, This filter intends to be a simpler alternative to amerge+pan for basic usage in a user's point of view. A good example would be adding a background music to a video.
Regards,
-- Nicolas George
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux)
iEUEARECAAYFAk93NFYACgkQsGPZlzblTJM5IQCgwRb/omxsbPFTPUsE3x5kHM4g eoMAmO61l5yd60b+1LIHi6Yf/z6gXBc= =606X -----END PGP SIGNATURE-----
_______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
Le duodi 12 germinal, an CCXX, Matthieu Bouron a écrit :
This filter intends to be a simpler alternative to amerge+pan for basic usage in a user's point of view.
Others might disagree, but I believe the maintenance burden of feature duplication far outweighs the benefits for the user.
A good example would be adding a background music to a video.
Assuming the input is stereo, you can do that with "[a] [b] amerge, pan=stereo:c0<c0+c2:c1<c1+c3". That's not very hard, and a copy-pastable example in the docs does not suffer from maintenance problems.
On Sat, Mar 31, 2012 at 07:55:00PM +0200, Nicolas George wrote:
Le duodi 12 germinal, an CCXX, Matthieu Bouron a écrit :
This filter intends to be a simpler alternative to amerge+pan for basic usage in a user's point of view.
Others might disagree, but I believe the maintenance burden of feature duplication far outweighs the benefits for the user.
A good example would be adding a background music to a video.
Assuming the input is stereo, you can do that with "[a] [b] amerge, pan=stereo:c0<c0+c2:c1<c1+c3". That's not very hard, and a copy-pastable example in the docs does not suffer from maintenance problems.
Adding the levels, it begins to look a bit insane IMO. Maybe the "mix" filter could be added in the "pan" one, so it could use the internals? (A pan cmd line could be generated in the mix init func and use amerge callbacks) -- Clément B.
Le tridi 13 germinal, an CCXX, Clément Bœsch a écrit :
Adding the levels, it begins to look a bit insane IMO.
That is a bit exaggerated, but I grant you it is somewhat overly verbose.
Maybe the "mix" filter could be added in the "pan" one, so it could use the internals?
Possibly. Another, IMHO cleaner solution, would be to tweak the parser to be able to write: pan=stereo:mix=0.4+0.6 and it would automatically fill the matrix as if the coefficients have been given as "stereo:c0=0.4*c0+0.6*c2:c1=0.4*c1+0.6*c3". A kind of macro-expansion, in fact (but IMHO, simpler written avoiding text operations).
(A pan cmd line could be generated in the mix init func and use amerge callbacks)
I am not sure what you mean, but it looks fragile and ad-hoc. Ad-hoc is my main problem with this patch: Matthieu wrote it for the case he needs to solve, but it stops working on most other case. For example mixing a mono commentary on a stereo music will not work. Adding a special case for a very common need is ok, of course, but I do not believe that mixing with the same channel layout is that much more common than mixing with different layouts, or mixing from more than two streams. Regards, -- Nicolas George
participants (4)
-
Clément Bœsch -
Matthieu Bouron -
Michael Niedermayer -
Nicolas George