Increased latency from filter_complex
Hi everyone, we are using ffmpeg to create hls streams in multiple resolutions and also languages from incoming rtmp streams. When encoding without the filter_complex for ducking we get a minimal latency of around 8 seconds, but the moment we add the autoducking we jump up to around 15 seconds. Also it seems that for every additional audio stream we add, it adds another 8 seconds of latency. The filter_complex is the folowing: ``` -filter_complex "format=nv12,hwupload,scale_vaapi=w=1280:h=720,split=2[720p][480];[480]scale_vaapi=w=854:h=480,split=2[480p][240];[240]scale_vaapi=w=426:h=240[240p]; \ [0:a:0] loudnorm=i=-16.0:lra=12.0:tp=-3.0, asplit=2 [normalize] [out]; \ [normalize] volume=0.1, asplit=2 [bg1] [bg2]; \ [1:a:0] agate, adelay=2900:all=1 [tmp1]; \ [bg1][tmp1] amix, dynaudnorm=b=1:p=0.35:r=1:f=150, loudnorm=i=-16.0:lra=12.0:tp=-3.0 [out1]; \ [2:a:0] agate, adelay=8000:all=1 [tmp2]; \ [bg2][tmp2] amix, dynaudnorm=b=1:p=0.35:r=1:f=150, loudnorm=i=-16.0:lra=12.0:tp=-3.0 [out2]" ´´´ To us it looks like the audio streams get processed sequentially. Each process needs to check the secondary audio levels for a few seconds, to be able to duck the main audio accordingly, resulting in the increased overall latency. Is this correct and if so, is there a way to achieve this in parallel? Or is there any other way to get the autoducking, that doesnt introduce additional latency? Thank you! Philipp PS: Full ffmpeg call in the appended textfile.
On Tue, Jul 27, 2021 at 12:36 PM <technik+ffmpeg@vistream.online> wrote:
Hi everyone,
we are using ffmpeg to create hls streams in multiple resolutions and also languages from incoming rtmp streams. When encoding without the filter_complex for ducking we get a minimal latency of around 8 seconds, but the moment we add the autoducking we jump up to around 15 seconds. Also it seems that for every additional audio stream we add, it adds another 8 seconds of latency. The filter_complex is the folowing: ``` -filter_complex "format=nv12,hwupload,scale_vaapi=w=1280:h=720,split=2[720p][480];[480]scale_vaapi=w=854:h=480,split=2[480p][240];[240]scale_vaapi=w=426:h=240[240p]; \ [0:a:0] loudnorm=i=-16.0:lra=12.0:tp=-3.0, asplit=2 [normalize] [out]; \ [normalize] volume=0.1, asplit=2 [bg1] [bg2]; \ [1:a:0] agate, adelay=2900:all=1 [tmp1]; \ [bg1][tmp1] amix, dynaudnorm=b=1:p=0.35:r=1:f=150, loudnorm=i=-16.0:lra=12.0:tp=-3.0 [out1]; \ [2:a:0] agate, adelay=8000:all=1 [tmp2]; \ [bg2][tmp2] amix, dynaudnorm=b=1:p=0.35:r=1:f=150, loudnorm=i=-16.0:lra=12.0:tp=-3.0 [out2]" ´´´ To us it looks like the audio streams get processed sequentially. Each process needs to check the secondary audio levels for a few seconds, to be able to duck the main audio accordingly, resulting in the increased overall latency.
Is this correct and if so, is there a way to achieve this in parallel?
Or is there any other way to get the autoducking, that doesnt introduce additional latency?
3 loudnorm calls and 2 dynaudnorm calls? for ducking there is sidechaincompress filter
Thank you!
Philipp
PS: Full ffmpeg call in the appended textfile. _______________________________________________ ffmpeg-user mailing list ffmpeg-user@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-user
To unsubscribe, visit link above, or email ffmpeg-user-request@ffmpeg.org with subject "unsubscribe".
Hi Paul, thanks for your answer.
3 loudnorm calls and 2 dynaudnorm calls? for ducking there is sidechaincompress filter
We tried sidechain of course, but will try this again. Our problem is, that the resulting audio from the sidechaincompress never reached the same quality. The audio from the primary channel also never got down to a level, where the secondary audio could be understood without problems. If you have any parameters or tips how to achieve this, please share. Thank you. But regarding the central difficulty, the increasing latency from - what we suspect serial instead of parallel processing of the audio channels - this does not help. Or is sidechaincompress being handled differently? Thank you for your time. Philipp
On Wed, Jul 28, 2021 at 10:11 PM <technik+ffmpeg@vistream.online> wrote:
Hi Paul, thanks for your answer.
3 loudnorm calls and 2 dynaudnorm calls? for ducking there is sidechaincompress filter
We tried sidechain of course, but will try this again. Our problem is, that the resulting audio from the sidechaincompress never reached the same quality. The audio from the primary channel also never got down to a level, where the secondary audio could be understood without problems. If you have any parameters or tips how to achieve this, please share. Thank you.
But regarding the central difficulty, the increasing latency from - what we suspect serial instead of parallel processing of the audio channels - this does not help. Or is sidechaincompress being handled differently?
Please show how you use sidechaincompress filter.
Thank you for your time.
Philipp _______________________________________________ ffmpeg-user mailing list ffmpeg-user@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-user
To unsubscribe, visit link above, or email ffmpeg-user-request@ffmpeg.org with subject "unsubscribe".
Hi Paul, thanks again for answering.
Please show how you use sidechaincompress filter.
Basically we did what is mentioned here: https://superuser.com/questions/1093607/ffmpeg-sidechaincompress-give-more-i... We too found no audible difference between ratio 1 and ratio 20. But this all doesn't touch the initial question I contacted the mailing list for. We found that applying any ducking increased latency. by ~8 seconds and for every additional audio track it added 8 seconds on top. So it seems like the analyzing of the the audio tracks happens sequentially at least for the filters we used. What I wanted to know is, if there are ways to write those differently to mitigate this problem. In case of the sidechain filter, if this is handled sequentially too, the problem would still be there, even if a sidechain might be less workload. @all Anybody else knows how to do this in parallel instead? Thanks Philipp
On Thu, Jul 29, 2021 at 7:13 PM <technik+ffmpeg@vistream.online> wrote:
Hi Paul,
thanks again for answering.
Please show how you use sidechaincompress filter.
Basically we did what is mentioned here:
https://superuser.com/questions/1093607/ffmpeg-sidechaincompress-give-more-i...
We too found no audible difference between ratio 1 and ratio 20.
But this all doesn't touch the initial question I contacted the mailing list for. We found that applying any ducking increased latency. by ~8 seconds and for every additional audio track it added 8 seconds on top.
Is sidechaincompress adding 8 seconds of latency too? ratio in processing is used with other options in parallel, especially threshold. Copying random commands with little or no understanding how filter is working will get you nowhere. Please read documentation of sidechaincompress filter, and experiment with command provided in documentation. Probably need to use amix filter instead of amerge as last filter in filtergraph.
So it seems like the analyzing of the the audio tracks happens sequentially at least for the filters we used. What I wanted to know is, if there are ways to write those differently to mitigate this problem. In case of the sidechain filter, if this is handled sequentially too, the problem would still be there, even if a sidechain might be less workload.
@all Anybody else knows how to do this in parallel instead?
Latency of loudnorm and dynaudnorm filters are extremely high compared to sidechaincompress, always couple of seconds. And keeping using those filters you will never expect less latency. You can control latency in some cases with asetnsamples filter as first filter in filtergraph. Also if you willing to provide input demo files I may look more into this, to help you get correct filtergraph. Also note that encoder may add additional latency.
Thanks
Philipp _______________________________________________ ffmpeg-user mailing list ffmpeg-user@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-user
To unsubscribe, visit link above, or email ffmpeg-user-request@ffmpeg.org with subject "unsubscribe".
participants (2)
-
Paul B Mahol -
technik+ffmpeg@vistream.online