[FFmpeg-devel] [RFC] How to deal with libavfilter buffer overflows?

[Michael Niedermayer]

On Wed, Oct 22, 2014 at 01:39:07PM +0200, Nicolas George wrote:
> Le decadi 30 vendémiaire, an CCXXIII, Stefano Sabatini a écrit :
> > $ ffmpeg -f lavfi -i "testsrc=d=10, split[t0][t1]; [t0]select='gte(t,5)',setpts=PTS-STARTPTS[s0]; [t1] select='lt(t,5)', setpts=PTS-STARTPTS[s1]; [s0][s1] concat" -y outtest.mp4
> > [...]
> > Input #0, lavfi, from 'testsrc=d=10, split[t0][t1]; [t0]select='gte(t,5)',setpts=PTS-STARTPTS[s0]; [t1] select='lt(t,5)', setpts=PTS-STARTPTS[s1]; [s0][s1] concat':
> >   Duration: N/A, start: 0.000000, bitrate: N/A
> >     Stream #0:0: Video: rawvideo (RGB[24] / 0x18424752), rgb24, 320x240 [SAR 1:1 DAR 4:3], 25 fps, 25 tbr, 1000k tbn, 1000k tbc
> > [...]
> > Output #0, mp4, to 'outtest.mp4':
> >   Metadata:
> >     encoder         : Lavf56.9.101
> >     Stream #0:0: Video: h264 (libx264) ([33][0][0][0] / 0x0021), yuv444p, 320x240 [SAR 1:1 DAR 4:3], q=-1--1, 25 fps, 12800 tbn, 25 tbc
> >     Metadata:
> >       encoder         : Lavc56.8.102 libx264
> > Stream mapping:
> >   Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (libx264))
> > Press [q] to stop, [?] for help
> > [out @ 0x3140560] 100 buffers queued in out, something may be wrong.
> > frame=  125 fps=0.0 q=-1.0 Lsize=      25kB time=00:00:04.92 bitrate=  42.4kbits/s    
> > video:23kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 9.620967%
> > 
> > I frequently stumble upon situations when buffer overflows happen, and
> > they are usually a showstopper in many applications. For example when
> > recombining streams with the concat filter, or when overlaying. The
> > output is usually playable but misses audio or video frames.  For
> > example in the case above, the command is trying to switch order of
> > the two input parts, but only the second part is displayed and the
> > user is left with the vague message "something may be wrong".
> > 
> > So the current buffering system sometimes fails, and the user has no
> > clue to understand where the problem lies. I want at least to improve
> > diagnostics, but I want to listen about other developers experience
> > first, and if you have ideas to tackle such issues.
> 
> Hi.
> 
> I am well aware of the problem too, and actually I have recently started
> writing a summary to try address it, along with the other problems of the
> current lavfi API.
> 
> There are several underlying issues that are raised by your example:
> 
> * You have AB, and you want to split and merge it into BA, but you only get
>   B: it looks like it is "just" a bug. The issue disappears if I test with
>   "ffmpeg -lavfi filter" instead of "ffmpeg -f lavfi -i filter". I have yet
>   to fully understand if the bug is in lavd/lavfi or in lavfi itself, but
>   even in the former case, there is still the problem that the API is almost
>   impossible to use correctly.
> 
> * Apart from this bug, this command works, but that is a bit of luck.
>   Someone raised the input buffer queue for concat to 256, and your graph
>   requires 128. The warning you see is just a warning. But if you were to
>   raise the duration of the first segment to something above 256 frames, it
>   would indeed fail. The message to look for is "Buffer queue overflow,
>   dropping.", not "100 buffers queued in output stream".
> 
> * You have AB from a non-replayable input and you want BA. That can not work
>   reliably. If your input were replayable, you could replay it as A1B1 and
>   A2B2 and produce B2A1. But since it is not replayable, you need to buffer
>   all A as decoded frames in memory. In this particular case, that is
>   5×25×320×240×3 = 27 Mo, still acceptable. But imagine the same thing with
>   a full-HD credits sequence: 5×90×1920×1080×3/2 = 1.3 Go.
> 
>   When it has already eaten half a gigaoctet of memory, ffmpeg/lavfi can
>   only guess whether this is expected and the memory consumption will soon
>   stop or if the graph has a bug (arithmetic mistake in setpts, links
>   connected to the wrong pad) that makes it eat memory indefinitely.
> 
>   Since the second case is likely to lead to memory and swap exhaustion,
>   making the host machine unusable, ffmpeg should not attempt it without
>   explicit user consent.

the fifo/buffer que should grow the ability to buffer frames to disk
and to do so using lossless and lossy compression
that way much larger delays can be achived with such buffers without
exhaustig memory

true seek support should be added to the source filters so that
one can just specify which sections to play in which order without the
need to use large buffers

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

The misfortune of the wise is better than the prosperity of the fool.
-- Epicurus
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: Digital signature
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20141022/207ec34c/attachment.asc>