[FFmpeg-trac] #3079(FFmpeg:open): libavfilter caches and drops frames with multiple desynched input streams

[FFmpeg]

#3079: libavfilter caches and drops frames with multiple desynched input streams
------------------------------------+----------------------------------
             Reporter:  richardpl   |                    Owner:
                 Type:  defect      |                   Status:  open
             Priority:  normal      |                Component:  FFmpeg
              Version:  git-master  |               Resolution:
             Keywords:  bounty      |               Blocked By:
             Blocking:              |  Reproduced by developer:  1
Analyzed by developer:  1           |
------------------------------------+----------------------------------

Comment (by Cigaes):

 I believe you misunderstood part of the problem. lavfi already have the
 required infrastructure to select the best input to get the data flowing.
 The problem is that ffmpeg does not respect that selection.

 More precisely, here is what happens:

 * Try to overlay 0:v on top of 1:v, muxed with audio from 1:a.

 * ffmpeg decides it needs a video packet, it requests a packet from the
 overlay filter graph.

 * overlay needs a frame on 0:v to progress, it does not have one, it marks
 it as needed.

 * ffmpeg tries to read from 0:v, '''XXX''' but 0:v is not ready.

 * ffmpeg marks the output video stream as temporarily unavailable and
 moves to the next stream: audio.

 * ffmpeg wants a frame for 1:a, so it reads from 1, possibly getting a
 frame for 1:v instead.

 * Repeat.

 The problem is in '''XXX''': if it happens occasionally due to thread
 scheduling randomness, it is not a problem because it is absorbed by the
 various buffers and ffmpeg catches up later.

 But if '''XXX''' happens repeatedly because 0 is slower than 1, then it
 becomes a problem, because decoded frames from 1:v will accumulate in
 overlay's input.

 But it has really nothing to do with lavfi and multiple inputs, the exact
 same happens if you just want to mux 0:v with 1:a, except this times it's
 encoded audio packets that accumulate in the muxer's queue, and that eats
 much less memory. But still, try this:
 {{{
 ./ffmpeg_g -f lavfi -i testsrc=s=10240x7680,scale=320:240 -f s16le -i
 /dev/zero -f framecrc -
 }}}
 {{{
 Error while decoding stream #1:0: Cannot allocate memory
 }}}

 IMHO, the good solution in this case is to stop being greedy: if ffmpeg
 decided it needs to read from input #‌0, then let it wait fro input #‌0;
 the demuxing threads are there to take care of input #‌1 in the meantime
 if necessary.

 I have sent a patch to that effect to the devel mailing-list, I suggest we
 continue the discussion there, with a decent user interface.

-- 
Ticket URL: <https://ffmpeg.org/trac/ffmpeg/ticket/3079#comment:20>
FFmpeg <http://ffmpeg.org>
FFmpeg issue tracker