[FFmpeg-devel] libavfilter and subtitles (was: [RFC] Talk about subtitles)

[Reimar Döffinger]

On Thu, Dec 22, 2011 at 02:32:09PM +0100, Nicolas George wrote:
> Le septidi 27 frimaire, an CCXX, Clément Bœsch a écrit :
> > First, writing the smovie source filter means adding the subtitles in the
> > filterchain.
> 
> There are big issues that will arise with subtitles filtering if the
> architecture is kept the way it is.
> 
> The problem I see is that subtitles, unlike audio and video, are sparse.
> Thus, with formats that mux the subtitles together with the video, accessing
> the next subtitle frame, as the result of request_frame, could trigger
> several megaoctets' worth of reads on the source file. For transcoding
> applications, this is a small drawback; for live playback this is a big no.
> 
> Solving this issue in a satisfactory manner seems like a lot of work, but I
> believe in the long term it would be worth it.
> 
> Filters with both continuous and sparse outputs should ignore request_frame
> and send on the sparse output when data is available. Filters with sparse
> inputs should not think that request_frame not triggering anything means the
> end of the stream.
> 
> But for that to work, it is necessary that video and subtitles are demuxed
> together. If the filterchain is something like this:
> 
> 	movie=input.vob [v]; smovie=input.mkv [s]; [v] [s] hardsub
> 
> then smovie has only the sparse output, it can not keep it in sync with the
> video stream.

What would be the point of that? If the inputs are different you can
just read up to the next subtitle, for most formats that should not be
that slow.

> IMHO, the real, clean, solution, would be to device a way for the movie
> source to be able to output both the video and the subtitles if necessary.
> And, while we are at it, the audio too.

That breaks down as soon as you _want_ to have a significant delay
between e.g. video and subtitle.
There are different options that seem more flexible to me:
1) request frame could have a time stamp. If there is no packet around
that time stamp, just return nothing. I think this will imply some
issues.
2) for sparse streams, we should automatically insert empty "virtual"
packets that have as pts the most recent time-stamp of which we know
there is no subtitle data for.
Though in light of seeking etc. I actually think that this would be much
easier if we re-introduced the end_pts again, but some people were
opposed enough to it to not want it despite its lack breaking MPlayer and
nobody really working on making things work properly without it.