[FFmpeg-devel] [PATCH 6/6] lavfi: make AVFilterLink opaque in two major bumps.

[Michael Niedermayer]

On Tue, Dec 20, 2016 at 07:24:44PM +0100, Nicolas George wrote:
> Le nonidi 29 frimaire, an CCXXV, Michael Niedermayer a écrit :
> > though i wonder what mistake we did to end here, the original design
> > was intended to be very simple ...
> 
> That was the mistake: trying for a simple design.
> 
> Ha ha! Only serious. Linear filter chains are simple, and could have
> worked easily with a simple design. But non-linear filter graphs are
> inherently more complex: the system needs to decide what filter to
> activate first from several parallel branches.

it shouldnt really be complex, not in concept, maybe in (efficient)
implementation
For example as a concept one could imagine each filter runs as its
own thread and waits on its inputs availability and output space.
if it gets input and has output space its woken up and works and
produces output and then wakes its surroundings up.
no difference between a linear chain or a complex graph here

and its simple as we use and need just a local view centered on a
filter and its neighbors

IIUC you already implemented the gory details of filters handing stuff
around and that working and not getting stuck nor bloating up too bad.
But as a concept ignoring the implementation details i would call it
simple.

iam not sure its usfull but
Another view from a different direction would be to see the filter
graph as a network of pipes with flowing water and interconnected
pumps. This too is a simple concept and intuition would suggest that
this would not easily end up with water being stuck nor too much
accumulating. It does not 1:1 match real filters though which work
on discrete frames, i wonder if this view is usefull
It would allow awnsering global questions about a graph, like what
inputs are useless when some output is welded shut or vice versa


> 
> Graphs with several inputs and/or outputs are even more difficult,
> because the application needs to decide which input needs a frame most
> urgently.
> 

> Both lavfi's original design (API and implementation) and its use in
> ffmpeg (actually, avconv) for complex filter graphs neglect partially or
> completely this issue.

i think the original lavfi design didnt really had any issue with graphs with
multiple inputs or outputs. A user app could decide were to in and out
put. but FFmpeg didnt support this at the time IIRC so the people working
on the original lavfi had nothing to implement.
the problems came when this support was added much later


[...]
> > You complain that noone else is working on libavfilter
> > how many people know what _NEEDS_ to be done to make the API good
> > enough for a first public and stable API ?
> > 
> > while i have a long todo and maybe no time for this, as it is i cant
> > even work on making libavfilter public+stable. Its not a technical
> > issue, its one of peoples requirements on what they require to be
> > changed first.
> 
> This is true, but there is another side to the human problem: explaining
> the plans and requirements of the API takes time; if nobody will read it
> carefully, that time is better spent actually coding (I am writing this
> mail by snippets while FATE is running).
> 
> Here is an outline, in approximate order (there are dependencies, but
> not everything):
> 

> - Add a callback AVFilter.activate() to replace filter_frame() on all
>   inputs and request_frame() on all outputs. Most non-trivial filters
>   are written that way in the first place.

ok thats mostly cosmetic id say.
I think we possibly even had something somewhat in that direction in
the original lavfi discussions MANY years ago
this makes more sense with non recursive stuff


> 
> - Change buffersink to implement that callback and peek directly in the
>   FIFO.

ok, "cosmetic"


> 
> - Write a generic activate callback for 1-to-1 filters and use it.

there were plans for simplifying generic filters
1 in 1 out frame filters
apply something to every pixel filters
some more cases i forgot

while different now with activate() the idea is old and IIRC was
supposed to be done long long ago. I think its a good idea if it does
happen


> 
> - Rewrite framesync (the utility system for filters with several video
>   inputs that need synchroized frames) to implement activate and use the
>   FIFO directly.

cosmetic :)


> 
> - Allow to give buffersrc a timestamp on EOF, make sure the timestamp is
>   forwarded by most filters and allow to retrieve it from buffersink.
> 
>   (If somebody has a suggestion of a good data structure for that...)

AVFrame.duration
This possibly is a big subject for discussion on its own, but maybe
not i dont know.


> 
> - Allow to set a callback on buffersinks and buffersrcs to be notified
>   when a frame arrives or is needed. It is much more convenient than
>   walking the buffers to check one by one.

agree that walking is bad.
cannot ATM argue on what is better as i dont feel that i have a clear
enough view of this and the surroundings.


> 
> - Allow to merge several filter graphs into one. This may be useful when
>   applications have several graphs to run simultaneously, so that they
>   do not need to decide which one to activate. Another option would be
>   to have the functions in the next step work on possibly several
>   graphs.

This may be orthogonal but i think a filter graph should be a filter
(maybe not at struct level litterally but it should be possible to have a
 AVFilter that is backed by a arbitrary user specified filter graph)


> 
> - Keep a priority queue of ready filters and use it instead of walking
>   the graph.

something better than O(n) yes, agree


> 
> - Add a function to run the graph until "something" happens; "something"
>   meaning a stop instruction called by the callbacks.

dont understand


> 
> - Add utility functions to easily run the graph. The most useful IMHO:
>   run until a frame comes out from any sink (we get the frame and
>   something useful identifying the sink) or one is needed from a source
>   (we get something identifying the source) or EOF.

if the API itself is simpler with utility functions then this sounds
like a good idea

thx

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

There will always be a question for which you do not know the correct answer.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: Digital signature
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20161220/5dc77db0/attachment.sig>