Evolution of lavfi's design and API
[ CCing Anton, as most that is written here also apply to libav too, and this would be a good occasion to try a cross-fork cooperation; if that is not wanted, please let us know so we can drop the cc. ] 1. Problems with the current design 1.1. Mixed input-/output-driven model Currently, lavfi is designed to work in a mixed input-driven and output-driven model. That means the application needs sometimes to add input to buffersources and sometimes request output to buffersinks. This is a bit of a nuisance, because it requires the application to do it properly: adding input on the wrong input or requesting a frame on the wrong output will cause extra memory consumption or latency. With the libav API, it can not work at all since there is no mechanism to determine which input needs a frame in order to proceed. The libav API is clearly designed for a more output-driven implementation, with FIFOs anywhere to prevent input-driven frames to reach unready filters. Unfortunately, since it is impossible from the outside to guess what output will get a frame next, that can cause frames to accumulate anywhere in the filter graph, eating a lot of memory unnecessarily. FFmpeg's API has eliminated FIFOs in favour of queues in filters that need it, but these queues can not be controlled for unusual filter graphs with extreme needs. Also, there still is an implicit FIFO inside buffersink. 1.2. Recursive implementation All work in a filter graph is triggered by recursive invocations of the filters' methods. It makes debugging harder. It also can lead to large stack usage and makes frame- and filter-level multithreading harder to implement. It also prevents some diagnosis from working reliably. 1.3. EOF handling Currently, EOF is propagated only through the return value of the request_frame() method. That means it only works in an output-driven scheme. It also means that it has no timestamp attached to it; this is an issue for filters where the duration of the last frame is relevant, like vf_fps. 1.4. Latency Some filters need to know the timestamp of the next frame in order to know when the current frame will stop and be able to process it: overlay, fps are two examples. These filters will introduce a latency of one input frame that could otherwise be avoided. 1.5. Timestamps Some filters do not care about timestamps at all. Some check and have a proper handling of NOPTS values. Some filters just assume the frames will have timestamps, and possibly make extra assumptions on that: monotony, consistency, etc. That is an inconsistent mess. 1.6. Sparse streams There is a more severe instance of the latency issue when the input comes from an interleaved sparse stream: in that case, waiting for the next frame in order to find the end of the current one may require demuxing a large chunk of input, in turn provoking a lot of activity on other inputs of the graph. 2. Proposed API changes To fix/enhance all these issues, I believe a complete rethink of the scheduling design of the library is necessary. I propose the following changes. Note: some of these changes are not 100% related to the issues I raised, but looked like a good idea while thinking on an API rework. 2.1. AVFrame.duration Add a duration field to AVFrame; if set, it indicates the duration of the frame. Thus, it becomes unnecessary to wait for the next frame to know when the current frame stops, reducing the latency. Another solution would be to add a dedicated function on buffersrc to inject a timestamp for end or activity on a link. That would avoid the need of adding a field to AVFrame. 2.2. Add some fields to AVFilterLink AVFilterLink.pts: current timestamp of the link, i.e. end timestamp of the last forwarede frame, assuming the duration was correct. This is somewhat redundant with the fields in AVFrame, but can carry the information even when there is no actual frame. AVFilterLink.status: if not 0, gives the return status of trying to pass a frame on this link. The typical use would be EOF. 2.3. AVFilterLink.need_ts Add a field to AVFilterLink to specify that the output filter requires reliable timestamps on its input. More precisely, specify how reliable the timestamps need to be: is the duration necessary? do the timestamps need to be monotonic? continuous? For audio streams, consistency between timestamps and the number of samples may also be tested. For video streams, constant frame rate may be enforced, but I am not sure about this one. A "fixpts" filter should be provided to allow the user to tweak how the timestamps are fixed (change the timestamps to match the duration or change the duration to match the timestamps?). When no explicit filter is inserted, the framework should do the work of fixing them automatically. I am not sure whether that should be done directly or by automatically inserting the fixpts filter. The later solution is more elegant, but it requires more changes to the framework and the filters (because the correctness of the timestamps would need to be merged just like formats), so I am rather for the former. Note that for a lot of filters, the actual duration or end timestamp is not required, only a lower bound for it. For sparse interleaved streams, that is very relevant as we may not know the exact time for the next frame until we reach it, but we can know it is later than the other streams' timestamps minus the interleaving delta. 2.4. Build FIFOs directly in AVFilterLink Instead of automatically insert an additional filter like libav, handle the FIFO operation directly in the framework using fields in AVFilterLink. The main benefit is that the framework can examine the inside of the FIFOs to make scheduling decisions. It can also do so to provide the user with more accurate diagnostics. An extra benefit: the memory pool for the FIFOed frames can more easily be shared, across the whole filter graph or the whole application. Memory management becomes easier: just take a good heuristics (half the RAM?), no need to guess what FIFOs will actually need a lot of memory and what FIFOs are just there mostly useless. Last but not least, FIFOs now become potential thread communication / synchronization points, making filter-level multithreading easier. For audio streams, framing (i.e. ensuring all frame have an exact / minimum / maximum number of samples) can be merged with FIFOs. 2.5. Allow status change pseudo-frames inside FIFOs To propagate EOF and possibly other status changes (errors) in input-driven model, allow FIFOs to contain not only frames but also kind of pseudo-frames with a timestamp and metadata attached. Depending on the filters, these pseudo-frames may be directly passed to the filter's methods, or they may be interpreted by the framework to just change fields on the AVFilterLink structure. 2.6. Change the scheduling logic for filters From the outside of a filter with several outputs, it is usually not possible to guess what output will get a frame next. Requesting a frame on output #0 may cause activity on the filter graph that produce a frame on output #1 instead, or possibly on a completely different filter. Therefore, having a request_frame() method on all outputs seems pointless. Instead, use a global AVFilter.activate() method that causes the filter to do one step of work if it can. This method is called each time something is changed to the filter: new frame on input, output ready, status change. It returns as soon at it could do something, either producing output and/or consuming input, or nothing if nothing can be done. 2.7. Add fields to AVFilterLink for flow control. Add to AVFilterLinks a few field to help filters decide if they need to process something, and if relevant in what order. The most obvious idea would be AVFilterLink.frames_needed, counting how many frames are probably needed on a link before anything can be done. For example, with concat, after input has been consumed, the frames_needed fields on the current input are set according to the corresponding output. 2.8. Activate the filters iteratively Keep a global (per graph) priority queue of filters that are supposed to be ready and call the activate() method on them. 2.9. AVFrame.stream_id Add an integer (or pointer: intptr_t maybe?) field to AVFrame to allow passing frames related to distinct streams on the same link. That would allow to multiplex all outputs of a graph into a single output, making the application simpler. Not sure this is really useful or necessary: for the graph outputs, a convenience function iterating on all of them and returning the frame and the output index separately would do the trick too. 2.10. buffersrc.callback and buffersink.callback Add a callback on both buffersource and buffersink, called respectively when a frame is necessary on input and a frame has arrived on output. This allows pure input-driven and pure output-driven design to work. 2.11. Links groups Links that carry frames from related interleaved streams should be explicitly connected together so that the framework can use the information. The typical use would be to group all the links from buffersrc that come from the same interleaved input file. When a frame is passed on a link, all links in the same group(s) that are too late (according to an interleaving tolerance that can be set) are activated using a dummy frame. 2.12. FIFOs with compression and external storage All FIFOs should be able to off-load some of their memory requirements by either compressing the frames (using a lossless or optionally lossy codec) and/or storing them on mass storage. The options for that should be changeable globally or on a per-link basis. 2.13. AVFrame.owner Add a owner field (probably with type "AVObject", i.e. "void *" pointing to AVClass *) to AVFrame, and update it whenever the frame is passed from one filter to the other. That way, inconsistent ref/unref operations can be detected. Regards, -- Nicolas George
On Wed, Oct 22, 2014 at 11:45:42PM +0200, Nicolas George wrote:
[ CCing Anton, as most that is written here also apply to libav too, and this would be a good occasion to try a cross-fork cooperation; if that is not wanted, please let us know so we can drop the cc. ]
1. Problems with the current design
1.1. Mixed input-/output-driven model
Currently, lavfi is designed to work in a mixed input-driven and output-driven model. That means the application needs sometimes to add input to buffersources and sometimes request output to buffersinks. This is a bit of a nuisance, because it requires the application to do it properly: adding input on the wrong input or requesting a frame on the wrong output will cause extra memory consumption or latency.
With the libav API, it can not work at all since there is no mechanism to determine which input needs a frame in order to proceed.
The libav API is clearly designed for a more output-driven implementation, with FIFOs anywhere to prevent input-driven frames to reach unready filters. Unfortunately, since it is impossible from the outside to guess what output will get a frame next, that can cause frames to accumulate anywhere in the filter graph, eating a lot of memory unnecessarily.
FFmpeg's API has eliminated FIFOs in favour of queues in filters that need it, but these queues can not be controlled for unusual filter graphs with extreme needs. Also, there still is an implicit FIFO inside buffersink.
1.2. Recursive implementation
All work in a filter graph is triggered by recursive invocations of the filters' methods. It makes debugging harder. It also can lead to large stack usage and makes frame- and filter-level multithreading harder to implement. It also prevents some diagnosis from working reliably.
This is definitely a huge hindrance and related to 1.1 I'd be curious to hear about how VapourSynth & friends handle that problem, because AFAIK it's only one way. It's likely they don't have to deal with the same problems we have though (the usage is more limited, no audio typically); typically because they don't seem stream but file based (so easy to index and exact seek etc.).
1.3. EOF handling
Currently, EOF is propagated only through the return value of the request_frame() method. That means it only works in an output-driven scheme. It also means that it has no timestamp attached to it; this is an issue for filters where the duration of the last frame is relevant, like vf_fps.
1.4. Latency
Some filters need to know the timestamp of the next frame in order to know when the current frame will stop and be able to process it: overlay, fps are two examples. These filters will introduce a latency of one input frame that could otherwise be avoided.
1.5. Timestamps
Some filters do not care about timestamps at all. Some check and have a proper handling of NOPTS values. Some filters just assume the frames will have timestamps, and possibly make extra assumptions on that: monotony, consistency, etc. That is an inconsistent mess.
1.6. Sparse streams
There is a more severe instance of the latency issue when the input comes from an interleaved sparse stream: in that case, waiting for the next frame in order to find the end of the current one may require demuxing a large chunk of input, in turn provoking a lot of activity on other inputs of the graph.
More still standing problems while we are at it: 1.7. Metadata Metadata are not available at "graph" level, or at least filter level, only at frame level. We also need to define how they can be injected and fetched from the users (think "rotate" metadata). 1.8. Seeking Way more troublesome: being able to request an exact frame in the past. This currently limits a lot the scope of the filters. thumbnail filter is a good example of this problem: the filter doesn't need to keep all the frames it analyzes in memory, it just needs statistics about them, and then fetches the best in the batch. Currently, it needs to keep them all because we are in a forward stream based logic. This model is kind of common and quite a pain to implement currently. I don't think the compression you propose at the end would really solve that. 1.9. Automatic I/O count "... [a] split [b][c] ..." should guess there is 2 outputs. "... [a][b][c] concat [d] ..." as well
2. Proposed API changes
To fix/enhance all these issues, I believe a complete rethink of the scheduling design of the library is necessary. I propose the following changes.
Did you already started some development? Do you need help? I'm asking because it looks like it could be split into small relatively easy tasks on the Trac and helps introducing new comers (and also track the progress if some people assign themselves to these tickets).
Note: some of these changes are not 100% related to the issues I raised, but looked like a good idea while thinking on an API rework.
2.1. AVFrame.duration
Add a duration field to AVFrame; if set, it indicates the duration of the frame. Thus, it becomes unnecessary to wait for the next frame to know when the current frame stops, reducing the latency.
Another solution would be to add a dedicated function on buffersrc to inject a timestamp for end or activity on a link. That would avoid the need of adding a field to AVFrame.
2.2. Add some fields to AVFilterLink
AVFilterLink.pts: current timestamp of the link, i.e. end timestamp of the last forwarede frame, assuming the duration was correct. This is somewhat redundant with the fields in AVFrame, but can carry the information even when there is no actual frame.
The timeline system seems to be able to workaround this. How is this going to help?
AVFilterLink.status: if not 0, gives the return status of trying to pass a frame on this link. The typical use would be EOF.
2.3. AVFilterLink.need_ts
Add a field to AVFilterLink to specify that the output filter requires reliable timestamps on its input. More precisely, specify how reliable the timestamps need to be: is the duration necessary? do the timestamps need to be monotonic? continuous?
For audio streams, consistency between timestamps and the number of samples may also be tested. For video streams, constant frame rate may be enforced, but I am not sure about this one.
A "fixpts" filter should be provided to allow the user to tweak how the timestamps are fixed (change the timestamps to match the duration or change the duration to match the timestamps?).
When no explicit filter is inserted, the framework should do the work of fixing them automatically. I am not sure whether that should be done directly or by automatically inserting the fixpts filter. The later solution is more elegant, but it requires more changes to the framework and the filters (because the correctness of the timestamps would need to be merged just like formats), so I am rather for the former.
Well, I believe it should be handled by the framework transparently somehow. Users can already fix the timestamps themselves with [a]setpts filters, but it's often not exactly obvious why they do need to. It doesn't need to be a filter and can be part of the framework itself.
Note that for a lot of filters, the actual duration or end timestamp is not required, only a lower bound for it. For sparse interleaved streams, that is very relevant as we may not know the exact time for the next frame until we reach it, but we can know it is later than the other streams' timestamps minus the interleaving delta.
[...] -- Clément B.
Le duodi 2 brumaire, an CCXXIII, Clement Boesch a écrit :
I'd be curious to hear about how VapourSynth & friends handle that problem, because AFAIK it's only one way. It's likely they don't have to deal with the same problems we have though (the usage is more limited, no audio typically); typically because they don't seem stream but file based (so easy to index and exact seek etc.).
I am not sure what you mean here. This was about recursive design. It is not very difficult to turn the recursive design into an iterative one: just handle the call stack manually. In this particular case, most of the filters use only tail-recursion ("return ff_filter_frame(outlink, frame);"), so there is even no need to handle a stack: just replace the function calls by message passing. People here seem impressed by VapourSynth's "nice" python syntax to build scripts, but it does not look extraordinary to me. I am convinced this is "just" a syntax to build the graph. For example, where lavfi requires to write "[v][s]overlay=x=42:y=12[vs]", VapourSynth would require something like "vs = v.overlay(s, 42, 12)". Just a syntactic matter. The actual work certainly happens when the graph is connected to its output. I believe it would be rather easy to have the same kind of interface for lavfi; all the filters are already introspectable, and that is the most major point. Unfortunately, two issues would make it of little use. First, lavfi handles only frames, not packets, so all the encoding and muxing would have to be handled separately. Second, lavfi is designed to make a lot of format decisions when the graph is complete, so the scripting capabilities would not have a lot of relevant information to work on. I wonder how VapourSynth handles that, but I am afraid to learn that it just do automatic conversions on the spot without globally optimizing them. Regards, -- Nicolas George
On Fri, 24 Oct 2014 13:07:22 +0200 Nicolas George <george@nsup.org> wrote:
Le duodi 2 brumaire, an CCXXIII, Clement Boesch a écrit :
I'd be curious to hear about how VapourSynth & friends handle that problem, because AFAIK it's only one way. It's likely they don't have to deal with the same problems we have though (the usage is more limited, no audio typically); typically because they don't seem stream but file based (so easy to index and exact seek etc.).
I am not sure what you mean here. This was about recursive design. It is not very difficult to turn the recursive design into an iterative one: just handle the call stack manually. In this particular case, most of the filters use only tail-recursion ("return ff_filter_frame(outlink, frame);"), so there is even no need to handle a stack: just replace the function calls by message passing.
People here seem impressed by VapourSynth's "nice" python syntax to build scripts, but it does not look extraordinary to me. I am convinced this is "just" a syntax to build the graph. For example, where lavfi requires to write "[v][s]overlay=x=42:y=12[vs]", VapourSynth would require something like "vs = v.overlay(s, 42, 12)". Just a syntactic matter. The actual work certainly happens when the graph is connected to its output.
But guess which syntax is nicer. And this is not only about syntax. VapourSynth is truly more flexible, and there's a damn good reason why avisynth is still alive (despite being a pile of crap code-wise), and why VapourSynth bases its design on it (hopefully only the good and sane parts). Also note that Python is not required for VapourSynth. I could hack up a Lua scripting frontend in 2 hours or so. Last but not least, the nicest part about VapourSynth is that you can use external filters. Nobody who writes a very sophisticated video filter wants to deal with FFmpeg development practices and the monolithic repo (except FFmpeg devs).
I believe it would be rather easy to have the same kind of interface for lavfi; all the filters are already introspectable, and that is the most major point. Unfortunately, two issues would make it of little use. First, lavfi handles only frames, not packets, so all the encoding and muxing would have to be handled separately.
That has nothing to do with VapourSynth, though.
Second, lavfi is designed to make a lot of format decisions when the graph is complete, so the scripting capabilities would not have a lot of relevant information to work on. I wonder how VapourSynth handles that, but I am afraid to learn that it just do automatic conversions on the spot without globally optimizing them.
Yes, lavfi refuses to work unless you "configure" the graph, which makes the whole thing very inflexible. As far as I know, VapourSynth has no automatic conversions. If it really matters, the script can manually insert conversion filters where it matters. Lavfi's format negotiation is a monster: how often did even FFmpeg devs wonder why the hell lavfi is converting to a specific format at a certain point? After all, the complexity tradeoff might not be worth the return of the "global optimization" (which often makes bad decisions too). VapourSynth's approach in this case is making all filters support a small set of sane formats. So you don't have to care about conversions, and in fact you don't need to convert at all, unless input or output uses fringe pixel formats.
Le duodi 2 brumaire, an CCXXIII, Clement Boesch a écrit :
More still standing problems while we are at it:
1.7. Metadata
Metadata are not available at "graph" level, or at least filter level, only at frame level. We also need to define how they can be injected and fetched from the users (think "rotate" metadata).
That is an interesting issue. At graph level, that is easy, but that would be mostly useless (the rotate filter changes the rotate metadata). At filter level, that is harder because it requires all filters to forward the metadata at init time, so extra code in a lot of filters. Furthermore, since our graphs are not constructed in order, and can even theoretically contain cycles, it requires another walk-over to ensure stabilization. The whole query_formats() / config_props() is already too complex IMHO. Actually, I believe I can propose a simple solution: inject the stream metadata as frame metadata on dummy frames. Filters that need them are changed to examine the dummy frames, filters that do not need them just ignore them and let the framework forward it. (Of course, the whole metadata system can never work perfectly: the scale filter does not update any "dpi" metadata; the crop filter would need to update the "aperture" metadata for photos, and if the crop is not centered I am not even sure this makes sense, etc. If someone adds "xmin", "xmax", "xscl" (no, not this one, bad work habit), "ymin", "ymax" to the frames produced by vsrc_mandelbrot, or the geographic equivalent to satellite images, how is the rotate filter supposed to handle that? The best answer would probably be "we do not care much".)
1.8. Seeking
Way more troublesome: being able to request an exact frame in the past. This currently limits a lot the scope of the filters.
thumbnail filter is a good example of this problem: the filter doesn't need to keep all the frames it analyzes in memory, it just needs statistics about them, and then fetches the best in the batch. Currently, it needs to keep them all because we are in a forward stream based logic. This model is kind of common and quite a pain to implement currently.
I don't think the compression you propose at the end would really solve that.
You raise an interesting point. Unlimited FIFOs (with or without external storage or compression: they are just means of handling larger FIFOs with smaller hardware) can be of some help in that case, but not much. In the particular example you indicate, I can imagine a solution with two filters: thumbnail-detect outputs just pseudo-frame metadata with the timestamp of the selected thumbnails images, and thumbnail-select use that metadata from one input, reading the actual frames from its second input connected to a large FIFO. But that is outright ugly. For actual seeking, I suppose we would need a mechanism to send messages backward on the graph. As for the actual implementation, I suppose that a filter that supports seeking would be required to advertise so on its output: "I can seek back to pts=42", and a filter that requires seeking from its input would give forewarning: "I may need to seek back to pts=12", so that the framework can buffer all frames from 12 to 42. That requires thinking.
1.9. Automatic I/O count
"... [a] split [b][c] ..." should guess there is 2 outputs. "... [a][b][c] concat [d] ..." as well
I believe this one to be pretty easy, design-wise, in fact: just decide on a standard name for the options that give the number of input and outputs, maybe just nb_inputs and nb_outputs, and then it is only a matter of tweaking the graph parser to set them if possible and necessary.
Did you already started some development? Do you need help?
I'm asking because it looks like it could be split into small relatively easy tasks on the Trac and helps introducing new comers (and also track the progress if some people assign themselves to these tickets).
I have not started writing code: for large re-design, I would not risk someone telling me "this is stupid, you can do the same thing ten times simpler like that". You are right, some of the points I raise are mostly stand-alone tasks.
AVFilterLink.pts: current timestamp of the link, i.e. end timestamp of the last forwarede frame, assuming the duration was correct. This is somewhat redundant with the fields in AVFrame, but can carry the information even when there is no actual frame.
The timeline system seems to be able to workaround this. How is this going to help?
I do not see how this is related. When the timeline system is invoked, there is a frame, with a timestamp. The timestamp may be NOPTS, but that is just a matter for the enable expression to handle correctly. The issue I am trying to address is the one raised in this example: suppose overlay detects EOF on its secondary input; the last secondary frames were at PTS 40, 41, 42, and now here comes a main frame at PTS 42.04: should overlay slap the last secondary frame on it or not? In this particular case, it is pretty obvious that EOF happens at PTS 43, but teaching a program to see the obvious is not easy, and that may actually be wrong. The previous filters, or the application (through the demuxer) may have more accurate information. What I propose for that issue is to have something like AVFilterLink.head_pts that records the PTS of the last activity on a link. When a frame is passed on the link, it is updated to frame.pts + frame.duration, but it may be updated by other circumstances too. The core idea is to have as much as possible information directly available to filters without requiring them to work for it. A filter could always update head_pts in its own private context, but then, if a new way of updating it is added, all filters may need to be updated.
Well, I believe it should be handled by the framework transparently somehow.
Yes, exactly.
Users can already fix the timestamps themselves with [a]setpts filters, but it's often not exactly obvious why they do need to.
Hum, I do not think that setpts is suitable in this case: you can not use it to set frame.duration = next_frame.pts - frame.pts because next_frame is not available. Even for cases it can handle, it requires complex formulas (with escaping; I still have to look at Michael's patch for balanced escaping); we do not want users to copy-paste half-broken expressions found in obsolete examples on the web. Plus it uses floats. A dedicated filter seems more correct: fixpts=delta2duration=1 for example.
It doesn't need to be a filter and can be part of the framework itself.
I believe the framework should do the work, but also expose it as an internal API to be used by the fixpts filter when explicit handling and user-settable options are necessary. Regards, -- Nicolas George
On Fri, Oct 24, 2014 at 04:22:23PM +0200, Nicolas George wrote:
Le duodi 2 brumaire, an CCXXIII, Clement Boesch a écrit : [...]
As for the actual implementation, I suppose that a filter that supports seeking would be required to advertise so on its output: "I can seek back to pts=42", and a filter that requires seeking from its input would give forewarning: "I may need to seek back to pts=12", so that the framework can buffer all frames from 12 to 42.
That requires thinking.
its easier for the input to seek to pts 2 in that case and discard data till pts 12. I assume pts42 is not the first point it can seek to files can generally be reopened to restart from the first frame [...] -- Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB You can kill me, but you cannot change the truth.
On Wed, 22 Oct 2014 23:45:42 +0200 Nicolas George <george@nsup.org> wrote:
[ CCing Anton, as most that is written here also apply to libav too, and this would be a good occasion to try a cross-fork cooperation; if that is not wanted, please let us know so we can drop the cc. ]
1. Problems with the current design
1.1. Mixed input-/output-driven model
Currently, lavfi is designed to work in a mixed input-driven and output-driven model. That means the application needs sometimes to add input to buffersources and sometimes request output to buffersinks. This is a bit of a nuisance, because it requires the application to do it properly: adding input on the wrong input or requesting a frame on the wrong output will cause extra memory consumption or latency.
With the libav API, it can not work at all since there is no mechanism to determine which input needs a frame in order to proceed.
The libav API is clearly designed for a more output-driven implementation, with FIFOs anywhere to prevent input-driven frames to reach unready filters. Unfortunately, since it is impossible from the outside to guess what output will get a frame next, that can cause frames to accumulate anywhere in the filter graph, eating a lot of memory unnecessarily.
FFmpeg's API has eliminated FIFOs in favour of queues in filters that need it, but these queues can not be controlled for unusual filter graphs with extreme needs. Also, there still is an implicit FIFO inside buffersink.
1.2. Recursive implementation
All work in a filter graph is triggered by recursive invocations of the filters' methods. It makes debugging harder. It also can lead to large stack usage and makes frame- and filter-level multithreading harder to implement. It also prevents some diagnosis from working reliably.
1.3. EOF handling
Currently, EOF is propagated only through the return value of the request_frame() method. That means it only works in an output-driven scheme. It also means that it has no timestamp attached to it; this is an issue for filters where the duration of the last frame is relevant, like vf_fps.
1.4. Latency
Some filters need to know the timestamp of the next frame in order to know when the current frame will stop and be able to process it: overlay, fps are two examples. These filters will introduce a latency of one input frame that could otherwise be avoided.
1.5. Timestamps
Some filters do not care about timestamps at all. Some check and have a proper handling of NOPTS values. Some filters just assume the frames will have timestamps, and possibly make extra assumptions on that: monotony, consistency, etc. That is an inconsistent mess.
1.6. Sparse streams
There is a more severe instance of the latency issue when the input comes from an interleaved sparse stream: in that case, waiting for the next frame in order to find the end of the current one may require demuxing a large chunk of input, in turn provoking a lot of activity on other inputs of the graph.
2. Proposed API changes
To fix/enhance all these issues, I believe a complete rethink of the scheduling design of the library is necessary. I propose the following changes.
Note: some of these changes are not 100% related to the issues I raised, but looked like a good idea while thinking on an API rework.
2.1. AVFrame.duration
Add a duration field to AVFrame; if set, it indicates the duration of the frame. Thus, it becomes unnecessary to wait for the next frame to know when the current frame stops, reducing the latency.
Another solution would be to add a dedicated function on buffersrc to inject a timestamp for end or activity on a link. That would avoid the need of adding a field to AVFrame.
2.2. Add some fields to AVFilterLink
AVFilterLink.pts: current timestamp of the link, i.e. end timestamp of the last forwarede frame, assuming the duration was correct. This is somewhat redundant with the fields in AVFrame, but can carry the information even when there is no actual frame.
AVFilterLink.status: if not 0, gives the return status of trying to pass a frame on this link. The typical use would be EOF.
2.3. AVFilterLink.need_ts
Add a field to AVFilterLink to specify that the output filter requires reliable timestamps on its input. More precisely, specify how reliable the timestamps need to be: is the duration necessary? do the timestamps need to be monotonic? continuous?
For audio streams, consistency between timestamps and the number of samples may also be tested. For video streams, constant frame rate may be enforced, but I am not sure about this one.
A "fixpts" filter should be provided to allow the user to tweak how the timestamps are fixed (change the timestamps to match the duration or change the duration to match the timestamps?).
When no explicit filter is inserted, the framework should do the work of fixing them automatically. I am not sure whether that should be done directly or by automatically inserting the fixpts filter. The later solution is more elegant, but it requires more changes to the framework and the filters (because the correctness of the timestamps would need to be merged just like formats), so I am rather for the former.
Note that for a lot of filters, the actual duration or end timestamp is not required, only a lower bound for it. For sparse interleaved streams, that is very relevant as we may not know the exact time for the next frame until we reach it, but we can know it is later than the other streams' timestamps minus the interleaving delta.
2.4. Build FIFOs directly in AVFilterLink
Instead of automatically insert an additional filter like libav, handle the FIFO operation directly in the framework using fields in AVFilterLink.
The main benefit is that the framework can examine the inside of the FIFOs to make scheduling decisions. It can also do so to provide the user with more accurate diagnostics.
An extra benefit: the memory pool for the FIFOed frames can more easily be shared, across the whole filter graph or the whole application. Memory management becomes easier: just take a good heuristics (half the RAM?), no need to guess what FIFOs will actually need a lot of memory and what FIFOs are just there mostly useless.
Last but not least, FIFOs now become potential thread communication / synchronization points, making filter-level multithreading easier.
For audio streams, framing (i.e. ensuring all frame have an exact / minimum / maximum number of samples) can be merged with FIFOs.
2.5. Allow status change pseudo-frames inside FIFOs
To propagate EOF and possibly other status changes (errors) in input-driven model, allow FIFOs to contain not only frames but also kind of pseudo-frames with a timestamp and metadata attached.
Depending on the filters, these pseudo-frames may be directly passed to the filter's methods, or they may be interpreted by the framework to just change fields on the AVFilterLink structure.
2.6. Change the scheduling logic for filters
From the outside of a filter with several outputs, it is usually not possible to guess what output will get a frame next. Requesting a frame on output #0 may cause activity on the filter graph that produce a frame on output #1 instead, or possibly on a completely different filter.
Therefore, having a request_frame() method on all outputs seems pointless.
Instead, use a global AVFilter.activate() method that causes the filter to do one step of work if it can. This method is called each time something is changed to the filter: new frame on input, output ready, status change. It returns as soon at it could do something, either producing output and/or consuming input, or nothing if nothing can be done.
2.7. Add fields to AVFilterLink for flow control.
Add to AVFilterLinks a few field to help filters decide if they need to process something, and if relevant in what order. The most obvious idea would be AVFilterLink.frames_needed, counting how many frames are probably needed on a link before anything can be done. For example, with concat, after input has been consumed, the frames_needed fields on the current input are set according to the corresponding output.
2.8. Activate the filters iteratively
Keep a global (per graph) priority queue of filters that are supposed to be ready and call the activate() method on them.
2.9. AVFrame.stream_id
Add an integer (or pointer: intptr_t maybe?) field to AVFrame to allow passing frames related to distinct streams on the same link. That would allow to multiplex all outputs of a graph into a single output, making the application simpler.
Not sure this is really useful or necessary: for the graph outputs, a convenience function iterating on all of them and returning the frame and the output index separately would do the trick too.
2.10. buffersrc.callback and buffersink.callback
Add a callback on both buffersource and buffersink, called respectively when a frame is necessary on input and a frame has arrived on output. This allows pure input-driven and pure output-driven design to work.
2.11. Links groups
Links that carry frames from related interleaved streams should be explicitly connected together so that the framework can use the information.
The typical use would be to group all the links from buffersrc that come from the same interleaved input file.
When a frame is passed on a link, all links in the same group(s) that are too late (according to an interleaving tolerance that can be set) are activated using a dummy frame.
2.12. FIFOs with compression and external storage
All FIFOs should be able to off-load some of their memory requirements by either compressing the frames (using a lossless or optionally lossy codec) and/or storing them on mass storage.
The options for that should be changeable globally or on a per-link basis.
2.13. AVFrame.owner
Add a owner field (probably with type "AVObject", i.e. "void *" pointing to AVClass *) to AVFrame, and update it whenever the frame is passed from one filter to the other. That way, inconsistent ref/unref operations can be detected.
Regards,
I'm fascinated how easy to use vapoursynth is (a modern and portable replacement for avisynth). You don't need to build graphs, but it allows arbitrary complexity. You can do your "own" stuff in-between, instead of being coerced into the closely coupled and claustrophobic concepts of lavfi. vapoursynth has no weird own filter description language, it just exports a list of options each filter supports and their types. (It's normally used through Python, but that is completely optional.) Maybe steal some ideas from them?
Sorry for the slow reply. On date Wednesday 2014-10-22 23:45:42 +0200, Nicolas George encoded:
[ CCing Anton, as most that is written here also apply to libav too, and this would be a good occasion to try a cross-fork cooperation; if that is not wanted, please let us know so we can drop the cc. ]
1. Problems with the current design
1.1. Mixed input-/output-driven model
Currently, lavfi is designed to work in a mixed input-driven and output-driven model. That means the application needs sometimes to add input to buffersources and sometimes request output to buffersinks. This is a bit of a nuisance, because it requires the application to do it properly: adding input on the wrong input or requesting a frame on the wrong output will cause extra memory consumption or latency.
With the libav API, it can not work at all since there is no mechanism to determine which input needs a frame in order to proceed.
The libav API is clearly designed for a more output-driven implementation, with FIFOs anywhere to prevent input-driven frames to reach unready filters. Unfortunately, since it is impossible from the outside to guess what output will get a frame next, that can cause frames to accumulate anywhere in the filter graph, eating a lot of memory unnecessarily.
FFmpeg's API has eliminated FIFOs in favour of queues in filters that need it, but these queues can not be controlled for unusual filter graphs with extreme needs. Also, there still is an implicit FIFO inside buffersink.
1.2. Recursive implementation
All work in a filter graph is triggered by recursive invocations of the filters' methods. It makes debugging harder. It also can lead to large stack usage and makes frame- and filter-level multithreading harder to implement. It also prevents some diagnosis from working reliably.
1.3. EOF handling
Currently, EOF is propagated only through the return value of the request_frame() method. That means it only works in an output-driven scheme. It also means that it has no timestamp attached to it; this is an issue for filters where the duration of the last frame is relevant, like vf_fps.
1.4. Latency
Some filters need to know the timestamp of the next frame in order to know when the current frame will stop and be able to process it: overlay, fps are two examples. These filters will introduce a latency of one input frame that could otherwise be avoided.
1.5. Timestamps
Some filters do not care about timestamps at all. Some check and have a proper handling of NOPTS values. Some filters just assume the frames will have timestamps, and possibly make extra assumptions on that: monotony, consistency, etc. That is an inconsistent mess.
1.6. Sparse streams
There is a more severe instance of the latency issue when the input comes from an interleaved sparse stream: in that case, waiting for the next frame in order to find the end of the current one may require demuxing a large chunk of input, in turn provoking a lot of activity on other inputs of the graph.
Other issues. S1. the filtergraph can't properly readapt to mid-stream changes involving assumed invariants (aspect ratio, size, timebase, pixel format, sample_rate). Indeed the framework was designed as though some of these properties (the ones defined by query_formats) were not allowed to change. S2. Another problem is that we initialize the filter before the filtergraph, so for example the single filter can't readapt to the filtergraph topology. For example it would be useful to have the split filter to change the number of outputs depending on the number of outputs specified, but this can't be easily achieved. (That's in my opinion a minor problem though). S3. It is not possible to direct commands towards a specific filter. For this we can add an ID to each filter instance. We could have something has: color:left_color=c=red [left] color:right_color=c=blue [right] then you can send commands (e.g. with zmq) with: echo left_color c yellow | tools/zmqsend S4. We should support output encoding movie. We got stuck designing the interface for that. ... About fifos and queues, we could add some options to control fifo filters to limit their size. For example we could specify the maximum number of allowed queued frames, or the total allowed size, and the dropping policy (drop last, drop first, drop random frame in the midst).
2. Proposed API changes
To fix/enhance all these issues, I believe a complete rethink of the scheduling design of the library is necessary. I propose the following changes.
Note: some of these changes are not 100% related to the issues I raised, but looked like a good idea while thinking on an API rework.
2.1. AVFrame.duration
Add a duration field to AVFrame; if set, it indicates the duration of the frame. Thus, it becomes unnecessary to wait for the next frame to know when the current frame stops, reducing the latency.
Another solution would be to add a dedicated function on buffersrc to inject a timestamp for end or activity on a link. That would avoid the need of adding a field to AVFrame.
Currently we have pkt_duration in AVFrame. The main problem is that we would need to rescale the duration accordingly in all filters when we change the timebase. Alternatively we should add a timebase value to AVFrame. Currently that information is stored externally, and this was never easy to handle. For example the timestamp in a filter is currently interpreted according to the input link time base. Unfortunately this will introduce redundancy.
2.2. Add some fields to AVFilterLink
AVFilterLink.pts: current timestamp of the link, i.e. end timestamp of the last forwarede frame, assuming the duration was correct. This is somewhat redundant with the fields in AVFrame, but can carry the information even when there is no actual frame.
AVFilterLink.status: if not 0, gives the return status of trying to pass a frame on this link. The typical use would be EOF.
2.3. AVFilterLink.need_ts
Add a field to AVFilterLink to specify that the output filter requires reliable timestamps on its input. More precisely, specify how reliable the timestamps need to be: is the duration necessary? do the timestamps need to be monotonic? continuous?
For audio streams, consistency between timestamps and the number of samples may also be tested. For video streams, constant frame rate may be enforced, but I am not sure about this one.
Yes, although these checks should be made optional.
A "fixpts" filter should be provided to allow the user to tweak how the timestamps are fixed (change the timestamps to match the duration or change the duration to match the timestamps?).
Yes, this could be properly done with a complex expression in setpts, but having an ad-hoc filter should ease user.
When no explicit filter is inserted, the framework should do the work of fixing them automatically. I am not sure whether that should be done directly or by automatically inserting the fixpts filter. The later solution is more elegant, but it requires more changes to the framework and the filters (because the correctness of the timestamps would need to be merged just like formats), so I am rather for the former.
Note that for a lot of filters, the actual duration or end timestamp is not required, only a lower bound for it. For sparse interleaved streams, that is very relevant as we may not know the exact time for the next frame until we reach it, but we can know it is later than the other streams' timestamps minus the interleaving delta.
2.4. Build FIFOs directly in AVFilterLink
Instead of automatically insert an additional filter like libav, handle the FIFO operation directly in the framework using fields in AVFilterLink.
The main benefit is that the framework can examine the inside of the FIFOs to make scheduling decisions. It can also do so to provide the user with more accurate diagnostics.
An extra benefit: the memory pool for the FIFOed frames can more easily be shared, across the whole filter graph or the whole application. Memory management becomes easier: just take a good heuristics (half the RAM?), no need to guess what FIFOs will actually need a lot of memory and what FIFOs are just there mostly useless.
Last but not least, FIFOs now become potential thread communication / synchronization points, making filter-level multithreading easier.
For audio streams, framing (i.e. ensuring all frame have an exact / minimum / maximum number of samples) can be merged with FIFOs.
2.5. Allow status change pseudo-frames inside FIFOs
To propagate EOF and possibly other status changes (errors) in input-driven model, allow FIFOs to contain not only frames but also kind of pseudo-frames with a timestamp and metadata attached.
Depending on the filters, these pseudo-frames may be directly passed to the filter's methods, or they may be interpreted by the framework to just change fields on the AVFilterLink structure.
2.6. Change the scheduling logic for filters
From the outside of a filter with several outputs, it is usually not possible to guess what output will get a frame next. Requesting a frame on output #0 may cause activity on the filter graph that produce a frame on output #1 instead, or possibly on a completely different filter.
Therefore, having a request_frame() method on all outputs seems pointless.
Instead, use a global AVFilter.activate() method that causes the filter to do one step of work if it can. This method is called each time something is changed to the filter: new frame on input, output ready, status change. It returns as soon at it could do something, either producing output and/or consuming input, or nothing if nothing can be done.
2.7. Add fields to AVFilterLink for flow control.
Add to AVFilterLinks a few field to help filters decide if they need to process something, and if relevant in what order. The most obvious idea would be AVFilterLink.frames_needed, counting how many frames are probably needed on a link before anything can be done. For example, with concat, after input has been consumed, the frames_needed fields on the current input are set according to the corresponding output.
2.8. Activate the filters iteratively
Keep a global (per graph) priority queue of filters that are supposed to be ready and call the activate() method on them.
2.9. AVFrame.stream_id
Add an integer (or pointer: intptr_t maybe?) field to AVFrame to allow passing frames related to distinct streams on the same link. That would allow to multiplex all outputs of a graph into a single output, making the application simpler.
Not sure this is really useful or necessary: for the graph outputs, a convenience function iterating on all of them and returning the frame and the output index separately would do the trick too.
2.10. buffersrc.callback and buffersink.callback
Add a callback on both buffersource and buffersink, called respectively when a frame is necessary on input and a frame has arrived on output. This allows pure input-driven and pure output-driven design to work.
2.11. Links groups
Links that carry frames from related interleaved streams should be explicitly connected together so that the framework can use the information.
The typical use would be to group all the links from buffersrc that come from the same interleaved input file.
When a frame is passed on a link, all links in the same group(s) that are too late (according to an interleaving tolerance that can be set) are activated using a dummy frame.
2.12. FIFOs with compression and external storage
All FIFOs should be able to off-load some of their memory requirements by either compressing the frames (using a lossless or optionally lossy codec) and/or storing them on mass storage.
The options for that should be changeable globally or on a per-link basis.
This is an interesting idea.
2.13. AVFrame.owner
Add a owner field (probably with type "AVObject", i.e. "void *" pointing to AVClass *) to AVFrame, and update it whenever the frame is passed from one filter to the other. That way, inconsistent ref/unref operations can be detected.
BTW, probably we should drop the rotting tracing code (I'm not using it since ages, don't know about others). -- FFmpeg = Free and Formidable Multipurpose Pitiless Enlightened Genius
On Thu, Oct 30, 2014 at 11:50:46AM +0100, Stefano Sabatini wrote:
Sorry for the slow reply.
On date Wednesday 2014-10-22 23:45:42 +0200, Nicolas George encoded:
[ CCing Anton, as most that is written here also apply to libav too, and this would be a good occasion to try a cross-fork cooperation; if that is not wanted, please let us know so we can drop the cc. ]
1. Problems with the current design
1.1. Mixed input-/output-driven model
Currently, lavfi is designed to work in a mixed input-driven and output-driven model. That means the application needs sometimes to add input to buffersources and sometimes request output to buffersinks. This is a bit of a nuisance, because it requires the application to do it properly: adding input on the wrong input or requesting a frame on the wrong output will cause extra memory consumption or latency.
With the libav API, it can not work at all since there is no mechanism to determine which input needs a frame in order to proceed.
The libav API is clearly designed for a more output-driven implementation, with FIFOs anywhere to prevent input-driven frames to reach unready filters. Unfortunately, since it is impossible from the outside to guess what output will get a frame next, that can cause frames to accumulate anywhere in the filter graph, eating a lot of memory unnecessarily.
FFmpeg's API has eliminated FIFOs in favour of queues in filters that need it, but these queues can not be controlled for unusual filter graphs with extreme needs. Also, there still is an implicit FIFO inside buffersink.
1.2. Recursive implementation
All work in a filter graph is triggered by recursive invocations of the filters' methods. It makes debugging harder. It also can lead to large stack usage and makes frame- and filter-level multithreading harder to implement. It also prevents some diagnosis from working reliably.
1.3. EOF handling
Currently, EOF is propagated only through the return value of the request_frame() method. That means it only works in an output-driven scheme. It also means that it has no timestamp attached to it; this is an issue for filters where the duration of the last frame is relevant, like vf_fps.
1.4. Latency
Some filters need to know the timestamp of the next frame in order to know when the current frame will stop and be able to process it: overlay, fps are two examples. These filters will introduce a latency of one input frame that could otherwise be avoided.
1.5. Timestamps
Some filters do not care about timestamps at all. Some check and have a proper handling of NOPTS values. Some filters just assume the frames will have timestamps, and possibly make extra assumptions on that: monotony, consistency, etc. That is an inconsistent mess.
1.6. Sparse streams
There is a more severe instance of the latency issue when the input comes from an interleaved sparse stream: in that case, waiting for the next frame in order to find the end of the current one may require demuxing a large chunk of input, in turn provoking a lot of activity on other inputs of the graph.
Other issues.
S1. the filtergraph can't properly readapt to mid-stream changes involving assumed invariants (aspect ratio, size, timebase, pixel format, sample_rate). Indeed the framework was designed as though some of these properties (the ones defined by query_formats) were not allowed to change.
no, no and no :) the filtergraph should be able adapt to some changes like aspect, resolution and pixel / sample format. These are not invariants, some of this definitly worked when i tried it long ago i posted a (incomplete) patchset that fixes bugs in this relation long ago There are filters that can perfectly well handle changes in resolution, aspect, formats, ... and there are filters which are buggy but could when they are fixed also equally well support such changes and there are filters which fundamentally do not support some changes, these need to either be reinited and loose state/history or a scale/aresample be inserted before them when changes on their input occur for some filters reinit is not appropriate, examples are things that are intended to collect global statistics Also scale / aresample filters can serve as "parameter change barriers", filters afterwards do not need to deal with such changes
S2. Another problem is that we initialize the filter before the filtergraph, so for example the single filter can't readapt to the filtergraph topology. For example it would be useful to have the split filter to change the number of outputs depending on the number of outputs specified, but this can't be easily achieved. (That's in my opinion a minor problem though).
S3. It is not possible to direct commands towards a specific filter. For this we can add an ID to each filter instance. We could have something has: color:left_color=c=red [left] color:right_color=c=blue [right]
then you can send commands (e.g. with zmq) with: echo left_color c yellow | tools/zmqsend
c sends to all filters C sends to a specific filter What problem is there with C ? [...]
2. Proposed API changes
To fix/enhance all these issues, I believe a complete rethink of the scheduling design of the library is necessary. I propose the following changes.
Note: some of these changes are not 100% related to the issues I raised, but looked like a good idea while thinking on an API rework.
2.1. AVFrame.duration
Add a duration field to AVFrame; if set, it indicates the duration of the frame. Thus, it becomes unnecessary to wait for the next frame to know when the current frame stops, reducing the latency.
Another solution would be to add a dedicated function on buffersrc to inject a timestamp for end or activity on a link. That would avoid the need of adding a field to AVFrame.
Currently we have pkt_duration in AVFrame. The main problem is that we would need to rescale the duration accordingly in all filters when we change the timebase.
Alternatively we should add a timebase value to AVFrame. Currently that information is stored externally, and this was never easy to handle. For example the timestamp in a filter is currently interpreted according to the input link time base. Unfortunately this will introduce redundancy.
yes, also theres a somewhat related patch: Benoit Fouet (4.5K) [FFmpeg-devel] [PATCH] libavu: add pkt_timebase to AVFrame. also speaking of patches, comments for [FFmpeg-devel] [PATCH] avutil/avstring: support asymmetric escaping in av_get_token() are welcome i cant just apply it as its not 100% compatible with past formating though its 99.9% and mayb 100% for actual used strings so this patch really needs comments from people ... [...] -- Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB While the State exists there can be no freedom; when there is freedom there will be no State. -- Vladimir Lenin
Hi, On Thu, Oct 30, 2014 at 1:59 PM, Michael Niedermayer <michaelni@gmx.at> wrote:
Sorry for the slow reply.
On date Wednesday 2014-10-22 23:45:42 +0200, Nicolas George encoded:
[ CCing Anton, as most that is written here also apply to libav too, and this would be a good occasion to try a cross-fork cooperation; if that is not wanted, please let us know so we can drop the cc. ]
1. Problems with the current design
1.1. Mixed input-/output-driven model
Currently, lavfi is designed to work in a mixed input-driven and output-driven model. That means the application needs sometimes to add input to buffersources and sometimes request output to buffersinks. This is a bit of a nuisance, because it requires the application to do it properly: adding input on the wrong input or requesting a frame on
wrong output will cause extra memory consumption or latency.
With the libav API, it can not work at all since there is no
mechanism
to determine which input needs a frame in order to proceed.
The libav API is clearly designed for a more output-driven implementation, with FIFOs anywhere to prevent input-driven frames
to
reach unready filters. Unfortunately, since it is impossible from
outside to guess what output will get a frame next, that can cause frames to accumulate anywhere in the filter graph, eating a lot of memory unnecessarily.
FFmpeg's API has eliminated FIFOs in favour of queues in filters
need it, but these queues can not be controlled for unusual filter graphs with extreme needs. Also, there still is an implicit FIFO
inside
buffersink.
1.2. Recursive implementation
All work in a filter graph is triggered by recursive invocations
of the
filters' methods. It makes debugging harder. It also can lead to
large
stack usage and makes frame- and filter-level multithreading
harder to
implement. It also prevents some diagnosis from working reliably.
1.3. EOF handling
Currently, EOF is propagated only through the return value of the request_frame() method. That means it only works in an
output-driven
scheme. It also means that it has no timestamp attached to it;
an issue for filters where the duration of the last frame is
relevant,
like vf_fps.
1.4. Latency
Some filters need to know the timestamp of the next frame in order
to
know when the current frame will stop and be able to process it: overlay, fps are two examples. These filters will introduce a
latency of
one input frame that could otherwise be avoided.
1.5. Timestamps
Some filters do not care about timestamps at all. Some check and
have a
proper handling of NOPTS values. Some filters just assume the
will have timestamps, and possibly make extra assumptions on that: monotony, consistency, etc. That is an inconsistent mess.
1.6. Sparse streams
There is a more severe instance of the latency issue when the input comes from an interleaved sparse stream: in that case, waiting for
On Thu, Oct 30, 2014 at 11:50:46AM +0100, Stefano Sabatini wrote: the the that this is frames the
next frame in order to find the end of the current one may require demuxing a large chunk of input, in turn provoking a lot of
activity on
other inputs of the graph.
Other issues.
S1. the filtergraph can't properly readapt to mid-stream changes involving assumed invariants (aspect ratio, size, timebase, pixel format, sample_rate). Indeed the framework was designed as though some of these properties (the ones defined by query_formats) were not allowed to change.
no, no and no :) the filtergraph should be able adapt to some changes like aspect, resolution and pixel / sample format. These are not invariants, some of this definitly worked when i tried it long ago i posted a (incomplete) patchset that fixes bugs in this relation long ago There are filters that can perfectly well handle changes in resolution, aspect, formats, ... and there are filters which are buggy but could when they are fixed also equally well support such changes and there are filters which fundamentally do not support some changes, these need to either be reinited and loose state/history or a scale/aresample be inserted before them when changes on their input occur for some filters reinit is not appropriate, examples are things that are intended to collect global statistics Also scale / aresample filters can serve as "parameter change barriers", filters afterwards do not need to deal with such changes
S2. Another problem is that we initialize the filter before the filtergraph, so for example the single filter can't readapt to the filtergraph topology. For example it would be useful to have the split filter to change the number of outputs depending on the number of outputs specified, but this can't be easily achieved. (That's in my opinion a minor problem though).
S3. It is not possible to direct commands towards a specific filter. For this we can add an ID to each filter instance. We could have something has: color:left_color=c=red [left] color:right_color=c=blue [right]
then you can send commands (e.g. with zmq) with: echo left_color c yellow | tools/zmqsend
c sends to all filters C sends to a specific filter What problem is there with C ?
2. Proposed API changes
To fix/enhance all these issues, I believe a complete rethink of the scheduling design of the library is necessary. I propose the following changes.
Note: some of these changes are not 100% related to the issues I raised, but looked like a good idea while thinking on an API rework.
2.1. AVFrame.duration
Add a duration field to AVFrame; if set, it indicates the duration of the frame. Thus, it becomes unnecessary to wait for the next frame to know when the current frame stops, reducing the latency.
Another solution would be to add a dedicated function on buffersrc to inject a timestamp for end or activity on a link. That would avoid
[...] the
need of adding a field to AVFrame.
Currently we have pkt_duration in AVFrame. The main problem is that we would need to rescale the duration accordingly in all filters when we change the timebase.
Alternatively we should add a timebase value to AVFrame. Currently that information is stored externally, and this was never easy to handle. For example the timestamp in a filter is currently interpreted according to the input link time base. Unfortunately this will introduce redundancy.
yes, also theres a somewhat related patch: Benoit Fouet (4.5K) [FFmpeg-devel] [PATCH] libavu: add pkt_timebase to AVFrame.
also speaking of patches, comments for [FFmpeg-devel] [PATCH] avutil/avstring: support asymmetric escaping in av_get_token() are welcome i cant just apply it as its not 100% compatible with past formating though its 99.9% and mayb 100% for actual used strings so this patch really needs comments from people ...
[...] -- Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB
While the State exists there can be no freedom; when there is freedom there will be no State. -- Vladimir Lenin
_______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
Nicolas, what is status of this? I'm currently interested in frame multithreading in lavfi.
Le duodi 12 fructidor, an CCXXIV, Paul B Mahol a écrit :
Nicolas, what is status of this?
I'm currently interested in frame multithreading in lavfi.
I am currently locked on a patch series to replace the recursive calls to filter_frames() with a FIFO on each link. I think this is an absolute necessity before considering inter-filter multithreading. Unfortunately, this is very tricky business because the filters' implementation are not ready for it. Filters must be activated, by actually calling their filter_frame() methods, when a frame is available, but not repeatedly so when they can not perform any work, but they often provide no visible test for that. Plus, the global API for activating a filtergraph was not designed with that kind of working in mind. I have made significant progress in the last weeks. I think I have got the propagating of frames working, and the propagating of EOF conditions on 1-to-1 filters too, but there is still an issue with filters with multiple inputs. I could post the whole patch as is at any time, but I do not think it would do much good as is. Regards, -- Nicolas George
On Tuesday, August 30, 2016, Nicolas George <george@nsup.org> wrote:
Le duodi 12 fructidor, an CCXXIV, Paul B Mahol a écrit :
Nicolas, what is status of this?
I'm currently interested in frame multithreading in lavfi.
I am currently locked on a patch series to replace the recursive calls to filter_frames() with a FIFO on each link.
I think this is an absolute necessity before considering inter-filter multithreading.
Unfortunately, this is very tricky business because the filters' implementation are not ready for it. Filters must be activated, by actually calling their filter_frame() methods, when a frame is available, but not repeatedly so when they can not perform any work, but they often provide no visible test for that. Plus, the global API for activating a filtergraph was not designed with that kind of working in mind.
I have made significant progress in the last weeks. I think I have got the propagating of frames working, and the propagating of EOF conditions on 1-to-1 filters too, but there is still an issue with filters with multiple inputs.
I could post the whole patch as is at any time, but I do not think it would do much good as is.
the filter frame multithreading would just internally, in filter context cache frames, once enough frames are in cache - call workers and be done, repeat. At eof call workers on remaining frames in cache.
Le quartidi 14 fructidor, an CCXXIV, Paul B Mahol a écrit :
the filter frame multithreading would just internally, in filter context cache frames, once enough frames are in cache - call workers and be done, repeat. At eof call workers on remaining frames in cache.
I have no idea how much thought you have already given to it, but I am pretty sure it is not as simple as that with the current architecture. By far. In the meantime, I finally got the non-recursive version passing FATE. Here are the raw patch, so that people can have an idea what this is all about. There is still a lot of cleanup and documentation to do, as you can see. Regards, -- Nicolas George
On Tue, Aug 30, 2016 at 09:08:18PM +0200, Nicolas George wrote:
Le quartidi 14 fructidor, an CCXXIV, Paul B Mahol a écrit :
the filter frame multithreading would just internally, in filter context cache frames, once enough frames are in cache - call workers and be done, repeat. At eof call workers on remaining frames in cache.
I have no idea how much thought you have already given to it, but I am pretty sure it is not as simple as that with the current architecture. By far.
In the meantime, I finally got the non-recursive version passing FATE. Here are the raw patch, so that people can have an idea what this is all about. There is still a lot of cleanup and documentation to do, as you can see.
Regards,
-- Nicolas George
./ffmpeg -i tickets//679/oversized_pgs_subtitles.mkv -filter_complex '[0:s:1]scale=848x480,[0:v]overlay=shortest=1' test.avi fails assertion: Assertion progress failed at libavfilter/avfilter.c:1391 https://trac.ffmpeg.org/attachment/ticket/679/oversized_pgs_subtitles.mkv ffmpeg -v 0 -i tickets/3539/crash.swf -map 0 -t 1 -f framecrc - output changes, not sure this is a bug but reporting it anyway as i noticed http://samples.ffmpeg.org/ffmpeg-bugs/trac/ticket3539/ doc/examples/filtering_video matrixbench_mpeg2.mpg also breaks [...] -- Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB Why not whip the teacher when the pupil misbehaves? -- Diogenes of Sinope
Le quintidi 15 fructidor, an CCXXIV, Michael Niedermayer a écrit :
./ffmpeg -i tickets//679/oversized_pgs_subtitles.mkv -filter_complex '[0:s:1]scale=848x480,[0:v]overlay=shortest=1' test.avi fails assertion: Assertion progress failed at libavfilter/avfilter.c:1391
https://trac.ffmpeg.org/attachment/ticket/679/oversized_pgs_subtitles.mkv
ffmpeg -v 0 -i tickets/3539/crash.swf -map 0 -t 1 -f framecrc - output changes, not sure this is a bug but reporting it anyway as i noticed
http://samples.ffmpeg.org/ffmpeg-bugs/trac/ticket3539/
doc/examples/filtering_video matrixbench_mpeg2.mpg also breaks
Thanks for the testing. I reproduced the first one (does not fail with "-f null -") and will try to fix all these. Regards, -- Nicolas George
Le quintidi 15 fructidor, an CCXXIV, Michael Niedermayer a écrit :
./ffmpeg -i tickets//679/oversized_pgs_subtitles.mkv -filter_complex '[0:s:1]scale=848x480,[0:v]overlay=shortest=1' test.avi fails assertion: Assertion progress failed at libavfilter/avfilter.c:1391
https://trac.ffmpeg.org/attachment/ticket/679/oversized_pgs_subtitles.mkv
This one was an easy fix.
ffmpeg -v 0 -i tickets/3539/crash.swf -map 0 -t 1 -f framecrc - output changes, not sure this is a bug but reporting it anyway as i noticed
This one is much more tricky. When a format change is detected, ffmpeg resets the graphs but does not drain them first. With this big change, frames inserted do not reach as far as previously: less immediate processing, more possibility to work in a thread. But the issue also happens with the current code: ffmpeg -i a%02d.png -vf fps=50 -f framecrc - with two input PNGs of different size, will only output the second one, because the first one is buffered by vf_fps and discarded when the graph is reconfigured. This is very tricky, because flushing the graph on format change is not the same thing as flushing it at EOF. Imagine the input format for an overlay change: flushing would mean finishing the background video with the last image before the format change, this is not at all what expected. I do not see a clean way of getting this particular example and all similar ones working. I can see an ugly solution, but I would rather avoid it and wait for a cleaner implementation of handling format changes. Regards, -- Nicolas George
On 9/4/16, Nicolas George <george@nsup.org> wrote:
Le quintidi 15 fructidor, an CCXXIV, Michael Niedermayer a ecrit :
./ffmpeg -i tickets//679/oversized_pgs_subtitles.mkv -filter_complex '[0:s:1]scale=848x480,[0:v]overlay=shortest=1' test.avi fails assertion: Assertion progress failed at libavfilter/avfilter.c:1391
https://trac.ffmpeg.org/attachment/ticket/679/oversized_pgs_subtitles.mkv
This one was an easy fix.
ffmpeg -v 0 -i tickets/3539/crash.swf -map 0 -t 1 -f framecrc - output changes, not sure this is a bug but reporting it anyway as i noticed
This one is much more tricky. When a format change is detected, ffmpeg resets the graphs but does not drain them first. With this big change, frames inserted do not reach as far as previously: less immediate processing, more possibility to work in a thread.
But the issue also happens with the current code:
ffmpeg -i a%02d.png -vf fps=50 -f framecrc -
with two input PNGs of different size, will only output the second one, because the first one is buffered by vf_fps and discarded when the graph is reconfigured.
This is very tricky, because flushing the graph on format change is not the same thing as flushing it at EOF. Imagine the input format for an overlay change: flushing would mean finishing the background video with the last image before the format change, this is not at all what expected.
I do not see a clean way of getting this particular example and all similar ones working. I can see an ugly solution, but I would rather avoid it and wait for a cleaner implementation of handling format changes.
And what would that cleaner implementation do? At current rate your lavfi changes will never get in, which sucks.
Le nonidi 19 fructidor, an CCXXIV, Paul B Mahol a écrit :
And what would that cleaner implementation do?
There is a rather simple implementation of format change in lavfi: have on each input a boolean flag "can_deal_with_format_change". If a frame with a different format arrives on a filter that does not have the flag, just insert the scale/aresample filter on the link to force the format to be the same. It is not entirely optimal, but it is quite simple and does the work. And it could probably be implemented independently of my changes. But I do not want to spend time on it before finishing this, or it will never end. (Actually, I think we had something like that for buffersrc but it disappeared at some time. avconv also has some very strange code to handle format changes.)
At current rate your lavfi changes will never get in, which sucks.
Which is why I would like to be authorized to ignore this kind of hiccups. Format change does not currently work, this particular case used to work only by chance. Can I break it and repair it later? Regards, -- Nicolas George
On 9/4/16, Nicolas George <george@nsup.org> wrote:
Which is why I would like to be authorized to ignore this kind of hiccups. Format change does not currently work, this particular case used to work only by chance. Can I break it and repair it later?
I agree, format change in filtergraph, when works it works by pure luck.
On Sun, Sep 04, 2016 at 10:16:57PM +0200, Nicolas George wrote:
Le nonidi 19 fructidor, an CCXXIV, Paul B Mahol a écrit :
And what would that cleaner implementation do?
There is a rather simple implementation of format change in lavfi: have on each input a boolean flag "can_deal_with_format_change". If a frame with a different format arrives on a filter that does not have the flag, just insert the scale/aresample filter on the link to force the format to be the same.
It is not entirely optimal, but it is quite simple and does the work.
And it could probably be implemented independently of my changes. But I do not want to spend time on it before finishing this, or it will never end.
(Actually, I think we had something like that for buffersrc but it disappeared at some time. avconv also has some very strange code to handle format changes.)
we have a few filters that should work fine with format changes i would have assumed that still works
At current rate your lavfi changes will never get in, which sucks.
Which is why I would like to be authorized to ignore this kind of hiccups. Format change does not currently work, this particular case used to work only by chance. Can I break it and repair it later?
I think there are 2 things First, filters simply supporting format changes without any magic or reinitialization, they just work if formats change. This should continue to be so but it also shouldnt really add any complication or am i missing something ? This was the type i was interrested in previously ... (until patch reviews made me loose interrest) Second, graph reinitialization, this is hard to get right and if its done right it still doesnt work for many usecases due to destroyed state. I dont think temporary worsening graph reinitialization is a problem but thats just my oppinion [...] -- Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB Breaking DRM is a little like attempting to break through a door even though the window is wide open and the only thing in the house is a bunch of things you dont want and which you would get tomorrow for free anyway
On 9/4/16, Michael Niedermayer <michael@niedermayer.cc> wrote:
On Sun, Sep 04, 2016 at 10:16:57PM +0200, Nicolas George wrote:
Le nonidi 19 fructidor, an CCXXIV, Paul B Mahol a ecrit :
And what would that cleaner implementation do?
There is a rather simple implementation of format change in lavfi: have on each input a boolean flag "can_deal_with_format_change". If a frame with a different format arrives on a filter that does not have the flag, just insert the scale/aresample filter on the link to force the format to be the same.
It is not entirely optimal, but it is quite simple and does the work.
And it could probably be implemented independently of my changes. But I do not want to spend time on it before finishing this, or it will never end.
(Actually, I think we had something like that for buffersrc but it disappeared at some time. avconv also has some very strange code to handle format changes.)
we have a few filters that should work fine with format changes i would have assumed that still works
At current rate your lavfi changes will never get in, which sucks.
Which is why I would like to be authorized to ignore this kind of hiccups. Format change does not currently work, this particular case used to work only by chance. Can I break it and repair it later?
I think there are 2 things First, filters simply supporting format changes without any magic or reinitialization, they just work if formats change. This should continue to be so but it also shouldnt really add any complication or am i missing something ? This was the type i was interrested in previously ... (until patch reviews made me loose interrest)
Second, graph reinitialization, this is hard to get right and if its done right it still doesnt work for many usecases due to destroyed state. I dont think temporary worsening graph reinitialization is a problem but thats just my oppinion
So everybody agrees, we should proceed.
Le quartidi 24 fructidor, an CCXXIV, Paul B Mahol a écrit :
So everybody agrees, we should proceed.
I am proceeding, but as you can see in the patch, there is still a fair amount of work to be done. Still, people can help if they want to speed things up, especially since a significant part of the work is design decisions that I can not do alone and will need to be discussed. What needs to be done (using this mail as a notepad, but including the tasks where help is required): - Finish documenting the scheduling and make sure the implementation matches the documentation. - Discuss if "private_fields.h" is acceptable or decide another solution. - Clearly identify and isolate the parts of the scheduling that are needed only for request_frame()/request_frame() compatibility. - Decide exactly what parts of the scheduling are the responsibility of filters (possibly in the compatibility activate function) and what parts are handled by the framework. - Think ahead about threading and use wrapper to access fields that will require locking or synchronization. - Think about features whose need I realized while trying to get it working: distinguish productive / processing activation, synchronize several filter graphs. Please feel free to ask details about any of these points: not only would getting interest help me stay motivated, but discussing implementation details and explaining the design would help me having a clear idea of the whole system. Regards, -- Nicolas George
On 9/10/16, Nicolas George <george@nsup.org> wrote:
Le quartidi 24 fructidor, an CCXXIV, Paul B Mahol a ecrit :
So everybody agrees, we should proceed.
I am proceeding, but as you can see in the patch, there is still a fair amount of work to be done. Still, people can help if they want to speed things up, especially since a significant part of the work is design decisions that I can not do alone and will need to be discussed.
What needs to be done (using this mail as a notepad, but including the tasks where help is required):
- Finish documenting the scheduling and make sure the implementation matches the documentation.
- Discuss if "private_fields.h" is acceptable or decide another solution.
- Clearly identify and isolate the parts of the scheduling that are needed only for request_frame()/request_frame() compatibility.
- Decide exactly what parts of the scheduling are the responsibility of filters (possibly in the compatibility activate function) and what parts are handled by the framework.
- Think ahead about threading and use wrapper to access fields that will require locking or synchronization.
- Think about features whose need I realized while trying to get it working: distinguish productive / processing activation, synchronize several filter graphs.
Please feel free to ask details about any of these points: not only would getting interest help me stay motivated, but discussing implementation details and explaining the design would help me having a clear idea of the whole system.
For start removal of recursiveness is mostly I'm interested in. What needs to be done for that, can I help somehow?
On 9/11/16, Paul B Mahol <onemda@gmail.com> wrote:
On 9/10/16, Nicolas George <george@nsup.org> wrote:
Le quartidi 24 fructidor, an CCXXIV, Paul B Mahol a ecrit :
So everybody agrees, we should proceed.
I am proceeding, but as you can see in the patch, there is still a fair amount of work to be done. Still, people can help if they want to speed things up, especially since a significant part of the work is design decisions that I can not do alone and will need to be discussed.
What needs to be done (using this mail as a notepad, but including the tasks where help is required):
- Finish documenting the scheduling and make sure the implementation matches the documentation.
- Discuss if "private_fields.h" is acceptable or decide another solution.
- Clearly identify and isolate the parts of the scheduling that are needed only for request_frame()/request_frame() compatibility.
- Decide exactly what parts of the scheduling are the responsibility of filters (possibly in the compatibility activate function) and what parts are handled by the framework.
- Think ahead about threading and use wrapper to access fields that will require locking or synchronization.
- Think about features whose need I realized while trying to get it working: distinguish productive / processing activation, synchronize several filter graphs.
Please feel free to ask details about any of these points: not only would getting interest help me stay motivated, but discussing implementation details and explaining the design would help me having a clear idea of the whole system.
For start removal of recursiveness is mostly I'm interested in. What needs to be done for that, can I help somehow?
Hi, So what's remain to be done to have frame threading in lavfi?
On 5/2/18, Paul B Mahol <onemda@gmail.com> wrote:
On 9/11/16, Paul B Mahol <onemda@gmail.com> wrote:
On 9/10/16, Nicolas George <george@nsup.org> wrote:
Le quartidi 24 fructidor, an CCXXIV, Paul B Mahol a ecrit :
So everybody agrees, we should proceed.
I am proceeding, but as you can see in the patch, there is still a fair amount of work to be done. Still, people can help if they want to speed things up, especially since a significant part of the work is design decisions that I can not do alone and will need to be discussed.
What needs to be done (using this mail as a notepad, but including the tasks where help is required):
- Finish documenting the scheduling and make sure the implementation matches the documentation.
- Discuss if "private_fields.h" is acceptable or decide another solution.
- Clearly identify and isolate the parts of the scheduling that are needed only for request_frame()/request_frame() compatibility.
- Decide exactly what parts of the scheduling are the responsibility of filters (possibly in the compatibility activate function) and what parts are handled by the framework.
- Think ahead about threading and use wrapper to access fields that will require locking or synchronization.
- Think about features whose need I realized while trying to get it working: distinguish productive / processing activation, synchronize several filter graphs.
Please feel free to ask details about any of these points: not only would getting interest help me stay motivated, but discussing implementation details and explaining the design would help me having a clear idea of the whole system.
For start removal of recursiveness is mostly I'm interested in. What needs to be done for that, can I help somehow?
Hi,
So what's remain to be done to have frame threading in lavfi?
Ping
On 6/2/18, Paul B Mahol <onemda@gmail.com> wrote:
On 5/2/18, Paul B Mahol <onemda@gmail.com> wrote:
On 9/11/16, Paul B Mahol <onemda@gmail.com> wrote:
On 9/10/16, Nicolas George <george@nsup.org> wrote:
Le quartidi 24 fructidor, an CCXXIV, Paul B Mahol a ecrit :
So everybody agrees, we should proceed.
I am proceeding, but as you can see in the patch, there is still a fair amount of work to be done. Still, people can help if they want to speed things up, especially since a significant part of the work is design decisions that I can not do alone and will need to be discussed.
What needs to be done (using this mail as a notepad, but including the tasks where help is required):
- Finish documenting the scheduling and make sure the implementation matches the documentation.
- Discuss if "private_fields.h" is acceptable or decide another solution.
- Clearly identify and isolate the parts of the scheduling that are needed only for request_frame()/request_frame() compatibility.
- Decide exactly what parts of the scheduling are the responsibility of filters (possibly in the compatibility activate function) and what parts are handled by the framework.
- Think ahead about threading and use wrapper to access fields that will require locking or synchronization.
- Think about features whose need I realized while trying to get it working: distinguish productive / processing activation, synchronize several filter graphs.
Please feel free to ask details about any of these points: not only would getting interest help me stay motivated, but discussing implementation details and explaining the design would help me having a clear idea of the whole system.
For start removal of recursiveness is mostly I'm interested in. What needs to be done for that, can I help somehow?
Hi,
So what's remain to be done to have frame threading in lavfi?
Ping
Ping
On Sun, Jun 03, 2018 at 07:43:24PM +0200, Paul B Mahol wrote:
On 6/2/18, Paul B Mahol <onemda@gmail.com> wrote:
On 5/2/18, Paul B Mahol <onemda@gmail.com> wrote:
On 9/11/16, Paul B Mahol <onemda@gmail.com> wrote:
On 9/10/16, Nicolas George <george@nsup.org> wrote:
Le quartidi 24 fructidor, an CCXXIV, Paul B Mahol a ecrit :
So everybody agrees, we should proceed.
I am proceeding, but as you can see in the patch, there is still a fair amount of work to be done. Still, people can help if they want to speed things up, especially since a significant part of the work is design decisions that I can not do alone and will need to be discussed.
What needs to be done (using this mail as a notepad, but including the tasks where help is required):
- Finish documenting the scheduling and make sure the implementation matches the documentation.
- Discuss if "private_fields.h" is acceptable or decide another solution.
- Clearly identify and isolate the parts of the scheduling that are needed only for request_frame()/request_frame() compatibility.
- Decide exactly what parts of the scheduling are the responsibility of filters (possibly in the compatibility activate function) and what parts are handled by the framework.
- Think ahead about threading and use wrapper to access fields that will require locking or synchronization.
- Think about features whose need I realized while trying to get it working: distinguish productive / processing activation, synchronize several filter graphs.
Please feel free to ask details about any of these points: not only would getting interest help me stay motivated, but discussing implementation details and explaining the design would help me having a clear idea of the whole system.
For start removal of recursiveness is mostly I'm interested in. What needs to be done for that, can I help somehow?
Hi,
So what's remain to be done to have frame threading in lavfi?
Ping
Ping
If noone, who has time to reply knows the awnser then you probably have to find it out from the code and any unfinished patchsets sending nicolas a private mail may also be more vissible to him than the ML in case he is busy -- Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB He who knows, does not speak. He who speaks, does not know. Lao Tzu
Michael Niedermayer (2018-06-04):
If noone, who has time to reply knows the awnser then you probably have to find it out from the code and any unfinished patchsets
sending nicolas a private mail may also be more vissible to him than the ML in case he is busy
In terms of design, the big thing missing from lavfi is a clean API to run a filter graph. Right now, it relies on requests on outputs, with a fragile heuristic to find the "oldest" one. A clean API would allow to run it as a whole and react to frames on output or requests on inputs, possibly with callbacks. This must come before threading, because it is what allows to control threading: threading is efficient when the system can start several threads and let them run, doing their work. If it is constantly stopping and re-starting because the calling API makes too small steps, much time is wasted. But more than that, it requires somebody working on it. Speaking for myself, the toxic ambiance in the project since a few months has destroyed my motivation for doing anything ambitious on it. And to be completely forthright, I feel that Paul is partly responsible for that toxic ambiance; see his interventions on the thread about enforcing the code of conduct for example. Regards, -- Nicolas George
On 6/6/18, Nicolas George <george@nsup.org> wrote:
Michael Niedermayer (2018-06-04):
If noone, who has time to reply knows the awnser then you probably have to find it out from the code and any unfinished patchsets
sending nicolas a private mail may also be more vissible to him than the ML in case he is busy
In terms of design, the big thing missing from lavfi is a clean API to run a filter graph. Right now, it relies on requests on outputs, with a fragile heuristic to find the "oldest" one. A clean API would allow to run it as a whole and react to frames on output or requests on inputs, possibly with callbacks.
This must come before threading, because it is what allows to control threading: threading is efficient when the system can start several threads and let them run, doing their work. If it is constantly stopping and re-starting because the calling API makes too small steps, much time is wasted.
But more than that, it requires somebody working on it. Speaking for myself, the toxic ambiance in the project since a few months has destroyed my motivation for doing anything ambitious on it. And to be completely forthright, I feel that Paul is partly responsible for that toxic ambiance; see his interventions on the thread about enforcing the code of conduct for example.
Your contributions will be missed. Good bye.
On Thu, Jun 07, 2018 at 11:23:40AM +0200, Paul B Mahol wrote:
On 6/6/18, Nicolas George <george@nsup.org> wrote:
Michael Niedermayer (2018-06-04):
If noone, who has time to reply knows the awnser then you probably have to find it out from the code and any unfinished patchsets
sending nicolas a private mail may also be more vissible to him than the ML in case he is busy
In terms of design, the big thing missing from lavfi is a clean API to run a filter graph. Right now, it relies on requests on outputs, with a fragile heuristic to find the "oldest" one. A clean API would allow to run it as a whole and react to frames on output or requests on inputs, possibly with callbacks.
This must come before threading, because it is what allows to control threading: threading is efficient when the system can start several threads and let them run, doing their work. If it is constantly stopping and re-starting because the calling API makes too small steps, much time is wasted.
But more than that, it requires somebody working on it. Speaking for myself, the toxic ambiance in the project since a few months has destroyed my motivation for doing anything ambitious on it. And to be completely forthright, I feel that Paul is partly responsible for that toxic ambiance; see his interventions on the thread about enforcing the code of conduct for example.
Your contributions will be missed.
Good bye.
Id like to see both you and nicolas work together on libavfilter. Thats a "WIN" for the community and FFmpeg. You two are the top 2 libavfilter develoepers currently. Its really _REALLY_ stupid if you two fight like this. Because no matter who "wins" this, everyone looses. [...] -- Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB Concerning the gods, I have no means of knowing whether they exist or not or of what sort they may be, because of the obscurity of the subject, and the brevity of human life -- Protagoras
On 8/30/16, Nicolas George <george@nsup.org> wrote:
Le quartidi 14 fructidor, an CCXXIV, Paul B Mahol a ecrit :
the filter frame multithreading would just internally, in filter context cache frames, once enough frames are in cache - call workers and be done, repeat. At eof call workers on remaining frames in cache.
I have no idea how much thought you have already given to it, but I am pretty sure it is not as simple as that with the current architecture. By far.
Yes, it is very simple. You just need to allocate own buffers that would be needed by filter. Than you just call ctx->internal->execute()...
In the meantime, I finally got the non-recursive version passing FATE. Here are the raw patch, so that people can have an idea what this is all about. There is still a lot of cleanup and documentation to do, as you can see.
Does this fixes buffer queue overflow for specific scenarios?
On Wed, Aug 31, 2016 at 10:18:31AM +0200, Paul B Mahol wrote:
On 8/30/16, Nicolas George <george@nsup.org> wrote:
Le quartidi 14 fructidor, an CCXXIV, Paul B Mahol a ecrit :
the filter frame multithreading would just internally, in filter context cache frames, once enough frames are in cache - call workers and be done, repeat. At eof call workers on remaining frames in cache.
I have no idea how much thought you have already given to it, but I am pretty sure it is not as simple as that with the current architecture. By far.
Yes, it is very simple. You just need to allocate own buffers that would be needed by filter. Than you just call ctx->internal->execute()...
I would have thought that filter_frame() would insert frames into a input fifo and remove & output frames from a output fifo and pass them on to the next filters ff_filter_frame() if the output fifo is empty and the input full it would block (implicitly gving its CPU time to the workers) and there would be backgound threads continuosly running that pick frames out of the input fifo and process them into an output fifo and wake up the main thread if the fifo state changes using execute() would IIUC (please correct me if i misunderstand) only execute when ff_filter_frame() is executed, this would restrict what can execute at the same time, also execute() needs to wait for all threads to finish before it can return, this too limits paralelism compared to continuously running workers that more independantly pick tasks and work on them but maybe i misuderstood how you intend to use execute() [...] -- Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB You can kill me, but you cannot change the truth.
On 8/31/16, Michael Niedermayer <michael@niedermayer.cc> wrote:
On Wed, Aug 31, 2016 at 10:18:31AM +0200, Paul B Mahol wrote:
On 8/30/16, Nicolas George <george@nsup.org> wrote:
Le quartidi 14 fructidor, an CCXXIV, Paul B Mahol a ecrit :
the filter frame multithreading would just internally, in filter context cache frames, once enough frames are in cache - call workers and be done, repeat. At eof call workers on remaining frames in cache.
I have no idea how much thought you have already given to it, but I am pretty sure it is not as simple as that with the current architecture. By far.
Yes, it is very simple. You just need to allocate own buffers that would be needed by filter. Than you just call ctx->internal->execute()...
I would have thought that filter_frame() would insert frames into a input fifo and remove & output frames from a output fifo and pass them on to the next filters ff_filter_frame() if the output fifo is empty and the input full it would block (implicitly gving its CPU time to the workers)
and there would be backgound threads continuosly running that pick frames out of the input fifo and process them into an output fifo and wake up the main thread if the fifo state changes
using execute() would IIUC (please correct me if i misunderstand) only execute when ff_filter_frame() is executed, this would restrict what can execute at the same time, also execute() needs to wait for all threads to finish before it can return, this too limits paralelism compared to continuously running workers that more independantly pick tasks and work on them but maybe i misuderstood how you intend to use execute()
How this one would work if for example first frame needs 10 seconds to generate and all other needs 1 second? How would you know from your output fifo that you got first frame, so you can pass first frame to next filter(s)? How do you know that your solution is always optimal? (Not saying that mine is anything better) How do you limit number of threads that will specifically work for this filter instance?
On Wed, Aug 31, 2016 at 02:20:27PM +0200, Paul B Mahol wrote:
On 8/31/16, Michael Niedermayer <michael@niedermayer.cc> wrote:
On Wed, Aug 31, 2016 at 10:18:31AM +0200, Paul B Mahol wrote:
On 8/30/16, Nicolas George <george@nsup.org> wrote:
Le quartidi 14 fructidor, an CCXXIV, Paul B Mahol a ecrit :
the filter frame multithreading would just internally, in filter context cache frames, once enough frames are in cache - call workers and be done, repeat. At eof call workers on remaining frames in cache.
I have no idea how much thought you have already given to it, but I am pretty sure it is not as simple as that with the current architecture. By far.
Yes, it is very simple. You just need to allocate own buffers that would be needed by filter. Than you just call ctx->internal->execute()...
I would have thought that filter_frame() would insert frames into a input fifo and remove & output frames from a output fifo and pass them on to the next filters ff_filter_frame() if the output fifo is empty and the input full it would block (implicitly gving its CPU time to the workers)
and there would be backgound threads continuosly running that pick frames out of the input fifo and process them into an output fifo and wake up the main thread if the fifo state changes
using execute() would IIUC (please correct me if i misunderstand) only execute when ff_filter_frame() is executed, this would restrict what can execute at the same time, also execute() needs to wait for all threads to finish before it can return, this too limits paralelism compared to continuously running workers that more independantly pick tasks and work on them but maybe i misuderstood how you intend to use execute()
How this one would work if for example first frame needs 10 seconds to generate and all other needs 1 second? How would you know from your output fifo that you got first frame, so you can pass first frame to next filter(s)?
one possibility: frame 1 comes in first worker adds a output entry to the end of the output linked list and starts working on frame 1 frame 2 comes in second worker adds a output entry to the end of the output linked list and starts working on frame 2 second worker finishes and replaces/adds its output to its output entry frame 3 comes in, the next output entry (from worker 1) is not finished so nothing can be output yet second worker adds a output entry to the end of the output linked list and starts working on frame 3 first worker finishes and replaces/adds its output to its output entry frame 4 comes in, the next output entry (from worker 1) is ready and is sent to the next filter
How do you know that your solution is always optimal? (Not saying that mine is anything better)
i dont know if its always optimal, in fact it likely is not always optimal. It seemed simple and reasonably good.
How do you limit number of threads that will specifically work for this filter instance?
i had assumed that a fixed number of worker threads would be used for each filter, some of these may be idle and consume no CPU if there is nothing to do I assumed a fixed number for simplicity, nothing in the design should depend on the number being fixed [...] -- Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB The real ebay dictionary, page 3 "Rare item" - "Common item with rare defect or maybe just a lie" "Professional" - "'Toy' made in china, not functional except as doorstop" "Experts will know" - "The seller hopes you are not an expert"
Le quintidi 15 fructidor, an CCXXIV, Paul B Mahol a écrit :
Yes, it is very simple. You just need to allocate own buffers that would be needed by filter. Than you just call ctx->internal->execute()...
I am sorry, but with the current API, to have something that really works with threads, not just pseudo-threading that is so full of blocking that it is mostly sequential, it is much more complicated than that. I think. Please prove me wrong.
Does this fixes buffer queue overflow for specific scenarios?
Not by itself, but then rewriting framesync to use the link's fifo becomes very easy and fix all the cases that can possibly work. Regards, -- Nicolas George
On 8/31/16, Nicolas George <george@nsup.org> wrote:
Le quintidi 15 fructidor, an CCXXIV, Paul B Mahol a ecrit :
Yes, it is very simple. You just need to allocate own buffers that would be needed by filter. Than you just call ctx->internal->execute()...
I am sorry, but with the current API, to have something that really works with threads, not just pseudo-threading that is so full of blocking that it is mostly sequential, it is much more complicated than that. I think. Please prove me wrong.
How would one use already available workers from lavfi/pthread.c ? If somebody wrote better implementation than me and its faster I will drop mine.
Does this fixes buffer queue overflow for specific scenarios?
Not by itself, but then rewriting framesync to use the link's fifo becomes very easy and fix all the cases that can possibly work.
Regards,
-- Nicolas George
participants (7)
-
Clément Bœsch -
Michael Niedermayer -
Michael Niedermayer -
Nicolas George -
Paul B Mahol -
Stefano Sabatini -
wm4