[RFC] Scripting filter and foreign language bindings
Hi, in many applications it is useful to perform very specific tasks e.g. to coordinate events in a filtergraph. For example, a filter may want to stop/start a stream depending on some external conditions (e.g. according to the result of inspecting the filesystem, or accessing an external DB, etc.). For such needs having a scripting engine in the filtergraph would be the best solution, since it would allow to write the logic in a compact way, relying on scripting and (possibly) external libraries with no need to hack/recompile the library. Thus I'm considering to write a scripting filter as a proof of concept. The filter should be able to load a program specified in the given script, which would provide hooks for the usual callbacks, should be able to process data, process and send events, call a given subsets of the public and private libav* API and such. I'm considering to try at first with Lua, since it seems it was designed with the objective of being easily embeddable in C, but if the experiment works nothing prevents us to extend the idea to other languages. Sending this email in case someone has ideas or recommendations to propose. -- FFmpeg = Forgiving and Frenzy Mysterious Perennial Erotic Gymnast
On date Monday 2013-01-14 00:43:49 +0100, Stefano Sabatini encoded:
Hi,
in many applications it is useful to perform very specific tasks e.g. to coordinate events in a filtergraph.
For example, a filter may want to stop/start a stream depending on some external conditions (e.g. according to the result of inspecting the filesystem, or accessing an external DB, etc.). For such needs having a scripting engine in the filtergraph would be the best solution, since it would allow to write the logic in a compact way, relying on scripting and (possibly) external libraries with no need to hack/recompile the library.
Thus I'm considering to write a scripting filter as a proof of concept. The filter should be able to load a program specified in the given script, which would provide hooks for the usual callbacks, should be able to process data, process and send events, call a given subsets of the public and private libav* API and such.
I'm considering to try at first with Lua, since it seems it was designed with the objective of being easily embeddable in C, but if the experiment works nothing prevents us to extend the idea to other languages.
Sending this email in case someone has ideas or recommendations to propose.
Very short followup after some time spent doing some hacks, in case someone is interested. First, it is clear that in order to be able to do useful stuff in a scripting filter it is required to use the FFmpeg API (e.g. handling frames and stuff), so the first very non trivial problem is to bind the subset of the FFmpeg (public) API to the target scripting language. I did some experiments with SWIG. SWIG can't be considered an universal solution, that is each language will need a specific maintainer and ad-hoc solutions, with or without SWIG, but SWIG can help to simplify the maintainance burden. See the following discussion for more insight: http://thread.gmane.org/gmane.comp.programming.swig/17211 One question is if the bindings should be maintained within FFmpeg or if they really should stay outside. I tend to lean towards the second solution. Indeed there are already several external wrappers in various languages, more or less maintained, and it is not realistic to think that there is enough manpower in FFmpeg to keep them udpated, and we should rather leverage the work done by these external contributors. Another point is that in order to do something useful with scripting we would need to call the internal API, that is what it can't do with an external C filter can't be done with a scripting filter as well, so in order to follow this path we would need to make many internal methods public (and in general make possible for a library user to define a filter and register, register and use it in its own code). -- FFmpeg = Funny & Free Magical Peaceful Enchanting Guru
On Sat, 16 Nov 2013 19:49:33 +0100 Stefano Sabatini <stefasab@gmail.com> wrote:
On date Monday 2013-01-14 00:43:49 +0100, Stefano Sabatini encoded:
Hi,
in many applications it is useful to perform very specific tasks e.g. to coordinate events in a filtergraph.
For example, a filter may want to stop/start a stream depending on some external conditions (e.g. according to the result of inspecting the filesystem, or accessing an external DB, etc.). For such needs having a scripting engine in the filtergraph would be the best solution, since it would allow to write the logic in a compact way, relying on scripting and (possibly) external libraries with no need to hack/recompile the library.
Thus I'm considering to write a scripting filter as a proof of concept. The filter should be able to load a program specified in the given script, which would provide hooks for the usual callbacks, should be able to process data, process and send events, call a given subsets of the public and private libav* API and such.
I'm considering to try at first with Lua, since it seems it was designed with the objective of being easily embeddable in C, but if the experiment works nothing prevents us to extend the idea to other languages.
Sending this email in case someone has ideas or recommendations to propose.
Very short followup after some time spent doing some hacks, in case someone is interested.
First, it is clear that in order to be able to do useful stuff in a scripting filter it is required to use the FFmpeg API (e.g. handling frames and stuff), so the first very non trivial problem is to bind the subset of the FFmpeg (public) API to the target scripting language.
That sounds a bit like overkill. Due to the way libavfilter works, maybe it would be best to restrict the beginning to 1.) making it possible to use scripting instead of av_expr_* strings to setup filters, and 2.) using scripting instead of the lavfi filter language to build a filter graph. Ideally, there should be some sort of high level API, which should basically allow the user to play with frames directly, but this wouldn't fit directly into libavfilter's architecture - but a hybrid approach might be possible.
I did some experiments with SWIG. SWIG can't be considered an universal solution, that is each language will need a specific maintainer and ad-hoc solutions, with or without SWIG, but SWIG can help to simplify the maintainance burden.
See the following discussion for more insight: http://thread.gmane.org/gmane.comp.programming.swig/17211
Yeah, SWIG sounds like overkill. In any case, if the underlying API is somewhat scripting-friendly (memory safety etc.), and if the API small, and if it's stable, then creating bindings manually is probably not _that_ hard. It might also make sense to pick only one scripting language, and stick to it - at least within ffmpeg.
One question is if the bindings should be maintained within FFmpeg or if they really should stay outside. I tend to lean towards the second solution. Indeed there are already several external wrappers in various languages, more or less maintained, and it is not realistic to think that there is enough manpower in FFmpeg to keep them udpated, and we should rather leverage the work done by these external contributors.
I was imagining that scripting support could allow handling new use cases, without adding weird things to ffmpeg.c and libav*.
Another point is that in order to do something useful with scripting we would need to call the internal API, that is what it can't do with an external C filter can't be done with a scripting filter as well, so in order to follow this path we would need to make many internal methods public (and in general make possible for a library user to define a filter and register, register and use it in its own code).
No, the API should improved. Of course that's hard work and easier said than done. Custom filters sound good, though. But AFAIK this is against FFmpeg's (unwritten?) policy of not allowing external plugins.
On date Monday 2013-11-18 17:43:43 +0100, wm4 encoded:
On Sat, 16 Nov 2013 19:49:33 +0100 Stefano Sabatini <stefasab@gmail.com> wrote: [...]
Very short followup after some time spent doing some hacks, in case someone is interested.
First, it is clear that in order to be able to do useful stuff in a scripting filter it is required to use the FFmpeg API (e.g. handling frames and stuff), so the first very non trivial problem is to bind the subset of the FFmpeg (public) API to the target scripting language.
That sounds a bit like overkill. Due to the way libavfilter works, maybe it would be best to restrict the beginning to 1.) making it possible to use scripting instead of av_expr_* strings to setup filters, and 2.) using scripting instead of the lavfi filter language to build a filter graph.
1. we don't use expr to setup filters in the first place (also: random language X should not be employed as substitute for expr) 2. this means binding the lavfi API to a high-level language (I don't think this was done for any of the available bindings).
Ideally, there should be some sort of high level API, which should basically allow the user to play with frames directly, but this wouldn't fit directly into libavfilter's architecture - but a hybrid approach might be possible.
But this is not strictly related to libavfilter.
I did some experiments with SWIG. SWIG can't be considered an universal solution, that is each language will need a specific maintainer and ad-hoc solutions, with or without SWIG, but SWIG can help to simplify the maintainance burden.
See the following discussion for more insight: http://thread.gmane.org/gmane.comp.programming.swig/17211
Yeah, SWIG sounds like overkill. In any case, if the underlying API is somewhat scripting-friendly (memory safety etc.), and if the API small, and if it's stable, then creating bindings manually is probably not _that_ hard. It might also make sense to pick only one scripting language, and stick to it - at least within ffmpeg.
My impression reading the discussion is that SWIG may be the tool, but it's not an universal tool. It can be used to automate binding creation, but in general every language has need different needs (for example SWIG doesn't support Haskell for reasons due to the nature of the language).
One question is if the bindings should be maintained within FFmpeg or if they really should stay outside. I tend to lean towards the second solution. Indeed there are already several external wrappers in various languages, more or less maintained, and it is not realistic to think that there is enough manpower in FFmpeg to keep them udpated, and we should rather leverage the work done by these external contributors.
I was imagining that scripting support could allow handling new use cases, without adding weird things to ffmpeg.c and libav*.
Indeed bindings should be transparent to the usual ffmpeg internal developer. What I propose is to rely on the work of external bindings, for example to extend the subset of the supported API. We may start by adding a section on the wiki, start to use the binding of our favourite language, and eventually improving the C API to simplify the binding process. Ideally this should end up improving both bindings and the library itself. But I don't think this should be done in the core of the project.
Another point is that in order to do something useful with scripting we would need to call the internal API, that is what it can't do with an external C filter can't be done with a scripting filter as well, so in order to follow this path we would need to make many internal methods public (and in general make possible for a library user to define a filter and register, register and use it in its own code).
No, the API should improved. Of course that's hard work and easier said than done.
Custom filters sound good, though. But AFAIK this is against FFmpeg's (unwritten?) policy of not allowing external plugins.
I don't think it makes much sense to have a "private" codec or muxer/demuxer. Filters are different, for custom logic you don't need much low-level optimizations but rather the availability of external modules to interface with external libraries and tools (e.g. to query databases, use widgets and such). -- FFmpeg = Fiendish and Fundamentalist Magic Powerful Evangelical Gnome
participants (2)
-
Stefano Sabatini -
wm4