[FFmpeg-devel] [RFC] Scripting filter and foreign language bindings

[Stefano Sabatini]

On date Monday 2013-01-14 00:43:49 +0100, Stefano Sabatini encoded:
> Hi,
> 
> in many applications it is useful to perform very specific tasks
> e.g. to coordinate events in a filtergraph.
> 
> For example, a filter may want to stop/start a stream depending on
> some external conditions (e.g. according to the result of inspecting
> the filesystem, or accessing an external DB, etc.). For such needs
> having a scripting engine in the filtergraph would be the best
> solution, since it would allow to write the logic in a compact way,
> relying on scripting and (possibly) external libraries with no need to
> hack/recompile the library.
> 
> Thus I'm considering to write a scripting filter as a proof of
> concept. The filter should be able to load a program specified in the
> given script, which would provide hooks for the usual callbacks,
> should be able to process data, process and send events, call a given
> subsets of the public and private libav* API and such.
> 
> I'm considering to try at first with Lua, since it seems it was
> designed with the objective of being easily embeddable in C, but if
> the experiment works nothing prevents us to extend the idea to other
> languages.
> 
> Sending this email in case someone has ideas or recommendations to
> propose.

Very short followup after some time spent doing some hacks, in case
someone is interested.

First, it is clear that in order to be able to do useful stuff in a
scripting filter it is required to use the FFmpeg API (e.g. handling
frames and stuff), so the first very non trivial problem is to bind
the subset of the FFmpeg (public) API to the target scripting
language.

I did some experiments with SWIG. SWIG can't be considered an
universal solution, that is each language will need a specific
maintainer and ad-hoc solutions, with or without SWIG, but SWIG can
help to simplify the maintainance burden.

See the following discussion for more insight:
http://thread.gmane.org/gmane.comp.programming.swig/17211

One question is if the bindings should be maintained within FFmpeg or
if they really should stay outside. I tend to lean towards the second
solution. Indeed there are already several external wrappers in
various languages, more or less maintained, and it is not realistic to
think that there is enough manpower in FFmpeg to keep them udpated,
and we should rather leverage the work done by these external
contributors.

Another point is that in order to do something useful with scripting
we would need to call the internal API, that is what it can't do with
an external C filter can't be done with a scripting filter as well, so
in order to follow this path we would need to make many internal
methods public (and in general make possible for a library user to
define a filter and register, register and use it in its own code).
-- 
FFmpeg = Funny & Free Magical Peaceful Enchanting Guru