[FFmpeg-devel] [RFC] Scripting filter and foreign language bindings

[wm4]

On Sat, 16 Nov 2013 19:49:33 +0100
Stefano Sabatini <stefasab at gmail.com> wrote:

> On date Monday 2013-01-14 00:43:49 +0100, Stefano Sabatini encoded:
> > Hi,
> > 
> > in many applications it is useful to perform very specific tasks
> > e.g. to coordinate events in a filtergraph.
> > 
> > For example, a filter may want to stop/start a stream depending on
> > some external conditions (e.g. according to the result of inspecting
> > the filesystem, or accessing an external DB, etc.). For such needs
> > having a scripting engine in the filtergraph would be the best
> > solution, since it would allow to write the logic in a compact way,
> > relying on scripting and (possibly) external libraries with no need to
> > hack/recompile the library.
> > 
> > Thus I'm considering to write a scripting filter as a proof of
> > concept. The filter should be able to load a program specified in the
> > given script, which would provide hooks for the usual callbacks,
> > should be able to process data, process and send events, call a given
> > subsets of the public and private libav* API and such.
> > 
> > I'm considering to try at first with Lua, since it seems it was
> > designed with the objective of being easily embeddable in C, but if
> > the experiment works nothing prevents us to extend the idea to other
> > languages.
> > 
> > Sending this email in case someone has ideas or recommendations to
> > propose.
> 
> Very short followup after some time spent doing some hacks, in case
> someone is interested.
> 
> First, it is clear that in order to be able to do useful stuff in a
> scripting filter it is required to use the FFmpeg API (e.g. handling
> frames and stuff), so the first very non trivial problem is to bind
> the subset of the FFmpeg (public) API to the target scripting
> language.

That sounds a bit like overkill. Due to the way libavfilter works,
maybe it would be best to restrict the beginning to 1.) making it
possible to use scripting instead of av_expr_* strings to setup
filters, and 2.) using scripting instead of the lavfi filter language to
build a filter graph.

Ideally, there should be some sort of high level API, which should
basically allow the user to play with frames directly, but this
wouldn't fit directly into libavfilter's architecture - but a hybrid
approach might be possible.

> I did some experiments with SWIG. SWIG can't be considered an
> universal solution, that is each language will need a specific
> maintainer and ad-hoc solutions, with or without SWIG, but SWIG can
> help to simplify the maintainance burden.
> 
> See the following discussion for more insight:
> http://thread.gmane.org/gmane.comp.programming.swig/17211

Yeah, SWIG sounds like overkill. In any case, if the underlying API is
somewhat scripting-friendly (memory safety etc.), and if the API small,
and if it's stable, then creating bindings manually is probably not
_that_ hard. It might also make sense to pick only one scripting
language, and stick to it - at least within ffmpeg.

> One question is if the bindings should be maintained within FFmpeg or
> if they really should stay outside. I tend to lean towards the second
> solution. Indeed there are already several external wrappers in
> various languages, more or less maintained, and it is not realistic to
> think that there is enough manpower in FFmpeg to keep them udpated,
> and we should rather leverage the work done by these external
> contributors.

I was imagining that scripting support could allow handling new
use cases, without adding weird things to ffmpeg.c and libav*.

> Another point is that in order to do something useful with scripting
> we would need to call the internal API, that is what it can't do with
> an external C filter can't be done with a scripting filter as well, so
> in order to follow this path we would need to make many internal
> methods public (and in general make possible for a library user to
> define a filter and register, register and use it in its own code).

No, the API should improved. Of course that's hard work and easier said
than done.

Custom filters sound good, though. But AFAIK this is against FFmpeg's
(unwritten?) policy of not allowing external plugins.