[FFmpeg-devel] [GSOC] [PATCH] TensorFlow backend introduction for DNN module

[Sergey Lavrushkin]

2018-06-02 22:13 GMT+03:00 James Almer <jamrial at gmail.com>:

> On 6/2/2018 3:45 PM, Sergey Lavrushkin wrote:
> > 2018-06-02 19:45 GMT+03:00 James Almer <jamrial at gmail.com
> > <mailto:jamrial at gmail.com>>:
> >
> >     On 5/31/2018 12:01 PM, Sergey Lavrushkin wrote:
> >     > diff --git a/Changelog b/Changelog
> >     > index df2024fb59..a667fd045d 100644
> >     > --- a/Changelog
> >     > +++ b/Changelog
> >     > @@ -11,6 +11,7 @@ version <next>:
> >     >  - support mbedTLS based TLS
> >     >  - DNN inference interface
> >     >  - Reimplemented SRCNN filter using DNN inference interface
> >     > +- TensorFlow DNN backend
> >
> >     This and the two entries you added earlier don't really belong here.
> >     It's enough with the line stating the filter was introduced back in
> >     ffmpeg 4.0
> >
> >
> > I should not add any line regarding introduced DNN inference module,
> > that can be usefull for someone writing another filter based on DNN?
>
> Changelog lists the newly added features of interest to users, not
> developers. When the CBS framework was added it wasn't mentioned here
> either, only the filters making use of it.
>
> Users don't care that the scrnn filter was reimplemented using this new
> framework, but are however interested to know that the filter was added,
> or at most if its features were considerably expanded.
> So based on that, I guess it's acceptable to add a line in this patch like
>
> - libtensorflow backend for DNN based filters like srcnn.
>
> Or similar. But the earlier two entries don't belong here.
>
> >
> >
> >     >
> >     >
> >     >  version 4.0:
> >     > diff --git a/configure b/configure
> >     > index 09ff0c55e2..47e21fec39 100755
> >     > --- a/configure
> >     > +++ b/configure
> >     > @@ -259,6 +259,7 @@ External library support:
> >     >    --enable-libspeex        enable Speex de/encoding via libspeex
> [no]
> >     >    --enable-libsrt          enable Haivision SRT protocol via
> >     libsrt [no]
> >     >    --enable-libssh          enable SFTP protocol via libssh [no]
> >     > +  --enable-libtensorflow   enable TensorFlow as a DNN module
> >     backend [no]
> >
> >     Maybe mention it's for the srcnn filter.
> >
> >     >    --enable-libtesseract    enable Tesseract, needed for ocr
> >     filter [no]
> >     >    --enable-libtheora       enable Theora encoding via libtheora
> [no]
> >     >    --enable-libtls          enable LibreSSL (via libtls), needed
> >     for https support
> >     > @@ -1713,6 +1714,7 @@ EXTERNAL_LIBRARY_LIST="
> >     >      libspeex
> >     >      libsrt
> >     >      libssh
> >     > +    libtensorflow
> >     >      libtesseract
> >     >      libtheora
> >     >      libtwolame
> >     > @@ -3453,7 +3455,7 @@ avcodec_select="null_bsf"
> >     >  avdevice_deps="avformat avcodec avutil"
> >     >  avdevice_suggest="libm"
> >     >  avfilter_deps="avutil"
> >     > -avfilter_suggest="libm"
> >     > +avfilter_suggest="libm libtensorflow"
> >
> >     Add instead
> >
> >     srcnn_filter_suggest="libtensorflow"
> >
> >     To the corresponding section.
> >
> >
> > But this DNN inference module can be used for other filters.
> > At least, I think, that after training more complicated models for super
> > resolution I'll have to add them as separate filters.
> > So, I thought, this module shouldn't be a part of srcnn filter from the
> > begining.
> > Or is it better to add  *_filter_suggest="libtensorflow" to the
> > configure script and
> > dnn_*.o to the Makefile for every new filter based on this module?
>
> Yes. If it's a non public framework/API used by one or more modules that
> can be disabled (as is the case with filters) then it should be added to
> the list of object dependencies for each of those optional modules. See
> framesync.o for another example of this.
>
> That being said, another way to implement this is making DNN its own
> independent module in configure, like for example cbs, fft and many
> others. This way, configure would look something like
>
> dnn_suggest="libtensorflow"
> srcnn_filter_select="dnn"
>
> With dnn being an entry added to the CONFIG_EXTRA list.
>
> Then in libavfilter/Makefile you'd do
>
> OBJS-$(CONFIG_DNN) += dnn_interface.o dnn_backend_native.o dnn_backend_tf.o
>
> Or even better, something like
>
> DNN-OBJS-$(CONFIG_LIBTENSORFLOW) += dnn_backend_tf.o
> OBJS-$(CONFIG_DNN) += dnn_interface.o dnn_backend_native.o $(DNN-OBJS-yes)
>
> Which would allow you to remove the #if preprocessor check in
> dnn_backend_tf.c, as it will only be compiled when both libtensorflow
> and at least one filter using dnn are enabled.
>
> The reason for all this is to have everything as modular as possible,
> reducing compiled objects and dependencies (like lavfi linking to lavf)
> to exactly what's strictly needed.


Here is the patch with suggested changes.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: adds-tensorflow-backend.patch
Type: text/x-patch
Size: 240505 bytes
Desc: not available
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20180603/95d39ccb/attachment.bin>