[FFmpeg-devel] [PATCH] libavfilter: Add more operation supports in FFmpeg dnn native mode.

Guo, Yejun yejun.guo at intel.com
Tue Apr 30 03:43:43 EEST 2019



> -----Original Message-----
> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces at ffmpeg.org] On Behalf Of
> Pedro Arthur
> Sent: Monday, April 29, 2019 11:26 PM
> To: FFmpeg development discussions and patches <ffmpeg-devel at ffmpeg.org>
> Subject: Re: [FFmpeg-devel] [PATCH] libavfilter: Add more operation supports in
> FFmpeg dnn native mode.
> 
> Em seg, 29 de abr de 2019 às 00:06, <xwmeng at pku.edu.cn> escreveu:
> >
> >
> >
> >
> > > -----原始邮件-----
> > > 发件人: "Pedro Arthur" <bygrandao at gmail.com>
> > > 发送时间: 2019-04-29 10:42:42 (星期一)
> > > 收件人: "FFmpeg development discussions and patches"
> <ffmpeg-devel at ffmpeg.org>
> > > 抄送:
> > > 主题: Re: [FFmpeg-devel] [PATCH] libavfilter: Add more operation supports
> in FFmpeg dnn native mode.
> > >
> > > I think for training these filters the preferred method is VALID as it
> > > uses only the data available (without filling the borders) and gives
> > > the best possible result.
> > > However for inference usually one expects to output an image with the
> > > same size of the original (imagine the case of chained filters where
> > > each one reduces the image by a few pixels, in the end one may have a
> > > useless output).
> > > Therefore it makes perfect sense to use different padding methods for
> > > training/inference.
> > >
> > > The clamp_to_edge padding was introduced before the TF backend thus it
> > > stayed in the native backend even after the introduction of the TF
> > > backend.
> > > Indeed the clamp_to_edge is simpler than the other padding methods and
> > > also gives a slight better result, If I remember correct the student
> > > which implemented the TF backend did not find an equivalent padding
> > > method in TF, thats why it uses different paddings.
> > >
> > Yes, I think clamp_to_edge is a good method to keep the output with the
> same size as input. However, I don't think "VALID" is the best method giving
> best possible result. So, for "VALID" mode, maybe we can use the
> clamp_to_edge method in the current dnn native mode? And then, we should
> also add "SAME" option to support other filters.

@xwmeng, We now can support 3 padding options for conv layer.
- valid:  the same definition as TF model. (the size of feature map changed).
- same:  the same definition as TF model. (zero filled for outside pixels).
- same_clamp_to_edge: just like 'same' option, but edge filled for outside pixels. So we don't need change SR code for the padding.

> >
> 
> I think it is best to not make any assumptions like VALID =>
> clamp_to_edge, but you can keep it for now.
> Ideally the model should have a padding layer which the backend
> properly implements. Currently the TF backend when reading a native
> model adds this padding layer implicitly, therefore it would be a
> matter of changing it to have an explicity padding layer in the model.
> 
> Maybe you can assume VALID => clamp_to_edge, so you can add what you
> need without changing the SR code and later you implement the
> explicity padding support and send a PR to the original repo
> (https://github.com/HighVoltageRocknRoll/sr) properly modifying the
> model.

yes, the ideal solution to make native SR model the same as TF model, is to:
- add explicit padding layer in SR native model. (the padding layer is explicitly added in TF model)
- export 'valid' option for conv layer in SR native model. (the TF model uses 'valid' option)
- add padding layer execution support in native mode.
- add padding option support in conv layer in native mode.

And, we can first keep most of SR code unchanged, by adding 'same_clamp_to_edge' option in conv layer
in native mode. This can be our current choice. And the ideal solution can be our next choice, just as you mentioned.

> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".


More information about the ffmpeg-devel mailing list