[FFmpeg-devel] [PATCH] libavfilter: Add more operation supports in FFmpeg dnn native mode.

xwmeng at pku.edu.cn xwmeng at pku.edu.cn
Wed May 1 12:08:51 EEST 2019




> -----原始邮件-----
> 发件人: "Guo, Yejun" <yejun.guo at intel.com>
> 发送时间: 2019-04-30 08:43:43 (星期二)
> 收件人: "FFmpeg development discussions and patches" <ffmpeg-devel at ffmpeg.org>
> 抄送: 
> 主题: Re: [FFmpeg-devel] [PATCH] libavfilter: Add more operation supports in FFmpeg dnn native mode.
> 
> 
> 
> > -----Original Message-----
> > From: ffmpeg-devel [mailto:ffmpeg-devel-bounces at ffmpeg.org] On Behalf Of
> > Pedro Arthur
> > Sent: Monday, April 29, 2019 11:26 PM
> > To: FFmpeg development discussions and patches <ffmpeg-devel at ffmpeg.org>
> > Subject: Re: [FFmpeg-devel] [PATCH] libavfilter: Add more operation supports in
> > FFmpeg dnn native mode.
> > 
> > Em seg, 29 de abr de 2019 às 00:06, <xwmeng at pku.edu.cn> escreveu:
> > >
> > >
> > >
> > >
> > > > -----原始邮件-----
> > > > 发件人: "Pedro Arthur" <bygrandao at gmail.com>
> > > > 发送时间: 2019-04-29 10:42:42 (星期一)
> > > > 收件人: "FFmpeg development discussions and patches"
> > <ffmpeg-devel at ffmpeg.org>
> > > > 抄送:
> > > > 主题: Re: [FFmpeg-devel] [PATCH] libavfilter: Add more operation supports
> > in FFmpeg dnn native mode.
> > > >
> > > > I think for training these filters the preferred method is VALID as it
> > > > uses only the data available (without filling the borders) and gives
> > > > the best possible result.
> > > > However for inference usually one expects to output an image with the
> > > > same size of the original (imagine the case of chained filters where
> > > > each one reduces the image by a few pixels, in the end one may have a
> > > > useless output).
> > > > Therefore it makes perfect sense to use different padding methods for
> > > > training/inference.
> > > >
> > > > The clamp_to_edge padding was introduced before the TF backend thus it
> > > > stayed in the native backend even after the introduction of the TF
> > > > backend.
> > > > Indeed the clamp_to_edge is simpler than the other padding methods and
> > > > also gives a slight better result, If I remember correct the student
> > > > which implemented the TF backend did not find an equivalent padding
> > > > method in TF, thats why it uses different paddings.
> > > >
> > > Yes, I think clamp_to_edge is a good method to keep the output with the
> > same size as input. However, I don't think "VALID" is the best method giving
> > best possible result. So, for "VALID" mode, maybe we can use the
> > clamp_to_edge method in the current dnn native mode? And then, we should
> > also add "SAME" option to support other filters.
> 
> @xwmeng, We now can support 3 padding options for conv layer.
> - valid:  the same definition as TF model. (the size of feature map changed).
> - same:  the same definition as TF model. (zero filled for outside pixels).
> - same_clamp_to_edge: just like 'same' option, but edge filled for outside pixels. So we don't need change SR code for the padding.

I think as long as we add different padding options, we need to change the SR generate code. 
> 
> > >
> > 
> > I think it is best to not make any assumptions like VALID =>
> > clamp_to_edge, but you can keep it for now.
> > Ideally the model should have a padding layer which the backend
> > properly implements. Currently the TF backend when reading a native
> > model adds this padding layer implicitly, therefore it would be a
> > matter of changing it to have an explicity padding layer in the model.
> > 
> > Maybe you can assume VALID => clamp_to_edge, so you can add what you
> > need without changing the SR code and later you implement the
> > explicity padding support and send a PR to the original repo
> > (https://github.com/HighVoltageRocknRoll/sr) properly modifying the
> > model.
> 
> yes, the ideal solution to make native SR model the same as TF model, is to:
> - add explicit padding layer in SR native model. (the padding layer is explicitly added in TF model)
> - export 'valid' option for conv layer in SR native model. (the TF model uses 'valid' option)
> - add padding layer execution support in native mode.
> - add padding option support in conv layer in native mode.
> 
> And, we can first keep most of SR code unchanged, by adding 'same_clamp_to_edge' option in conv layer
> in native mode. This can be our current choice. And the ideal solution can be our next choice, just as you mentioned.
> 
> > _______________________________________________
> > ffmpeg-devel mailing list
> > ffmpeg-devel at ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> > 
> > To unsubscribe, visit link above, or email
> > ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".


More information about the ffmpeg-devel mailing list