[FFmpeg-trac] #10586(avfilter:new): Impossible to retrieve libvmaf result from AVFrame at the output of filter graph

[FFmpeg]

#10586: Impossible to retrieve libvmaf result from AVFrame at the output of filter
graph
-------------------------------------+-------------------------------------
             Reporter:  ichlubna     |                    Owner:  (none)
                 Type:  defect       |                   Status:  new
             Priority:  normal       |                Component:  avfilter
              Version:  git-master   |               Resolution:
             Keywords:  libvmaf      |               Blocked By:
  frame filter libav psnr            |
             Blocking:               |  Reproduced by developer:  0
Analyzed by developer:  0            |
-------------------------------------+-------------------------------------
Description changed by ichlubna:

Old description:

> There is no way to retrieve vmaf score from the resulting frame produced
> at the end of the filter graph. The same approach which works for psnr
> and ssim filters is not working.
>
> When I use [https://ffmpeg.org/ffmpeg-filters.html#psnr psnr] or
> [https://ffmpeg.org/ffmpeg-filters.html#psnr ssim] filter I can list the
> result from the [https://ffmpeg.org/doxygen/4.1/structAVFrame.html frame]
> read from the sink buffer of the filter graph which is stored in the
> metadata [https://ffmpeg.org/doxygen/4.1/structAVDictionary.html
> dictionary] as:
> {{{
> char *entries = nullptr;
> av_dict_get_string(resultFrame->metadata, &entries, ',', '|');
> std::cout << entries;
> }}}
>
> I use this filter:
> {{{
> const AVFilter *vmafFilter  = avfilter_get_by_name("libvmaf");
> }}}
>
> The same cannot be used for [https://ffmpeg.org/ffmpeg-
> filters.html#libvmaf libvmaf]. The metadata dictionary is empty. I
> searched the whole frame structure and couldn't find the result anywhere.
> Shouldn't this be unified?
>
> I don't see the storing of the result in the frame anywhere in the
> [https://github.com/FFmpeg/FFmpeg/blob/13a3e2a9b4697a1403d3e58eb5e8ef9c72561d54/libavfilter/vf_libvmaf.c
> source code]. Storing to the frame is visible in the
> [https://github.com/FFmpeg/FFmpeg/blob/13a3e2a9b4697a1403d3e58eb5e8ef9c72561d54/libavfilter/vf_psnr.c#L223
> code of psnr] and
> [https://github.com/FFmpeg/FFmpeg/blob/13a3e2a9b4697a1403d3e58eb5e8ef9c72561d54/libavfilter/vf_ssim.c#L388
> code of ssim]. I think that the storing of the vmaf score is just missing
> in the code.
>
> The only workaround is to call the
> [https://github.com/FFmpeg/FFmpeg/blob/9e531370b3da27aa72703b9bb40cdd697e88607e/libavfilter/vf_libvmaf.c#L654
> uninit] function of the vmaf filter which triggers the log message with
> vmaf score (having set the log level to AV_LOG_INFO) but this is not
> usable for per-frame vmaf result which would be similar to psnr and ssim.
> It would require a new construction of the filters in the graph after
> processing each frame.
>
> I think that something like this this is missing in the
> [https://github.com/FFmpeg/FFmpeg/blob/9e531370b3da27aa72703b9bb40cdd697e88607e/libavfilter/vf_libvmaf.c#L142
> do_vmaf] function:
>
> {{{
> double vmaf_score;
> vmaf_score_pooled(s->vmaf, s->model[0], pool_method_map(s->pool),
> &vmaf_score, 0, s->frame_cnt - 1);
> set_meta(&dist->metadata, "lavfi.vmaf", 0, vmaf_score);
> }}}

New description:

 There is no way to retrieve vmaf score from the resulting frame produced
 at the end of the filter graph. The same approach which works for psnr and
 ssim filters is not working.

 When I use [https://ffmpeg.org/ffmpeg-filters.html#psnr psnr] or
 [https://ffmpeg.org/ffmpeg-filters.html#psnr ssim] filter I can list the
 result from the [https://ffmpeg.org/doxygen/4.1/structAVFrame.html frame]
 read from the sink buffer of the filter graph which is stored in the
 metadata [https://ffmpeg.org/doxygen/4.1/structAVDictionary.html
 dictionary] as:
 {{{
 char *entries = nullptr;
 av_dict_get_string(resultFrame->metadata, &entries, ',', '|');
 std::cout << entries;
 }}}

 I use this filter:
 {{{
 const AVFilter *vmafFilter  = avfilter_get_by_name("libvmaf");
 }}}

 The same cannot be used for [https://ffmpeg.org/ffmpeg-
 filters.html#libvmaf libvmaf]. The metadata dictionary is empty. I
 searched the whole frame structure and couldn't find the result anywhere.
 Shouldn't this be unified?

 I don't see the storing of the result in the frame anywhere in the
 [https://github.com/FFmpeg/FFmpeg/blob/13a3e2a9b4697a1403d3e58eb5e8ef9c72561d54/libavfilter/vf_libvmaf.c
 source code]. Storing to the frame is visible in the
 [https://github.com/FFmpeg/FFmpeg/blob/13a3e2a9b4697a1403d3e58eb5e8ef9c72561d54/libavfilter/vf_psnr.c#L223
 code of psnr] and
 [https://github.com/FFmpeg/FFmpeg/blob/13a3e2a9b4697a1403d3e58eb5e8ef9c72561d54/libavfilter/vf_ssim.c#L388
 code of ssim]. I think that the storing of the vmaf score is just missing
 in the code.

 The only workaround is to call the
 [https://github.com/FFmpeg/FFmpeg/blob/9e531370b3da27aa72703b9bb40cdd697e88607e/libavfilter/vf_libvmaf.c#L654
 uninit] function of the vmaf filter which triggers the log message with
 vmaf score (having set the log level to AV_LOG_INFO) but this is not
 usable for per-frame vmaf result which would be similar to psnr and ssim.
 It would require a new construction of the filters in the graph after
 processing each frame.

 I think that something like this this is missing in the
 [https://github.com/FFmpeg/FFmpeg/blob/9e531370b3da27aa72703b9bb40cdd697e88607e/libavfilter/vf_libvmaf.c#L142
 do_vmaf] function:

 {{{
 double vmaf_score;
 vmaf_score_pooled(s->vmaf, s->model[0], pool_method_map(s->pool),
 &vmaf_score, 0, s->frame_cnt - 1);
 set_meta(&dist->metadata, "lavfi.vmaf", 0, vmaf_score);
 }}}

 I have tried doing so but got this error with increasing index for each
 frame:

 {{{
 libvmaf ERROR vmaf_predict_score_at_index(): no feature '��%�}U' at index
 1
 }}}

--
-- 
Ticket URL: <https://trac.ffmpeg.org/ticket/10586#comment:5>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker