[FFmpeg-trac] #6006(avfilter:new): FPS filter not selecting right frames from input

[FFmpeg]

#6006: FPS filter not selecting right frames from input
----------------------------------+--------------------------------------
             Reporter:  Misaki    |                     Type:  defect
               Status:  new       |                 Priority:  normal
            Component:  avfilter  |                  Version:  git-master
             Keywords:            |               Blocked By:
             Blocking:            |  Reproduced by developer:  0
Analyzed by developer:  0         |
----------------------------------+--------------------------------------
 I am seeing the FPS choose the wrong frames. YouTube videos sometimes
 demonstrate this problem in both mp4 (with x264) and webm (with vp9)
 formats, usually only one of the formats instead of both. This can happen
 when the input video file is 60 fps, and the user is looking at a 30 fps
 version. The usual pattern is that every 3 frames, one of them is wrong.
 If the source video actually had a duplicate frame every other frame, then
 when this happens, one out of three frames will be a duplicate of one of
 the other two, and a frame that was doubled in the 60 fps video is missing
 entirely from the 30 fps one.

 I thought this was because YouTube was using an fps filter that selected
 the 'next' frame or something, because in recent years it also started
 handling variable-fps videos incorrectly. But I just found it, and was
 able to replicate it, in ffmpeg.

 The manual says out of the different frame selection methods, the fps
 filter chooses the nearest frame.

 It's safe to say I don't understand exactly why this is happening. My
 first attempt to replicate it failed.

 This command replicates it:

  ffmpeg -filter_complex
 testsrc=rate=30,settb=1/1000,setpts=PTS+18,showinfo,fps=15,showinfo -t 0.2
 -hide_banner -f null -


 So these time values were used just because they're very similar to the
 video where I encountered this problem.

 Output from showinfo filters:

  [Parsed_showinfo_3 @ 0x3c15980] n:   0 pts:     18 pts_time:0.018   pos:
 -1 fmt:rgb24 sar:1/1 s:320x240 i:P iskey:1 type:I checksum:88C4D19A
 plane_checksum:[88C4D19A] mean:[127] stdev:[125.7]
  [Parsed_showinfo_3 @ 0x3c15980] n:   1 pts:     51 pts_time:0.051   pos:
 -1 fmt:rgb24 sar:1/1 s:320x240 i:P iskey:1 type:I checksum:FD48FF60
 plane_checksum:[FD48FF60] mean:[127] stdev:[125.7]
  [Parsed_showinfo_3 @ 0x3c15980] n:   2 pts:     85 pts_time:0.085   pos:
 -1 fmt:rgb24 sar:1/1 s:320x240 i:P iskey:1 type:I checksum:5BBC2F63
 plane_checksum:[5BBC2F63] mean:[127] stdev:[125.7]
  [Parsed_showinfo_5 @ 0x3c16d80] n:   0 pts:      0 pts_time:0       pos:
 -1 fmt:rgb24 sar:1/1 s:320x240 i:P iskey:1 type:I checksum:FD48FF60
 plane_checksum:[FD48FF60] mean:[127] stdev:[125.7]
  [Parsed_showinfo_3 @ 0x3c15980] n:   3 pts:    118 pts_time:0.118   pos:
 -1 fmt:rgb24 sar:1/1 s:320x240 i:P iskey:1 type:I checksum:39A855A7
 plane_checksum:[39A855A7] mean:[127] stdev:[125.7]
  [Parsed_showinfo_5 @ 0x3c16d80] n:   1 pts:      1 pts_time:0.0666667
 pos:       -1 fmt:rgb24 sar:1/1 s:320x240 i:P iskey:1 type:I
 checksum:5BBC2F63 plane_checksum:[5BBC2F63] mean:[127] stdev:[125.7]
  [Parsed_showinfo_3 @ 0x3c15980] n:   4 pts:    151 pts_time:0.151   pos:
 -1 fmt:rgb24 sar:1/1 s:320x240 i:P iskey:1 type:I checksum:6B3D7C29
 plane_checksum:[6B3D7C29] mean:[128] stdev:[125.7]
  [Parsed_showinfo_3 @ 0x3c15980] n:   5 pts:    185 pts_time:0.185   pos:
 -1 fmt:rgb24 sar:1/1 s:320x240 i:P iskey:1 type:I checksum:7E969AAD
 plane_checksum:[7E969AAD] mean:[128] stdev:[125.7]
  [Parsed_showinfo_5 @ 0x3c16d80] n:   2 pts:      2 pts_time:0.133333 pos:
 -1 fmt:rgb24 sar:1/1 s:320x240 i:P iskey:1 type:I checksum:6B3D7C29
 plane_checksum:[6B3D7C29] mean:[128] stdev:[125.7]
  [Parsed_showinfo_3 @ 0x3c15980] n:   6 pts:    218 pts_time:0.218   pos:
 -1 fmt:rgb24 sar:1/1 s:320x240 i:P iskey:1 type:I checksum:0930B896
 plane_checksum:[0930B896] mean:[128] stdev:[125.6]
  [Parsed_showinfo_3 @ 0x3c15980] n:   7 pts:    251 pts_time:0.251   pos:
 -1 fmt:rgb24 sar:1/1 s:320x240 i:P iskey:1 type:I checksum:CED6CEBF
 plane_checksum:[CED6CEBF] mean:[128] stdev:[125.6]
  [Parsed_showinfo_3 @ 0x3c15980] n:   8 pts:    285 pts_time:0.285   pos:
 -1 fmt:rgb24 sar:1/1 s:320x240 i:P iskey:1 type:I checksum:3F25E3D1
 plane_checksum:[3F25E3D1] mean:[128] stdev:[125.6]
  [Parsed_showinfo_5 @ 0x3c16d80] n:   3 pts:      3 pts_time:0.2     pos:
 -1 fmt:rgb24 sar:1/1 s:320x240 i:P iskey:1 type:I checksum:CED6CEBF
 plane_checksum:[CED6CEBF] mean:[128] stdev:[125.6]

 The result is slightly different from the original file I found this in,
 but it still shows the problem. The filter selects frames with n= 1, 2, 4,
 7, instead of every other frame.

 This is somehow related to the offset. In the original file, this is
 apparently because the audio starts sooner. Removing the offset might fix
 the problem but cause slight desync issues. If YouTube is in fact using
 ffmpeg to process input, video streams not starting at 0 could be the
 reason it's sometimes bugged.

 In the original file, I tried looking at the detailed output from -v
 trace, and finding the frame that was nearest to an interval based on the
 output framerate, but could not understand why a frame was being dropped
 when it was nearest to the interval's middle.

--
Ticket URL: <https://trac.ffmpeg.org/ticket/6006>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker