[FFmpeg-devel] FIX: HLS Discontinuity / Non-Monotonous DTS

[wm4]

On Fri, 16 Mar 2018 09:58:17 -0400
Joe Koberg <joe at koberg.us> wrote:

> Hello,
> 
> This patch adds code to track and correct timestamp discontinuities, fixing
> "non-monotonous dts" errors and timing issues with HLS playlists.
> 
> For some time, FFmpeg has not properly handled discontinuous timestamps in
> the MPEG-TS stream from Apple HLS playlists. Symptoms include inability to
> remux to containers like MP4 or stream to RTMP endpoints; choppy, broken,
> or sped-up playback; and voluminous "Non-monotonous DTS", "went backwards",
> and "out of order" messages logged.  This is particularly troublesome in
> the context of HLS, as the format allows insertion of independently-sourced
> fragments for ad-insertion or to concatenate ongoing video to a stream.
> 
> Over the years, many users have encountered and reported this issue in
> different ways.
> 
>   * https://trac.ffmpeg.org/ticket/5419 "HLS EXT-X-DISCONTINUITY tag is not
> supported"
>   * https://trac.ffmpeg.org/ticket/6613
>   * https://trac.ffmpeg.org/ticket/6810
>   * https://trac.ffmpeg.org/ticket/6709
>   * https://trac.ffmpeg.org/ticket/6365
>   * https://trac.ffmpeg.org/ticket/5236
>   * https://github.com/Bilibili/ijkplayer/issues/2565
>   * https://stackoverflow.com/questions/49289394/downloading-ts-stream-with-
> ext-x-discontinuity-sequence-ffmpeg
>   * etc...
> 
> 
> It appears that the -dts_delta_threshold checking code is quite similar in
> intent, and adjacent to this code, but the HLS demuxer lacks the
> AVFMT_TS_DISCONT flag, effectively disabling that option.  Even when
> AVFMT_TS_DISCONT is set on the input format (e.g. MPEG-TS),
> "-dts_delta_threshold 0" produces out-of-sync audio in our tests.  That
> code has been left as-is for now.  The flag probably should be set on the
> HLS format, though.
> 
> In this fix, any backwards DTS jump is considered an error, as is any
> forward jump of DTS or PTS by more than `dts_monotonicity_threshold`
> microseconds (really, AV_TIME_BASE units, defaulting to 1000000 = 1
> second). The delta from the predicted value is updated and applied to all
> subsequent packets, with the assumption that video was stitched together
> from continuous runs.
> 
> With this patch in place and the -force_dts_monotonicity flag set, we can
> successfuly transcode arbitrary HLS playlists to MP4, FLV, or other
> timestamped containers when we could not before.  The playback presentation
> of the result is smooth, properly timed, and in-sync in Quicktime Player,
> VLC, and ffplay with correctly reported and the file seekable.
> 
> Interaction with flags such as -copyts, -copytb, and others has not been
> fully analyzed.
> 
> We have prepared a test HLS playlist that reveals the issue.
> 
> How to reproduce:
> 
>   ffmpeg -i https://s3.amazonaws.com/playon-test-videos/discont_
> test_new/discont_test.m3u8 -c copy -y output.mp4
> 
> With patch:
> 
>   ffmpeg -force_dts_monotonicity -i https://s3.amazonaws.com/
> playon-test-videos/discont_test_new/discont_test.m3u8 -c copy -y output.mp4
> 
> 
> Please let me know if I've missed anything!

So I haven't looked too closely at this. It's possible that ffmpeg.c
should handle discontinuities better, or has a regression with handling
it. But besides that, shouldn't the HLS demuxer just be changed to
parse the discontinuity markers, and add them as packet flags? Then
these "clever" heuristics (which are just really hard to maintain)
wouldn't be needed, and also applications using libavformat as library
would benefit from it.

Your patch may or may not have a benefit on top of it (for normal .ts
files etc.), and the HLS demuxer may or may not have additional issues,
but I think that's orthogonal.

There isn't such a discontinuity packet flag - that would have to be
added.