[FFmpeg-devel] [PATCH v7] Add SUP/PGS subtitle demuxer

[Petri Hintukainen]

On ti, 2014-09-30 at 16:09 +0200, wm4 wrote:
> On Tue, 30 Sep 2014 12:01:33 +0300
> Petri Hintukainen <phintuka at gmail.com> wrote:
> 
> > Hello,
> > 
> > Missing DTS combined with non-monotonic PTS seems to cause some
> > problems. This is with the file from ticket #2208 (I added dumping
> > timestamps to demuxer and muxer):
> > 
> > 
> > $ ../ffmpeg -i orig/track_06\ -\ Subtitle.sup -scodec copy 6.sup
> > ffmpeg version N-66546-gea74007 Copyright (c) 2000-2014 the FFmpeg
> > developers
> >   built on Sep 30 2014 11:10:08 with gcc 4.8 (Ubuntu 4.8.2-19ubuntu1)
> >   configuration: 
> >   libavutil      54.  7.101 / 54.  7.101
> >   libavcodec     56.  1.101 / 56.  1.101
> >   libavformat    56.  7.101 / 56.  7.101
> >   libavdevice    56.  1.100 / 56.  1.100
> >   libavfilter     5.  1.102 /  5.  1.102
> >   libswscale      3.  1.100 /  3.  1.100
> >   libswresample   1.  1.100 /  1.  1.100
> > IN  pts 00255314 dts 00000000
> > Input #0, sup, from 'orig/track_06 - Subtitle.sup':
> >   Duration: N/A, start: 2.836822, bitrate: N/A
> >     Stream #0:0: Subtitle: hdmv_pgs_subtitle, 1920x1080
> > Output #0, sup, to '6.sup':
> >   Metadata:
> >     encoder         : Lavf56.7.101
> >     Stream #0:0: Subtitle: hdmv_pgs_subtitle, 1920x1080
> > Stream mapping:
> >   Stream #0:0 -> #0:0 (copy)
> > Press [q] to stop, [?] for help
> > OUT pts 00000000 dts 00000000
> > IN  pts 00255037 dts 00000000
> > OUT pts 00000000 dts 00000000
> > IN  pts 00249205 dts 00000000
> > [sup @ 0x241f4e0] Application provided invalid, non monotonically
> > increasing dts to muxer in stream 0: -277 >= -6109
> > av_interleaved_write_frame(): Invalid argument
> > size=       0kB time=00:00:00.00 bitrate=N/A    
> > video:0kB audio:0kB subtitle:1kB other streams:0kB global headers:0kB
> > muxing overhead: unknown
> > Conversion failed!
> 
> The DTS in this file are indeed decreasing sometimes.

No, there are no DTS timestamps in this file.

> First packet:
> 
> pts=255314
> pts_time=2.836822
> dts=255314
> dts_time=2.836822
> 
> Second packet:
> 
> pts=255037
> pts_time=2.833744
> dts=255037
> dts_time=2.833744

those dts's are generated by ffmpeg, probably in mux.c
compute_pkt_fields2().
  
> > Also note that timestamps are re-positioned to start from 0. This may be
> > problematic, it requires manually synchronizing subtitles to the video.
> 
> "Rebasing" the timestamps is apparently the normal operation of
> ffmpeg.c, there's probably a way to make it stop doing that.

Yes, -copyts seems to do the trick.
I think all pts timestamps in .sup file should be relative to video
start, i.e. if video starts at pts 12345, corresponding .sup file starts
at pts = pts_of_first_subtitle - 12345.

> > Remuxing from the original .m2ts file works just well:
> > 
> > 
> > $ ffmpeg -i orig/00038.m2ts -map 0:6 -scodec copy 6.sup
> > ffmpeg version N-66546-gea74007 Copyright (c) 2000-2014 the FFmpeg developers
> >   built on Sep 30 2014 11:10:08 with gcc 4.8 (Ubuntu 4.8.2-19ubuntu1)
> >   configuration: 
> >   libavutil      54.  7.101 / 54.  7.101
> >   libavcodec     56.  1.101 / 56.  1.101
> >   libavformat    56.  7.101 / 56.  7.101
> >   libavdevice    56.  1.100 / 56.  1.100
> >   libavfilter     5.  1.102 /  5.  1.102
> >   libswscale      3.  1.100 /  3.  1.100
> >   libswresample   1.  1.100 /  1.  1.100
> > [NULL @ 0x1bf9ba0] start time for stream 6 is not set in estimate_timings_from_pts
> > [NULL @ 0x1c146e0] start time for stream 7 is not set in estimate_timings_from_pts
> > [NULL @ 0x1c151c0] start time for stream 8 is not set in estimate_timings_from_pts
> > [mpegts @ 0x1bf1ca0] Could not find codec parameters for stream 6 (Subtitle: hdmv_pgs_subtitle ([144][0][0][0] / 0x0090)): unspecified size
> > Consider increasing the value for the 'analyzeduration' and 'probesize' options
> > [mpegts @ 0x1bf1ca0] Could not find codec parameters for stream 7 (Subtitle: hdmv_pgs_subtitle ([144][0][0][0] / 0x0090)): unspecified size
> > Consider increasing the value for the 'analyzeduration' and 'probesize' options
> > [mpegts @ 0x1bf1ca0] Could not find codec parameters for stream 8 (Subtitle: hdmv_pgs_subtitle ([144][0][0][0] / 0x0090)): unspecified size
> > Consider increasing the value for the 'analyzeduration' and 'probesize' options
> > Input #0, mpegts, from 'orig/00038.m2ts':
> >   Duration: 00:00:43.90, start: 11.650667, bitrate: 35145 kb/s
> >   Program 1 
> >     Stream #0:0[0x1011]: Video: h264 (High) (HDMV / 0x564D4448), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 23.98 fps, 23.98 tbr, 90k tbn, 47.95 tbc
> >     Stream #0:1[0x1100]: Audio: truehd (AC-3 / 0x332D4341), 48000 Hz, 7.1, s32 (24 bit)
> >     Stream #0:2[0x1100]: Audio: ac3 (AC-3 / 0x332D4341), 48000 Hz, 5.1(side), fltp, 640 kb/s
> >     Stream #0:3[0x1101]: Audio: ac3 (AC-3 / 0x332D4341), 48000 Hz, stereo, fltp, 320 kb/s
> >     Stream #0:4[0x1102]: Audio: ac3 (AC-3 / 0x332D4341), 48000 Hz, stereo, fltp, 320 kb/s
> >     Stream #0:5[0x1200]: Subtitle: hdmv_pgs_subtitle ([144][0][0][0] / 0x0090), 1920x1080
> >     Stream #0:6[0x1201]: Subtitle: hdmv_pgs_subtitle ([144][0][0][0] / 0x0090)
> >     Stream #0:7[0x1202]: Subtitle: hdmv_pgs_subtitle ([144][0][0][0] / 0x0090)
> >     Stream #0:8[0x1203]: Subtitle: hdmv_pgs_subtitle ([144][0][0][0] / 0x0090)
> > Output #0, sup, to '6.sup':
> >   Metadata:
> >     encoder         : Lavf56.7.101
> >     Stream #0:0: Subtitle: hdmv_pgs_subtitle ([144][0][0][0] / 0x0090)
> > Stream mapping:
> >   Stream #0:6 -> #0:0 (copy)
> > Press [q] to stop, [?] for help
> > OUT pts 00255255 dts 00249146
> > OUT pts 00254978 dts 00249146
> > OUT pts 00249146 dts 00249146
> > OUT pts 00249699 dts 00249146
> > OUT pts 00249699 dts 00249699
> > ...
> > 
> > 
> > Also, remuxing this file again (.sup->.sup) with ffmpeg works. Well,
> > except that timestamps are again repositioned.
> > 
> > 
> > Is this something that can be handled in muxer ?
>
> Not sure what's going on here at all. Normally, DTS as stored in a file
> should never decrease, except on timestamp resets, which doesn't happen
> here.

ffmpeg uses PTS as DTS when there is no DTS. This does not work well
with non-monotonic PTS (that's why DTS should be kept in those
files ...). I think there are also some players that do not display PGS
subtitles without DTS.

Using AVFMT_NOTIMESTAMPS in .sup muxer kind of works, but incorrect DTS
is still written to output file, and there are lot of error messages. I
could probably hack the muxer to drop those invalid DTS timestamps, but
it doesn't feel the right solution (and does not work with other output
formats).

One possible solution in demuxer (or parser) could be reading all
segments until segment 0x80 is found, and using the smallest PTS value
found as DTS for every segment in display set. I'm not sure if this
would be 100% compilant, but I can try to check it.
In any case, there's no way one could guess/calculate DTS from PTS of
first segment. Some upper limit / "safety distance" for PTS-DTS
difference could possibly be found from full-screen display update, but
I don't know if that could cause overflowing graphics buffers with high
frame rate animations.

Probably the best solution would be to not output DTS when it is not
present in input (and maybe refusing to mux such files to containers
that require DTS), but I have no idea how to implement it :)


- Petri