Unresolved concatenation and subtitle problems
Investigation of unresolved concatenation and subtitle problems. Two scripts, one running '-vf showinfo', the other, '-show_streams'. echo 1.1 00305 M2TS SHOWINFO - h:\BDMV\STREAM\00305.m2ts set SOURCE=h:\BDMV\STREAM\00305.m2ts set PREP=-report -analyzeduration 5772725000 -probesize 15091101696 set ARGS=-map 0 -vf showinfo -c:v rawvideo -sn -an -f null c:\void.mkv set TARGET=e:\00305.m2ts_showinfo.txt ffmpeg %PREP% -i %SOURCE% %ARGS% 2>%TARGET% echo 1.2 00305 M2TS SHOW STREAMS - h:\BDMV\STREAM\00305.m2ts set SOURCE=h:\BDMV\STREAM\00305.m2ts set PREP=-analyzeduration 5772725000 -probesize 15091101696 set ARGS=-show_streams set TARGET=e:\00305.m2ts_streams.txt ffprobe %PREP% -i %SOURCE% %ARGS% >%TARGET% Both of them provoke "Failed to allocate buffers for seekback" message, and 165 "DTS discontinuity in stream 2" (the subs) messages. packet 12 with DTS 12859568, packet 13 with DTS 12970433 packet 20 with DTS 13302507, packet 21 with DTS 13417133 packet 28 with DTS 14965402, packet 29 with DTS 15080059 : packet 6026 with DTS 495381711, packet 6027 with DTS 495687565 COMPARING WHAT MPV SHOWS VERSUS 'showinfo' VERSUS 'show_streams' MPV first = 0 MPV total = 1:36:12.725291[6..] = 5772.725291[6..] seconds MPV final = 5772.725291[6..]-1.001/24 = 5772.68358[3..] seconds MPV 0..519541522.4[9..] ticks --> 0..138405.[9..] frames : : showinfo [note1] 0..519541522 0..138406 show_streams 1048560 = video start_pts 1048560 = audio start_pts 7685190 = subs start_pts [note2] (PREP shown above) 1048560 = subs start_pts [note3] (no PREP) [note1] First & final '-vf showinfo' listings: [Parsed_showinfo_0 @ 00000000d91a1f00] n: 0 pts: 0 -snip- [Parsed_showinfo_0 @ 00000000d91a1f00] n:138406 pts:519541522 -snip- [note2] 7685190 ticks 'says' 1st subtitle is at 1:13.740 - confirmed. [note3] 1048560 ticks 'says' 1st subtitle is at 0:00.000 - incorrect. Q1: Why do 'ffmpeg -vf showinfo' and 'ffprobe -show_streams' differ? Q2: Is '-analyzeduration 5772725000 -probesize 15091101696' catching all? Q3: What is the effect of 'no PREP' on concatination? If you have questions, kindly ask. I have the results saved, and I can explain the "COMPARING" section. Thank you -- Mark. (stopping now to get some sleep).
Maybe that was too much information. I'll try again. When ffmpeg's '-vf showinfo' and ffprobe's '-show_streams' disagree, which should I trust?
On 12/26/23 17:30, Mark Filipak wrote:
Maybe that was too much information. I'll try again.
When ffmpeg's '-vf showinfo' and ffprobe's '-show_streams' disagree, which should I trust?
How about this? I'm trying to remux-join all 4 episodes of Bergman's 5-hour "FANNY AND ALEXANDER". I want to cut the intros of 2 3 & 4 and the credits of 1 2 & 3 so it's continuous. I first tried it via an "ffconcat version 1.0" text that did all the remuxing-trimming-joining. I made bean soup. Not only was the join messed up, the episodes behaved like they'd been put through a temporal blender. I tried MKVToolNix and made a slightly different flavor of bean soup, one that had no ending PTS so running time was listed as over 5000 hours. Yikes! Obviously, something about the packet PTSs or how they were handled was not right. Now I'm trying to trim-remux one at a time, then craft smoothed & shifted PTSs, then join them via a simple "ffconcat version 1.0" text. Combining segments 1 & 2, all is Joy except the join. The join is 2 seconds of 5 stop-start 'stutters' showing 3 flashes of the opening frame of segment 2. Rather than add a few black frames to the beginning of segment 2, I'm actually trying to fix things. I thought I'd made it so that segment 1 would join seamlessly with segment 2 without ffmpeg having to do anything. It appears the packets are out of order. But so what? FFmpeg can handle that, can't it? Then I discovered that ffmpeg's '-vf showinfo' and ffprobe's '-show_streams' disagree on the starting PTSs for video. How about the audio, and subs? They appear to be okay: No sync issues. Is the 2 seconds of 'stutters' and flashes taken into account in the audio & sub PTSs? Apparently, yes. Will they be in sync if I overcome the 'stutters' and flashes? Who knows? If the 2 seconds is real, then probably. If the 2 seconds is bogus, then maybe not. But first, I need to find what the 'true' start_pts is. It's been my experience that ffmpeg may be right, or ffprobe may be right, or they may both be wrong. If the sources were VOBs, then I could look in the packet headers and find out. But they're M2TSs. Have YOU been through this? Do you have any experience/advice to share? I'll send you a mental hug if you do. I'm getting frazzled. -- Mark.
When ffmpeg's '-vf showinfo' and ffprobe's '-show_streams' disagree, which should I trust?
You should usually trust both, but their meanings are different. showinfo is a filter, so apply to a running filter graph, I don't think this information could be very helpful for your use case. show_streams will give you some start_time/duration using the same core components (avformat_find_stream_info) as the concat demuxer, which actually can help. Another option is to use a kind of "plain raw dump" and guess start_time/duration by yourself. This can be achieved with an ffmpeg -copyts -c copy -f framecrc - Note that in some cases, it might happen that the real "first pts" comes before the ff reported start_time. And in that case, you will see mpv (based on ff) play your file at a point that is earlier than the start_time reported by ff. So, be careful, these start_time/duration things are often much trickier than they look like. It seems you want to join segments without reencoding. First, you have to take care of the available features of your output muxer : an mp4 is more powerful to handle stream discontinuities than an mpegts format. And there are two things to consider: - "timestamp joining" : care about start/end points; but also, if your input audio/video is not aligned/locked (any usual compressed audio codec), you will have to reencode audio at least if you want to avoid a hole (or overlap) in the timeline. - keyframes : open-gop is not editable with a straight stream copy if you don't allow some overlap; and you would certainly experience some difficulties to handle this overlap in a muxer So many things, very dependent on your exact files and workflow. At the end, if you encounter an issue, try to reproduce it on short segments to document a case in trac, with a very exhaustive report and corresponding media samples. Nicolas
On 12/28/23 07:47, Nicolas Gaullier wrote:
When ffmpeg's '-vf showinfo' and ffprobe's '-show_streams' disagree, which should I trust?
You should usually trust both, but their meanings are different. showinfo is a filter, so apply to a running filter graph, I don't think this information could be very helpful for your use case. show_streams will give you some start_time/duration using the same core components (avformat_find_stream_info) as the concat demuxer, which actually can help. Another option is to use a kind of "plain raw dump" and guess start_time/duration by yourself. This can be achieved with an ffmpeg -copyts -c copy -f framecrc - -continued below-
Thank you, Nicolas. I ran this: <windows script> set SOURCE=h:\BDMV\STREAM\00306.m2ts set TARGET=e:\00306.m2ts_framecrc.txt ffmpeg -i %SOURCE% -copyts -c copy -f framecrc %TARGET% </windows script> Here's 00306.m2ts, packets 0..16, as reported by '-framecrc': stream number ('0' is the video stream, '1' is the audio stream) P ¦ DTS(P) PTS(P) T(P) B(P)/8 CRC(P) unknown -- ¦ ------- ------- ---- ------ ---------- ------------- 0: 0, 1044806, 1048560, 3753, 178583, 0x34cc62a3, S=1, 1 1: 0, 1048560, 1056067, 3753, 92709, 0xfa7b5edb, F=0x0, S=1, 1 2: 1, 1048560, 1048560, 960, 1084, 0xe54d2b6f, S=1, 1 3: 1, 1049520, 1049520, 960, 1084, 0xe54d2b6f, S=1, 1 4: 1, 1050480, 1050480, 960, 1084, 0xe54d2b6f, S=1, 1 5: 1, 1051440, 1051440, 960, 1084, 0xe54d2b6f, S=1, 1 6: 0, 1052313, 1052313, 3753, 145206, 0x10576cdb, F=0x0, S=1, 1 7: 1, 1052400, 1052400, 960, 1084, 0xe54d2b6f, S=1, 1 8: 1, 1053360, 1053360, 960, 1084, 0xe54d2b6f, S=1, 1 9: 1, 1054320, 1054320, 960, 1084, 0xe54d2b6f, S=1, 1 10: 1, 1055280, 1055280, 960, 1084, 0xe54d2b6f, S=1, 1 11: 0, 1056067, 1063575, 3753, 141288, 0xb70f376e, F=0x0, S=1, 1 12: 1, 1063920, 1063920, 960, 1084, 0xe54d2b6f, S=1, 1 13: 1, 1064880, 1064880, 960, 1084, 0xe54d2b6f, S=1, 1 14: 1, 1065840, 1065840, 960, 1084, 0xe54d2b6f, S=1, 1 15: 1, 1066800, 1066800, 960, 1084, 0xe54d2b6f, S=1, 1 16: 0, 1067328, 1067328, 3753, 146136, 0xc75a68fb, F=0x0, S=1, 1 T(P) for stream 0 is reported erroneously: reported T(P) actual T(P) N(P) P0 3753 3754 1 P1 3753 3753 1 P6 3753 3754 1 P11 3753 11261 3 So '-framecrc' & '-showinfo' agree but they disagree with '-show_streams'. They may agree because they 'drink the same beer'. This is all indeterminate without the ability to parse actual m2ts headers. That is where the truth lies.
Note that in some cases, it might happen that the real "first pts" comes before the ff reported start_time. And in that case, you will see mpv (based on ff) play your file at a point that is earlier than the start_time reported by ff.
I've seen that.
So, be careful, these start_time/duration things are often much trickier than they look like.
It seems you want to join segments without reencoding. First, you have to take care of the available features of your output muxer : an mp4 is more powerful to handle stream discontinuities than an mpegts format.
Umm... (No, I'm not going to comment on why that might be at this time).
And there are two things to consider: - "timestamp joining" : care about start/end points;
I cut and trim solely on I-frames, preserving (doubling) the I-frame on both sides of a cut.
but also, if your input audio/video is not aligned/locked (any usual compressed audio codec), you will have to reencode audio at least if you want to avoid a hole (or overlap) in the timeline.
So video packs and audio packs are not correlated, eh? Too bad.
- keyframes : open-gop is not editable with a straight stream copy if you don't allow some overlap; and you would certainly experience some difficulties to handle this overlap in a muxer
So many things, very dependent on your exact files and workflow. At the end, if you encounter an issue, try to reproduce it on short segments to document a case in trac, with a very exhaustive report and corresponding media samples.
Will do. ----- I have designed hundreds of hardware finite state machines (FSM). Every one of them worked exactly as designed, zero bugs. I was taught by Dr. John Battocletti at Ohio State University. FSMs assure 100% coverage and 100% testability. I have taught FSM methods to codesmiths in Silicon Valley. The 'C' language has all the elements needed for FSM design. I will help with FSM design in 'C' for free with anyone who asks. OO FSM (object-oriented FSM) would be perfect for AV processing. (The MPEG specifications utilize an informal form of FSM.) I've been casually working on OOFFmepg for about a year. OOFFmpeg could be implemented in 'C' using existing FFmpeg libraries. Based on my experiences, I think OOFFmpeg could be made fairly quickly. ----- Cheers -- Mark.
On 12/28/23 07:47, Nicolas Gaullier wrote:
When ffmpeg's '-vf showinfo' and ffprobe's '-show_streams' disagree, which should I trust?
You should usually trust both, but their meanings are different.
You should trust neither. I have more info that includes corrections to what I previously wrote -- gad, this is confusing and error prone work. COMPUTE (exact) MPV <------1:18:19.236208[3..]-----> end to end 0..4699.1945 seconds, first..last 0..112668 frames, computed from seconds : SHOWINFO 0..112668 frames 0..422927505 ticks <-----------+ FRAMECRC 1044806..423972311 ticks vid ¦ :\________\__ 422927505 <------+ 1048560..423979440 ticks aud :\________\__ 422930880 STREAMS 1048560..? vid start_pts <- not in showinfo, not vid PTS 1048560..? aud start_pts \ 3143152..? sub start_pts \_\__ 1st sub at 0:23.273 - confirmed. if -framecrc is right, then -showinfo "n: 0 pts: 0" <- wrong -- no equivocation, wrong -show_streams vid "start_pts=1048560" <- wrong -- no equivocation, wrong I previously wrote, "So '-framecrc' & '-showinfo' agree but they disagree with '-show_streams'." That was erroneous. They all disagree with each other. Apparently, ffmpeg/ffprobe should be fixed so that my chart would then be: MPV <------1:18:19.236208[3..]-----> end to end 0..4699.1945 seconds, first..last 0..112668 frames, computed from seconds : SHOWINFO 0..112668 frames 1044806..423972311 ticks <== when fixed FRAMECRC 1044806..423972311 ticks vid :\________\__ 422927505 1048560..423979440 ticks aud :\________\__ 422930880 STREAMS 1044806..? vid start_pts <== when fixed 1048560..? aud start_pts \ 3143152..? sub start_pts \_\__ 1st sub at 0:23.273 - confirmed. But is '-framecrc' actually correct? I will use the '-framecrc' numbers to do the DTS-PTS shifting and packet concatenations and will report back. -- Mark.
The FFmpeg project is using PTSs to report relative measurements. DTS & PTS are in PES headers. They are metadata, NOT measurements. Please do not use DTSs/PTSs as metrics. Use "ticks", instead. For example, PTS=2854113..504223732 should be reported as "501369619 ticks", not as PTS=0..501369619. Thank you. -- Mark.
The saga continues... I'm trimming h:\BDMV\STREAM\00305.m2ts, making e:\00305.mkv FFmpeg appears to be adding soundless audio frames before the 1st video frame. Concatenating e:\00306.mkv, made the same way, produces stutter & flashing at the join. This report concentrates on the '-ss' trim of e:\00305.mkv. About 00305.m2ts: The 1st video packet is DTS=1044806, PTS=1048560. +---------+ ... DTS precedes PTS by ¦ 1 frame ¦ <== remember this. +---------+ --------------------------------+ Now, I want to trim to ¦ 20.061708[3..]..1:33:10.83525 ¦ exactly. +-------------------------------+ +-------------------------------+ Therefore, I want ¦ PTS=1805553.75..503175172.5 ¦ exactly. +-------------------------------+ (At least, I think I do.) '-showinfo' says this: "n: 481 pts: 1805553 ... type:I". +-------------------------------+ Therefore, I want frames ¦ 481..134046 ¦ exactly. +-------------------------------+ But is frame 481 actually at PTS=1805553? No. Why do I say that? Because '-framecrc' says this: +-----------------------------------------------+ ¦ 1, 1805040, 1805040, 960, 1084 ¦ ¦ 1, 1806000, 1806000, 960, 1084 ¦ +-----------------------------------------------+ There is no packet with PTS=1805553 -- doesn't exist. So, what if I ignore that and just do this: <windows script> set SOURCE=h:\BDMV\STREAM\00305.m2ts set TRIM=-ss 20.061708333333333333333333333333 -to 1:33:10.83525 set ARGS=-map 0 -c copy -dn set TARGET=e:\00305.mkv ffmpeg -report %TRIM% -i %SOURCE% %ARGS% %TARGET% </windows script> MPV plays it but shows 0:00.616 as the start time. '-framecrc' shows 43 (presumably soundless) audio packets (DTS=0..448, PTS=0..448) preceding the 1st video packet (DTS=449, PTS=616). +----------+ ... DTS precedes PTS by ¦ 4 frames ¦ +----------+ The same thing probably happens at the '-to' trim point of 00305.m2ts, and it does happen at the '-ss' trim point of 00306.m2ts, so when I concat 00305.mkv & 00306.mkv, the join is messed up. For 2 seconds, MPV stutters 5 times and flashes the first image frame of 00306.mkv 3 times: stop-pause-go- -stop-pause-go- -stop-flash-pause-go- -stop-flash-pause-go- -stop-flash-pause-go. Then MPV continues as though nothing had happened. Each of the 5 stutters are like micro-movies in which running time stops but frames 'run'. Audio & subs flow though all that without a hitch. That tells me that audio packets are actually 'running the show'. What '-showinfo' is calling "PTS" is not PTS, it's ticks. Ticks relative to the 1st video packet, but not to _all_ packets. Since '-showinfo' lies about PTSs, I can't know 'N'. And 'framecrc' doesn't show 'N's, so I'm in the dark regarding the true identity of the frame and its true PTS. I also created e:\00305DTS-STARTDTS÷90.mkv <windows script> set SOURCE=h:\BDMV\STREAM\00305.m2ts set TRIM=-ss 20.061708333333333333333333333333 -to 1:33:10.83525 set BSFV=-bsf:v setts=dts=DTS-STARTDTS/90:pts=PTS-STARTPTS/90 set BSFA=-bsf:a setts=dts=DTS-STARTDTS/90:pts=PTS-STARTPTS/90 set ARGS=-map 0 %BSFV% %BSFA% -c copy -dn set TARGET=e:\00305_DTS-STARTDTS÷90.mkv ffmpeg -report %TRIM% -i %SOURCE% %ARGS% %TARGET% </windows script> and got similar results. I'm like you. I just want perfect video. I'll really appreciate help. -- Mark.
This is 00305+00306.mkv_framecrc.txt. It is headed by this analysis of the concatenation of 00305.mkv & 00306.mkv. 00305+00306.mkv contains either 241520 or 241552 or 241554 frames. The differences cannot be explained away as rounding differences. 00305+00306.mkv has a span of 1188892 packets: • 944567 audio packets PTS=0..10075411. 48000 samp-per-sec, 483618304 samples, 512 samp-per-pack. • 2782 subtitle packets PTS=54295..10073292. • 241543 video packets PTS=617..10075378. • That's 617.5±0.4[9..]..10075378.5±0.4[9..] = 10074761±0.9[9..] video ticks, exactly. = 241552.[711288..]±0.[023976..] frames, exactly. • '-showinfo' reports N=0..241519 = 241520 frames. • MPV shows 0.617..2:47:55.460 end to end = 0.6175±0.0004[9..]..2:47:55.4605±0.0004[9..] = 10074843.0±0.9[9..] ticks, exactly. = 241554.[677322..]±0.[023976..] frames, exactly. So, are there 241520 frames, as '-showinfo' claims, or 241552 frames, as '-framecrc' claims, or 241554 frames, as MPV shows? I favor 241552 as '-framecrc' claims for no particular reason. I will post 00305+00306.mkv_framecrc.txt on request. It is 70 MB. I will post 00305+00306.mkv_showinfo.txt on request. It is 124 MB. No, I cannot assume '-framecrc' is correct. In the face of inconsistent FFmpeg tools, I'm giving up. -- Mark.
One last note. The concatenation is I-frame..I-frame+I-frame..I-frame. So I should find a pair of back-to-back I-frames. Not found. Instead there appears to be a few hundred mixed up packets (I think ...can't be sure due to the unreliability-primitive nature of the tools). I remind the reader about the 5 stutters and 3 flashes at the join. I previously wrote that the stutter duration is 2 seconds. It is not. It is approximately 6 seconds. I made clean trims, but the concatenation is flawed. -- Mark.
participants (2)
-
Mark Filipak -
Nicolas Gaullier