Can't get a/v synchronisation right with mpeg transport stream that has gaps in it
Hi people, I'm having this problem for years and years (so it's not a recent problem or regression). I make recordings on my enigma2 based settopbox. It saves the recordings as the original transport stream, but with filtering of only the requested service pids. There is a feature to edit the recorded services, in a basic way. Very helpful to edit out commercials. Problem is that it makes cuts in a very "rude" way, it justs cuts the file at a certain position, in the middle of all kinds of open packets and streams. On playback it isn't such a big deal, because the damaged packets are discarded and the DVB hardware resyncs on the timestamps pretty quickly. After the crude editing, I like to perform a real editing run, using kdenlive (or any other NLVE, for that case). I've found that all of them have problems when fed with a "damaged" transport stream, like this. So what I do, is transcode the stream to something with mjpeg and pcm codecs, intra frames only, very easy on the editor. This is the problem: after every timestamp discontinuity ("cut" or "gap"), ffmpeg resyncs audio and video BUT it doesn't do it 100% correctly. Sound remains a bit "in front" of the video, skewed a bit. Also, it gets worse at every cut, everytime about 200-300 ms is added to the skew. That's not what I'd expect? I've tried these options to force synchronisation: - -vsync 1 (not per sé required, but I want the audio and video streams in the container in such a way that they're not skewed, so if the editor decides to demux and throw away the timestamps, it still works) - -r 25 (material is 25 fps, tried with and without, doesn't matter) - filter:a " aresample=async=10000:min_comp=0.1:min_hard_comp=0.1:max_soft_comp=10000:first_pts=0" None of this helps. I can see ffmpeg dropping and dupping frames, but it doesn't help to fix the slow drift. I was under the impression that ffmpeg would automatically base the synchronisation of the (single) audio stream on the (single) video stream. Is this a correct assumption or do I need to specify an explicit "-map" statement? It looks like ffmpeg takes care that frames are dupped/dropped so the frame rate keeps "solid" 25 fps (or whatever the source frame rate is specified as or is specified with -r), but it seems it does not keep the audio stream synchronised. Thanks for help, Erik.
2017-03-28 13:17 GMT+02:00 Erik Slagter <erik@slagter.name>: [...] Without reading your report that contains a lot of information but not what we absolutely need to help you: Needs -async 1
I was under the impression that ffmpeg would automatically base the synchronisation of the (single) audio stream on the (single) video stream. Is this a correct assumption or do I need to specify an explicit "-map" statement?
(After reading parts of your mail) Unfortunately, it is not that simple: Players can simply drop frames (or increase video playback speed) for "missing" audio, this doesn't work with ffmpeg (the application), silence has to be created and inserted. Carl Eugen
Hi Carl,
not what we absolutely need to help you: Needs -async 1
Have a look at this part of the command line: -filter:a " aresample=async=10000:min_comp=0.1:min_hard_comp=0.1:max_soft_comp=10000:first_pts=0" This is what -async 1 does (-async is declared "obsolete").
I was under the impression that ffmpeg would automatically base the synchronisation of the (single) audio stream on the (single) video stream. Is this a correct assumption or do I need to specify an explicit "-map" statement?
(After reading parts of your mail) Unfortunately, it is not that simple: Players can simply drop frames (or increase video playback speed) for "missing" audio, this doesn't work with ffmpeg (the application), silence has to be created and inserted.
Yeah, fine by me, I don't care. As far as I understand: - -vsync vfr just adjusts the timestamps - -vsync cfr drops/dups frames, which is fine, no need to fiddle with the audio stream - -async 1 (and variants) stretch/shrink audio samples - asresample stretch/shrink/dup/drop audio samples All of them are fine by me, although dupping/dropping video frames is preferred. With vsync=1 is do see frame dropping/dupping taking place. The problem is that after each "desync" event, gaps in the transport stream that have timestamp discontinueties, audio and video get skewed just a little bit more, every gap makes them skew another ~100 ms. So that's why I asked if the explicit "-map 0:v,0:a -map 0:a" (or another method) is actially required, to have the video synced to the audio (and not the to nominal frame rate, I think that is the actual problem).
2017-03-29 16:44 GMT+02:00 Erik Slagter <erik@slagter.name>:
Needs -async 1
Have a look at this part of the command line:
-filter:a " aresample=async=10000:min_comp=0.1:min_hard_comp=0.1:max_soft_comp=10000:first_pts=0"
This is what -async 1 does
(-async is declared "obsolete").
At least I find "-async 1" easier to remember so I hope it will stay... Please provide a sample if it doesn't work for you. Carl Eugen
On 30-03-17 01:01, Carl Eugen Hoyos wrote:
2017-03-29 16:44 GMT+02:00 Erik Slagter <erik@slagter.name>:
Needs -async 1
Have a look at this part of the command line:
-filter:a " aresample=async=10000:min_comp=0.1:min_hard_comp=0.1:max_soft_comp=10000:first_pts=0"
This is what -async 1 does
(-async is declared "obsolete").
At least I find "-async 1" easier to remember so I hope it will stay...
Please provide a sample if it doesn't work for you.
The "sample" is 1.5 hours long, with the gaps spread over the complete running time. It's quite easy to create one yourself though. - create an mpeg transport stream with h264 video and ac3 audio (even better if you can have one completely from a DVB source) - at multiple locations, cut out a piece, doesn't really matter how far spread out or how long the gaps are; use a file-level cutter, like dd, it's important that the gaps have incomplete packets (so don't use a tool that is aware of mpeg stream structure, like ffmpeg itself) That's all. Now convert it to something else, using ffmpeg, like mjpeg+pcm in mkv, using the flags mentioned. You'll that most of the a/v desync created by the gap is worked around by ffmpeg, but a tiny bit is left, making the audio and video get out of sync just a little bit more with every gap, 100-200 ms.
On 31-03-17 18:14, Carl Eugen Hoyos wrote:
2017-03-31 9:50 GMT+02:00 Erik Slagter <erik@slagter.name>:
The "sample" is 1.5 hours long, with the gaps spread over the complete running time.
Please provide a link.
It's quite easy to create one yourself though.
Feel free to create one and point me to that.
It's 3.4 Gb. A bit too much for my upstream.
[ ffmpeg not syncing a/v properly with an mpeg transport stream, that has gaps in it, as input ] It seems nobody experiences the same issues. As said before it's not specific to a certain file, it happens with all such streams. So uploading such a huge stream is imho pointless. I've done some experimentation in the meantime, the results may or may not help others, who knows. This is the exact command I'm using. Note the vaapi hwaccel, yes I am aware and it does not have any impact on the synchronisation (as it should). ffmpeg -y -frame_drop_threshold -0.1 -hwaccel vaapi -vaapi_device /dev/dri/card0 -i test.ts -sn -dn -codec:v mjpeg -q:v 3 -codec:a pcm_s16le -ac 2 -ar 48000 -af 'aresample=och=2:async=1:osr=48000:min_comp=0.02:min_hard_comp=0.02:max_soft_comp=10000:first_pts=0' -vsync cfr -shortest -map 0:v,0:a -map 0:a -f matroska test.mkv The "-af aresample..." is similar to -async with some extra user settable thresholds. They don't seem to make any difference. The -vsync cfr makes sure the audio and video streams are physically aligned within the container (i.e. when some was to demux the container stream, the audio and video stream would still align to each other without a skew). My experience: - most gaps are recovered from quite well, although a tiny bit of desync remains, usually about 100 ms that the video lags behind the audio, just enough to be noticable - sometimes the desync is considerately larger, up to a few seconds; my guess that is around a mpeg transport stream timestamp wrapround (every 27 hours), it may be that ffmpeg doesn't handle that too well. What really surprises me is that mplayer plays these streams perfectly, as do my DVB settopboxes. So it's not something that's impossible.
2017-04-03 15:47 GMT+02:00 Erik Slagter <erik@slagter.name>:
[ ffmpeg not syncing a/v properly with an mpeg transport stream, that has gaps in it, as input ]
It seems nobody experiences the same issues. As said before it's not specific to a certain file, it happens with all such streams. So uploading such a huge stream is imho pointless.
I do not understand: You do not agree that without samples, bugs are very unlikely to get fixed? Please try to avoid uploading >10G, that is our informal limit. Carl Eugen
On 03-04-17 18:43, Carl Eugen Hoyos wrote:
2017-04-03 15:47 GMT+02:00 Erik Slagter <erik@slagter.name>:
[ ffmpeg not syncing a/v properly with an mpeg transport stream, that has gaps in it, as input ]
It seems nobody experiences the same issues. As said before it's not specific to a certain file, it happens with all such streams. So uploading such a huge stream is imho pointless.
I do not understand: You do not agree that without samples, bugs are very unlikely to get fixed?
Please try to avoid uploading >10G, that is our informal limit.
I am still hoping it's not a bug but a simple operating error on my side i.o.w. something I didn't understand correctly. I have great respect for the ffmpeg developers (no pun).
2017-04-03 18:54 GMT+02:00 Erik Slagter <erik@slagter.name>:
On 03-04-17 18:43, Carl Eugen Hoyos wrote:
2017-04-03 15:47 GMT+02:00 Erik Slagter <erik@slagter.name>:
[ ffmpeg not syncing a/v properly with an mpeg transport stream, that has gaps in it, as input ]
It seems nobody experiences the same issues. As said before it's not specific to a certain file, it happens with all such streams. So uploading such a huge stream is imho pointless.
I do not understand: You do not agree that without samples, bugs are very unlikely to get fixed?
Please try to avoid uploading >10G, that is our informal limit.
I am still hoping it's not a bug but a simple operating error on my side i.o.w. something I didn't understand correctly.
I will do my best to test your sample (assuming it is <10G). Carl Eugen
On 03-04-17 19:01, Carl Eugen Hoyos wrote:
2017-04-03 18:54 GMT+02:00 Erik Slagter <erik@slagter.name>:
On 03-04-17 18:43, Carl Eugen Hoyos wrote:
2017-04-03 15:47 GMT+02:00 Erik Slagter <erik@slagter.name>:
[ ffmpeg not syncing a/v properly with an mpeg transport stream, that has gaps in it, as input ]
It seems nobody experiences the same issues. As said before it's not specific to a certain file, it happens with all such streams. So uploading such a huge stream is imho pointless.
I do not understand: You do not agree that without samples, bugs are very unlikely to get fixed?
Please try to avoid uploading >10G, that is our informal limit.
I am still hoping it's not a bug but a simple operating error on my side i.o.w. something I didn't understand correctly.
I will do my best to test your sample (assuming it is <10G).
I was at the virge of finally being conviced that upload was necessary, then I realised this is "heavily copyrighted" material. It's not a film but a (full) concert registration. And of course it's also available on DVD's from the store. So I'm having this hunch the producing company won't really like me for it when I put it in the public domain. Maybe there is a way to share it in a very limited group?
On 3 April 2017 at 15:47, Erik Slagter <erik@slagter.name> wrote:
[ ffmpeg not syncing a/v properly with an mpeg transport stream, that has gaps in it, as input ]
It seems nobody experiences the same issues. As said before it's not specific to a certain file, it happens with all such streams. So uploading such a huge stream is imho pointless.
IMHO you're making you're life hard unnecessarily. In the first place, why do you edit in two steps, first a raw cut, then a more precise? Let me recommend you MPEG Streamclip, which is a free software that can connect directly to your Enigma box, access its recordings, edit at keyframes to avoid A/V sync issues and save the edited file losslessly. I can't think of a more pristine workflow. Cheers
participants (3)
-
Carl Eugen Hoyos -
Carles Vila -
Erik Slagter