[FFmpeg-devel] Audio Video Sync issue with dvr-ms files

Jonathon Elwood jaelwood
Fri May 11 00:26:08 CEST 2007


On 5/10/07, Jonathon Elwood <jaelwood at gmail.com> wrote:
> Hi
>
> On 5/2/07, Michael Niedermayer <michaelni at gmx.at> wrote:
> > Hi
> >
> > On Tue, May 01, 2007 at 11:02:15PM -0400, Jonathon Elwood wrote:
> > > Hi
> > >
> > > On 5/1/07, Michael Niedermayer <michaelni at gmx.at> wrote:
> > > >Hi
> > > >
> > > >On Tue, May 01, 2007 at 02:47:31PM -0400, Jonathon Elwood wrote:
> > > >> On 4/3/07, Jonathon Elwood <jaelwood at gmail.com> wrote:
> > > >> >
> > > >> >
> > > >> >There is an AV sync issue with the latest SVN (8618).   I've uploaded a
> > > >> >sample file that demonstrates this issue to:
> > > >> >
> > > >> >ftp://upload.mplayerhq.hu/MPlayer/incoming/Backpackers -
> > > >> >_23_03_2007_23_00_03_comms.dvr-ms
> > > >> >
> > > >> >Michael Niedermayer recommended that I test with previous versions of
> > > >> >ffmpeg
> > > >> >and find the patch that broke the AV sync.   I traced the problem to
> > > >patch
> > > >> >#7905.  ffmpeg 7904 worked fine, but 7905 and above demonstrate this AV
> > > >> >sync
> > > >> >issue.   I modified the latest SVN to reverse the changes made in patch
> > > >> >7905, and the AV sync looks good.  However, this obviously isn't a
> > > >proper
> > > >> >fix.   Also without the changes in patch 7905 I have other dvr-ms shows
> > > >> >where the fps is detected incorrectly, so this patch is obviously
> > > >needed.
> > > >> >Could you please look at this?   I'm not sure what the code in patch
> > > >7905
> > > >> >is
> > > >> >doing, so I won't be able to create the appropriate patch.
> > > >> >
> > > >>
> > > >> I have been working with John Donaghy trying to solve the problems I
> > > >> reported above (thanks John for all your help).  I've made some
> > > >> progress fixing the timestamps (the video isn't "freezing" and the fps
> > > >> is detected correctly on all shows now); however, the video frames
> > > >> before the first keyframe are being discarded.  This is leading to a
> > > >> bad AV sync issue because the audio starts from the beginning of the
> > > >> stream, but the video starts from the first Key Frame.
> > > >> I've attached two patches (one to asf.c and the other to asf.h)
> > > >> showing the changes I've made so far.  These patches obviously aren't
> > > >> ready for submission so please be kind, this is obviously still a work
> > > >> in progress.  I've also attached a debug log file showing the output
> > > >> of my debug statements.
> > > >> My questions are:
> > > >> 1) If you look at the debug.txt file you will see the timestamps I'm
> > > >> now allocating to each segment.  However, I'm not sure what to do with
> > > >> the video segments before the first Key Frame.  My guess is I need to
> > > >> do something special to these timestamps so the frames aren't
> > > >> discarded, but what?  Right now I'm just setting them to
> > > >> AV_NOPTS_VALUE,
> > > >
> > > >AV_NOPTS_VALUE is correct if you dont know their timestamp
> > > >
> > > >
> > > >> however, they are being discarded.  I tried setting a
> > > >> value on the new segments, however, there was a big jump in the
> > > >> timestamp values once I got to the first Key Frame and they were
> > > >> discarded anyway.  So, I need to know what to do with these sequences.
> > > >> John suggested I just discard both the audio and video sequences
> > > >> until I get to the Video Key Frame.  I don't know how to do this, is
> > > >> there a mechanism in place to make this easy?
> > > >
> > > >a demuxer should not discard frames at random, if this causes a problem
> > > >then the bug is not in the demuxer
> > > >
> > > >
> > > >> 2) If you look at the segments marked "audio timestamp" you will
> > > >> notice that the audio is split into two segments.  I'm assigning the
> > > >> timestamp to the first segment, and setting the second segment to
> > > >> AV_NOPTS_VALUE.  Is this correct, or should the second segment also
> > > >> have a timestamp?  If the second segment has to have a timestamp, then
> > > >> what timestamp do I use (if I read the timestamp from the file, it is
> > > >> the same timestamp as the first segment)?
> > > >
> > > >i dont know that, but i dont think setting the timestamp for both is
> > > >correct, either first or second ... but iam just guessing, it depends on
> > > >what dvr-ms really does ...
> > > >
> > > >
> > > >> 3) Should both audio segments have the Key Frame flag set, or should I
> > > >> remove the Key Frame flag from the second segment?
> > > >>
> > > >> If there is anything else you see wrong regarding the timestamps I've
> > > >> allocated please let me know.  I've learned a lot, but all of this is
> > > >> still new to me.
> > > >
> > > >yes i will NOT accept a patch with some odd interpolation code without
> > > >fully understanding why that is needed
> > > >that is if your only explanation is like "it doesnt work without" then
> > > >the patch is rejected
> > > >
> > > >also all float/double calculations must be removed they are unacceptable
> > > >due to breaking binary reproduceable output between different architectures
> > > >
> > > >the debug output you provide below is fairly hard to read its not obvious
> > > >from it what timestamps where in the file, what would be wrong with them or
> > > >what better timestamps you set
> > > >
> > > >also without having looked at the files, did you try to simply set the
> > > >frame rate correctly if it was wrong? the internal timestamp code uses
> > > >the frame rate and will not work if its wrong ...
> > >
> > > Given your input, I decided to start simple.  I've removed all the
> > > interpolation code and only set the timestamps on the first segment of
> > > a multi-segment payload.  I know my debug is a little hard to read,
> > > but I did modify it a little so hopefully you will be able to see
> > > what's happing with the timestamps.  Also, I'm setting the Key Frame
> > > based on the flag in the Replic Data (before this change every segment
> > > was marked as a Key Frame).  This fixes the AV sync issues on 60% of
> > > my samples (a big improvement).  Now I only two of my three origional
> > > outstanding issues.  First, I have some shows where the fps isn't
> > > being detected correctly.  I get the message "Seems stream 2 codec
> > > frame rate differs from container frame rate: 29.97 (30000/1001) ->
> > > 30.33 (91/3)".  I know I can fix this with all the interpolation code,
> > > but I'm going to take your advice and try setting the correct frame
> > > rate.
> > >
> > > Sorry for asking such a basic question, but I'm not sure what
> > > variables I should look at to set the frame rate from the demuxer.
> > > After searching I found the variables below.  Which set would be
> > > appropriate for the demuxer?
> > > frame_rate and frame_rate_base
> > > r_frame_rate.num and r_frame_rate.den
> > > rfps and rfps_base
> > > time_base.den and time_base.num
> >
> > try setting AVCodecContext.time_base and AVStream.r_frame_rate
> > that said this is for experimentation and might not be the final
> > solution ...
>
> The Good news is I didn't have to set the frame rate.  I found out the
> segment_marker isn't in the same place for all samples.  The attached
> patch accounts for this and now the fps is detected correctly for all
> dvr-ms files.
>
> >
> >
> > >
> > > Lastly, I have a problem where the first frame is "frozen" until the
> > > video gets to the first Key Frame (of course after that the AV is
> > > completly out of sync).  I can fix this problem with the interpolation
> > > code as well, so maybe once I set the frame rate this problem will go
> > > away...
> > >
> > > >> +            audio_video_type = get_byte(pb); //This value appears to
> > > >identify the type of audio and video.
> > > >
> > > >can this be used to identify ac3/mp3 properly?
> > > On 99% of my samples yes.  However, I do have one sample where this
> > > value is very strange.  Having said that, I'm not convinced this value
> > > is unreliable because something else is going on with this sample.
> > > The timestamps are all messed up as well (for example: the stamps will
> > > be 192467, 189164, 193264).  Plus the Key Frame markers arn't there.
> > > I haven't look into this yet, but my guess is values have shifted in
> > > the header, so we are just looking in the wrong spot.  Once I figure
> > > out what's going on with this sample I'll be able to say whether this
> > > value will reliably identify the audio and video.  However, John
> > > Donaghy's code to probe for the audio type is working well for me so
> > > I'm not sure we even need this.
> >
> > i dont like the current probe hack ...
> > if we can detected the the audio codec from some field in the header this
> > would be greatly preferred
> >
> >
> > [...]
> >
> > > +                     if (s->streams[asf->stream_index]->codec->codec_type == CODEC_TYPE_AUDIO) {
> > > +                        //only assign the timestamp to the first segment in a multiple segment payload
> > > +                        if (ts0 != -1) {
> > > +                                if (asf->last_aud_seq == asf->packet_seq)
> > > +                                        asf->packet_frag_timestamp = AV_NOPTS_VALUE;
> > > +                                else
> > > +                                    asf->packet_frag_timestamp= ts0/10000;
> >
> > isnt a check based on packet_frag_offset possible and simpler?
> >
> > also tabs are forbidden in svn
> >
> Unfortunately, I'm stuck?  With the attached patch the fps is now
> detected correctly on all samples.  Also the AV sync is correct on 80%
> of my samples.  This is a nice improvement, but I still have the
> problem where the video is "stuck" on the first frame (this accounts
> for the 20% where the AV sync is incorrect).  Unfortunately, I don't
> know where to look.  Any help or guidance would be greatly
> appreciated.
> If anyone would like to see a sample that exhibits the "stuck frame"
> problem, I uploaded the sample on April 3rd that demonstrated this
> issue.   ftp://upload.mplayerhq.hu/MPlayer/incoming/ NBA
> Basketball_WPTA-DT_28_01_2007_14_59_45.dvr-ms.  If this sample doesn't
> exist on the ftp site any longer, please let me know and I'll upload
> it again.
>
>
Sorry for the double post, but there was one piece of information I
forgot.  I compiled MPlayer with these patches and used the -demuxer
35 option and the video played great.  I know this is the ffmpeg dev
list, and this piece of information doesn't help ffmpeg, but I thought
it might prove that the "frozen frame" problem isn't in the demuxer.
Unfortunately, I just don't know where to look...




More information about the ffmpeg-devel mailing list