[FFmpeg-trac] #9776(avdevice:new): pulse and x11grab: audio is around 2 seconds behind video

[FFmpeg]

#9776: pulse and x11grab: audio is around 2 seconds behind video
-------------------------------------+-------------------------------------
             Reporter:  merulasnyde  |                    Owner:  (none)
                 Type:  defect       |                   Status:  new
             Priority:  normal       |                Component:  avdevice
              Version:  git-master   |               Resolution:
             Keywords:  pulse        |               Blocked By:
  pulseaudio avdevice delay desync   |
  audio audio/video                  |
             Blocking:               |  Reproduced by developer:  0
Analyzed by developer:  0            |
-------------------------------------+-------------------------------------

Old description:

> Summary of the bug: When recording audio+video of my screen, the captured
> audio is around 2 seconds behind the video.
> How to reproduce:
> {{{
> % ffmpeg -f pulse -ac 2 -channel_layout stereo -i default -r 60
> -video_size 640x480 -f x11grab -i :0.0+0,0 -c:a libvorbis -b:a 192k -c:v
> libx264 -b:v 2M delayed_audio.mkv
>
> * Start the capture (possibly with some audio already playing so
> pavucontrol can be checked to see whether audio is being recorded)
> * Move something that can produce audio at demand into the captured
> region, e.g. a media player with a play/pause button.
> * Periodically push the play/pause button to start and stop audio.
> * Check the recorded video and see whether the audio start+stop matches
> the pushes of the button
> }}}
>
> {{{
> ffmpeg version f77ac5131cd1c623aa54ec432cba0d922e9aa470
> ffmpeg version N-106835-gf77ac5131c Copyright (c) 2000-2022 the FFmpeg
> developers
> built with gcc 11.2.0 (Gentoo Hardened 11.2.0 p1)
>
> configuration: --enable-libx264 --enable-libvorbis --enable-libpulse
> --enable-gpl --enable-ffmpeg --enable-version3 --enable-nonfree
> --disable-ffplay --disable-ffprobe --disable-libx265 --disable-libv4l2
> --disable-libvpx --disable-nvenc --disable-nvdec --disable-cuda
> --disable-vaapi --disable-librtmp --disable-ffnvcodec --disable-hwaccels
> --disable-filters --enable-filter=scale --enable-filter=aresample
> --disable-decoders --disable-encoders --enable-decoder=rawvideo --enable-
> decoder=pcm_s16le --enable-encoder=libx264 --enable-encoder=libvorbis
> --disable-doc
> libavutil      57. 24.101 / 57. 24.101
> libavcodec     59. 27.100 / 59. 27.100
> libavformat    59. 23.100 / 59. 23.100
> libavdevice    59.  6.100 / 59.  6.100
> libavfilter     8. 38.100 /  8. 38.100
> libswscale      6.  6.100 /  6.  6.100
> libswresample   4.  6.100 /  4.  6.100
> libpostproc    56.  5.100 / 56.  5.100
>

> }}}
>
> It was fine in 4.3 and stopped working in 4.4-something.
> I ran a bisect between 4.3 and some bad commit in 4.4, using the strategy
> outlined above to determine whether the commit is good or bad.
> The bisect revealed this to be a (first) bad commit:
> b2d0826513c5e76f9bad2f1f0c809bc5c8e58b0c
> {{{
> Author: Marton Balint <cus at passwd.hu>
> Date:   Sat Feb 6 19:48:51 2021 +0100
>
>     avdevice/pulse_audio_dec: do not read undersized frames
>
>     Keep on reading fragments until we got fragment_size amount of data,
> otherwise
>     we might get frames with 1-2 samples only if pa_stream_peek is called
> slightly
>     less frequently than sample rate.
>
>     Note that fragments might contain a lot less data than fragment_size,
> so
>     reading multiple fragments to get fragment_size amount of data is
> intentional.
>
>     Signed-off-by: Marton Balint <cus at passwd.hu>
> }}}
>

> I was also able to reproduce it on a precompiled binary of (the same)
> master N-106835-gf77ac5131c-20220509 prior to the bisect.
>
> Because it seems to be a pulse issue I imagine there is some extra info
> that could be useful (e.g. pertaining to my pulseaudio installation).

New description:

 Summary of the bug: When recording audio+video of my screen, the captured
 audio is around 2 seconds behind the video.
 How to reproduce:
 {{{
 % ffmpeg -f pulse -ac 2 -channel_layout stereo -i default -r 60
 -video_size 640x480 -f x11grab -i :0.0+0,0 -c:a libvorbis -b:a 192k -c:v
 libx264 -b:v 2M delayed_audio.mkv

 * Start the capture (possibly with some audio already playing so
 pavucontrol can be checked to see whether audio is being recorded)
 * Move something that can produce audio at demand into the captured
 region, e.g. a media player with a play/pause button.
 * Periodically push the play/pause button to start and stop audio.
 * Check the recorded video and see whether the audio start+stop matches
 the pushes of the button
 }}}

 {{{
 ffmpeg version f77ac5131cd1c623aa54ec432cba0d922e9aa470
 ffmpeg version N-106835-gf77ac5131c Copyright (c) 2000-2022 the FFmpeg
 developers
 built with gcc 11.2.0 (Gentoo Hardened 11.2.0 p1)

 configuration: --enable-libx264 --enable-libvorbis --enable-libpulse
 --enable-gpl --enable-ffmpeg --enable-version3 --enable-nonfree --disable-
 ffplay --disable-ffprobe --disable-libx265 --disable-libv4l2 --disable-
 libvpx --disable-nvenc --disable-nvdec --disable-cuda --disable-vaapi
 --disable-librtmp --disable-ffnvcodec --disable-hwaccels --disable-filters
 --enable-filter=scale --enable-filter=aresample --disable-decoders
 --disable-encoders --enable-decoder=rawvideo --enable-decoder=pcm_s16le
 --enable-encoder=libx264 --enable-encoder=libvorbis --disable-doc
 libavutil      57. 24.101 / 57. 24.101
 libavcodec     59. 27.100 / 59. 27.100
 libavformat    59. 23.100 / 59. 23.100
 libavdevice    59.  6.100 / 59.  6.100
 libavfilter     8. 38.100 /  8. 38.100
 libswscale      6.  6.100 /  6.  6.100
 libswresample   4.  6.100 /  4.  6.100
 libpostproc    56.  5.100 / 56.  5.100


 }}}

 It was fine in 4.3 and stopped working in 4.4-something.
 I ran a bisect between 4.3 and some bad commit in 4.4, using the strategy
 outlined above to determine whether the commit is good or bad.
 The bisect revealed this to be a (first) bad commit:
 b2d0826513c5e76f9bad2f1f0c809bc5c8e58b0c
 {{{
 Author: Marton Balint <cus at passwd.hu>
 Date:   Sat Feb 6 19:48:51 2021 +0100

     avdevice/pulse_audio_dec: do not read undersized frames

     Keep on reading fragments until we got fragment_size amount of data,
 otherwise
     we might get frames with 1-2 samples only if pa_stream_peek is called
 slightly
     less frequently than sample rate.

     Note that fragments might contain a lot less data than fragment_size,
 so
     reading multiple fragments to get fragment_size amount of data is
 intentional.

     Signed-off-by: Marton Balint <cus at passwd.hu>
 }}}


 I was also able to reproduce it on a precompiled binary of (the same)
 master N-106835-gf77ac5131c-20220509 prior to the bisect.

 Because it seems to be a pulse issue I imagine there is some extra info
 that could be useful (e.g. pertaining to my pulseaudio installation).

 While I reproduced it on master, it occurred during normal use after
 upgrading to various versions of 4.4.

 Reverting the bad commit on master seems to fix the issue.

--
Comment (by merulasnyde):

 Also
-- 
Ticket URL: <https://trac.ffmpeg.org/ticket/9776#comment:1>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker