#11398(undetermined:new): AVFoundation audio recordings have audible ticks and are missing samples
#11398: AVFoundation audio recordings have audible ticks and are missing samples -------------------------------------+------------------------------------- Reporter: drobinson | Type: defect Status: new | Priority: normal Component: | Version: 7.1 undetermined | Keywords: | Blocked By: avfoundation audio dropouts | Blocking: | Reproduced by developer: 0 Analyzed by developer: 0 | -------------------------------------+------------------------------------- Summary of the bug: I am using macOS 15.1 (Sequoia) on a 2021 Apple M1 Pro MacBook Pro 14-inch laptop, running FFmpeg version 7.1 installed via Homebrew. Audio captured using AVFoundation through any audio codec (AAC, MP3, PCM, or -c:a copy) produces audible glitches in the form of missing samples. This issue occurs regardless of the sound device selected (built-in audio, USB audio, etc.). Originally, I ran into this issue while attempting to capture both video and audio and encode them using libx264 and AAC. However, I found that the issue persists even in the most basic use case—just recording audio without video encoding. Steps to Reproduce: 1. Record a known waveform with the following command: {{{ ffmpeg -f avfoundation -i ":0" -c:a copy output.wav }}} 2. Listen to output.wav. I notice frequent random audible ticks. 3. Inspect the waveform of output.wav in an audio editing tool (e.g., Reaper). I observed that the waveform exhibits discontinuities, where a sequence of input samples, such as (1, 2, 3, 4, 5, 6, 7, 8, 9), results in an output sequence like (1, 2, 3, 8, 9). In this example, samples 4, 5, 6, and 7 are missing, and sample 8 immediately follows sample 3. These missing samples occur randomly, typically every 3000-10000 samples, with no discernible pattern. Here is an example of a ramp waveform I used with the glitch circled in red: [[Image(https://i.ibb.co/rysLqqW/glitch.png)]] And up close: [[Image(https://i.ibb.co/hmKp3hT/glitch-close.png)]] Expected Behavior: The output audio should accurately represent the input without any missing or skipped samples. Actual Behavior: There are consistent audio dropouts, where specific sections of audio are completely missing, resulting in a glitchy sound. This happens with every audio codec tested (AAC, MP3, PCM) and with different sound devices (built-in, USB, etc.). Details: {{{ % ffmpeg -f avfoundation -i ":0" -c:a copy output.wav ffmpeg version 7.1 Copyright (c) 2000-2024 the FFmpeg developers built with Apple clang version 16.0.0 (clang-1600.0.26.4) configuration: --prefix=/opt/homebrew/Cellar/ffmpeg/7.1_4 --enable- shared --enable-pthreads --enable-version3 --cc=clang --host-cflags= --host-ldflags='-Wl,-ld_classic' --enable-ffplay --enable-gnutls --enable- gpl --enable-libaom --enable-libaribb24 --enable-libbluray --enable- libdav1d --enable-libharfbuzz --enable-libjxl --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librist --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtesseract --enable-libtheora --enable-libvidstab --enable- libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable- libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable- libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable- libopenjpeg --enable-libspeex --enable-libsoxr --enable-libzmq --enable- libzimg --disable-libjack --disable-indev=jack --enable-videotoolbox --enable-audiotoolbox --enable-neon libavutil 59. 39.100 / 59. 39.100 libavcodec 61. 19.100 / 61. 19.100 libavformat 61. 7.100 / 61. 7.100 libavdevice 61. 3.100 / 61. 3.100 libavfilter 10. 4.100 / 10. 4.100 libswscale 8. 3.100 / 8. 3.100 libswresample 5. 3.100 / 5. 3.100 libpostproc 58. 3.100 / 58. 3.100 }}} {{{ Input #0, avfoundation, from ':0': Duration: N/A, start: 10114.532792, bitrate: 3072 kb/s Stream #0:0: Audio: pcm_f32le, 48000 Hz, stereo, flt, 3072 kb/s File 'output.wav' already exists. Overwrite? [y/N] y Stream mapping: Stream #0:0 -> #0:0 (copy) Output #0, wav, to 'output.wav': Metadata: ISFT : Lavf61.7.100 Stream #0:0: Audio: pcm_f32le ([3][0][0][0] / 0x0003), 48000 Hz, stereo, flt, 3072 kb/s Press [q] to stop, [?] for help size= 1792KiB time=00:00:07.71 bitrate=1903.5kbits/s speed=1.28x }}} -- Ticket URL: <https://trac.ffmpeg.org/ticket/11398> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#11398: "avfoundation" audio capture had missing samples randomly --------------------------------------+------------------------------------ Reporter: drobinson | Owner: (none) Type: defect | Status: new Priority: normal | Component: avdevice Version: 7.1 | Resolution: Keywords: avfoundation | Blocked By: Blocking: | Reproduced by developer: 0 Analyzed by developer: 1 | --------------------------------------+------------------------------------ Changes (by MasterQuestionable): * analyzed: 0 => 1 * cc: MasterQuestionable (added) * component: undetermined => avdevice * keywords: avfoundation audio dropouts => avfoundation * summary: AVFoundation audio recordings have audible ticks and are missing samples => "avfoundation" audio capture had missing samples randomly Comment: ͏ Reference: ͏ https://ffmpeg.org/ffmpeg-devices.html#avfoundation ͏ AVFoundation appears to be Apple specific. -- Ticket URL: <https://trac.ffmpeg.org/ticket/11398#comment:1> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#11398: "avfoundation" audio capture had missing samples randomly --------------------------------------+------------------------------------ Reporter: drobinson | Owner: (none) Type: defect | Status: new Priority: normal | Component: avdevice Version: 7.1 | Resolution: Keywords: avfoundation | Blocked By: Blocking: | Reproduced by developer: 0 Analyzed by developer: 1 | --------------------------------------+------------------------------------ Changes (by drobinson): * Attachment "avfoundation_audio_fifo.patch" added. -- Ticket URL: <https://trac.ffmpeg.org/ticket/11398> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#11398: "avfoundation" audio capture had missing samples randomly --------------------------------------+------------------------------------ Reporter: drobinson | Owner: (none) Type: defect | Status: new Priority: normal | Component: avdevice Version: 7.1 | Resolution: Keywords: avfoundation | Blocked By: Blocking: | Reproduced by developer: 0 Analyzed by developer: 1 | --------------------------------------+------------------------------------ Comment (by drobinson): I had a chance to do some more investigation. It seems like the issue is related to `captureOutput:didOutputSampleBuffer:fromConnection` callback in the `AVFAudioReceiver` class being called before the `avf_read_packet` function and freeing/overwriting the `current_audio_frame` before it is read. I am able to avoid the dropouts by creating a FIFO for the received frames. Not being that familiar with this codebase or AVFoundation, I am not sure if this is a reasonable fix or not (I suppose there could be latency/alignment fallout), but the avfoundation_audio_fifo.patch attached patch seems to resolve the issue for me. -- Ticket URL: <https://trac.ffmpeg.org/ticket/11398#comment:2> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#11398: "avfoundation" audio capture had missing samples randomly --------------------------------------+------------------------------------ Reporter: drobinson | Owner: (none) Type: defect | Status: new Priority: normal | Component: avdevice Version: 7.1 | Resolution: Keywords: avfoundation | Blocked By: Blocking: | Reproduced by developer: 0 Analyzed by developer: 1 | --------------------------------------+------------------------------------ Changes (by drobinson): * Attachment "avfoundation_audio_fifo-v2.patch" added. When the frame list overflows, advance the read index by half the FIFO length -- Ticket URL: <https://trac.ffmpeg.org/ticket/11398> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#11398: "avfoundation" audio capture had missing samples randomly --------------------------------------+------------------------------------ Reporter: drobinson | Owner: (none) Type: defect | Status: new Priority: normal | Component: avdevice Version: 7.1 | Resolution: Keywords: avfoundation | Blocked By: Blocking: | Reproduced by developer: 0 Analyzed by developer: 1 | --------------------------------------+------------------------------------ Comment (by MasterQuestionable): ͏ Difference: {{{#!diff --- avfoundation_audio_fifo.patch 2025-01-05T16:19:29Z +++ avfoundation_audio_fifo-v2.patch 2025-01-06T01:18:05Z @@ -18,10 +18,12 @@ +static void push_frame(AVFFrameList* list, CMSampleBufferRef frame) +{ + list->frames[list->write_index] = (CMSampleBufferRef)CFRetain(frame); + list->write_index = (list->write_index + 1) % AVF_FRAME_LIST_LENGTH; + + if (list->write_index == list->read_index) { -+ CFRelease(list->frames[list->read_index]); -+ list->read_index = (list->read_index + 1) % AVF_FRAME_LIST_LENGTH; ++ for (int i = 0; i < (AVF_FRAME_LIST_LENGTH / 2); ++i) { ++ CFRelease(list->frames[list->read_index]); ++ list->read_index = (list->read_index + 1) % AVF_FRAME_LIST_LENGTH; ++ } + } +} }}} -- Ticket URL: <https://trac.ffmpeg.org/ticket/11398#comment:3> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#11398: "avfoundation" audio capture had missing samples randomly --------------------------------------+------------------------------------ Reporter: drobinson | Owner: (none) Type: defect | Status: new Priority: normal | Component: avdevice Version: 7.1 | Resolution: Keywords: avfoundation | Blocked By: Blocking: | Reproduced by developer: 0 Analyzed by developer: 1 | --------------------------------------+------------------------------------ Comment (by Justin Taylor): Hi there, also experiencing this issue. The only way to resolve it is to roll back to ffmpeg 4.1 which isn't ideal. Has this patch been integrated into the main build yet? -- Ticket URL: <https://trac.ffmpeg.org/ticket/11398#comment:4> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#11398: "avfoundation" audio capture had missing samples randomly --------------------------------------+------------------------------------ Reporter: drobinson | Owner: (none) Type: defect | Status: new Priority: normal | Component: avdevice Version: 7.1 | Resolution: Keywords: avfoundation | Blocked By: Blocking: | Reproduced by developer: 0 Analyzed by developer: 1 | --------------------------------------+------------------------------------ Comment (by Aman): drobinson could you submit this patch to the ffmpeg-devel mailing list so it can be reviewed -- Ticket URL: <https://trac.ffmpeg.org/ticket/11398#comment:5> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#11398: "avfoundation" audio capture had missing samples randomly -------------------------------------+------------------------------------- Reporter: drobinson | Owner: Thilo | Borgmann Type: defect | Status: open Priority: normal | Component: avdevice Version: 7.1 | Resolution: Keywords: | Blocked By: avfoundation | Blocking: | Reproduced by developer: 0 Analyzed by developer: 1 | -------------------------------------+------------------------------------- Changes (by Thilo Borgmann): * owner: (none) => Thilo Borgmann * status: new => open -- Ticket URL: <https://trac.ffmpeg.org/ticket/11398#comment:6> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
participants (1)
-
FFmpeg