#4437(avdevice:new): AVFoundation recording has consistent intermittent frame drops
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Type: defect Status: new | Priority: normal Component: avdevice | Version: git- Keywords: | master avfoundation | Blocked By: Blocking: | Reproduced by developer: 0 Analyzed by developer: 0 | -------------------------------------+------------------------------------- '''Summary''': 60 FPS screen capture on OS X results in intermittent drops every 5-8 seconds. '''How to reproduce''': Record a video with AVFoundation using `-framerate 60`. {{{ ffmpeg -f avfoundation -pix_fmt 0rgb -framerate 60 -video_device_index 3 -i “” -c:v libx264rgb -crf 0 -vf crop=426:240:507:339 ~/Desktop/avfoundation.avi }}} Additional notes: * I have a SATA 3 SSD; plenty of disk write speed. * The source does not drop frames during recording, only the video does. * The source frame rate is exactly 60 FPS. * The frame drops have a fairly consistent timing. * Recording the same source in Windows does not produce these frame drops. * Media players will report an average rate of around 57.13 (it varies). * The `-r 60` output option will of course force the rate to 60, but not prevent the frame drops. * Example recording: https://www.dropbox.com/s/0dthf652g644ldq/avfoundation.avi.xz?dl=0 -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops --------------------------------------+------------------------------------ Reporter: LordHDL | Owner: Type: defect | Status: new Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: avfoundation | Blocked By: Blocking: | Reproduced by developer: 0 Analyzed by developer: 0 | --------------------------------------+------------------------------------ Comment (by thilo.borgmann): Please provide the full uncut console output of the command you show to reproduce this behaviour. What device is it, you are capturing from? -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:1> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops --------------------------------------+------------------------------------ Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: avfoundation | Blocked By: Blocking: | Reproduced by developer: 0 Analyzed by developer: 0 | --------------------------------------+------------------------------------ Changes (by mateo): * cc: matthieu.bouron@… (added) * owner: => mateo * status: new => open -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:2> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops --------------------------------------+------------------------------------ Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: avfoundation | Blocked By: Blocking: | Reproduced by developer: 0 Analyzed by developer: 0 | --------------------------------------+------------------------------------ Comment (by LordHDL): Replying to [comment:1 thilo.borgmann]:
Please provide the full uncut console output of the command you show to reproduce this behaviour.
What device is it, you are capturing from?
The full output is in the attached text file. "Capture screen 0" is the selected device. -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:3> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops --------------------------------------+------------------------------------ Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: avfoundation | Blocked By: Blocking: | Reproduced by developer: 0 Analyzed by developer: 0 | --------------------------------------+------------------------------------ Comment (by thilo.borgmann): I cannot reproduce with libx264rgb... Please strip the command line / features down to the minimum required to trigger these frameskips you're reporting. Try a different internal video codec an/or raw video and to skip the crop filter. -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:4> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops --------------------------------------+------------------------------------ Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: avfoundation | Blocked By: Blocking: | Reproduced by developer: 0 Analyzed by developer: 0 | --------------------------------------+------------------------------------ Comment (by mateo): I'm not able to reproduce the issue either. What is your OSX version ? Maybe the CPU is the bottleneck at some point. Is your machine under heavy load ? -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:5> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops --------------------------------------+------------------------------------ Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: avfoundation | Blocked By: Blocking: | Reproduced by developer: 0 Analyzed by developer: 0 | --------------------------------------+------------------------------------ Comment (by LordHDL): I'm on 10.10.2. My CPU usage is around 15-18% while recording. I have an i7 2720QM 2.2 GHz and a Radeon 6750M (MacBook Pro 8,2). As I said in my notes, the drops don't happen when recording with ffmpeg in Windows on the same hardware. On OS X the drops also happen when using huffyuv. Also I know I specified 60 FPS but this happens even if I don't use the `-framerate` option. The recording defaults to 30 FPS and will still exhibit the intermittent drops. The only time hardware appears to be a bottleneck is if I don't use the crop filter and record full screen. -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:6> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops --------------------------------------+------------------------------------ Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: avfoundation | Blocked By: Blocking: | Reproduced by developer: 0 Analyzed by developer: 0 | --------------------------------------+------------------------------------ Comment (by LordHDL): Here's a test without the `-vf` option: https://www.dropbox.com/s/a982jatnw7n6d45/nofilter.tar.xz?dl=0 I didn't include the lossless recording since, even compressed via xz, my connection is far too slow to upload within a reasonable time. But you can get the idea from the encoded video and included Terminal outputs. I'm actually surprised the recording is using that much CPU. I thought lossless was supposed to be more taxing on disk write speed, while compression depends more on CPU? -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:7> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops --------------------------------------+------------------------------------ Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: avfoundation | Blocked By: Blocking: | Reproduced by developer: 0 Analyzed by developer: 0 | --------------------------------------+------------------------------------ Comment (by LordHDL): Just an update to say that I've added a bounty to this on Bountysource; I'm still interested in a fix for this. I also have recent videos here: https://www.dropbox.com/s/0ovhm3h3dl1fhx9/Tests.tar.xz?dl=0 -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:8> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops --------------------------------------+------------------------------------ Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: avfoundation | Blocked By: Blocking: | Reproduced by developer: 0 Analyzed by developer: 0 | --------------------------------------+------------------------------------ Comment (by mateo): I'll take another look at the issue in one week as I won't have access to a mac until then (and if no one fixes the issue). -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:9> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops --------------------------------------+------------------------------------ Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: avfoundation | Blocked By: Blocking: | Reproduced by developer: 0 Analyzed by developer: 0 | --------------------------------------+------------------------------------ Comment (by LordHDL): I increased the bounty for this. Let me know if you could use any other tests or information from my part. -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:10> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Reproduced by developer: 0 Blocking: | Analyzed by developer: 0 | -------------------------------------+------------------------------------- Changes (by llogan): * keywords: avfoundation => avfoundation bounty Comment: Bountysource link: https://www.bountysource.com/issues/10804663-avfoundation-recording-has- consistent-intermittent-frame-drops -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:11> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Reproduced by developer: 0 Blocking: | Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by mateo): I'm currently working on the issue. A patch can be expected at the end of the week. -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:12> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Reproduced by developer: 0 Blocking: | Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by mateo): A patch has been sent to the mailing list [1], however it does not get rid of the issue but might help a bit. I beleive the issue is on the avfoundation side as the screen recording feature might not be in sync with the display (or the display compositor might not be able to stay at a constant 60fps ?). Can you try the patch and let me know if it improves a bit your issue ? ffmpeg will output a warning if a drop happens on the avdevice side. [1] http://ffmpeg.org/pipermail/ffmpeg-devel/2015-August/177845.html -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:13> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Reproduced by developer: 0 Blocking: | Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by thilo.borgmann): Thanks for your patch, mateo! If you extend your buffer size to way bigger than 4 frames you should be able to see if your assumptions are correct and we just do not consume the frames fast enough (and your buffer warning occurs later). Otherwise, if there is something out of sync on the system-side, your buffering extension could be useless and memory requirements raise. In that case we have to figure out if we can somehow acquire these missing frames in any way or if we don't even have a chance. I'm eager to hear feedback on the patch from LordHDL. -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:14> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
Thanks for your patch, mateo!
If you extend your buffer size to way bigger than 4 frames you should be able to see if your assumptions are correct and we just do not consume the
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Reproduced by developer: 0 Blocking: | Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by mateo): Replying to [comment:14 thilo.borgmann]: frames fast enough (and your buffer warning occurs later). While testing the patch, ffmpeg did not output the new warning telling the user that a frame is being dropped because the queue is full while still seeing the issue on the output file (what LordHDL has described / what is mentionned in the commit message).
Otherwise, if there is something out of sync on the system-side, your
buffering extension could be useless and memory requirements raise. In that case we have to figure out if we can somehow acquire these missing frames in any way or if we don't even have a chance. I've tried to set the setAlwaysDiscardsLateVideoFrames to NO but it was leading to a performance decrease, ie: the capture was running at 54fps instead of 60. I also experience the same issue using gstreamer. Anyway, I don't think the buffering extension is useless as the new code reduce the locking area and give more guarantee that in case of a sudden slowdown we don't drop any frames (unless the queue gets full). If the memory requirements raise is an issue i can reduce the default queue size. Even add an option to configure it. What do you think ? [...] -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:15> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Reproduced by developer: 0 Blocking: | Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by LordHDL): I've never applied patches in this manner before so I'm not actually sure how to do it. Could you give me a brief rundown? Additionally as I said in comment 6, this issue does also happen with the default 30 FPS. I initially thought it was specific to 60 but it happens regardless. -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:16> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Reproduced by developer: 0 Blocking: | Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by thilo.borgmann): Briefly: You might want to do it in a seperate branch, so create one with {{{git branch}}}, next you apply the patch file with {{{git am}}}. Then you rerun {{{configure}}} and {{{make}}}. After that you can test the new behaviour. -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:17> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Reproduced by developer: 0 Blocking: | Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by LordHDL): What option am I supposed to use with `git am`? This is my first time using this command. -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:18> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Reproduced by developer: 0 Blocking: | Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by thilo.borgmann): You might use {{{git am < file.patch}}}. -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:19> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Reproduced by developer: 0 Blocking: | Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by LordHDL): That does nothing for me. I'm not a git expert so I'm likely just doing it wrong. Can you give an example of how that is supposed to look? -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:20> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Reproduced by developer: 0 Blocking: | Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by thilo.borgmann): Use: {{{git apply ../patches/avf.patch}}} If you further want it to be committed to your local repository: {{{git add libavdevice/avfoundation.m}}} {{{git commit -m "avf patch"}}} Then {{{git log -1}}} shows you your patch to be the last change. The avf.patch is just a copy&paste from the mail on ffmpeg-devel. -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:21> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Reproduced by developer: 0 Blocking: | Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by thilo.borgmann): You can validate that your patch is applied if you lastly type: {{{git diff}}} The output you get should look (almost) similar to the patch file. -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:22> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Reproduced by developer: 0 Blocking: | Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by LordHDL): New test with patch: https://www.dropbox.com/s/wbp01bcxhrrauxi/AVFTest.tar.xz?dl=0 Ignore the registration prompt (click outside or the X), it's not needed to download. The source chosen is a recent PC game (not emulated, low CPU & GPU demands) so it runs at 60 FPS exactly as opposed to older standards like 59.94. Drops in smoothness are very obvious. The patch does seem to help with the audio tempo issue I've experienced, however. It doesn't seem to cut off or warp anymore. -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:23> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Reproduced by developer: 0 Blocking: | Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by thilo.borgmann): Unfortunately I will be offline for the whole weekend so maybe mateo can comment on how the patch work for him. If it just fixes an audio issue (got a ticket for that?), then it might help with that but does not fix this issue. Still I wonder if the FFmpeg side causes the frame drops or if we never come to see these frames and Apple drops them. I will also need some more time to review the patch on the ml. -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:24> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Reproduced by developer: 0 Blocking: | Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by LordHDL): I'm hoping it's not a rooted issue in AV Foundation itself. As I said before, recordings in Windows do not exhibit the drops (on the same hardware). I know OS X is capable of perfectly smooth capture, as evidenced by the recording app ScreenFlow, so it's not a limitation of the OS so much as something janky in the software somewhere. The audio bug ticket is #4463. -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:25> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Reproduced by developer: 1 Blocking: | Analyzed by developer: 0 | -------------------------------------+------------------------------------- Changes (by thilo.borgmann): * reproduced: 0 => 1 Comment: I can reproduce it. There are missing frames and duplicated frames during my testing. However, I'm still not sure what causes these drops. If I understand it correctly, the patch sent does not fix the issue even partially. I will have a look at it and review it on the ML asap. And I will look at #4463 again and see what this patch achieves there. So, LordHDL, you possess the ScreenFlow software and it captures the very same resolution at the very same framerate with unique frames? Can you link to another software capable of that, maybe one that's open- source (or at least free)? -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:26> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Reproduced by developer: 1 Blocking: | Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by LordHDL): Correct, ScreenFlow seems to nail both frame and lossless RGB capture (too bad the export options suck). The only other software I've used that gets frames right is Norichan, but that won't help much. It's only usable with a certain capture device and was designed to record interlaced video so that the resulting deinterlaced encoding would be perfect. Some notable mentions, however: [http://cocoasplit.com/releases/coreimage.html CocoaSplit], [https://github.com/jp9000/obs-studio/releases OBS], and [http://syphon.v002.info/recorder/ Syphon Recorder]. All do exhibit some drops (Syphon Recorder has a similar problem to ffmpeg), but so far these are the best of the bunch that I've encountered. As far as video capture (not recording), [http://camtwiststudio.com/ CamTwist] is also excellent. These are all free. If you're looking to test Syphon Recorder a good complement to it would be [http://openemu.org/ OpenEmu], as it has Syphon output built-in. -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:27> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Reproduced by developer: 1 Blocking: | Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by LordHDL): I recorded more examples that compare the current git master to the patch: https://www.dropbox.com/s/p3na7szw6d8ds9r/PatchComparison.tar.xz?dl=0 You'll notice that the audio plays correctly in the patched version, but video behaves the same. Commands and outputs are in the archive for each. -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:28> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Reproduced by developer: 1 Blocking: | Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by LordHDL): As per Thilo's request I'm posting my test results for his mirror here. This is what I got: https://www.dropbox.com/s/apkdhi4liixgdl8/avf2.tar.xz?dl=0 Full console outputs included in the archive. Some observations: * Recording now forces my device's sample rate to max, which is 192,000 Hz. I'm using Soundflower (2ch) for audio and usually leave it at 48,000. * Audio is just all static, probably due to the sample rate thing, although when I use `-ar 48000` it's still static. * The audio/video queue warning messages are gone. * Frame rate capture seems unaffected. -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:29> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Reproduced by developer: 1 Blocking: | Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by thilo.borgmann): For the audio: I forgot to mentioned there are new options for that. "-list_audio_formats true" shows the formats available. "-audio_format_index N" selects the one you want. I tested with my phone connected to line-in and it works for me with any of the formats given there (which are not marked as unsupported). Please post the output of -list_audio_formats here and try some another formats for recording - static (silence) seems to be just another problem I did only encounter with microphone so far. For the video: I don't see any problems in the first few hundred frames, visually checked. However, there are duplicated frames like 790-791. This looks to me that the applications window was not redrawn by the window server. I encounter similar behaviour when capturing the entire screen where some parts were redrawn while the 60fps application shows the very same pixels. The PTS timestamps from the Apple provided sample buffers show different values in these cases. Thus I think that it is not the case that ffmpeg is loosing frames for latency somewhere. To prove this assumption, another capture of more than one 60fps application should be helpful. Is this application you're using somehting I can download for free somewhere (are you actually playing or is it a self-running demo)? I would like to capture it next to PixelToy to be sure that there is nothing more we could do here. -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:30> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Reproduced by developer: 1 Blocking: | Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by thilo.borgmann): I found it non-free on Steam so I can't test with that. My tests with PixelToy validate my window server assumption. I would appreciate it if you would reevaluate using PixelToy (free at Mac App Store, the snowfall set works pretty well). For the audio part I will have another look at higher bitrates than my current devices offer. I would also appreciate to see the format listing of you audio input devices. -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:31> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Reproduced by developer: 1 Blocking: | Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by thilo.borgmann): I tested with Soundflower and it seems like not being capable of doing more than 96kHz during passthrough - means it is not the avfoundation device to blame for 192kHz not working. I will polish up the patch on github and post it to devel asap (means next week, unfortunately). Thanks. -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:32> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Reproduced by developer: 1 Blocking: | Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by LordHDL): I am not trying to get 192 kHz working, I am using 48 but ffmpeg is forcing it to change to 192. It only does this in the build from the mirror. -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:33> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Reproduced by developer: 1 Blocking: | Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by thilo.borgmann): Did you try to use -audio_format_index N to chose a 49kHz format? Does this work for you? Fro me, FFmpeg choses the first format available if no index is given which corresponds to 192kHz for the SoundFlower device. It sounds like FFmpeg resets the format within the SoundFlower audio configuration for you? -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:34> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Reproduced by developer: 1 Blocking: | Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by fthiery): For what it's worth, the single file patch here solves the issue in our case, but not the bigger overhaul 3-part patch that is suggested in #4513 and available here: http://thread.gmane.org/gmane.comp.video.ffmpeg.devel/203959/focus=203966 -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:35> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Reproduced by developer: 1 Blocking: | Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by thilo.borgmann): Thanks for the feedback! The overhaul patch is going to be void soon since we're currently trying to get a common codebase with Libav. This will unfortunately postpone a real update on this device for some time. However, thanks to your feedback, I will reevaluate again the corresponding tickets for the avfoundation device. Thanks! -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:36> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Reproduced by developer: 1 Blocking: | Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by LordHDL): Replying to [comment:36 thilo.borgmann]:
Thanks for the feedback!
The overhaul patch is going to be void soon since we're currently trying to get a common codebase with Libav. This will unfortunately postpone a real update on this device for some time.
However, thanks to your feedback, I will reevaluate again the corresponding tickets for the avfoundation device.
Thanks!
Is there currently any way to test? I am unfamiliar with Libav. I'm also curious if this problem (this ticket) is the result of limitations within AVFoundation itself. Thanks. -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:37> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Reproduced by developer: 1 Blocking: | Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by thilo.borgmann): Unfortunately there has been no progress for a long time. You can expect me to get back to the overhaul patch soon. -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:38> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Reproduced by developer: 1 Blocking: | Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by LordHDL): Came across something curious in a recent test. I did a lossless RGB recording at 720p and when played back in ffplay the FPS doesn't seem smooth (as much as usual, anyway). However when played back in VLC it seems fine. Still has the intermittent drops but the rest was smooth. Here's an example: https://www.dropbox.com/s/du4bxjltwi0uae3/720p60.mkv?dl=0 -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:39> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Reproduced by developer: 1 Blocking: | Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by LordHDL): I know this issue is on the back burner for now but I have some more information that might be helpful to you guys. I have a couple recorded tests using Pixel Toy here: http://puu.sh/pX4FU/8119e38474.xz Included are the console outputs and screen shots that show the write speed during the screen capture as well as what my SSD is capable of. Now here's another tidbit that might be a factor. I have an app called RefreshRate that displays the rate of the GPU currently selected. I forget where I got it but I've uploaded it here: http://puu.sh/pX4GV/594298bee7.tar According to that, my AMD Radeon 6750M has a rate of 59.901460. I'm not sure if this is playing a part but there it is just in case. -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:40> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Reproduced by developer: 1 Blocking: | Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by LordHDL): All right so I have yet ''more'' information that might be helpful. I have these 2 videos recorded via Syphon Recorder (couldn't use PixelToy because it didn't work with Syphon Inject): (Ignore registration prompt and download directly.) https://www.dropbox.com/s/2wg1hwydxomdgaa/60.mkv?dl=0 https://www.dropbox.com/s/l56iuma9hrz9uum/AFAP.mkv?dl=0 The first one is very similar to the videos I get when recording using ffmpeg. The second is perfectly smooth. The difference is in choosing a number frame rate versus "As Fast As Possible" in the preferences. Here are their average rates respectively: [[Image(https://puu.sh/rgumQ/57d2f5abed.png)]] [[Image(https://puu.sh/rgumk/7e86c11bd3.png)]] The source in question does not run higher than 60 FPS anyway, so choosing "As Fast As Possible" nets every possible frame it can deliver, resulting in a perfect video. Is there a way to set ffmpeg to record "as fast as possible" with AVFoundation screen capture? I think that may be a good workaround for this issue for the time being. -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:41> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Reproduced by developer: 1 Blocking: | Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by eugeneware): Hi Thilo, I know this on the backburner, but do you have an estimate on when you're likely to get time to get back to this? There's quite a few projects like https://github.com/wulkano/aperture.js which could be using `ffmpeg` and thus produce a platform-independent screen recording code-base which are having to fork their code for OS X and use Swift/Objective-C directly to get a quality screen recording with audio which is a shame for both performance reasons and also the audio glitching. Hope everything is going well with the code-base merge. Good luck, and thanks so much for your incredible contributions. My mind was blown when I discovered last week that this was even possible! -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:42> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Reproduced by developer: 1 Blocking: | Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by thilo.borgmann): Expect me to continue this year. Unfortunately, I can't make any more promises. -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:43> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Reproduced by developer: 1 Blocking: | Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by LordHDL): I wasn't aware at the time of posting this ticket, but I've recently learned about VFR, CFR, and how they relate to each other. Based on my testing with other software, it seems I can only capture every frame properly when recording VFR, as CFR will guarantee that frames are lost as the sources in question (contrary to what I previously thought) are ''not'' a constant rate. Their advertised rate is 60 but in practice the FPS always fluctuates, making CFR unreliable for recording. I believe recording with the native input frame rate as variable might be the solution (as it was in other software) but I haven't been able to test it. I tried `-vsync 0` (input, output, and also both at once) but the resulting video is always considered constant. How do I tell ffmpeg to use VFR for input (and really that should be the default behavior)? -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:44> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Reproduced by developer: 1 Blocking: | Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by thilo.borgmann): I'm note sure if this actually comes into the game here. If so, I would why frame drops only happen so "rarely". -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:45> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Blocking: | Reproduced by developer: 1 Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by Correa): It's not just the video frames that has drops but the audio as well (see #4513), and then it's very noticeable. Been trying to find a solution to the problem for a week in a program using ffmpeg as a library and only applying this patch that I was able to hear intelligible audio. Unfortunately after the commit with support for raw video this patch doesn't apply cleanly anymore on top of HEAD. -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:46> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: important | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Blocking: | Reproduced by developer: 1 Analyzed by developer: 0 | -------------------------------------+------------------------------------- Changes (by bgreen965): * priority: normal => important Comment: This is still an issue on 4.3. Is there an update for the patch?The bug causes audio dropouts that make AVFoundation audio unusable. -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:47> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Blocking: | Reproduced by developer: 1 Analyzed by developer: 0 | -------------------------------------+------------------------------------- Changes (by cehoyos): * priority: important => normal -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:48> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Blocking: | Reproduced by developer: 1 Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by varenc): For those struggling with the avfoundation audio stuttering issue, I resolved it just using the older ffmpeg 4.2-tessus to record. That predates this ticket so clearly YMMV. Specifically I was getting audio stuttering with avfoundation only when using a virtual audio loopback device. (Loopback/Soundflower/etc) -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:49> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#4437: AVFoundation recording has consistent intermittent frame drops -------------------------------------+------------------------------------- Reporter: LordHDL | Owner: mateo Type: defect | Status: open Priority: normal | Component: avdevice Version: git-master | Resolution: Keywords: | Blocked By: avfoundation bounty | Blocking: | Reproduced by developer: 1 Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by varenc): For those struggling with the avfoundation audio stuttering issue, I resolved it just using the older ffmpeg 4.2-tessus to record. That predates this ticket so clearly YMMV. Specifically I was getting audio stuttering with avfoundation only when using a virtual audio loopback device. (Loopback/Soundflower/etc) -- Ticket URL: <https://trac.ffmpeg.org/ticket/4437#comment:50> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
participants (1)
-
FFmpeg