#10642(avcodec:new): [hwaccel] AV1 hardware decoding for Apple M3
#10642: [hwaccel] AV1 hardware decoding for Apple M3 -------------------------------------+------------------------------------- Reporter: Nomis101 | Type: | enhancement Status: new | Priority: normal Component: avcodec | Version: git- Keywords: hwaccel | master videotoolbox av1 | Blocked By: Blocking: | Reproduced by developer: 0 Analyzed by developer: 0 | -------------------------------------+------------------------------------- Now that the Apple M3 family does support AV1 hardware decoding, it would be nice to add this to the hwaccel framework (videotoolbox.c). There is kCMVideoCodecType_AV1 for this. https://www.apple.com/newsroom/2023/10/apple-unveils-m3-m3-pro-and-m3-max- the-most-advanced-chips-for-a-personal-computer/ https://developer.apple.com/documentation/coremedia/kcmvideocodectype_av1/ -- Ticket URL: <https://trac.ffmpeg.org/ticket/10642> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#10642: [hwaccel] AV1 hardware decoding for Apple M3 -------------------------------------+------------------------------------- Reporter: Nomis101 | Owner: (none) Type: enhancement | Status: new Priority: normal | Component: avcodec Version: git-master | Resolution: Keywords: hwaccel | Blocked By: videotoolbox av1 | Blocking: | Reproduced by developer: 0 Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by jeeb): Unfortunately Apple has not added an AV1 software decoder to VT, even though they ship dav1d as part of Safari for AVIF. Thus this (very hacky) code was written completely blind, and the only verification done was that the CFDataRef should contain what appears to be a valid av1C: https://github.com/jeeb/ffmpeg/commits/videotoolbox_av1 Only builds as static due to the av1C generation not being properly shared between libraries (as I don't want to start that work until I at least know that things work on some level). Could be checked with: {{{ ffmpeg -v verbose -hwaccel videotoolbox -i INPUT -map 0:v -f null - }}} On my 2015 Intel Macbook pro I just get {{{ [av1 @ 0x7fbf6e00a1c0] avc1C: size: 17, first bytes: 81 08 4d 00 [av1 @ 0x7fbf6e00a1c0] VideoToolbox decoder for this format not found. [av1 @ 0x7fbf6e00a1c0] Failed setup for format videotoolbox_vld: hwaccel initialisation returned error. [av1 @ 0x7fbf6e00a1c0] Your platform doesn't support hardware accelerated AV1 decoding. [av1 @ 0x7fbf6e00a1c0] Failed to get pixel format. }}} -- Ticket URL: <https://trac.ffmpeg.org/ticket/10642#comment:1> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#10642: [hwaccel] AV1 hardware decoding for Apple M3 -------------------------------------+------------------------------------- Reporter: Nomis101 | Owner: (none) Type: enhancement | Status: new Priority: normal | Component: avcodec Version: git-master | Resolution: Keywords: hwaccel | Blocked By: videotoolbox av1 | Blocking: | Reproduced by developer: 0 Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by Nomis101): Thanks for your input. This will not work on Intel, only on M3 and later (and maybe latest iPhone). Would you be able to test this on Apple M3 hardware? -- Ticket URL: <https://trac.ffmpeg.org/ticket/10642#comment:2> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#10642: [hwaccel] AV1 hardware decoding for Apple M3 -------------------------------------+------------------------------------- Reporter: Nomis101 | Owner: (none) Type: enhancement | Status: new Priority: normal | Component: avcodec Version: git-master | Resolution: Keywords: hwaccel | Blocked By: videotoolbox av1 | Blocking: | Reproduced by developer: 0 Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by Arkelic): I was able to set up this branch on my M3 MacBook pro and here is the output when using [https://github.com/SPBTV/video_av1_samples/blob/master/spbtv_sample_bipbop_a... this sample] file: {{{ ffmpeg -v verbose -hwaccel videotoolbox -i /Users/arkelic/Downloads/spbtv_sample_bipbop_av1_960x540_25fps.mp4 -map 0:v -f null - ffmpeg version N-112731-g8d79cdeccb Copyright (c) 2000-2023 the FFmpeg developers built with Apple clang version 15.0.0 (clang-1500.1.0.2.2) configuration: libavutil 58. 32.100 / 58. 32.100 libavcodec 60. 33.100 / 60. 33.100 libavformat 60. 17.100 / 60. 17.100 libavdevice 60. 4.100 / 60. 4.100 libavfilter 9. 13.100 / 9. 13.100 libswscale 7. 6.100 / 7. 6.100 libswresample 4. 13.100 / 4. 13.100 Selecting decoder 'av1' because of requested hwaccel method videotoolbox Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/Users/arkelic/Downloads/spbtv_sample_bipbop_av1_960x540_25fps.mp4': Metadata: major_brand : mp42 minor_version : 0 compatible_brands: avc1isommp42avc1 creation_time : 2018-10-05T14:40:45.000000Z Duration: 00:00:15.04, start: 0.000000, bitrate: 130 kb/s Stream #0:0[0x1](und): Audio: aac (LC) (mp4a / 0x6134706D), 32000 Hz, mono, fltp, 8 kb/s (default) Metadata: handler_name : snd vendor_id : [0][0][0][0] Stream #0:1[0x2](und): Video: av1 (Main), 1 reference frame (av01 / 0x31307661), yuv420p(tv), 960x540, 116 kb/s, 25 fps, 25 tbr, 90k tbn (default) Metadata: handler_name : vid vendor_id : [0][0][0][0] [out#0/null @ 0x6000038506c0] Adding streams from explicit maps... [vost#0:0/wrapped_avframe @ 0x12e605db0] Created video stream from input stream 0:1 Stream mapping: Stream #0:1 -> #0:0 (av1 (native) -> wrapped_avframe (native)) Press [q] to stop, [?] for help [av1 @ 0x12e609fb0] avc1C: size: 17, first bytes: 81 04 0c 00 [av1 @ 0x12e609fb0] vt decoder cb: output image buffer is null: -12911 [av1 @ 0x12e609fb0] HW accel end frame fail. [vist#0:1/av1 @ 0x12e609e60] Error submitting packet to decoder: Unknown error occurred [av1 @ 0x12e609fb0] VideoToolbox decoder needs reconfig, restarting.. [av1 @ 0x12e609fb0] avc1C: size: 17, first bytes: 81 04 0c 00 [av1 @ 0x12e609fb0] vt decoder cb: output image buffer is null: -12911 [av1 @ 0x12e609fb0] HW accel end frame fail. [vist#0:1/av1 @ 0x12e609e60] Error submitting packet to decoder: Unknown error occurred [1] 64153 segmentation fault ffmpeg -v verbose -hwaccel videotoolbox -i -map 0:v -f null - }}} -- Ticket URL: <https://trac.ffmpeg.org/ticket/10642#comment:3> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#10642: [hwaccel] AV1 hardware decoding for Apple M3 -------------------------------------+------------------------------------- Reporter: Nomis101 | Owner: (none) Type: enhancement | Status: new Priority: normal | Component: avcodec Version: git-master | Resolution: Keywords: hwaccel | Blocked By: videotoolbox av1 | Blocking: | Reproduced by developer: 0 Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by NG): Will we get AV1 on M3 soon ? Personally for Moonlight gamestream -- Ticket URL: <https://trac.ffmpeg.org/ticket/10642#comment:4> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#10642: [hwaccel] AV1 hardware decoding for Apple M3 -------------------------------------+------------------------------------- Reporter: Nomis101 | Owner: (none) Type: enhancement | Status: new Priority: normal | Component: avcodec Version: git-master | Resolution: Keywords: hwaccel | Blocked By: videotoolbox av1 | Blocking: | Reproduced by developer: 0 Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by Aaron Graham): It seems like FFmpeg doesn't support AV1 decoding using the macOS VideoToolbox API yet. There are few Merge Requests and suggestions here but no final call to merge them: https://github.com/jeeb/ffmpeg/commits/videotoolbox_av1 -- Ticket URL: <https://trac.ffmpeg.org/ticket/10642#comment:5> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#10642: [hwaccel] AV1 hardware decoding for Apple M3 -------------------------------------+------------------------------------- Reporter: Nomis101 | Owner: (none) Type: enhancement | Status: new Priority: normal | Component: avcodec Version: git-master | Resolution: Keywords: hwaccel | Blocked By: videotoolbox av1 | Blocking: | Reproduced by developer: 0 Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by quinkblack): The most difficult thing for developer is we don't have the hardware for test. Apple doesn't provide a software implementation for products before M3. It's hard if not impossible for personal developer to has access to the latest hardware. -- Ticket URL: <https://trac.ffmpeg.org/ticket/10642#comment:6> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#10642: [hwaccel] AV1 hardware decoding for Apple M3 -------------------------------------+------------------------------------- Reporter: Nomis101 | Owner: (none) Type: enhancement | Status: new Priority: normal | Component: avcodec Version: git-master | Resolution: Keywords: hwaccel | Blocked By: videotoolbox av1 | Blocking: | Reproduced by developer: 0 Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by Aaron Graham): Any luck on this port ? -- Ticket URL: <https://trac.ffmpeg.org/ticket/10642#comment:7> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#10642: [hwaccel] AV1 hardware decoding for Apple M3 -------------------------------------+------------------------------------- Reporter: Nomis101 | Owner: (none) Type: enhancement | Status: new Priority: normal | Component: avcodec Version: git-master | Resolution: Keywords: hwaccel | Blocked By: videotoolbox av1 | Blocking: | Reproduced by developer: 0 Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by nenkoru): Currently working on this. Trying to overcome the -12911 issue, but not luck so far. Already took a look into how chromium & moonlight implemented this + one forun on apple developer's platform. Tried adding more info into configuration that is being passed to VT, no luck. One thing that I noticed that getting Sequence OBU Header always return the same length and bytes for different sample videos: av1C = {length = 17, bytes = 0x81040c000a0b00000024cf7f0dbfff3008}; Buf I am not sure what length and bytes should be returned, I suppose it should be 20, but not sure what each bit refers to, reading ISO documentation for now. -- Ticket URL: <https://trac.ffmpeg.org/ticket/10642#comment:8> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#10642: [hwaccel] AV1 hardware decoding for Apple M3 -------------------------------------+------------------------------------- Reporter: Nomis101 | Owner: (none) Type: enhancement | Status: new Priority: normal | Component: avcodec Version: git-master | Resolution: Keywords: hwaccel | Blocked By: videotoolbox av1 | Blocking: | Reproduced by developer: 0 Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by padenot): Hi, we've recently done this in Firefox in https://bugzilla.mozilla.org/show_bug.cgi?id=1871796 (diff at https://phabricator.services.mozilla.com/D205841, complete source at https://searchfox.org/mozilla- central/source/dom/media/platforms/apple/AppleVTDecoder.cpp), let us know (e.g. on matrix at matrix.to/#/#media:mozilla.org or email padenot@mozilla.com) if you need us to dump some values e.g. some extradata or something. Firefox is fairly easy to build and work with if you want to do it yourself to compare, please don't hesitate to ask (for that or anything else). -- Ticket URL: <https://trac.ffmpeg.org/ticket/10642#comment:9> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#10642: [hwaccel] AV1 hardware decoding for Apple M3 -------------------------------------+------------------------------------- Reporter: Nomis101 | Owner: (none) Type: enhancement | Status: new Priority: normal | Component: avcodec Version: git-master | Resolution: Keywords: hwaccel | Blocked By: videotoolbox av1 | Blocking: | Reproduced by developer: 0 Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by nenkoru): Made this to work. Essentially the only thing that was missing is properly copying the sequence header in a decode_params function, and copying the frame data into the bitstream from the start_frame func. Preparing the patch, will send it asap. Huge thanks to @padenot who dumped the frames from the firefox. -- Ticket URL: <https://trac.ffmpeg.org/ticket/10642#comment:10> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#10642: [hwaccel] AV1 hardware decoding for Apple M3 -------------------------------------+------------------------------------- Reporter: Nomis101 | Owner: (none) Type: enhancement | Status: new Priority: normal | Component: avcodec Version: git-master | Resolution: Keywords: hwaccel | Blocked By: videotoolbox av1 | Blocking: | Reproduced by developer: 0 Analyzed by developer: 0 | -------------------------------------+------------------------------------- Comment (by nenkoru): Here are the links to patchwork of ffmpeg: https://patchwork.ffmpeg.org/project/ffmpeg/patch/1e1173f1-9ba6-40cc-b491-e0... https://patchwork.ffmpeg.org/project/ffmpeg/patch/821cf833-ed3d-4068-92fb- 01b247b767ee@gmail.com/ -- Ticket URL: <https://trac.ffmpeg.org/ticket/10642#comment:11> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
#10642: [hwaccel] AV1 hardware decoding for Apple M3 -------------------------------------+------------------------------------- Reporter: Nomis101 | Owner: (none) Type: enhancement | Status: closed Priority: normal | Component: avcodec Version: git-master | Resolution: fixed Keywords: hwaccel | Blocked By: videotoolbox av1 | Blocking: | Reproduced by developer: 0 Analyzed by developer: 0 | -------------------------------------+------------------------------------- Changes (by Nomis101): * status: new => closed * resolution: => fixed Comment: Closed by https://github.com/FFmpeg/FFmpeg/commit/f9c5c5358cfef3847674c6f3b3ded9611ebc... -- Ticket URL: <https://trac.ffmpeg.org/ticket/10642#comment:12> FFmpeg <https://ffmpeg.org> FFmpeg issue tracker
participants (1)
-
FFmpeg