[FFmpeg-devel] [PATCH] avcodec: add new Videotoolbox hwaccel.

Zhang Rui bbcallen at gmail.com
Thu Jul 30 11:12:21 CEST 2015


2015-07-30 16:37 GMT+08:00 wm4 <nfxjfg at googlemail.com>:
> On Thu, 30 Jul 2015 16:08:36 +0800
> Zhang Rui <bbcallen at gmail.com> wrote:
>
>> 2015-07-30 0:01 GMT+08:00 Clément Bœsch <u at pkh.me>:
>> > On Tue, Jul 28, 2015 at 09:01:50PM +0800, Zhang Rui wrote:
>> >> 2015-07-28 16:04 GMT+08:00 Clément Bœsch <u at pkh.me>:
>> >> > On Mon, Jul 27, 2015 at 10:10:12PM +0800, Zhang Rui wrote:
>> >> >> 2015-07-27 20:14 GMT+08:00 Clément Bœsch <u at pkh.me>:
>> >> >> > On Mon, Jul 27, 2015 at 06:13:30PM +0800, Zhang Rui wrote:
>> >> >> >> 2015-07-15 21:51 GMT+08:00 Clément Bœsch <u at pkh.me>:
>> >> >> >> > On Sat, Jun 20, 2015 at 01:33:00PM +0200, Sebastien Zwickert wrote:
>> >> >> >> >> Old videtotoolbox patch rebased and updated to target the new HWAccel API.
>> >> >> >> >> As VDA is a wrapper of VideoToolbox framework, the changes base vda implementation
>> >> >> >> >> upon the videotoolbox implementation to factorize common part of code.
>> >> >> >> >>
>> >> >> >> >> ---
>> >> >> >> >>  Changelog                    |   1 +
>> >> >> >> >>  MAINTAINERS                  |   1 +
>> >> >> >> >>  Makefile                     |   5 +-
>> >> >> >> >>  configure                    |  19 +-
>> >> >> >> >>  ffmpeg.h                     |   2 +
>> >> >> >> >>  ffmpeg_opt.c                 |   5 +-
>> >> >> >> >>  ffmpeg_vda.c                 | 134 --------
>> >> >> >> >>  ffmpeg_videotoolbox.c        | 147 +++++++++
>> >> >> >> >>  libavcodec/Makefile          |  12 +-
>> >> >> >> >>  libavcodec/allcodecs.c       |   5 +
>> >> >> >> >>  libavcodec/h263dec.c         |   3 +
>> >> >> >> >>  libavcodec/h264_slice.c      |   4 +
>> >> >> >> >>  libavcodec/mpeg12dec.c       |   3 +
>> >> >> >> >>  libavcodec/vda.c             |   2 +-
>> >> >> >> >>  libavcodec/vda_h264.c        | 154 +---------
>> >> >> >> >>  libavcodec/vda_internal.h    |  33 --
>> >> >> >> >>  libavcodec/vda_vt_internal.h |  57 ++++
>> >> >> >> >>  libavcodec/version.h         |   2 +-
>> >> >> >> >>  libavcodec/videotoolbox.c    | 705 +++++++++++++++++++++++++++++++++++++++++++
>> >> >> >> >>  libavcodec/videotoolbox.h    | 126 ++++++++
>> >> >> >> >>  libavutil/pixdesc.c          |   4 +
>> >> >> >> >>  libavutil/pixfmt.h           |   2 +
>> >> >> >> >
>> >> >> >> > I tried on iOS (iPhone 6+ so last one unless I'm mistaken) with a 240fps
>> >> >> >> > footage, and it seems the decode calls take more than 5ms, which seems not
>> >> >> >> > enough. Do you have any idea what could cause this? (I'm running a timer
>> >> >> >> > after/before avcodec_decode_video*())
>> >> >> >>
>> >> >> >> 1. kVTDecodeFrame_EnableAsynchronousDecompression can make it faster,
>> >> >> >> but not suitable for sync-style API (avcodec_decode_video*())
>> >> >> >
>> >> >> > not suitable in the sense that it wouldn't work, or it would have
>> >> >> > undesirable side effects?
>> >> >>
>> >> >> It would work, but it would be better and easier if decoder's input
>> >> >> and output are decoupled.
>> >> >
>> >> >> Anyhow, as wm4 said, it's probably not worth the trouble.
>> >> >>
>> >> >
>> >> > Well, if it makes possible for ffmpeg to decode 240fps footage @ 240Hz
>> >> > that's kind of worth the trouble IMO. The HW decoder seems to be able to
>> >> > decode frames between 700 to 1100µs, which makes it possible to handle
>> >> > such material. A factor 5 to 6 with FFmpeg is kind of a problem IMO.
>> >>
>> >> Would you mind to share your 240fps test sample?
>> >> I'd like to take a benchmark in my own player with async VideoToolbox.
>> >> https://github.com/Bilibili/ijkplayer
>> >
>> > Well, you can just capture a video with the slowmo mode in the camera of
>> > the iphone (that's where you plan to bench, right?).
>>
>> Thank you.
>>
>> This is my test result on iPhone6 (iOS 8.4):
>>
>>   Metadata:
>>     major_brand     : qt
>>     minor_version   : 0
>>     compatible_brands: qt
>>     creation_time   : 2015-07-30 06:02:29
>>     encoder         : 8.4
>>     encoder-zho     : 8.4
>>     date            : 2015-07-30T14:02:29+0800
>>     date-zho        : 2015-07-30T14:02:29+0800
>>     location        : +31.2423+121.5203+020.306/
>>     location-zho    : +31.2423+121.5203+020.306/
>>     model           : iPhone 6
>>     model-zho       : iPhone 6
>>     make            : Apple
>>     make-zho        : Apple
>>   Duration: 00:00:31.12, start: 0.000000, bitrate: 40163 kb/s
>>     Stream #0:0(und), 1, 1/2400: Video: h264, 1 reference frame (avc1
>> / 0x31637661), yuv420p(tv, bt709, left), 1280x720, 1/4800, 40069 kb/s,
>> 239.84 fps, 240 tbr, 2400 tbn, 4800 tbc (default)
>>     Metadata:
>>       creation_time   : 2015-07-30 06:02:29
>>       handler_name    : Core Media Data Handler
>>       encoder         : H.264
>>     Stream #0:1(und), 87, 1/44100: Audio: aac (LC) (mp4a /
>> 0x6134706D), 44100 Hz, mono, fltp, 62 kb/s (default)
>>     Metadata:
>>       creation_time   : 2015-07-30 06:02:29
>>       handler_name    : Core Media Data Handler
>>
>> 1. Frames and time are count in VTB callback.
>> 2. Without rendering.
>> 3. At most 3 AVPackets are sent to VTB before any decoded frame being
>> received in callback.
>>
>> ----- sync -----
>> 192.886031 fps, 5.184409 ms/frame, 7440 frames
>> 202.669572 fps, 4.934140 ms/frame, 7440 frames
>> 207.768997 fps, 4.813038 ms/frame, 7440 frames
>> 204.986913 fps, 4.878360 ms/frame, 7440 frames
>> 200.836820 fps, 4.979167 ms/frame, 7440 frames
>> 221.152131 fps, 4.521774 ms/frame, 7440 frames
>>
>> ----- async  -----
>> 451.840155 fps, 2.213172 ms/frame, 7440 frames
>> 450.117974 fps, 2.221640 ms/frame, 7440 frames
>> 450.663275 fps, 2.218952 ms/frame, 7440 frames
>> 453.686200 fps, 2.204167 ms/frame, 7440 frames
>> 454.517686 fps, 2.200134 ms/frame, 7440 frames
>> 454.739930 fps, 2.199059 ms/frame, 7440 frames
>
> Could this be fixed by using the async API in lavc and adding more
> buffering?

Only easy if libavformat gurantee that an AVPacket contains one and
only one frame.
I don't know how VTB would act if AVPacket contains multiple frames.

And I don't know how to retrieve last delayed frames with
avcodec_decode_video*() after EOF.
Maybe call it with a zero-len packet?

BTW:
I think current patch is OK enough.
Async-mode could be an experimental option.


More information about the ffmpeg-devel mailing list