[FFmpeg-devel] [Internet][PATCH 00/12] Add MediaCodec encoder and NDK MediaCodec support
Olivier Ayache
olivier.ayache at gmail.com
Mon Nov 14 18:43:31 EET 2022
Hello here is the work I have pushed in a fork of FFmpeg integrated in
source code of xuggler (a Java/C++ wrapper of FFmpeg)
https://github.com/olivierayache/xuggle-xuggler/commit/98469ebea59c921f46a114bd954f7a27a7c95a41
For the limitations of NDK I agree with you and that is why I have made
some backport to be able to support all needed features for FFmpeg (when I
worked on that it was FFmpeg 3.4.x version).
This backport allows support to these functions for Android devices since
API 21 (which is more than 99% of devices).
- AMediaCodec_createInputSurface
- AMediaCodec_setParameters
- ANativeWindow_toSurface
For testing reviewing the patch I will make it as soon as I update FFmpeg
to the current version. I will be able to do it this week.
Olivier
Le dim. 13 nov. 2022, 12:49, Zhao Zhili <quinkblack at foxmail.com> a écrit :
>
> > From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of
> Olivier Ayache
> > Sent: 2022年11月12日 18:00
> > To: FFmpeg development discussions and patches <ffmpeg-devel at ffmpeg.org>
> > Cc: matthieu.bouron at stupeflix.com; aman at tmm1.net
> > Subject: Re: [FFmpeg-devel] [Internet][PATCH 00/12] Add MediaCodec
> encoder and NDK MediaCodec support
> >
> > Hello there I implemented that few years ago in
> > https://github.com/olivierayache/xuggle-xuggler/
>
> I can't find where is the patch or source code of FFmpeg.
>
> > And when I tried to submit a patch to FFmpeg for add support for NDK I
> had
> > received this answer.
> > I think using NDK directly rather than JNI can be better in case of
> > developing native application on Android.
>
> NDK is still missing some important features, and some features only
> supported recently.
> So the NDK implementation doesn't meant to be a replacement of the JNI
> version:
> 1. JNI version has more features and supports more devices
> 2. NDK implementation combined with some JNI functions has almost the same
> feature
> As JNI version on new devices.
> 3. In the environment where JVM is unavailable, the NDK version is the
> only choice.
>
> > If I can help on these subjects I would be really happy
>
> Welcome to review and/or test the patch.
>
> >
> > Olivier Ayache
> >
> >
> >
> > ---------- Forwarded message ---------
> > From: Olivier Ayache <olivier.ayache at gmail.com>
> > Date: Sun, Jun 28, 2020 at 2:48 PM
> > Subject: Re: [FFmpeg-devel] [PATCH] add support to Android ndk MediaCodec
> > for encoding/decoding
> > To: <ffmpeg-devel at ffmpeg.org>
> >
> >
> > Thank for your replies I choose the NDK approach in order to be able to
> be
> > independant from the JVM, to maximize performance and to be sure to
> detect
> > compatibility issues (using JNI approach can compile and crash at
> runtime).
> > Concerning the version of Android it is compatible from API 21 (94% of
> > devices)
> > I agree that JNI overhead is not big but in combinaison with Xuggler it
> > seems to me a little weird because Xuggler (Java part) already use JNI to
> > interact with Xuggler (C++) and FFmpeg. If FFmpeg uses JNI in the other
> way
> > (from c to Java) it seems to me an anti pattern.
> >
> > If you think it is better to begin to use JNI I could transform my
> encoder
> > and we could discuss about NDK after
> >
> > Olivier
> >
> >
> >
> > Le sam. 27 juin 2020 à 20:55, Martin Storsjö <martin at martin.st> a écrit
> :
> >
> > > On Sat, 27 Jun 2020, Olivier Ayache wrote:
> > >
> > > > Hi everyone this is the first time I post on this mailing list. I am
> > > > working since several years on a fork of Xuggler for manipulating
> ffmpeg
> > > > API with Java/Kotlin.
> > > > This work leads me to develop encoder and decoder based on Android
> NDK
> > > > MediaCodec.
> > > >
> > > > This work can be found on my Github repository
> > > >
> > > > https://github.com/olivierayache/xuggle-xuggler
> > > >
> > > >
> > > > I know that FFmpeg already integrate MediaCodec for decoding via Jni
> > > > wrappers since version 3.1. I began this work on FFmpeg 2.8.x and I
> > > choose
> > > > the NDK in order to achieve optimum performance.
> > >
> > > If you mean you used the NDK MediaCodec API, I'd suggest you use the
> same
> > > JNI wrappers as ffmpeg has already, for consistency. The overhead of a
> few
> > > JNI calls per encoded frame is generally negligible. If it, at a later
> > > time, is decided to drop support for older versions at some point, it
> > > should be straightforward to convert it to use the NDK MediaCodec API
> > > instead.
> > >
> > > // Martin
> > >
> > > _______________________________________________
> > > ffmpeg-devel mailing list
> > > ffmpeg-devel at ffmpeg.org
> > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> > >
> > > To unsubscribe, visit link above, or email
> > > ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
> >
> >
> > On Sat, Nov 12, 2022 at 6:07 AM Zhao Zhili <quinkblack at foxmail.com>
> wrote:
> >
> > >
> > >
> > > > -----Original Message-----
> > > > From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of
> Chema
> > > Gonzalez
> > > > Sent: 2022年11月12日 0:55
> > > > To: FFmpeg development discussions and patches <
> ffmpeg-devel at ffmpeg.org>
> > > > Cc: matthieu.bouron at stupeflix.com; aman at tmm1.net
> > > > Subject: Re: [FFmpeg-devel] [Internet][PATCH 00/12] Add MediaCodec
> > > encoder and NDK MediaCodec support
> > > >
> > > > On Thu, Nov 10, 2022 at 7:36 PM "zhilizhao(赵志立)" <
> quinkblack at foxmail.com>
> > > wrote:
> > > > >
> > > > > Ping for review.
> > > > >
> > > > > > On Oct 24, 2022, at 11:16, Zhao Zhili <quinkblack at foxmail.com>
> > > wrote:
> > > > > >
> > > > > > From: Zhao Zhili <zhilizhao at tencent.com>
> > > > > >
> > > > > > Firstly, some bugs were fixed (patch 1-4).
> > > > > >
> > > > > > Patch 5 and 6 make mediacodec_wrapper support Java MediaCodec
> and NDK
> > > > > > MediaCodec. The use case I'm considering is run FFmpeg on cmdline
> > > without JVM,
> > > > > > for example, run FFmpeg inside of termux (an Android terminal
> > > emulator). It's
> > > > > > well known that NDK MediaCodec missing some important functions,
> > > like get the
> > > > > > list of codecs, but still useable.
> > > > > >
> > > > > > Patch 7 add NDK MediaCodec decoder support. It can be enabled via
> > > options,
> > > > > > and enabled automatically if no JVM is available.
> > > > > >
> > > > > > Patch 8 add ANativeWindow support to hwcontext_mediacodec. It
> can be
> > > set by
> > > > > > user, and can be created via
> AMediaCodec_createPersistentInputSurface
> > > > > > automatically. This is a preparation for encoder.
> > > > > >
> > > > > > Patch 9 makes MediaCodec decoder to support ANativeWindow
> directly.
> > > It worth
> > > > > > to note that AVMediaCodecContext has only surface. Although we
> > > provided
> > > > > > av_mediacodec_alloc_context(), we didn't strictly prevent users
> to
> > > allocate
> > > > > > AVMediaCodecContext on stack. I'm not sure if it's OK to add new
> > > field to
> > > > > > AVMediaCodecContext.
> > > > > >
> > > > > > Patch 10 add MediaCodec encoder support. Frame can be feed to
> > > encoder via
> > > > > > buffer, or via Surface/ANativeWindow. If Surface/ANativeWindow is
> > > used, and
> > > > > > the frames come from our MediaCodec decoder wrapper, we can
> control
> > > it's
> > > > > > 'render' (send to encoder's surface) via
> > > av_mediacodec_release_buffer(). A DTS
> > > > > > generation strategy works in this case. However, if frames comes
> > > from other
> > > > > > sources, like a camera, there is no way to control the 'render'
> yet,
> > > so DTS is
> > > > > > missing in this case.
> > > > > >
> > > > > > Finally, we can do mediacodec transcoding with FFmpeg cmdline on
> > > Android.
> > > > > > More importantly, we can do MediaCodec decoder to encoder without
> > > copy frames,
> > > > > > although it's very limited since most of avfilters doesn't work.
> For
> > > example:
> > > > > >
> > > > > > ./ffmpeg -hwaccel mediacodec -hwaccel_output_format mediacodec -i
> > > /sdcard/test.mp4 -an -c:v h264_mediacodec -y
> > > > /sdcard/out.mp4
> > > >
> > > > How do you build the ffmpeg binary you're using for experiments in
> the
> > > > android device?
> > >
> > > Just configure with --enable-mediacodec and --enable-jni. Here is the
> build
> > > script I used for the development.
> > >
> > > https://github.com/quink-black/ffmpeg-ci/blob/master/android_ffmpeg.sh
> > >
> > > Here is the Android Studio project which can be used to test the
> mediacodec
> > > wrapper implemented via JNI.
> > >
> > > >
> > > >
> > > >
> > > >
> > > > > >
> > > > > > Since there is no real AVHWFrameContext implementation in
> > > hwcontext_mediacodec.
> > > > > > there is no hwframe_ctx for mediacodec and
> > > av_hwframe_transfer_data() doesn't
> > > > > > work. So if -hwaccel_output_format isn't being specified like:
> > > > > >
> > > > > > ./ffmpeg -hwaccel mediacodec -i /sdcard/test.mp4 -an -c:v
> > > h264_mediacodec -y /sdcard/out.mp4
> > > > > >
> > > > > > It will trigger a crash in av_hwframe_transfer_data. Patch 11
> add a
> > > check on
> > > > > > hwframe_ctx. Patch 12 set hwaccel_output_format automatically to
> > > avoid such
> > > > > > case.
> > > > > >
> > > > > >
> > > > > > Zhao Zhili (12):
> > > > > > avcodec/mediacodec: fix incorrect crop info
> > > > > > avcodec/mediacodecdec: don't break out if both input and output
> port
> > > > > > return try again
> > > > > > avcodec/mediacodecdec_common: fix misuse av_free/av_freep
> > > > > > avcodec/mediacodecdec_common: fix useless av_buffer_unref
> > > > > > avcodec/mediacodec_wrapper: separate implementation from
> interface
> > > > > > avcodec/mediacodec: add NDK media codec wrapper
> > > > > > avcodec/mediacodecdec: enable NDK mediacodec
> > > > > > avutil/hwcontext_mediacodec: add ANativeWindow support
> > > > > > avcodec/mediacodec: add ANativeWindow support
> > > > > > avcodec: add MediaCodec encoder
> > > > > > avutil/hwcontext: verify hw_frames_ctx in transfer_data_alloc
> > > > > > fftools/ffmpeg_opt: set default hwaccel_output_format for
> mediacodec
> > > > > >
> > > > > > Changelog | 2 +
> > > > > > configure | 6 +
> > > > > > fftools/ffmpeg_opt.c | 4 +
> > > > > > libavcodec/Makefile | 2 +
> > > > > > libavcodec/allcodecs.c | 2 +
> > > > > > libavcodec/mediacodec_surface.c | 46 +-
> > > > > > libavcodec/mediacodec_surface.h | 8 +-
> > > > > > libavcodec/mediacodec_wrapper.c | 942
> > > +++++++++++++++++++++++++++---
> > > > > > libavcodec/mediacodec_wrapper.h | 275 +++++++--
> > > > > > libavcodec/mediacodecdec.c | 21 +-
> > > > > > libavcodec/mediacodecdec_common.c | 35 +-
> > > > > > libavcodec/mediacodecdec_common.h | 1 +
> > > > > > libavcodec/mediacodecenc.c | 495 ++++++++++++++++
> > > > > > libavcodec/version.h | 2 +-
> > > > > > libavutil/hwcontext.c | 6 +-
> > > > > > libavutil/hwcontext_mediacodec.c | 56 +-
> > > > > > libavutil/hwcontext_mediacodec.h | 11 +
> > > > > > libavutil/version.h | 4 +-
> > > > > > 18 files changed, 1780 insertions(+), 138 deletions(-)
> > > > > > create mode 100644 libavcodec/mediacodecenc.c
> > > > > >
> > > > > > --
> > > > > > 2.25.1
> > > > > >
> > > > > >
> > > > >
> > > > > _______________________________________________
> > > > > ffmpeg-devel mailing list
> > > > > ffmpeg-devel at ffmpeg.org
> > > > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> > > > >
> > > > > To unsubscribe, visit link above, or email
> > > > > ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
> > > > _______________________________________________
> > > > ffmpeg-devel mailing list
> > > > ffmpeg-devel at ffmpeg.org
> > > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> > > >
> > > > To unsubscribe, visit link above, or email
> > > > ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
> > >
> > > _______________________________________________
> > > ffmpeg-devel mailing list
> > > ffmpeg-devel at ffmpeg.org
> > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> > >
> > > To unsubscribe, visit link above, or email
> > > ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
> > >
> > _______________________________________________
> > ffmpeg-devel mailing list
> > ffmpeg-devel at ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >
> > To unsubscribe, visit link above, or email
> > ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
>
More information about the ffmpeg-devel
mailing list