[FFmpeg-devel] [PATCH] lavc/mediacodec: add hwaccel support

[Matthieu Bouron]

On Fri, Mar 18, 2016 at 5:50 PM, Matthieu Bouron <matthieu.bouron at gmail.com>
wrote:

> From: Matthieu Bouron <matthieu.bouron at stupeflix.com>
>
> ---
>
> Hello,
>
> The following patch add hwaccel support to the mediacodec (h264) decoder
> by allowing
> the user to render the output frames directly on a surface.
>
> In order to do so the user needs to initialize the hwaccel through the use
> of
> av_mediacodec_alloc_context and av_mediacodec_default_init functions. The
> later
> takes a reference to an android/view/Surface as parameter.
>
> If the hwaccel successfully initialize, the decoder output frames pix fmt
> will be
> AV_PIX_FMT_MEDIACODEC. The following snippet of code demonstrate how to
> render
> the frames on the surface:
>
>     AVMediaCodecBuffer *buffer = (AVMediaCodecBuffer *)frame->data[3];
>     av_mediacodec_release_buffer(buffer, 1);
>
> The last argument of av_mediacodec_release_buffer enable rendering of the
> buffer on the surface (or not if set to 0).
>
> Regarding the internal changes in the mediacodec decoder:
>
> MediaCodec.flush() discards both input and output buffers meaning that if
> MediaCodec.flush() is called all output buffers the user has a reference
> on are
> now invalid (and cannot be used).
> This behaviour does not fit well in the avcodec API.
>
> When the decoder is configured to output software buffers, there is no
> issue as
> the buffers are copied.
>
> Now when the decoder is configured to output to a surface, the user might
> not
> want to render all the frames as fast as the decoder can go and might want
> to
> control *when* the frame are rendered, so we need to make sure that the
> MediaCodec.flush() call is delayed until all the frames the user retains
> has
> been released or rendered.
>
> Delaying the call to MediaCodec.flush() means buffering any inputs that
> come
> the decoder until the user has released/renderer the frame he retains.
>
> This is a limitation of this hwaccel implementation, if the user retains a
> frame (a), then issue a flush command to the decoder, the packets he feeds
> to
> the decoder at that point will be queued in the internal decoder packet
> queue
> (until he releases the frame (a)). This scenario leads to a memory usage
> increase to say the least.
>
> Currently there is no limitation on the size of the internal decoder packet
> queue but this is something that can be added easily. Then, if the queue is
> full, what would be the behaviour of the decoder ? Can it block ? Or
> should it
> returns something like AVERROR(EAGAIN) ?
>
> About the other internal decoder changes I introduced:
>
> The MediaCodecDecContext is now refcounted (using the lavu/atomic api)
> since
> the (hwaccel) frames can be retained by the user, we need to delay the
> destruction of the codec until the user has released all the frames he has
> a
> reference on.
> The reference counter of the MediaCodecDecContext is incremented each time
> an
> (hwaccel) frame is outputted by the decoder and decremented each time a
> (hwaccel) frame is released.
>
> Also, when the decoder is configured to output to a surface the pts that
> are
> given to the MediaCodec API are now rescaled based on the codec_timebase as
> those timestamps values are propagated to the frames rendered on the
> surface
> since Android M. Not sure if it's really useful though.
>
> On the performance side:
>
> On a nexus 5, decoding an h264 stream (main profile) 1080p at 60fps:
>   - software output + rgba conversion goes at 59~60fps
>   - surface output + render on a surface goes at 100~110fps
>
>
[...]

Patch updated with the following differences:
  * the public mediacodec api is now always built (not only when mediacodec
is available) (and the build when mediacodec is not available has been
fixed)
  * the documentation of av_mediacodec_release_buffer has been improved a
bit

The development branch is located here:
https://github.com/mbouron/FFmpeg/tree/feature/mediacodec-hwaccel
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-lavc-add-mediacodec-hwaccel-support.patch
Type: text/x-diff
Size: 32403 bytes
Desc: not available
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20160322/48100520/attachment.patch>