[FFmpeg-devel] [PATCH] added support for hardware assist H264 video encoding for the Raspberry Pi
Amancio Hasty
ahasty at gmail.com
Wed Jun 22 21:11:59 CEST 2016
> On Jun 22, 2016, at 10:20 AM, Aman Gupta <ffmpeg at tmm1.net> wrote:
>
> This patch should fix the assertion failure you see:
> https://github.com/FFmpeg/FFmpeg/commit/1087f0dc172a9adf779e41bf2dc82639fb04ebd4 <https://github.com/FFmpeg/FFmpeg/commit/1087f0dc172a9adf779e41bf2dc82639fb04ebd4>
>
> Aman
> On Sat, Jun 18, 2016 at 8:43 AM Amancio Hasty <ahasty at gmail.com <mailto:ahasty at gmail.com>> wrote:
>
>>
>>> On Jun 16, 2016, at 11:16 AM, Aman Gupta <ffmpeg at tmm1.net> wrote:
>>>
>>> The patchset that was merged into libav is now available in ffmpeg as
>> well:
>>> https://github.com/FFmpeg/FFmpeg/blob/master/libavcodec/omx.c
>>>
>>> You can compile ffmpeg from the master branch
>>> with --enable-omx --enable-omx-rpi
>>>
>>> Aman
>>>
>>> On Thu, Jun 16, 2016 at 2:16 AM, Amancio Hasty <ahasty at gmail.com> wrote:
>>>
>>>>
>>>>> On May 9, 2016, at 7:55 AM, Amancio Hasty <ahasty at gmail.com> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> So what is the next step?
>>>>>
>>>>> If you want testers I suggest posting to ccrisan’s motionpie mailing
>>>> list.
>>>>> Cheers
>>>>> Amancio
>>>>>
>>>>>
>>>>>> On Mar 31, 2016, at 7:27 PM, Amancio Hasty <ahasty at gmail.com> wrote:
>>>>>>
>>>>>> I am not a lawyer…
>>>>>>
>>>>>>
>>>>>> I updated the patch. vc264.c now has a the copyright notice embedded
>> in
>>>>>> a volatile global so if a binary is compiled against vc264.o , the
>>>> copyright notice
>>>>>> can be displayed by:
>>>>>> strings ffmpeg | grep -i copyright
>>>>>>
>>>>>> LICENSE.md has been updated to include Broadcom’s copyright notice.
>>>>>>
>>>>>> A distribution of a binary that includes vc264.o should include
>>>> LICENSE.md and if
>>>>>> that is missing, the copyright notice can be displayed via the shell
>>>>>> command ‘strings’ .
>>>>>>
>>>>>> Amancio
>>>>>> <c-0001-added-support-for-hardware-assist-H264-video-encodin.patch>
>>>>>>> On Mar 22, 2016, at 12:12 PM, Lou Logan <lou at lrcd.com> wrote:
>>>>>>>
>>>>>>> On Mon, 21 Mar 2016 20:07:01 -0700, Amancio Hasty wrote:
>>>>>>>
>>>>>>>> From 874a72eec2a78f4935fea091003e534b5f8d5413 Mon Sep 17 00:00:00
>> 2001
>>>>>>>> From: Amancio Hasty <ahasty at gmail.com>
>>>>>>>> Date: Mon, 21 Mar 2016 18:56:05 -0700
>>>>>>>> Subject: [PATCH] added support for hardware assist H264 video
>>>> encoding for
>>>>>>>> the Raspberry Pi
>>>>>>>>
>>>>>>>> ---
>>>>>>>> configure | 12 ++
>>>>>>>> libavcodec/Makefile | 1 +
>>>>>>>> libavcodec/allcodecs.c | 2 +
>>>>>>>> libavcodec/vc264.c | 387
>>>> +++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>> 4 files changed, 402 insertions(+)
>>>>>>>> create mode 100644 libavcodec/vc264.c
>>>>>>>>
>>>>>>> [...]
>>>>>>>> diff --git a/libavcodec/allcodecs.c b/libavcodec/allcodecs.c
>>>>>>>> index 2a25d66..3c7bd9b 100644
>>>>>>>> --- a/libavcodec/allcodecs.c
>>>>>>>> +++ b/libavcodec/allcodecs.c
>>>>>>>> @@ -74,6 +74,7 @@ void avcodec_register_all(void)
>>>>>>>> initialized = 1;
>>>>>>>>
>>>>>>>
>>>>>>> Nit: Whitespace on the line above should be removed.
>>>>>>>
>>>>>>> [...]
>>>>>>>> --- /dev/null
>>>>>>>> +++ b/libavcodec/vc264.c
>>>>>>>> @@ -0,0 +1,387 @@
>>>>>>>> +/* H.264 hardware assist video encoding code taken from
>>>>>>>> + * raspberry's os :
>>>>>>>> + * /opt/vc/src/hello_pi/hello_encode/encode.c
>>>>>>>> + */
>>>>>>>> +
>>>>>>>> +/*
>>>>>>>> +Copyright (c) 2012, Broadcom Europe Ltd
>>>>>>>> +Copyright (c) 2012, Kalle Vahlman <zuh at iki>
>>>>>>>> + Tuomas Kulve <tuomas at kulve.fi>
>>>>>>>> +All rights reserved.
>>>>>>>> +
>>>>>>>> +Redistribution and use in source and binary forms, with or without
>>>>>>>> +modification, are permitted provided that the following conditions
>>>> are met:
>>>>>>>> +* Redistributions of source code must retain the above copyright
>>>>>>>> + notice, this list of conditions and the following disclaimer.
>>>>>>>> + * Redistributions in binary form must reproduce the above
>>>> copyright
>>>>>>>> + notice, this list of conditions and the following disclaimer
>>>> in the
>>>>>>>> + documentation and/or other materials provided with the
>>>> distribution.
>>>>>>>> + * Neither the name of the copyright holder nor the
>>>>>>>> + names of its contributors may be used to endorse or promote
>>>> products
>>>>>>>> + derived from this software without specific prior written
>>>> permission.
>>>>>>>> +
>>>>>>>> +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
>>>> "AS IS" AND
>>>>>>>> +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
>>>> THE IMPLIED
>>>>>>>> +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
>>>> ARE
>>>>>>>> +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS
>>>> BE LIABLE FOR ANY
>>>>>>>> +DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
>>>> DAMAGES
>>>>>>>> +(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
>>>> SERVICES;
>>>>>>>> +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
>>>> CAUSED AND
>>>>>>>> +ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
>>>> OR TORT
>>>>>>>> +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
>>>> USE OF THIS
>>>>>>>> +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
>>>>>>>
>>>>>>> I wonder if any of the above legalese is compatible. Granted, I see a
>>>>>>> similar paragraph in "libavformat/aadec.c".
>>>>>>>
>>>>>>>> + * ffmpeg driver for hardware assist video H.264 encoding using
>>>> Broadcom's GPU
>>>>>>>> + * Copyright (C) 2016 Amancio Hasty ahasty at gmail.com
>>>>>>>> + *
>>>>>>>> + *
>>>>>>>> + * This file is part of FFmpeg.
>>>>>>>> + *
>>>>>>>> + * FFmpeg is free software; you can redistribute it and/or
>>>>>>>> + * modify it under the terms of the GNU Lesser General Public
>>>>>>>> + * License as published by the Free Software Foundation; either
>>>>>>>> + * version 2.1 of the License, or (at your option) any later
>> version.
>>>>>>>> + *
>>>>>>>> + * FFmpeg is distributed in the hope that it will be useful,
>>>>>>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>>>>>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> GNU
>>>>>>>> + * Lesser General Public License for more details.
>>>>>>>> + *
>>>>>>>> + * You should have received a copy of the GNU Lesser General Public
>>>>>>>> + * License along with FFmpeg; if not, write to the Free Software
>>>>>>>> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
>>>> 02110-1301 USA
>>>>>>>> + *
>>>>>>>> + */
>>>>>>>> +
>>>>>>>> +
>>>>>>>> +/**
>>>>>>>> + * @ file vc264.c
>>>>>>>> + * Broadcom bm2865's Visual Core hardware assist h264 using
>>>>>>>> + openMax interface to the GPU.
>>>>>>>> +
>>>>>>>> +*/
>>>>>>>> +
>>>>>>>> +#include <stdio.h>
>>>>>>>> +#include <stdlib.h>
>>>>>>>> +#include <string.h>
>>>>>>>> +#define OMX_SKIP64BIT
>>>>>>>> +#include "bcm_host.h"
>>>>>>>> +#include "ilclient.h"
>>>>>>>> +#include "avcodec.h"
>>>>>>>> +#include "internal.h"
>>>>>>>> +
>>>>>>>> +typedef struct VC264Context {
>>>>>>>> + OMX_VIDEO_PARAM_PORTFORMATTYPE format;
>>>>>>>> + OMX_PARAM_PORTDEFINITIONTYPE def;
>>>>>>>> + COMPONENT_T *video_encode;
>>>>>>>> + COMPONENT_T *list[5];
>>>>>>>> + OMX_BUFFERHEADERTYPE *buf;
>>>>>>>> + OMX_BUFFERHEADERTYPE *out;
>>>>>>>> + ILCLIENT_T *client;
>>>>>>>> + OMX_VIDEO_PARAM_BITRATETYPE bitrateType;
>>>>>>>> + int width;
>>>>>>>> + int height;
>>>>>>>> + int bit_rate;
>>>>>>>> +} VC264Context;
>>>>>>>> +
>>>>>>>> +
>>>>>>>> +static int vc264_init(AVCodecContext *avctx) {
>>>>>>>> +
>>>>>>>> +
>>>>>>>> +
>>>>>>>> + OMX_ERRORTYPE r;
>>>>>>>> + int error;
>>>>>>>> +
>>>>>>>> +
>>>>>>>> +
>>>>>>>> + VC264Context *vc = avctx->priv_data;
>>>>>>>> +
>>>>>>>> + vc->width = avctx->width;
>>>>>>>> + vc->height = avctx->height;
>>>>>>>> + vc->bit_rate = avctx->bit_rate;
>>>>>>>> + printf("vc264: bit rate %d \n", avctx->bit_rate);
>>>>>>>> +#if FF_API_CODED_FRAME
>>>>>>>> +FF_DISABLE_DEPRECATION_WARNINGS
>>>>>>>> +
>>>>>>>> + avctx->coded_frame = av_frame_alloc();
>>>>>>>> + avctx->coded_frame->pict_type = AV_PICTURE_TYPE_I;
>>>>>>>> +FF_ENABLE_DEPRECATION_WARNINGS
>>>>>>>> +#endif
>>>>>>>> +
>>>>>>>> +
>>>>>>>> + memset(&vc->list, 0, sizeof(vc->list));
>>>>>>>> + bcm_host_init();
>>>>>>>> + if ((vc->client = ilclient_init()) == NULL) {
>>>>>>>> + return -3;
>>>>>>>> + }
>>>>>>>> + error = OMX_Init();
>>>>>>>> +
>>>>>>>> + if (error != OMX_ErrorNone) {
>>>>>>>> + ilclient_destroy(vc->client);
>>>>>>>> + av_log(avctx,AV_LOG_ERROR,"in vc264_init OMX_Init failed ");
>>>>>>>> + return -4;
>>>>>>>> + }
>>>>>>>> +
>>>>>>>> + // create video_encode
>>>>>>>> + r = ilclient_create_component(vc->client, &vc->video_encode,
>>>> (char *) "video_encode",
>>>>>>>> + ILCLIENT_DISABLE_ALL_PORTS |
>>>>>>>> + ILCLIENT_ENABLE_INPUT_BUFFERS |
>>>>>>>> + ILCLIENT_ENABLE_OUTPUT_BUFFERS);
>>>>>>>
>>>>>>> Tabs should be converted to spaces. There are many instances of tabs
>>>>>>> being used in this patch.
>>>>>>>
>>>>>>> Others will have to provide a more technical review (not to mention
>>>>>>> possible additions docs, Changelog, MAINTAINERS, and
>>>>>>> libavcodec/version.h).
>>>>>>> _______________________________________________
>>>>>>> ffmpeg-devel mailing list
>>>>>>> ffmpeg-devel at ffmpeg.org
>>>>>>> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>>>>>>
>>>>>
>>>>
>>>> Any thoughts on how we may proceed forward?
>>>>
>>>> Thank You,
>>>> Amancio
>>>>
>>>>
>>>> _______________________________________________
>>>> ffmpeg-devel mailing list
>>>> ffmpeg-devel at ffmpeg.org
>>>> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>>>>
>>> _______________________________________________
>>> ffmpeg-devel mailing list
>>> ffmpeg-devel at ffmpeg.org
>>> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel <
>> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel <http://ffmpeg.org/mailman/listinfo/ffmpeg-devel>>
>>
>>
>> This is what I getting over here with RPI3:
>>
>> ./ffmpeg_g -loglevel debug -f v4l2 -input_format yuv420p -framerate 25
>> -video_size 640x480 -i /dev/video0 output.h264
>> ffmpeg version N-80363-g403a53c Copyright (c) 2000-2016 the FFmpeg
>> developers
>> built with gcc 4.9.2 (Raspbian 4.9.2-10)
>> configuration: --enable-gpl --enable-omx --enable-omx-rpi
>> --enable-nonfree --extra-libs=-ldl --extra-cflags=' -pipe '
>> libavutil 55. 24.100 / 55. 24.100
>> libavcodec 57. 46.100 / 57. 46.100
>> libavformat 57. 38.100 / 57. 38.100
>> libavdevice 57. 0.101 / 57. 0.101
>> libavfilter 6. 46.101 / 6. 46.101
>> libswscale 4. 1.100 / 4. 1.100
>> libswresample 2. 1.100 / 2. 1.100
>> libpostproc 54. 0.100 / 54. 0.100
>> Splitting the commandline.
>> Reading option '-loglevel' ... matched as option 'loglevel' (set logging
>> level) with argument 'debug'.
>> Reading option '-f' ... matched as option 'f' (force format) with argument
>> 'v4l2'.
>> Reading option '-input_format' ... matched as AVOption 'input_format' with
>> argument 'yuv420p'.
>> Reading option '-framerate' ... matched as AVOption 'framerate' with
>> argument '25'.
>> Reading option '-video_size' ... matched as AVOption 'video_size' with
>> argument '640x480'.
>> Reading option '-i' ... matched as input file with argument '/dev/video0'.
>> Reading option 'output.h264' ... matched as output file.
>> Finished splitting the commandline.
>> Parsing a group of options: global .
>> Applying option loglevel (set logging level) with argument debug.
>> Successfully parsed a group of options.
>> Parsing a group of options: input file /dev/video0.
>> Applying option f (force format) with argument v4l2.
>> Successfully parsed a group of options.
>> Opening an input file: /dev/video0.
>> [video4linux2,v4l2 @ 0x2189260] fd:3 capabilities:85200005
>> [video4linux2,v4l2 @ 0x2189260] Current input_channel: 0, input_name:
>> Camera 0, input_std: 0
>> [video4linux2,v4l2 @ 0x2189260] Setting time per frame to 1/25
>> [video4linux2,v4l2 @ 0x2189260] All info found
>> Input #0, video4linux2,v4l2, from '/dev/video0':
>> Duration: N/A, start: -140423796.316413, bitrate: 92160 kb/s
>> Stream #0:0, 1, 1/1000000: Video: rawvideo, 1 reference frame (I420 /
>> 0x30323449), yuv420p, 640x480, 0/1, 92160 kb/s, 25 fps, 25 tbr, 1000k tbn,
>> 1000k tbc
>> Successfully opened the file.
>> Parsing a group of options: output file output.h264.
>> Successfully parsed a group of options.
>> Opening an output file: output.h264.
>> [file @ 0x218c930] Setting default whitelist 'file,crypto'
>> Successfully opened the file.
>> detected 4 logical cores
>> [graph 0 input from stream 0:0 @ 0x2195270] Setting 'video_size' to value
>> '640x480'
>> [graph 0 input from stream 0:0 @ 0x2195270] Setting 'pix_fmt' to value '0'
>> [graph 0 input from stream 0:0 @ 0x2195270] Setting 'time_base' to value
>> '1/1000000'
>> [graph 0 input from stream 0:0 @ 0x2195270] Setting 'pixel_aspect' to
>> value '0/1'
>> [graph 0 input from stream 0:0 @ 0x2195270] Setting 'sws_param' to value
>> 'flags=2'
>> [graph 0 input from stream 0:0 @ 0x2195270] Setting 'frame_rate' to value
>> '25/1'
>> [graph 0 input from stream 0:0 @ 0x2195270] w:640 h:480 pixfmt:yuv420p
>> tb:1/1000000 fr:25/1 sar:0/1 sws_param:flags=2
>> [format @ 0x2195530] compat: called with args=[yuv420p]
>> [format @ 0x2195530] Setting 'pix_fmts' to value 'yuv420p'
>> [AVFilterGraph @ 0x2194b40] query_formats: 4 queried, 3 merged, 0 already
>> done, 0 delayed
>> [h264_omx @ 0x218c2c0] Using OMX.broadcom.video_encode
>> [h264_omx @ 0x218c2c0] OMX state changed to 2
>> [h264_omx @ 0x218c2c0] OMX state changed to 3
>> [h264_omx @ 0x218c2c0] OMX port 201 settings changed
>> [h264 @ 0x218b010] Using AVStream.codec to pass codec parameters to muxers
>> is deprecated, use AVStream.codecpar instead.
>> Output #0, h264, to 'output.h264':
>> Metadata:
>> encoder : Lavf57.38.100
>> Stream #0:0, 0, 1/25: Video: h264 (h264_omx), 1 reference frame,
>> yuv420p, 640x480, 0/1, q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc
>> Metadata:
>> encoder : Lavc57.46.100 h264_omx
>> Stream mapping:
>> Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (h264_omx))
>> Press [q] to stop, [?] for help
>> cur_dts is invalid (this is harmless if it occurs once at the start per
>> stream)
>> [rawvideo @ 0x218ac30] PACKET SIZE: 460800, STRIDE: 960
>> Clipping frame in rate conversion by 0.000008
>> Assertion ret <= 0 failed at libavcodec/utils.c:1956
>> Aborted
>>
>> How did you test h264 encoding?
>>
>> Thanks!
>> Amancio
>>
>>
>>
>>
>> _______________________________________________
>> ffmpeg-devel mailing list
>> ffmpeg-devel at ffmpeg.org <mailto:ffmpeg-devel at ffmpeg.org>
>> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel <http://ffmpeg.org/mailman/listinfo/ffmpeg-devel>
>>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org <mailto:ffmpeg-devel at ffmpeg.org>
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel <http://ffmpeg.org/mailman/listinfo/ffmpeg-devel>
Hello Aman,
Thanks for the patch. I noticed that in libav/utils.c:
ret = avctx->codec->encode2(avctx, avpkt, frame, got_packet_ptr);
1325 <https://git.libav.org/?p=libav.git;a=blob;f=libavcodec/utils.c;h=bc1beee4626d339717d328c605b60bd564580447;hb=HEAD#l1325> if (!ret) {
1326 <https://git.libav.org/?p=libav.git;a=blob;f=libavcodec/utils.c;h=bc1beee4626d339717d328c605b60bd564580447;hb=HEAD#l1326> if (*got_packet_ptr) {
1327 <https://git.libav.org/?p=libav.git;a=blob;f=libavcodec/utils.c;h=bc1beee4626d339717d328c605b60bd564580447;hb=HEAD#l1327> if (!(avctx->codec->capabilities & AV_CODEC_CAP_DELAY)) {
1328 <https://git.libav.org/?p=libav.git;a=blob;f=libavcodec/utils.c;h=bc1beee4626d339717d328c605b60bd564580447;hb=HEAD#l1328> if (avpkt->pts == AV_NOPTS_VALUE)
1329 <https://git.libav.org/?p=libav.git;a=blob;f=libavcodec/utils.c;h=bc1beee4626d339717d328c605b60bd564580447;hb=HEAD#l1329> avpkt->pts = frame->pts;
1330 <https://git.libav.org/?p=libav.git;a=blob;f=libavcodec/utils.c;h=bc1beee4626d339717d328c605b60bd564580447;hb=HEAD#l1330> if (!avpkt->duration)
1331 <https://git.libav.org/?p=libav.git;a=blob;f=libavcodec/utils.c;h=bc1beee4626d339717d328c605b60bd564580447;hb=HEAD#l1331> avpkt->duration = ff_samples_to_time_base(avctx,
1332 <https://git.libav.org/?p=libav.git;a=blob;f=libavcodec/utils.c;h=bc1beee4626d339717d328c605b60bd564580447;hb=HEAD#l1332> frame->nb_samples);
1333 <https://git.libav.org/?p=libav.git;a=blob;f=libavcodec/utils.c;h=bc1beee4626d339717d328c605b60bd564580447;hb=HEAD#l1333> }
1334 <https://git.libav.org/?p=libav.git;a=blob;f=libavcodec/utils.c;h=bc1beee4626d339717d328c605b60bd564580447;hb=HEAD#l1334> avpkt->dts = avpkt->pts;
1335 <https://git.libav.org/?p=libav.git;a=blob;f=libavcodec/utils.c;h=bc1beee4626d339717d328c605b60bd564580447;hb=HEAD#l1335> } else {
1336 <https://git.libav.org/?p=libav.git;a=blob;f=libavcodec/utils.c;h=bc1beee4626d339717d328c605b60bd564580447;hb=HEAD#l1336> avpkt->size = 0;
1337 <https://git.libav.org/?p=libav.git;a=blob;f=libavcodec/utils.c;h=bc1beee4626d339717d328c605b60bd564580447;hb=HEAD#l1337> }
Which is different than ffmpeg’s libavcodec/utils.c ….
Last but not least I get a bunch or warnings in ffmpeg when I run it
with your patch.
./ffmpeg_g -f v4l2 -input_format yuv420p -framerate 25 -video_size 640x480 -i /dev/video0 a.h264
DTS 140462612887087, next:1760000 st:0 invalid dropping
PTS 140462612887087, next:1760000 invalid dropping st:0
….
Thanks again!
Amancio
More information about the ffmpeg-devel
mailing list