2009/11/3 Carl Eugen Hoyos <cehoyos at ag.or.at>

Please run tools/patcheck on your patch and/or read http://ffmpeg.org/developer.html#SEC4 (esp. the following paragraph, in short:

Ok, here is fixed patch. But i think it's temporary, until we have encoding support in hwaccels, or maybe it would be better to move all crusher functionality into libx264.c?

Sergiy <piratfm <at> gmail.com> writes:

Sorry, it was not full. Here is full patch.

There are still some cosmetic issues (alphabetical order in configure, Makefile and allcodecs.c, variable declarations of same type on same line), but before sending another patch, please wait for an actual review by Michael (or another developer). Carl Eugen

On Wed, Nov 04, 2009 at 04:54:17PM +0200, Sergiy wrote:

Sorry, it was not full. Here is full patch.

...

Ok, here is fixed patch.

configure | 7 + libavcodec/Makefile | 1 libavcodec/allcodecs.c | 1 libavcodec/libcrusher264.c | 264 +++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 273 insertions(+) f4a6f6e34161bf9d5c90fd20491aa308037e57c7 libcrusher264.patch diff --git a/configure b/configure index b876e66..eca21e3 100755 --- a/configure +++ b/configure [...] --- a/libavcodec/Makefile +++ b/libavcodec/Makefile

not maintained by me ... [...]

+/* parse every qbox, received in codec flushed event */ +static int add_nals(AVCodecContext *ctx, uint8_t *out_buff, int in_bufsize)

comments should be doxygen compatible [...]

+ +static int CrusherEnc_frame(AVCodecContext *ctx, uint8_t *buf, + int bufsize, void *data) +{ + CrusherEncContext *c4 = ctx->priv_data; + AVFrame *frame = data; + int ret = -1; + int datalen = 0; + + assert(ctx->pix_fmt == PIX_FMT_YUV420P); + + if(frame){ + /* Convert planes to the crusher format (UV12 blocked) */

what is UV12 blocked?

+ if (frame->linesize[1] != frame->linesize[2]) { + av_log(ctx, AV_LOG_ERROR, "U and V stride differ\n"); + return -1; + } + + c4->converter.y_stride = frame->linesize[0]; + c4->converter.uv_stride = frame->linesize[1]; + c4->converter.y = frame->data[0]; + c4->converter.u = frame->data[1]; + c4->converter.v = frame->data[2]; + if (imgConverter_convert(&c4->converter) != CONVERTER_FINISHED) { + av_log(ctx, AV_LOG_ERROR, "Converter failed!\n"); + return -1; + } + + do { + ret = crusher_encode(&c4->enc, c4->converter.iyuv_blocked, c4->converter.iyuv_frame_size_d); + if(ret == CODEC_FLUSHED) { + datalen+= add_nals(ctx, buf+datalen, bufsize-datalen); + /* hack: maybe qbox cts can be usable here? */ + ctx->coded_frame->pts = frame->pts;

whats a qbox cts ?

+ } + } while (ret == CODEC_FLUSHED); + } else if(!c4->enc.finished) {

+ av_log(ctx, AV_LOG_ERROR, "Codec finishing!\n");

doesnt sound like an error [...]

+static av_cold int CrusherEnc_init(AVCodecContext *avctx) +{ + CrusherEncContext *c4 = avctx->priv_data; + + crusher_defaults(&c4->enc); + c4->enc.out_format = OUT_FORMAT_QBOX; + c4->enc.width = avctx->width; + c4->enc.height = avctx->height; + c4->enc.bitrate = avctx->bit_rate; + c4->enc.framerate = avctx->time_base.den; + c4->enc.gopsize = avctx->gop_size; + if(avctx->rc_buffer_size) + c4->enc.rc_size = avctx->rc_buffer_size; + + c4->enc.scenecut_threshold = avctx->scenechange_threshold; + c4->enc.deblock = (avctx->flags & CODEC_FLAG_LOOP_FILTER) ? 1 : 0; + c4->enc.deblock_alpha = avctx->deblockalpha; + c4->enc.deblock_beta = avctx->deblockbeta; +

+ av_log(avctx, AV_LOG_ERROR, "fmt=%d, w=%d, h=%d, br=%d, fr=%d, gop_s=%d, rc_s=%d, sc_newgop=%d, db=%d, db_a=%d, db_b=%d", + c4->enc.out_format, c4->enc.width, c4->enc.height, c4->enc.bitrate, c4->enc.framerate, c4->enc.gopsize, + c4->enc.rc_size, c4->enc.scenecut_threshold, c4->enc.deblock, c4->enc.deblock_alpha, c4->enc.deblock_beta );

dont misuse AV_LOG_ERROR [...] -- Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB It is not what we do, but why we do it that matters.

On Mon, Nov 09, 2009 at 02:29:14PM +0200, Sergiy wrote:

Michael Niedermayer ?????:

...

what is UV12 blocked?

It's bug, must be NV12 (YUV 4:2:0 image with a plane of 8 bit Y samples followed by an interleaved U/V plane containing 8 bit 2x2 subsampled colour difference samples). But width must be divisible by 64 and height - by 32. It also converted by some (unknown to me) algorithm, looks like chess desk with 32x64 cells. Some description of converting given at the top of crusher264/src/imgConverter.c

and where do i find that file?

...

whats a qbox cts ?

This is "current timestamp", generated by device, here is values, generated

is there some documentation about it somewhere?

for 25fps: 0 - for sps/pps. 0 - for first frame (idr). next frames: 100, 200, 300, 400, 500 and so on...

does the hardware support b frames? [....] -- Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB Everything should be made as simple as possible, but not simpler. -- Albert Einstein

On Mon, Nov 09, 2009 at 11:55:47PM +0000, Carl Eugen Hoyos wrote:

Michael Niedermayer <michaelni <at> gmx.at> writes:

...

...

It also converted by some (unknown to me) algorithm, looks like chess desk with 32x64 cells. Some description of converting given at the top of crusher264/src/imgConverter.c

and where do i find that file?

http://tipok.org.ua/node/13

thanks [...] -- Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB Avoid a single point of failure, be that a person or equipment.

Sergiy <piratfm <at> gmail.com> writes:

Here is new patch for MG1264-hardware support via crusher264.

Two files are missing license headers (I don't think you have to re-send, but this should be fixed in a future version). Carl Eugen

Sergiy <piratfm <at> gmail.com> writes:

I'm removed that files in my working copy for now (that was yuv420 -> nv12 converter). Now codec accepts nv12 as native input.

While I have no idea if the patch is acceptable at all, that is a reason to resubmit, imo. Carl Eugen

On Fri, Apr 16, 2010 at 07:58:24PM +0300, Sergiy wrote:

Hi all

As i said before, now encoder accepts NV12 natively.

Long time I'd tried to understand why PTS wrapping does not works, but seems it's known ffmpeg's issue,

which issue on roundup is it? [...]

+/* parse first 6x4 bytes of header + * TODO: parse headers with extensions (may be longer) + * + */

not doxy compatible

+int ff_qbox_parse(qboxContext *qbox, uint8_t *input_data, int data_size) +{ + const uint8_t *src = input_data; + + if(data_size < QBOX_HDR_SIZE) + return 0; + + + qbox->qbox_size = bytestream_get_be32(&src); + if(qbox->qbox_size < QBOX_HDR_SIZE) + return 0; + + qbox->qbox_size-=QBOX_HDR_SIZE;

+ bytestream_get_be32(&src); //qbox + qbox->version = bytestream_get_byte(&src); + qbox->boxflags = bytestream_get_be24(&src); + qbox->sample_stream_type = bytestream_get_be16(&src); + qbox->sample_stream_id = bytestream_get_be16(&src); + qbox->sample_flags = bytestream_get_be32(&src); + qbox->sample_cts = bytestream_get_be32(&src);

vertical align [...]

+ + +/* + * demuxer probe: + * first 4 bytes of every qbox - is full qbox length + * next 4 bytes - chars "qbox", try find them + */

missing /** also please try tools/patcheck [...]

+/* + * extradata set, create packets from qboxes + * */ +static int qbox_read_packet(AVFormatContext *s, AVPacket *pkt) { + qboxDemux *d = s->priv_data; + qboxContext *c = &d->qbox; + ByteIOContext *pb = s->pb; + uint8_t qbox_header[QBOX_HDR_SIZE]; + int ret = 0; + + if(!d->from_header) { + if(!get_buffer(pb, qbox_header, QBOX_HDR_SIZE)) + return AVERROR(EIO); + + if(!ff_qbox_parse(c, qbox_header, QBOX_HDR_SIZE)) + return AVERROR(EIO); + } else { + d->from_header=0; + } +#ifdef QBOX_EXTRA_DEBUG + av_log(s, AV_LOG_DEBUG, "qbox: s:%d t:0x%08x id:%08x f:%08x cts:%d\n", + c->qbox_size, c->sample_stream_type, c->sample_stream_id, c->sample_flags, c->sample_cts); +#endif + + if ((ret = av_get_packet(pb, pkt, c->qbox_size)) <= 0) + return AVERROR(EIO); + + pkt->dts = c->sample_cts; + if(c->sample_flags & SAMPLE_FLAGS_SYNC_POINT) + pkt->flags |= PKT_FLAG_KEY; + + + switch (c->sample_stream_type) { + case SAMPLE_TYPE_QMA: + case SAMPLE_TYPE_PCM: + case SAMPLE_TYPE_AAC: + pkt->stream_index = 1; + break; + case SAMPLE_TYPE_H264: + pkt->stream_index = 0;

missing checks that these streams exist at all [...]

+enum { + SAMPLE_TYPE_AAC = 0x1, + SAMPLE_TYPE_H264, + SAMPLE_TYPE_PCM, + SAMPLE_TYPE_DEBUG, + SAMPLE_TYPE_H264_SLICE, + SAMPLE_TYPE_QMA, + SAMPLE_TYPE_VIN_STATS_GLOBAL, + SAMPLE_TYPE_VIN_STATS_MB, + SAMPLE_TYPE_Q711, + SAMPLE_TYPE_Q728, + SAMPLE_TYPE_MAX, +};

the enum should be named and the type should then be used [...]

+/** parse every qbox, received from codec, when have "flush" event + */ +static int add_nals(AVCodecContext *ctx, int alloc_buff) +{ + CrusherEncContext *c4 = ctx->priv_data; + qboxContext qbox; + int i; + int oldptr = c4->frames_buff_fill; + + if(!c4->enc.out_blocks) + return 0; + + /* in most cases we have 6 qboxes at one event */ + for(i=c4->curr_block; i < c4->enc.out_blocks; i++) { + if(ff_qbox_parse(&qbox, c4->enc.out_data[i].data, c4->enc.out_data[i].len)) { +#ifdef CRUSHER_EXTRA_DEBUG + av_log(ctx, AV_LOG_DEBUG, "QBOX: sst:0x%08x sf:0x%08x l:%d ts:%d\n", + qbox.sample_stream_type, qbox.sample_flags, qbox.qbox_size, qbox.sample_cts); +#endif + if(qbox.sample_stream_type != SAMPLE_TYPE_H264) { + av_log(ctx, AV_LOG_ERROR, "Unknown sample flags 0x%08x (maybe audio?)\n", qbox.sample_flags); + return -1; + } + + if(!ff_qbox_make_startcode(qbox.data, qbox.qbox_size)){ + av_log(ctx, AV_LOG_ERROR, "can't create H264 startcodes\n"); + return -1; + } + + /* payload started ? */ + if(qbox.sample_flags & SAMPLE_FLAGS_CONFIGURATION_INFO) { + if(alloc_buff) { + c4->sps_pps_size = qbox.qbox_size; + c4->sps_pps = av_malloc(c4->sps_pps_size); + memcpy(c4->sps_pps, qbox.data, c4->sps_pps_size); + } else { + av_log(ctx, AV_LOG_ERROR, "SPS/PPS received in the middle of encoding\n"); + } + } else { + if(c4->frames_buff_skip) { + c4->frames_buff_skip--; + continue; + } + if(c4->frames_buff_fill == c4->frames_buff_size) { + if(alloc_buff) { + c4->frames_buff_size++; + c4->frames_buff = av_realloc(c4->frames_buff, sizeof(qboxContext) * c4->frames_buff_size);

missing check for malloc fail [...]

+/** encode 1 frame + */ +static int CrusherEnc_frame(AVCodecContext *ctx, uint8_t *buf, + int bufsize, void *data) +{ + CrusherEncContext *c4 = ctx->priv_data; + AVFrame *frame = data; + int ret = -1; + int datalen = 0; + + assert(ctx->pix_fmt == PIX_FMT_NV12); + + if(frame) {

+ /* canvas size & copy frame + * can't use av_picture_copy due uv width must be equal y width */ + ff_img_copy_plane(c4->conv_input.data[0], c4->conv_input.linesize[0], + frame->data[0], frame->linesize[0], ctx->width, ctx->height); + ff_img_copy_plane(c4->conv_input.data[1], c4->conv_input.linesize[1], + frame->data[1], frame->linesize[1], ctx->width, ctx->height>>1); + + /* Convert planes to the crusher format (sequence of 32x16 blocks in NV12 format) */ + conv_frame(c4);

you copy data twice, thats surely not needed

+ do { + ret = crusher_encode(&c4->enc, c4->conv_output.data[0], c4->enc.inputFrameLen); + if(ret == CODEC_FLUSHED) + add_nals(ctx, 0); + } while (ret == CODEC_FLUSHED); + } else if(!c4->enc.finished) { + av_log(ctx, AV_LOG_DEBUG, "Codec finishing!\n"); + do { + ret = crusher_encode(&c4->enc, NULL, 0); + if (ret == CODEC_FLUSHED) + add_nals(ctx, 0); + } while(ret != CODEC_FAIL && ret != CODEC_FINISHED); + } + + if(ret == CODEC_FAIL) { + av_log(ctx, AV_LOG_ERROR, "Failed\n"); + return -1; + } + + /* send frame from buffer */ + if(c4->frames_buff_fill) { + assert(bufsize >= c4->frames_buff[0].size); +#ifdef CRUSHER_EXTRA_DEBUG + av_log(ctx, AV_LOG_DEBUG, "Frame out[0/%d/%d]:" + "len:%d, type:%d, ts:%d\n", + c4->frames_buff_fill, c4->frames_buff_size, + c4->frames_buff[0].qbox_size, + c4->frames_buff[0].sample_stream_type, + c4->frames_buff[0].sample_cts); +#endif + datalen = c4->frames_buff[0].qbox_size; + memcpy(buf, c4->frames_buff[0].data, datalen); + av_freep(&c4->frames_buff[0].data); + + if(c4->frames_buff[0].sample_flags & SAMPLE_FLAGS_SYNC_POINT) { + ctx->coded_frame->key_frame = 1; + ctx->coded_frame->pict_type = FF_I_TYPE; + } else { + ctx->coded_frame->key_frame = 0; + ctx->coded_frame->pict_type = FF_P_TYPE; + } + //just return number of frame? + if (c4->frames_buff[0].sample_flags & SAMPLE_FLAGS_CTS_PRESENT) {

+ ctx->coded_frame->pts = c4->frames_buff[0].sample_cts/c4->enc.framerate_den - c4->frames_delay;

the division looks strange [...] -- Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB Many things microsoft did are stupid, but not doing something just because microsoft did it is even more stupid. If everything ms did were stupid they would be bankrupt already.

2009/11/3 Gwenole Beauchesne <gbeauchesne at splitted-desktop.com>

Hi,

On Tue, 3 Nov 2009, Sergiy wrote:

Also I integrate it with ffmpeg as libcrusher264, it's in very beta state,

...

but usable.

Looks interesting. Due to the lack of "native" FFmpeg H.264 encoder,

I would be interested in extending the hwaccel hooks to support encoding

This would be best solution, i think.

but I'd rather add those to the libx264 interface instead of an extra libcrusher264 one.

We can add some flag to ffmpeg "try to detect hw h264 accel". In libx264.c crusher264 code can be selected, instead of x264 code (so we don't need to add one more codec to ffmpeg).

I can help if I have the relevant HW.

I bought one on ebay it cost ~30$.