- ffmpeg-devel - lists.ffmpeg.org

[PR] avcodec/videotoolboxenc: write A53 data in standalone SEI (PR #24056)
by Qingzheng Li 09 Aug '26

09 Aug '26

PR #24056 opened by Qingzheng Li (iSoldLeo) URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24056 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24056.patch ## Problem When VideoToolbox emits its own H.264 SEI NAL, the current A53 path tries to append the caption payload to that NAL. The insertion code scans already escaped EBSP as if it were RBSP, so emulation-prevention bytes can move the apparent message boundary. In longer caption-bearing streams this can corrupt the existing SEI, lose A53 data, or make following NAL units undecodable. ## Changes The first patch moves H.264-without-A53 and HEVC length-prefix conversion to a checked reader-based helper. It uses identical count and write passes, allocates the exact Annex B packet size, accepts only 1-, 2-, and 4-byte length prefixes, and rejects zero, truncated, or overlong NAL units. It also confines codec- and A53-specific code to the corresponding build configurations and releases an allocated packet if the adjacent raw CoreMedia copy path fails. Its only cosmetic change removes braces from the nearby single-statement keyframe flag check. The second patch keeps every VideoToolbox-produced H.264 NAL body byte-exact and writes A53 data in a standalone `user_data_registered_itu_t_t35` SEI before the first Annex A VCL NAL. The A53 path requires exactly one `CMSampleBuffer` sample and rejects samples without VCL or with a definite new-access-unit boundary after VCL. H.264 without A53 and HEVC use the generic converter; ProRes retains its raw-copy path. Portable FATE coverage checks NAL ordering, preservation of existing NAL bodies, 1/2/4-byte length prefixes, SEI size fields and escaping, exact and undersized capacities, malformed samples, definite post-VCL AU boundaries, and reader failures. ## Testing The broader matrix, sanitizer/fuzz/stress, and build-configuration results below were collected from the pre-rebase validation build (`N-125927-g25663e6924`). The relevant patched production and test sources used by that build are byte-identical to this series. The focused six-case VideoToolbox smoke and selected Homebrew reproduction were then rerun with the committed two-patch head (`N-125996-g458fb81bc7f`). - `make fate-videotoolboxenc-rewrite` - `make checkheaders` - ASan+UBSan unit test, 500,000-case independent-reference fuzz test, 32 MiB NAL copy stress, and 16 MiB synthetic SEI-writer capacity stress - H.264-only, HEVC-only, ProRes-only, static, shared, LTO, no-VideoToolbox, minimal, and full build configurations - A broader VideoToolbox matrix ran 179 cases per engine. The candidate passed all 157 reached cases. Reached coverage included caption-bearing AV1/H.264/HEVC/MPEG-2 inputs, A53 on/off, Baseline/Constrained Baseline/Main/High/Constrained High H.264, MP4/MKV/MOV/MPEG-TS/raw/fMP4/HLS, 23.976/59.94 fps, all-intra, B-frames, 360p through 4K, HEVC, and ProRes including alpha. Twenty Extended-profile cases, one real-multi-slice probe, and one HEVC-alpha probe were not reached and are not counted as passes. - On the committed two-patch head, a focused six-case VideoToolbox rerun passed H.264 at 60000/1001 fps, H.264 B-frames, H.264 with A53 disabled, a full-duration A53 input, HEVC, and ProRes 4444 with alpha. Homebrew reproduced the three selected old H.264/A53 failures. - Across the 150 cases where both candidate and Homebrew succeeded, normalized coded-picture content was identical (109 H.264, 15 HEVC, 26 ProRes). The seven candidate-only successes that reproduced the baseline A53/SEI failures contained 1,881/1,881 matching standalone A53 events under an independent AVCC/RBSP parser. For those seven files, AVFoundation frame counts, decoded NV12 active-pixel hashes, and microsecond-normalized PTS hashes were identical before and after removing all H.264 type-6 SEI NAL units. From 556c626f0b2404aed8436b33592141559ad242f3 Mon Sep 17 00:00:00 2001 From: Qingzheng Li <s(a)qwqlog.com> Date: Sat, 8 Aug 2026 23:07:21 +0800 Subject: [PATCH 1/2] avcodec/videotoolboxenc: use checked NAL length conversion The current packet path counts NAL units by advancing over unchecked length prefixes, then estimates the Annex B allocation from that count. A malformed sample may therefore pass the count stage and fail only after allocation. For packets without A53 data, move length-prefix conversion to a reader-based helper. Use the same checked parser for count and write passes, allocate the exact packet size, accept only 1-, 2-, and 4-byte prefixes, and reject zero, truncated, or overlong NAL units. Keep the existing A53 insertion path unchanged for now. Add portable FATE coverage for prefix widths, malformed samples, capacity limits, and reader failures. Signed-off-by: Qingzheng Li <s(a)qwqlog.com> --- libavcodec/Makefile | 6 +- libavcodec/tests/videotoolboxenc_rewrite.c | 154 +++++++++++++++++ libavcodec/videotoolboxenc.c | 190 +++++++++++++++------ libavcodec/videotoolboxenc_rewrite.c | 123 +++++++++++++ libavcodec/videotoolboxenc_rewrite.h | 39 +++++ tests/fate/libavcodec.mak | 5 + 6 files changed, 465 insertions(+), 52 deletions(-) create mode 100644 libavcodec/tests/videotoolboxenc_rewrite.c create mode 100644 libavcodec/videotoolboxenc_rewrite.c create mode 100644 libavcodec/videotoolboxenc_rewrite.h diff --git a/libavcodec/Makefile b/libavcodec/Makefile index e464811af6..c194a1d11f 100644 --- a/libavcodec/Makefile +++ b/libavcodec/Makefile @@ -454,7 +454,7 @@ OBJS-$(CONFIG_H264_VAAPI_ENCODER) += vaapi_encode_h264.o h264_levels.o \ OBJS-$(CONFIG_H264_VULKAN_ENCODER) += vulkan_encode.o vulkan_encode_h264.o \ hw_base_encode.o hw_base_encode_h264.o \ h264_levels.o h2645data.o -OBJS-$(CONFIG_H264_VIDEOTOOLBOX_ENCODER) += videotoolboxenc.o +OBJS-$(CONFIG_H264_VIDEOTOOLBOX_ENCODER) += videotoolboxenc.o videotoolboxenc_rewrite.o OBJS-$(CONFIG_H264_V4L2M2M_DECODER) += v4l2_m2m_dec.o OBJS-$(CONFIG_H264_V4L2M2M_ENCODER) += v4l2_m2m_enc.o OBJS-$(CONFIG_HAP_DECODER) += hapdec.o hap.o @@ -486,7 +486,7 @@ OBJS-$(CONFIG_HEVC_VULKAN_ENCODER) += vulkan_encode.o vulkan_encode_h265.o \ h265_profile_level.o h2645data.o OBJS-$(CONFIG_HEVC_V4L2M2M_DECODER) += v4l2_m2m_dec.o OBJS-$(CONFIG_HEVC_V4L2M2M_ENCODER) += v4l2_m2m_enc.o -OBJS-$(CONFIG_HEVC_VIDEOTOOLBOX_ENCODER) += videotoolboxenc.o +OBJS-$(CONFIG_HEVC_VIDEOTOOLBOX_ENCODER) += videotoolboxenc.o videotoolboxenc_rewrite.o OBJS-$(CONFIG_HNM4_VIDEO_DECODER) += hnm4video.o OBJS-$(CONFIG_HQ_HQA_DECODER) += hq_hqa.o hq_hqadsp.o hq_common.o canopus.o OBJS-$(CONFIG_HQX_DECODER) += hqx.o hqxdsp.o hq_common.o canopus.o @@ -1382,6 +1382,7 @@ TESTPROGS = avcodec \ htmlsubtitles \ jpeg2000dwt \ mathops \ + videotoolboxenc_rewrite \ TESTPROGS-$(CONFIG_APV_DECODER) += apv TESTPROGS-$(CONFIG_AV1_VAAPI_ENCODER) += av1_levels @@ -1417,6 +1418,7 @@ HOSTPROGS = aacps_tablegen \ CLEANFILES = *_tables.c *_tables.h *_tablegen$(HOSTEXESUF) $(SUBDIR)tests/dct$(EXESUF): $(SUBDIR)dctref.o $(SUBDIR)aandcttab.o +$(SUBDIR)tests/videotoolboxenc_rewrite$(EXESUF): $(SUBDIR)videotoolboxenc_rewrite.o $(SUBDIR)dv_tablegen$(HOSTEXESUF): $(SUBDIR)dvdata_host.o ifdef CONFIG_SMALL diff --git a/libavcodec/tests/videotoolboxenc_rewrite.c b/libavcodec/tests/videotoolboxenc_rewrite.c new file mode 100644 index 0000000000..c5fa53d150 --- /dev/null +++ b/libavcodec/tests/videotoolboxenc_rewrite.c @@ -0,0 +1,154 @@ +/* + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include <stdio.h> +#include <string.h> + +#include "libavutil/error.h" + +#include "libavcodec/videotoolboxenc_rewrite.h" + +typedef struct BufferReader { + const uint8_t *data; + size_t size; + int calls; + int fail_at; + int error; +} BufferReader; + +static int read_buffer(void *opaque, size_t offset, uint8_t *dst, size_t size) +{ + BufferReader *reader = opaque; + int call = reader->calls++; + + if (reader->fail_at >= 0 && call == reader->fail_at) + return reader->error; + if (offset > reader->size || size > reader->size - offset) + return AVERROR_INVALIDDATA; + if (size) + memcpy(dst, reader->data + offset, size); + return 0; +} + +static int rewrite_lengths(const uint8_t *src, size_t src_size, + size_t length_size, uint8_t *dst, + size_t capacity, size_t *written) +{ + BufferReader reader = { src, src_size, 0, -1, 0 }; + + return ff_vt_rewrite_length_prefixes(read_buffer, &reader, src_size, + length_size, dst, capacity, written); +} + +static int test_plain_converter(void) +{ + static const uint8_t src1[] = { 1, 0x65, 2, 0x61, 0xaa }; + static const uint8_t src2[] = { 0, 1, 0x65, 0, 2, 0x61, 0xaa }; + static const uint8_t src4[] = { + 0, 0, 0, 1, 0x65, 0, 0, 0, 2, 0x61, 0xaa, + }; + static const uint8_t expected[] = { + 0, 0, 0, 1, 0x65, + 0, 0, 0, 1, 0x61, 0xaa, + }; + static const uint8_t zero[] = { 0 }; + static const uint8_t truncated[] = { 0, 1 }; + static const uint8_t overlong[] = { 2, 0x65 }; + static const uint8_t trailing[] = { 1, 0x65, 0 }; + const uint8_t *sources[] = { src1, src2, src4 }; + const size_t source_sizes[] = { sizeof(src1), sizeof(src2), sizeof(src4) }; + const size_t length_sizes[] = { 1, 2, 4 }; + uint8_t dst[sizeof(expected) + 16]; + size_t needed = 0, written = 0, capacity; + int i; + + for (i = 0; i < 3; i++) { + if (rewrite_lengths(sources[i], source_sizes[i], length_sizes[i], + NULL, 0, &needed) < 0 || + needed != sizeof(expected) || + rewrite_lengths(sources[i], source_sizes[i], length_sizes[i], + dst, needed, &written) < 0 || written != needed || + memcmp(dst, expected, sizeof(expected))) { + fprintf(stderr, "plain converter length size %zu failed\n", + length_sizes[i]); + return 1; + } + } + + for (capacity = 0; capacity < sizeof(expected); capacity++) { + size_t j; + + memset(dst, 0xa5, sizeof(dst)); + if (rewrite_lengths(src1, sizeof(src1), 1, dst, capacity, + &written) != AVERROR_BUFFER_TOO_SMALL) + return 1; + for (j = capacity; j < sizeof(dst); j++) { + if (dst[j] != 0xa5) { + fprintf(stderr, "plain converter canary changed at %zu\n", j); + return 1; + } + } + } + + if (rewrite_lengths(NULL, 0, 1, NULL, 0, &written) < 0 || written || + rewrite_lengths(zero, sizeof(zero), 1, NULL, 0, + &written) != AVERROR_INVALIDDATA || + rewrite_lengths(truncated, sizeof(truncated), 2, NULL, 0, + &written) != AVERROR_INVALIDDATA || + rewrite_lengths(overlong, sizeof(overlong), 1, NULL, 0, + &written) != AVERROR_INVALIDDATA || + rewrite_lengths(trailing, sizeof(trailing), 1, NULL, 0, + &written) != AVERROR_INVALIDDATA || + rewrite_lengths(src1, sizeof(src1), 0, NULL, 0, + &written) != AVERROR_INVALIDDATA || + rewrite_lengths(src1, sizeof(src1), 3, NULL, 0, + &written) != AVERROR_INVALIDDATA || + rewrite_lengths(src1, sizeof(src1), 5, NULL, 0, + &written) != AVERROR_INVALIDDATA || + ff_vt_rewrite_length_prefixes(NULL, NULL, 0, 1, NULL, 0, + &written) != AVERROR(EINVAL) || + ff_vt_rewrite_length_prefixes(read_buffer, NULL, 0, 1, NULL, 0, + NULL) != AVERROR(EINVAL)) { + fprintf(stderr, "plain converter malformed policy mismatch\n"); + return 1; + } + return 0; +} + +static int test_reader_failures(void) +{ + static const uint8_t src[] = { 1, 0x65 }; + BufferReader reader = { src, sizeof(src), 0, 0, AVERROR_EXTERNAL }; + uint8_t dst[8]; + size_t written; + + if (ff_vt_rewrite_length_prefixes(read_buffer, &reader, sizeof(src), 1, + NULL, 0, &written) != AVERROR_EXTERNAL) + return 1; + reader = (BufferReader) { src, sizeof(src), 0, 1, AVERROR_EXTERNAL }; + if (ff_vt_rewrite_length_prefixes(read_buffer, &reader, sizeof(src), 1, + dst, sizeof(dst), + &written) != AVERROR_EXTERNAL) + return 1; + return 0; +} + +int main(void) +{ + return test_plain_converter() || test_reader_failures(); +} diff --git a/libavcodec/videotoolboxenc.c b/libavcodec/videotoolboxenc.c index 85561c0479..7563c8ba0c 100644 --- a/libavcodec/videotoolboxenc.c +++ b/libavcodec/videotoolboxenc.c @@ -26,6 +26,8 @@ #include <TargetConditionals.h> #include <VideoToolbox/VideoToolbox.h> +#include "config_components.h" + #include "libavutil/attributes.h" #include "libavutil/avassert.h" #include "libavutil/imgutils.h" @@ -42,6 +44,10 @@ #include "hwconfig.h" #include "internal.h" +#if CONFIG_H264_VIDEOTOOLBOX_ENCODER || CONFIG_HEVC_VIDEOTOOLBOX_ENCODER +#include "videotoolboxenc_rewrite.h" +#endif + #if !HAVE_KCMVIDEOCODECTYPE_HEVC enum { kCMVideoCodecType_HEVC = 'hvc1' }; #endif @@ -475,6 +481,7 @@ static void vtenc_q_push(VTEncContext *vtctx, BufNode *info) pthread_mutex_unlock(&vtctx->lock); } +#if CONFIG_H264_VIDEOTOOLBOX_ENCODER || CONFIG_HEVC_VIDEOTOOLBOX_ENCODER static int count_nalus(size_t length_code_size, CMSampleBufferRef sample_buffer, int *count) @@ -517,6 +524,7 @@ static int count_nalus(size_t length_code_size, *count = nalu_ct; return 0; } +#endif static CMVideoCodecType get_cm_codec_type(AVCodecContext *avctx, int profile, @@ -780,6 +788,7 @@ static void vtenc_output_callback( vtenc_q_push(vtctx, info); } +#if CONFIG_H264_VIDEOTOOLBOX_ENCODER || CONFIG_HEVC_VIDEOTOOLBOX_ENCODER static int get_length_code_size( AVCodecContext *avctx, CMSampleBufferRef sample_buffer, @@ -810,6 +819,7 @@ static int get_length_code_size( *size = isize; return 0; } +#endif /* * Returns true on success. @@ -1833,6 +1843,7 @@ static void vtenc_get_frame_info(CMSampleBufferRef buffer, bool *is_key_frame) } } +#if CONFIG_H264_VIDEOTOOLBOX_ENCODER || CONFIG_HEVC_VIDEOTOOLBOX_ENCODER static int is_post_sei_nal_type(int nal_type){ return nal_type != H264_NAL_SEI && nal_type != H264_NAL_SPS && @@ -2204,6 +2215,26 @@ static int get_sei_msg_bytes(const ExtraSEI* sei, int type){ return copied_size + sei->size / 255 + 1 + type / 255 + 1; } +typedef struct VTEncByteReader { + AVCodecContext *avctx; + CMBlockBufferRef block; +} VTEncByteReader; + +static int vtenc_read_bytes(void *opaque, size_t offset, + uint8_t *dst, size_t size) +{ + VTEncByteReader *reader = opaque; + OSStatus status = CMBlockBufferCopyDataBytes(reader->block, offset, + size, dst); + + if (status != kCMBlockBufferNoErr) + av_log(reader->avctx, AV_LOG_ERROR, + "Cannot copy VideoToolbox sample data: %d\n", status); + + return status == kCMBlockBufferNoErr ? 0 : AVERROR_EXTERNAL; +} +#endif + static int vtenc_cm_to_avpacket( AVCodecContext *avctx, CMSampleBufferRef sample_buffer, @@ -2214,26 +2245,31 @@ static int vtenc_cm_to_avpacket( int status; bool is_key_frame; - bool add_header; - size_t length_code_size; - size_t header_size = 0; - size_t in_buf_size; - size_t out_buf_size; - size_t sei_nalu_size = 0; int64_t dts_delta; - int nalu_count; CMTime pts; CMTime dts; - CMVideoFormatDescriptionRef vid_fmt; +#if !CONFIG_H264_VIDEOTOOLBOX_ENCODER && !CONFIG_HEVC_VIDEOTOOLBOX_ENCODER + (void)sei; +#endif vtenc_get_frame_info(sample_buffer, &is_key_frame); +#if CONFIG_H264_VIDEOTOOLBOX_ENCODER || CONFIG_HEVC_VIDEOTOOLBOX_ENCODER if (vtctx->get_param_set_func) { + CMVideoFormatDescriptionRef vid_fmt = NULL; + size_t length_code_size; + size_t header_size = 0; + size_t in_buf_size = CMSampleBufferGetTotalSampleSize(sample_buffer); + bool add_header; + status = get_length_code_size(avctx, sample_buffer, &length_code_size); - if (status) return status; + if (status < 0) + return status; + if (length_code_size != 1 && length_code_size != 2 && + length_code_size != 4) + return AVERROR_INVALIDDATA; add_header = is_key_frame && !(avctx->flags & AV_CODEC_FLAG_GLOBAL_HEADER); - if (add_header) { vid_fmt = CMSampleBufferGetFormatDescription(sample_buffer); if (!vid_fmt) { @@ -2242,49 +2278,95 @@ static int vtenc_cm_to_avpacket( } status = get_params_size(avctx, vid_fmt, &header_size); - if (status) return status; + if (status < 0) + return status; } - status = count_nalus(length_code_size, sample_buffer, &nalu_count); - if(status) - return status; + if (!sei) { + CMBlockBufferRef block = CMSampleBufferGetDataBuffer(sample_buffer); + VTEncByteReader reader; + size_t body_size; + size_t out_buf_size; + size_t written; - if (sei) { - size_t msg_size = get_sei_msg_bytes(sei, - SEI_TYPE_USER_DATA_REGISTERED_ITU_T_T35); + if (!block) + return AVERROR_EXTERNAL; + reader = (VTEncByteReader) { avctx, block }; - sei_nalu_size = sizeof(start_code) + 1 + msg_size + 1; + status = ff_vt_rewrite_length_prefixes( + vtenc_read_bytes, &reader, in_buf_size, length_code_size, + NULL, 0, &body_size); + if (status < 0) + return status; + if (body_size > SIZE_MAX - header_size) + return AVERROR_INVALIDDATA; + out_buf_size = header_size + body_size; + if (out_buf_size > INT_MAX - AV_INPUT_BUFFER_PADDING_SIZE) + return AVERROR_INVALIDDATA; + + status = ff_get_encode_buffer(avctx, pkt, out_buf_size, 0); + if (status < 0) + return status; + + if (add_header) { + status = copy_param_sets(avctx, vid_fmt, pkt->data, + header_size); + if (status < 0) + goto fail_packet; + } + + status = ff_vt_rewrite_length_prefixes( + vtenc_read_bytes, &reader, in_buf_size, length_code_size, + pkt->data + header_size, body_size, &written); + if (status < 0) { + av_log(avctx, AV_LOG_ERROR, + "Error copying packet data: %d\n", status); + goto fail_packet; + } + if (written != body_size) { + status = AVERROR_BUG; + goto fail_packet; + } + } else { + size_t sei_nalu_size; + size_t out_buf_size; + int nalu_count; + + status = count_nalus(length_code_size, sample_buffer, &nalu_count); + if (status < 0) + return status; + + sei_nalu_size = sizeof(start_code) + 1 + + get_sei_msg_bytes( + sei, + SEI_TYPE_USER_DATA_REGISTERED_ITU_T_T35) + 1; + out_buf_size = header_size + in_buf_size + sei_nalu_size + + nalu_count * ((int)sizeof(start_code) - + (int)length_code_size); + + status = ff_get_encode_buffer(avctx, pkt, out_buf_size, 0); + if (status < 0) + return status; + + if (add_header) { + status = copy_param_sets(avctx, vid_fmt, pkt->data, + header_size); + if (status < 0) + goto fail_packet; + } + + status = copy_replace_length_codes( + avctx, length_code_size, sample_buffer, sei, + pkt->data + header_size, pkt->size - header_size); + if (status < 0) { + av_log(avctx, AV_LOG_ERROR, + "Error copying packet data: %d\n", status); + goto fail_packet; + } } - - in_buf_size = CMSampleBufferGetTotalSampleSize(sample_buffer); - out_buf_size = header_size + - in_buf_size + - sei_nalu_size + - nalu_count * ((int)sizeof(start_code) - (int)length_code_size); - - status = ff_get_encode_buffer(avctx, pkt, out_buf_size, 0); - if (status < 0) - return status; - - if (add_header) { - status = copy_param_sets(avctx, vid_fmt, pkt->data, out_buf_size); - if(status) return status; - } - - status = copy_replace_length_codes( - avctx, - length_code_size, - sample_buffer, - sei, - pkt->data + header_size, - pkt->size - header_size - ); - - if (status) { - av_log(avctx, AV_LOG_ERROR, "Error copying packet data: %d\n", status); - return status; - } - } else { + } else +#endif + { size_t len; CMBlockBufferRef buf = CMSampleBufferGetDataBuffer(sample_buffer); if (!buf) { @@ -2301,13 +2383,13 @@ static int vtenc_cm_to_avpacket( status = CMBlockBufferCopyDataBytes(buf, 0, len, pkt->data); if (status) { av_log(avctx, AV_LOG_ERROR, "Error copying packet data: %d\n", status); + av_packet_unref(pkt); return AVERROR_EXTERNAL; } } - if (is_key_frame) { + if (is_key_frame) pkt->flags |= AV_PKT_FLAG_KEY; - } pts = CMSampleBufferGetPresentationTimeStamp(sample_buffer); dts = CMSampleBufferGetDecodeTimeStamp (sample_buffer); @@ -2332,6 +2414,12 @@ static int vtenc_cm_to_avpacket( - dts_delta; return 0; + +#if CONFIG_H264_VIDEOTOOLBOX_ENCODER || CONFIG_HEVC_VIDEOTOOLBOX_ENCODER +fail_packet: + av_packet_unref(pkt); + return status; +#endif } static int copy_avframe_to_pixel_buffer(AVCodecContext *avctx, @@ -2482,7 +2570,9 @@ static int vtenc_send_frame(AVCodecContext *avctx, CMTime time; CFDictionaryRef frame_dict = NULL; CVPixelBufferRef cv_img = NULL; +#if CONFIG_ATSC_A53 AVFrameSideData *side_data = NULL; +#endif BufNode *node = av_mallocz(sizeof(*node)); int status; diff --git a/libavcodec/videotoolboxenc_rewrite.c b/libavcodec/videotoolboxenc_rewrite.c new file mode 100644 index 0000000000..e23792df3f --- /dev/null +++ b/libavcodec/videotoolboxenc_rewrite.c @@ -0,0 +1,123 @@ +/* + * VideoToolbox length-prefixed sample rewriting helpers + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include <stdint.h> +#include <string.h> + +#include "libavutil/error.h" + +#include "videotoolboxenc_rewrite.h" + +typedef struct ByteWriter { + uint8_t *dst; + size_t capacity; + size_t position; +} ByteWriter; + +static int writer_reserve(const ByteWriter *writer, size_t size) +{ + if (size > SIZE_MAX - writer->position) + return AVERROR_INVALIDDATA; + if (writer->dst && + (writer->position > writer->capacity || + size > writer->capacity - writer->position)) + return AVERROR_BUFFER_TOO_SMALL; + return 0; +} + +static int writer_bytes(ByteWriter *writer, const uint8_t *src, size_t size) +{ + int ret; + + if (size && !src) + return AVERROR(EINVAL); + ret = writer_reserve(writer, size); + if (ret < 0) + return ret; + if (writer->dst && size) + memcpy(writer->dst + writer->position, src, size); + writer->position += size; + return 0; +} + +static int copy_source(ByteWriter *writer, FFVTByteReader read, void *opaque, + size_t offset, size_t size) +{ + int ret = writer_reserve(writer, size); + + if (ret < 0) + return ret; + if (writer->dst && size) { + ret = read(opaque, offset, writer->dst + writer->position, size); + if (ret < 0) + return ret; + } + + writer->position += size; + return 0; +} + +int ff_vt_rewrite_length_prefixes(FFVTByteReader read, void *opaque, + size_t src_size, size_t length_code_size, + uint8_t *dst, size_t dst_capacity, + size_t *written) +{ + static const uint8_t start_code[] = { 0, 0, 0, 1 }; + ByteWriter writer = { dst, dst_capacity, 0 }; + size_t src_offset = 0; + + if (!read || !written) + return AVERROR(EINVAL); + if (length_code_size != 1 && length_code_size != 2 && + length_code_size != 4) + return AVERROR_INVALIDDATA; + + while (src_offset < src_size) { + uint8_t length_bytes[4]; + size_t nal_size = 0; + size_t remaining = src_size - src_offset; + size_t i; + int ret; + + if (length_code_size > remaining) + return AVERROR_INVALIDDATA; + ret = read(opaque, src_offset, length_bytes, length_code_size); + if (ret < 0) + return ret; + src_offset += length_code_size; + remaining -= length_code_size; + + for (i = 0; i < length_code_size; i++) + nal_size = (nal_size << 8) | length_bytes[i]; + if (!nal_size || nal_size > remaining) + return AVERROR_INVALIDDATA; + + ret = writer_bytes(&writer, start_code, sizeof(start_code)); + if (ret < 0) + return ret; + ret = copy_source(&writer, read, opaque, src_offset, nal_size); + if (ret < 0) + return ret; + src_offset += nal_size; + } + + *written = writer.position; + return 0; +} diff --git a/libavcodec/videotoolboxenc_rewrite.h b/libavcodec/videotoolboxenc_rewrite.h new file mode 100644 index 0000000000..7db9bc221b --- /dev/null +++ b/libavcodec/videotoolboxenc_rewrite.h @@ -0,0 +1,39 @@ +/* + * VideoToolbox length-prefixed sample rewriting helpers + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#ifndef AVCODEC_VIDEOTOOLBOXENC_REWRITE_H +#define AVCODEC_VIDEOTOOLBOXENC_REWRITE_H + +#include <stddef.h> +#include <stdint.h> + +typedef int (*FFVTByteReader)(void *opaque, size_t offset, + uint8_t *dst, size_t size); + +/* + * Rewrite length-prefixed NAL units as Annex B without interpreting their + * contents. Passing a null dst performs a checked count-only pass. + */ +int ff_vt_rewrite_length_prefixes(FFVTByteReader read, void *opaque, + size_t src_size, size_t length_code_size, + uint8_t *dst, size_t dst_capacity, + size_t *written); + +#endif /* AVCODEC_VIDEOTOOLBOXENC_REWRITE_H */ diff --git a/tests/fate/libavcodec.mak b/tests/fate/libavcodec.mak index e2d616e307..5e400bda8b 100644 --- a/tests/fate/libavcodec.mak +++ b/tests/fate/libavcodec.mak @@ -96,6 +96,11 @@ fate-mathops: libavcodec/tests/mathops$(EXESUF) fate-mathops: CMD = run libavcodec/tests/mathops$(EXESUF) fate-mathops: CMP = null +FATE_LIBAVCODEC-yes += fate-videotoolboxenc-rewrite +fate-videotoolboxenc-rewrite: libavcodec/tests/videotoolboxenc_rewrite$(EXESUF) +fate-videotoolboxenc-rewrite: CMD = run libavcodec/tests/videotoolboxenc_rewrite$(EXESUF) +fate-videotoolboxenc-rewrite: CMP = null + FATE_LIBAVCODEC-$(CONFIG_JPEG2000_ENCODER) += fate-j2k-dwt fate-j2k-dwt: libavcodec/tests/jpeg2000dwt$(EXESUF) fate-j2k-dwt: CMD = run libavcodec/tests/jpeg2000dwt$(EXESUF) -- 2.52.0 From 458fb81bc7f39361aa7f90c6d719471fe06228b6 Mon Sep 17 00:00:00 2001 From: Qingzheng Li <s(a)qwqlog.com> Date: Sat, 8 Aug 2026 23:10:45 +0800 Subject: [PATCH 2/2] avcodec/videotoolboxenc: write A53 data in standalone SEI The current A53 path appends caption data to the first H.264 SEI NAL after converting its length prefix. Existing data is EBSP, while the append code locates the insertion point as if it were RBSP. This can corrupt the SEI and lose captions when VideoToolbox supplies its own SEI. Keep every VideoToolbox NAL body byte-exact instead and insert A53 data in a standalone user_data_registered_itu_t_t35 SEI before the first Annex A VCL NAL. Require exactly one CMSampleBuffer sample, and reject samples without VCL or with a definite new-access-unit boundary after VCL. Add FATE coverage for NAL ordering, preservation of existing NAL bodies, SEI size fields and escaping, malformed samples, capacity limits, and reader failures. Signed-off-by: Qingzheng Li <s(a)qwqlog.com> --- libavcodec/tests/videotoolboxenc_rewrite.c | 356 +++++++++++++- libavcodec/videotoolboxenc.c | 541 +++------------------ libavcodec/videotoolboxenc_rewrite.c | 166 ++++++- libavcodec/videotoolboxenc_rewrite.h | 15 + 4 files changed, 592 insertions(+), 486 deletions(-) diff --git a/libavcodec/tests/videotoolboxenc_rewrite.c b/libavcodec/tests/videotoolboxenc_rewrite.c index c5fa53d150..e7299f5899 100644 --- a/libavcodec/tests/videotoolboxenc_rewrite.c +++ b/libavcodec/tests/videotoolboxenc_rewrite.c @@ -17,9 +17,12 @@ */ #include <stdio.h> +#include <stdlib.h> #include <string.h> #include "libavutil/error.h" +#include "libavutil/macros.h" +#include "libavutil/mem.h" #include "libavcodec/videotoolboxenc_rewrite.h" @@ -45,6 +48,17 @@ static int read_buffer(void *opaque, size_t offset, uint8_t *dst, size_t size) return 0; } +static int rewrite(const uint8_t *src, size_t src_size, size_t length_size, + const uint8_t *a53, size_t a53_size, + uint8_t *dst, size_t capacity, size_t *written) +{ + BufferReader reader = { src, src_size, 0, -1, 0 }; + + return ff_vt_h264_rewrite_annexb(read_buffer, &reader, src_size, + length_size, a53, a53_size, + dst, capacity, written); +} + static int rewrite_lengths(const uint8_t *src, size_t src_size, size_t length_size, uint8_t *dst, size_t capacity, size_t *written) @@ -55,6 +69,46 @@ static int rewrite_lengths(const uint8_t *src, size_t src_size, length_size, dst, capacity, written); } +static int check_equal(const char *name, const uint8_t *actual, + const uint8_t *expected, size_t size) +{ + if (!memcmp(actual, expected, size)) + return 0; + fprintf(stderr, "%s: byte mismatch\n", name); + return 1; +} + +static int test_length_sizes(void) +{ + static const uint8_t src1[] = { 1, 0x65 }; + static const uint8_t src2[] = { 0, 1, 0x65 }; + static const uint8_t src4[] = { 0, 0, 0, 1, 0x65 }; + static const uint8_t a53[] = { 0x7f }; + const uint8_t *sources[] = { src1, src2, src4 }; + const size_t sizes[] = { sizeof(src1), sizeof(src2), sizeof(src4) }; + const size_t lengths[] = { 1, 2, 4 }; + static const uint8_t expected[] = { + 0, 0, 0, 1, 0x06, 0x04, 0x01, 0x7f, 0x80, + 0, 0, 0, 1, 0x65, + }; + int i; + + for (i = 0; i < 3; i++) { + uint8_t dst[32]; + size_t needed = 0, written = 0; + int ret = rewrite(sources[i], sizes[i], lengths[i], a53, sizeof(a53), + NULL, 0, &needed); + if (ret < 0 || needed > sizeof(dst) || + rewrite(sources[i], sizes[i], lengths[i], a53, sizeof(a53), + dst, needed, &written) < 0 || written != needed || + needed != sizeof(expected) || memcmp(dst, expected, needed)) { + fprintf(stderr, "length size %zu failed\n", lengths[i]); + return 1; + } + } + return 0; +} + static int test_plain_converter(void) { static const uint8_t src1[] = { 1, 0x65, 2, 0x61, 0xaa }; @@ -119,18 +173,14 @@ static int test_plain_converter(void) rewrite_lengths(src1, sizeof(src1), 3, NULL, 0, &written) != AVERROR_INVALIDDATA || rewrite_lengths(src1, sizeof(src1), 5, NULL, 0, - &written) != AVERROR_INVALIDDATA || - ff_vt_rewrite_length_prefixes(NULL, NULL, 0, 1, NULL, 0, - &written) != AVERROR(EINVAL) || - ff_vt_rewrite_length_prefixes(read_buffer, NULL, 0, 1, NULL, 0, - NULL) != AVERROR(EINVAL)) { + &written) != AVERROR_INVALIDDATA) { fprintf(stderr, "plain converter malformed policy mismatch\n"); return 1; } return 0; } -static int test_reader_failures(void) +static int test_plain_converter_reader_failures(void) { static const uint8_t src[] = { 1, 0x65 }; BufferReader reader = { src, sizeof(src), 0, 0, AVERROR_EXTERNAL }; @@ -148,7 +198,299 @@ static int test_reader_failures(void) return 0; } +static int test_order_and_existing_bodies(void) +{ + static const uint8_t src[] = { + 1, 0x09, + 2, 0x67, 0xaa, + 2, 0x68, 0xbb, + 2, 0x6d, 0xcc, + 2, 0x6f, 0xdd, + 2, 0x70, 0xee, + 2, 0x71, 0xe1, + 2, 0x72, 0xe2, + 5, 0x06, 0x00, 0x00, 0x03, 0x01, + 2, 0x6e, 0xf0, + 2, 0x61, 0xf1, + 2, 0x61, 0xf2, + }; + static const uint8_t a53[] = { 0x00, 0x00, 0x01 }; + static const uint8_t expected[] = { + 0, 0, 0, 1, 0x09, + 0, 0, 0, 1, 0x67, 0xaa, + 0, 0, 0, 1, 0x68, 0xbb, + 0, 0, 0, 1, 0x6d, 0xcc, + 0, 0, 0, 1, 0x6f, 0xdd, + 0, 0, 0, 1, 0x70, 0xee, + 0, 0, 0, 1, 0x71, 0xe1, + 0, 0, 0, 1, 0x72, 0xe2, + 0, 0, 0, 1, 0x06, 0x00, 0x00, 0x03, 0x01, + 0, 0, 0, 1, 0x06, 0x04, 0x03, + 0x00, 0x00, 0x03, 0x01, 0x80, + 0, 0, 0, 1, 0x6e, 0xf0, + 0, 0, 0, 1, 0x61, 0xf1, + 0, 0, 0, 1, 0x61, 0xf2, + }; + uint8_t dst[sizeof(expected)]; + size_t written = 0; + + if (rewrite(src, sizeof(src), 1, a53, sizeof(a53), + dst, sizeof(dst), &written) < 0 || written != sizeof(expected)) + return 1; + return check_equal("order/existing bodies", dst, expected, + sizeof(expected)); +} + +static int test_prefix_before_idr(void) +{ + static const uint8_t src[] = { 1, 0x6e, 1, 0x65 }; + static const uint8_t a53[] = { 0x7f }; + static const uint8_t expected[] = { + 0, 0, 0, 1, 0x06, 0x04, 0x01, 0x7f, 0x80, + 0, 0, 0, 1, 0x6e, + 0, 0, 0, 1, 0x65, + }; + uint8_t dst[sizeof(expected)]; + size_t written = 0; + + if (rewrite(src, sizeof(src), 1, a53, sizeof(a53), + dst, sizeof(dst), &written) < 0 || written != sizeof(expected)) + return 1; + return check_equal("prefix before IDR", dst, expected, sizeof(expected)); +} + +static int check_size_case(const char *name, size_t a53_size, + const uint8_t *prefix, size_t prefix_size) +{ + static const uint8_t src[] = { 1, 0x65 }; + uint8_t *a53 = av_malloc(a53_size); + uint8_t *dst = NULL; + size_t needed = 0, written = 0; + int result = 1; + + if (!a53) + return 1; + memset(a53, 0x7f, a53_size); + if (rewrite(src, sizeof(src), 1, a53, a53_size, + NULL, 0, &needed) < 0) + goto end; + dst = av_malloc(needed); + if (!dst) + goto end; + if (rewrite(src, sizeof(src), 1, a53, a53_size, + dst, needed, &written) < 0 || written != needed || + needed < 5 + prefix_size || memcmp(dst + 5, prefix, prefix_size)) { + fprintf(stderr, "%s: payload-size encoding mismatch\n", name); + goto end; + } + result = 0; +end: + av_free(dst); + av_free(a53); + return result; +} + +static int test_size_fields(void) +{ + static const uint8_t size1[] = { 0x04, 0x01, 0x7f }; + static const uint8_t size254[] = { 0x04, 0xfe, 0x7f }; + static const uint8_t size255[] = { 0x04, 0xff, 0x00, 0x7f }; + static const uint8_t size256[] = { 0x04, 0xff, 0x01, 0x7f }; + static const uint8_t size510[] = { 0x04, 0xff, 0xff, 0x00, 0x7f }; + + return check_size_case("size 1", 1, size1, sizeof(size1)) || + check_size_case("size 254", 254, size254, sizeof(size254)) || + check_size_case("size 255", 255, size255, sizeof(size255)) || + check_size_case("size 256", 256, size256, sizeof(size256)) || + check_size_case("size 510", 510, size510, sizeof(size510)); +} + +static int test_epb_and_trailing(void) +{ + static const uint8_t src[] = { 1, 0x65 }; + static const uint8_t a53[] = { + 0, 0, 0, 0, 0, 1, 0, 0, 2, 0, 0, 3, 0, 0, 4, + }; + static const uint8_t escaped_and_trailing[] = { + 0, 0, 3, 0, 0, 3, 0, 1, 0, 0, 3, 2, + 0, 0, 3, 3, 0, 0, 4, 0x80, + }; + uint8_t dst[64]; + size_t written; + + if (rewrite(src, sizeof(src), 1, a53, sizeof(a53), + dst, sizeof(dst), &written) < 0 || + written < 7 + sizeof(escaped_and_trailing) || + memcmp(dst + 7, escaped_and_trailing, + sizeof(escaped_and_trailing))) { + fprintf(stderr, "EPB/trailing mismatch\n"); + return 1; + } + return 0; +} + +static int test_fail_closed_and_malformed(void) +{ + static const uint8_t leading_only[] = { + 1, 0x09, 1, 0x67, 1, 0x68, 1, 0x06, 1, 0x6d, 1, 0x6f, 1, 0x70, + }; + static const uint8_t prefixed_vcl[] = { 1, 0x6e, 1, 0x61 }; + static const uint8_t auxiliary_only[] = { 1, 0x73 }; + static const uint8_t au_boundary_types[] = { + 6, 7, 8, 9, 14, 15, 16, 17, 18, + }; + static const uint8_t bad_zero[] = { 0 }; + static const uint8_t bad_long[] = { 2, 0x65 }; + static const uint8_t a53[] = { 0x7f }; + uint8_t dst[32]; + uint8_t multiple_aus[] = { 1, 0x65, 1, 0, 1, 0x61 }; + size_t written = 123; + size_t i; + + if (rewrite(leading_only, sizeof(leading_only), 1, a53, sizeof(a53), + NULL, 0, &written) != AVERROR_INVALIDDATA || + rewrite(bad_zero, sizeof(bad_zero), 1, a53, sizeof(a53), + NULL, 0, &written) != AVERROR_INVALIDDATA || + rewrite(bad_long, sizeof(bad_long), 1, a53, sizeof(a53), + NULL, 0, &written) != AVERROR_INVALIDDATA || + rewrite(auxiliary_only, sizeof(auxiliary_only), 1, a53, sizeof(a53), + NULL, 0, &written) != AVERROR_INVALIDDATA || + rewrite(prefixed_vcl, sizeof(prefixed_vcl), 1, a53, sizeof(a53), + dst, sizeof(dst), &written) < 0 || + rewrite(prefixed_vcl, sizeof(prefixed_vcl), 3, a53, sizeof(a53), + NULL, 0, &written) != AVERROR_INVALIDDATA || + rewrite(prefixed_vcl, sizeof(prefixed_vcl), 1, NULL, 0, + NULL, 0, &written) != AVERROR(EINVAL)) { + fprintf(stderr, "fail-closed/malformed policy mismatch\n"); + return 1; + } + + for (i = 0; i < FF_ARRAY_ELEMS(au_boundary_types); i++) { + multiple_aus[3] = au_boundary_types[i]; + if (rewrite(multiple_aus, sizeof(multiple_aus), 1, + a53, sizeof(a53), NULL, 0, + &written) != AVERROR_INVALIDDATA) { + fprintf(stderr, "post-VCL NAL type %u did not fail closed\n", + au_boundary_types[i]); + return 1; + } + } + return 0; +} + +static int test_capacity_edges(void) +{ + static const uint8_t src[] = { 1, 0x65 }; + static const uint8_t a53[] = { 0, 0, 1 }; + uint8_t dst[64]; + size_t needed = 0, written = 0, capacity; + + if (rewrite(src, sizeof(src), 1, a53, sizeof(a53), + NULL, 0, &needed) < 0) + return 1; + + for (capacity = 0; capacity < needed; capacity++) { + size_t i; + int ret; + + memset(dst, 0xa5, sizeof(dst)); + ret = rewrite(src, sizeof(src), 1, a53, sizeof(a53), + dst, capacity, &written); + if (ret != AVERROR_BUFFER_TOO_SMALL) { + fprintf(stderr, "capacity %zu returned %d\n", capacity, ret); + return 1; + } + for (i = capacity; i < sizeof(dst); i++) { + if (dst[i] != 0xa5) { + fprintf(stderr, "capacity canary changed at %zu\n", i); + return 1; + } + } + } + return 0; +} + +static int test_direct_large_nal_copy(void) +{ + static const uint8_t a53[] = { 0x7f }; + const size_t nal_size = 1024 * 1024 + 1; + const size_t src_size = 4 + nal_size; + BufferReader count_reader; + BufferReader write_reader; + uint8_t *src = av_malloc(src_size); + uint8_t *dst = NULL; + size_t needed = 0, written = 0; + int result = 1; + + if (!src) + return 1; + src[0] = (uint8_t)(nal_size >> 24); + src[1] = (uint8_t)(nal_size >> 16); + src[2] = (uint8_t)(nal_size >> 8); + src[3] = (uint8_t)nal_size; + src[4] = 0x65; + memset(src + 5, 0xa5, nal_size - 1); + + count_reader = (BufferReader) { src, src_size, 0, -1, 0 }; + if (ff_vt_h264_rewrite_annexb(read_buffer, &count_reader, + src_size, 4, a53, sizeof(a53), + NULL, 0, &needed) < 0) + goto end; + + dst = av_malloc(needed); + if (!dst) + goto end; + write_reader = (BufferReader) { src, src_size, 0, -1, 0 }; + if (ff_vt_h264_rewrite_annexb(read_buffer, &write_reader, + src_size, 4, a53, sizeof(a53), + dst, needed, &written) < 0 || + written != needed || needed < nal_size || + memcmp(dst + needed - nal_size, src + 4, nal_size)) { + fprintf(stderr, "large NAL copy mismatch\n"); + goto end; + } + + result = 0; +end: + av_free(dst); + av_free(src); + return result; +} + +static int test_reader_failures(void) +{ + static const uint8_t src[] = { 1, 0x65 }; + static const uint8_t a53[] = { 0x7f }; + BufferReader reader = { src, sizeof(src), 0, 0, AVERROR_EXTERNAL }; + uint8_t dst[32]; + size_t written; + + if (ff_vt_h264_rewrite_annexb(read_buffer, &reader, sizeof(src), 1, + a53, sizeof(a53), NULL, 0, + &written) != AVERROR_EXTERNAL) + return 1; + + reader = (BufferReader) { src, sizeof(src), 0, 2, AVERROR_EXTERNAL }; + if (ff_vt_h264_rewrite_annexb(read_buffer, &reader, sizeof(src), 1, + a53, sizeof(a53), dst, sizeof(dst), + &written) != AVERROR_EXTERNAL) { + fprintf(stderr, "body-reader failure mismatch\n"); + return 1; + } + return 0; +} + int main(void) { - return test_plain_converter() || test_reader_failures(); + return test_length_sizes() || + test_plain_converter() || + test_plain_converter_reader_failures() || + test_order_and_existing_bodies() || + test_prefix_before_idr() || + test_size_fields() || + test_epb_and_trailing() || + test_fail_closed_and_malformed() || + test_capacity_edges() || + test_direct_large_nal_copy() || + test_reader_failures(); } diff --git a/libavcodec/videotoolboxenc.c b/libavcodec/videotoolboxenc.c index 7563c8ba0c..c109977276 100644 --- a/libavcodec/videotoolboxenc.c +++ b/libavcodec/videotoolboxenc.c @@ -30,6 +30,7 @@ #include "libavutil/attributes.h" #include "libavutil/avassert.h" +#include "libavutil/bswap.h" #include "libavutil/imgutils.h" #include "libavutil/mem.h" #include "libavutil/opt.h" @@ -39,8 +40,6 @@ #include "atsc_a53.h" #include "codec_internal.h" #include "encode.h" -#include "h264.h" -#include "h264_sei.h" #include "hwconfig.h" #include "internal.h" @@ -481,51 +480,6 @@ static void vtenc_q_push(VTEncContext *vtctx, BufNode *info) pthread_mutex_unlock(&vtctx->lock); } -#if CONFIG_H264_VIDEOTOOLBOX_ENCODER || CONFIG_HEVC_VIDEOTOOLBOX_ENCODER -static int count_nalus(size_t length_code_size, - CMSampleBufferRef sample_buffer, - int *count) -{ - size_t offset = 0; - int status; - int nalu_ct = 0; - uint8_t size_buf[4]; - size_t src_size = CMSampleBufferGetTotalSampleSize(sample_buffer); - CMBlockBufferRef block = CMSampleBufferGetDataBuffer(sample_buffer); - - if (length_code_size > 4) - return AVERROR_INVALIDDATA; - - while (offset < src_size) { - size_t curr_src_len; - size_t box_len = 0; - size_t i; - - status = CMBlockBufferCopyDataBytes(block, - offset, - length_code_size, - size_buf); - - if (status != kCMBlockBufferNoErr) { - return AVERROR_EXTERNAL; - } - - for (i = 0; i < length_code_size; i++) { - box_len <<= 8; - box_len |= size_buf[i]; - } - - curr_src_len = box_len + length_code_size; - offset += curr_src_len; - - nalu_ct++; - } - - *count = nalu_ct; - return 0; -} -#endif - static CMVideoCodecType get_cm_codec_type(AVCodecContext *avctx, int profile, double alpha_quality) @@ -1844,377 +1798,6 @@ static void vtenc_get_frame_info(CMSampleBufferRef buffer, bool *is_key_frame) } #if CONFIG_H264_VIDEOTOOLBOX_ENCODER || CONFIG_HEVC_VIDEOTOOLBOX_ENCODER -static int is_post_sei_nal_type(int nal_type){ - return nal_type != H264_NAL_SEI && - nal_type != H264_NAL_SPS && - nal_type != H264_NAL_PPS && - nal_type != H264_NAL_AUD; -} - -/* - * Finds the sei message start/size of type find_sei_type. - * If more than one of that type exists, the last one is returned. - */ -static int find_sei_end(AVCodecContext *avctx, - uint8_t *nal_data, - size_t nal_size, - uint8_t **sei_end) -{ - int nal_type; - size_t sei_payload_size = 0; - uint8_t *nal_start = nal_data; - *sei_end = NULL; - - if (!nal_size) - return 0; - - nal_type = *nal_data & 0x1F; - if (nal_type != H264_NAL_SEI) - return 0; - - nal_data++; - nal_size--; - - if (nal_data[nal_size - 1] == 0x80) - nal_size--; - - while (nal_size > 0 && *nal_data > 0) { - do{ - nal_data++; - nal_size--; - } while (nal_size > 0 && *nal_data == 0xFF); - - if (!nal_size) { - av_log(avctx, AV_LOG_ERROR, "Unexpected end of SEI NAL Unit parsing type.\n"); - return AVERROR_INVALIDDATA; - } - - do{ - sei_payload_size += *nal_data; - nal_data++; - nal_size--; - } while (nal_size > 0 && *nal_data == 0xFF); - - if (nal_size < sei_payload_size) { - av_log(avctx, AV_LOG_ERROR, "Unexpected end of SEI NAL Unit parsing size.\n"); - return AVERROR_INVALIDDATA; - } - - nal_data += sei_payload_size; - nal_size -= sei_payload_size; - } - - *sei_end = nal_data; - - return nal_data - nal_start + 1; -} - -/** - * Copies the data inserting emulation prevention bytes as needed. - * Existing data in the destination can be taken into account by providing - * dst with a dst_offset > 0. - * - * @return The number of bytes copied on success. On failure, the negative of - * the number of bytes needed to copy src is returned. - */ -static int copy_emulation_prev(const uint8_t *src, - size_t src_size, - uint8_t *dst, - ssize_t dst_offset, - size_t dst_size) -{ - int zeros = 0; - int wrote_bytes; - uint8_t* dst_start; - uint8_t* dst_end = dst + dst_size; - const uint8_t* src_end = src + src_size; - int start_at = dst_offset > 2 ? dst_offset - 2 : 0; - int i; - for (i = start_at; i < dst_offset && i < dst_size; i++) { - if (!dst[i]) - zeros++; - else - zeros = 0; - } - - dst += dst_offset; - dst_start = dst; - for (; src < src_end; src++, dst++) { - if (zeros == 2) { - int insert_ep3_byte = *src <= 3; - if (insert_ep3_byte) { - if (dst < dst_end) - *dst = 3; - dst++; - } - - zeros = 0; - } - - if (dst < dst_end) - *dst = *src; - - if (!*src) - zeros++; - else - zeros = 0; - } - - wrote_bytes = dst - dst_start; - - if (dst > dst_end) - return -wrote_bytes; - - return wrote_bytes; -} - -static int write_sei(const ExtraSEI *sei, - int sei_type, - uint8_t *dst, - size_t dst_size) -{ - uint8_t *sei_start = dst; - size_t remaining_sei_size = sei->size; - size_t remaining_dst_size = dst_size; - int header_bytes; - int bytes_written; - ssize_t offset; - - if (!remaining_dst_size) - return AVERROR_BUFFER_TOO_SMALL; - - while (sei_type && remaining_dst_size != 0) { - int sei_byte = sei_type > 255 ? 255 : sei_type; - *dst = sei_byte; - - sei_type -= sei_byte; - dst++; - remaining_dst_size--; - } - - if (!dst_size) - return AVERROR_BUFFER_TOO_SMALL; - - while (remaining_sei_size && remaining_dst_size != 0) { - int size_byte = remaining_sei_size > 255 ? 255 : remaining_sei_size; - *dst = size_byte; - - remaining_sei_size -= size_byte; - dst++; - remaining_dst_size--; - } - - if (remaining_dst_size < sei->size) - return AVERROR_BUFFER_TOO_SMALL; - - header_bytes = dst - sei_start; - - offset = header_bytes; - bytes_written = copy_emulation_prev(sei->data, - sei->size, - sei_start, - offset, - dst_size); - if (bytes_written < 0) - return AVERROR_BUFFER_TOO_SMALL; - - bytes_written += header_bytes; - return bytes_written; -} - -/** - * Copies NAL units and replaces length codes with - * H.264 Annex B start codes. On failure, the contents of - * dst_data may have been modified. - * - * @param length_code_size Byte length of each length code - * @param sample_buffer NAL units prefixed with length codes. - * @param sei Optional A53 closed captions SEI data. - * @param dst_data Must be zeroed before calling this function. - * Contains the copied NAL units prefixed with - * start codes when the function returns - * successfully. - * @param dst_size Length of dst_data - * @return 0 on success - * AVERROR_INVALIDDATA if length_code_size is invalid - * AVERROR_BUFFER_TOO_SMALL if dst_data is too small - * or if a length_code in src_data specifies data beyond - * the end of its buffer. - */ -static int copy_replace_length_codes( - AVCodecContext *avctx, - size_t length_code_size, - CMSampleBufferRef sample_buffer, - ExtraSEI *sei, - uint8_t *dst_data, - size_t dst_size) -{ - size_t src_size = CMSampleBufferGetTotalSampleSize(sample_buffer); - size_t remaining_src_size = src_size; - size_t remaining_dst_size = dst_size; - size_t src_offset = 0; - int wrote_sei = 0; - int status; - uint8_t size_buf[4]; - uint8_t nal_type; - CMBlockBufferRef block = CMSampleBufferGetDataBuffer(sample_buffer); - - if (length_code_size > 4) { - return AVERROR_INVALIDDATA; - } - - while (remaining_src_size > 0) { - size_t curr_src_len; - size_t curr_dst_len; - size_t box_len = 0; - size_t i; - - uint8_t *dst_box; - - status = CMBlockBufferCopyDataBytes(block, - src_offset, - length_code_size, - size_buf); - if (status) { - av_log(avctx, AV_LOG_ERROR, "Cannot copy length: %d\n", status); - return AVERROR_EXTERNAL; - } - - status = CMBlockBufferCopyDataBytes(block, - src_offset + length_code_size, - 1, - &nal_type); - - if (status) { - av_log(avctx, AV_LOG_ERROR, "Cannot copy type: %d\n", status); - return AVERROR_EXTERNAL; - } - - nal_type &= 0x1F; - - for (i = 0; i < length_code_size; i++) { - box_len <<= 8; - box_len |= size_buf[i]; - } - - if (sei && !wrote_sei && is_post_sei_nal_type(nal_type)) { - //No SEI NAL unit - insert. - int wrote_bytes; - - memcpy(dst_data, start_code, sizeof(start_code)); - dst_data += sizeof(start_code); - remaining_dst_size -= sizeof(start_code); - - *dst_data = H264_NAL_SEI; - dst_data++; - remaining_dst_size--; - - wrote_bytes = write_sei(sei, - SEI_TYPE_USER_DATA_REGISTERED_ITU_T_T35, - dst_data, - remaining_dst_size); - - if (wrote_bytes < 0) - return wrote_bytes; - - remaining_dst_size -= wrote_bytes; - dst_data += wrote_bytes; - - if (remaining_dst_size <= 0) - return AVERROR_BUFFER_TOO_SMALL; - - *dst_data = 0x80; - - dst_data++; - remaining_dst_size--; - - wrote_sei = 1; - } - - curr_src_len = box_len + length_code_size; - curr_dst_len = box_len + sizeof(start_code); - - if (remaining_src_size < curr_src_len) { - return AVERROR_BUFFER_TOO_SMALL; - } - - if (remaining_dst_size < curr_dst_len) { - return AVERROR_BUFFER_TOO_SMALL; - } - - dst_box = dst_data + sizeof(start_code); - - memcpy(dst_data, start_code, sizeof(start_code)); - status = CMBlockBufferCopyDataBytes(block, - src_offset + length_code_size, - box_len, - dst_box); - - if (status) { - av_log(avctx, AV_LOG_ERROR, "Cannot copy data: %d\n", status); - return AVERROR_EXTERNAL; - } - - if (sei && !wrote_sei && nal_type == H264_NAL_SEI) { - //Found SEI NAL unit - append. - int wrote_bytes; - int old_sei_length; - int extra_bytes; - uint8_t *new_sei; - old_sei_length = find_sei_end(avctx, dst_box, box_len, &new_sei); - if (old_sei_length < 0) - return old_sei_length; - - wrote_bytes = write_sei(sei, - SEI_TYPE_USER_DATA_REGISTERED_ITU_T_T35, - new_sei, - remaining_dst_size - old_sei_length); - if (wrote_bytes < 0) - return wrote_bytes; - - if (new_sei + wrote_bytes >= dst_data + remaining_dst_size) - return AVERROR_BUFFER_TOO_SMALL; - - new_sei[wrote_bytes++] = 0x80; - extra_bytes = wrote_bytes - (dst_box + box_len - new_sei); - - dst_data += extra_bytes; - remaining_dst_size -= extra_bytes; - - wrote_sei = 1; - } - - src_offset += curr_src_len; - dst_data += curr_dst_len; - - remaining_src_size -= curr_src_len; - remaining_dst_size -= curr_dst_len; - } - - return 0; -} - -/** - * Returns a sufficient number of bytes to contain the sei data. - * It may be greater than the minimum required. - */ -static int get_sei_msg_bytes(const ExtraSEI* sei, int type){ - int copied_size; - if (sei->size == 0) - return 0; - - copied_size = -copy_emulation_prev(sei->data, - sei->size, - NULL, - 0, - 0); - - if ((sei->size % 255) == 0) //may result in an extra byte - copied_size++; - - return copied_size + sei->size / 255 + 1 + type / 255 + 1; -} - typedef struct VTEncByteReader { AVCodecContext *avctx; CMBlockBufferRef block; @@ -2248,7 +1831,10 @@ static int vtenc_cm_to_avpacket( int64_t dts_delta; CMTime pts; CMTime dts; -#if !CONFIG_H264_VIDEOTOOLBOX_ENCODER && !CONFIG_HEVC_VIDEOTOOLBOX_ENCODER +#if CONFIG_H264_VIDEOTOOLBOX_ENCODER + bool rewrite_h264_a53 = avctx->codec_id == AV_CODEC_ID_H264 && + sei && sei->size > 0; +#else (void)sei; #endif @@ -2259,8 +1845,13 @@ static int vtenc_cm_to_avpacket( CMVideoFormatDescriptionRef vid_fmt = NULL; size_t length_code_size; size_t header_size = 0; + size_t body_size; size_t in_buf_size = CMSampleBufferGetTotalSampleSize(sample_buffer); + size_t out_buf_size; bool add_header; + CMBlockBufferRef block = NULL; + VTEncByteReader reader; + size_t written; status = get_length_code_size(avctx, sample_buffer, &length_code_size); if (status < 0) @@ -2269,6 +1860,11 @@ static int vtenc_cm_to_avpacket( length_code_size != 4) return AVERROR_INVALIDDATA; + block = CMSampleBufferGetDataBuffer(sample_buffer); + if (!block) + return AVERROR_EXTERNAL; + reader = (VTEncByteReader) { avctx, block }; + add_header = is_key_frame && !(avctx->flags & AV_CODEC_FLAG_GLOBAL_HEADER); if (add_header) { vid_fmt = CMSampleBufferGetFormatDescription(sample_buffer); @@ -2282,39 +1878,62 @@ static int vtenc_cm_to_avpacket( return status; } - if (!sei) { - CMBlockBufferRef block = CMSampleBufferGetDataBuffer(sample_buffer); - VTEncByteReader reader; - size_t body_size; - size_t out_buf_size; - size_t written; - - if (!block) - return AVERROR_EXTERNAL; - reader = (VTEncByteReader) { avctx, block }; +#if CONFIG_H264_VIDEOTOOLBOX_ENCODER + if (rewrite_h264_a53) { + /* Keep VideoToolbox's existing EBSP SEI NALs byte-exact. */ + if (CMSampleBufferGetNumSamples(sample_buffer) != 1) { + av_log(avctx, AV_LOG_ERROR, + "Cannot attach A53 data unless the sample buffer " + "contains exactly one sample.\n"); + return AVERROR_INVALIDDATA; + } + status = ff_vt_h264_rewrite_annexb( + vtenc_read_bytes, &reader, in_buf_size, length_code_size, + sei->data, sei->size, NULL, 0, &body_size); + if (status < 0) + return status; + } else { +#endif status = ff_vt_rewrite_length_prefixes( vtenc_read_bytes, &reader, in_buf_size, length_code_size, NULL, 0, &body_size); if (status < 0) return status; - if (body_size > SIZE_MAX - header_size) - return AVERROR_INVALIDDATA; - out_buf_size = header_size + body_size; - if (out_buf_size > INT_MAX - AV_INPUT_BUFFER_PADDING_SIZE) - return AVERROR_INVALIDDATA; +#if CONFIG_H264_VIDEOTOOLBOX_ENCODER + } +#endif - status = ff_get_encode_buffer(avctx, pkt, out_buf_size, 0); + if (body_size > SIZE_MAX - header_size) + return AVERROR_INVALIDDATA; + out_buf_size = header_size + body_size; + if (out_buf_size > INT_MAX - AV_INPUT_BUFFER_PADDING_SIZE) + return AVERROR_INVALIDDATA; + + status = ff_get_encode_buffer(avctx, pkt, out_buf_size, 0); + if (status < 0) + return status; + + if (add_header) { + status = copy_param_sets(avctx, vid_fmt, pkt->data, header_size); if (status < 0) - return status; + goto fail_packet; + } - if (add_header) { - status = copy_param_sets(avctx, vid_fmt, pkt->data, - header_size); - if (status < 0) - goto fail_packet; +#if CONFIG_H264_VIDEOTOOLBOX_ENCODER + if (rewrite_h264_a53) { + status = ff_vt_h264_rewrite_annexb( + vtenc_read_bytes, &reader, in_buf_size, length_code_size, + sei->data, sei->size, pkt->data + header_size, body_size, + &written); + if (status < 0) + goto fail_packet; + if (written != body_size) { + status = AVERROR_BUG; + goto fail_packet; } - + } else { +#endif status = ff_vt_rewrite_length_prefixes( vtenc_read_bytes, &reader, in_buf_size, length_code_size, pkt->data + header_size, body_size, &written); @@ -2327,43 +1946,9 @@ static int vtenc_cm_to_avpacket( status = AVERROR_BUG; goto fail_packet; } - } else { - size_t sei_nalu_size; - size_t out_buf_size; - int nalu_count; - - status = count_nalus(length_code_size, sample_buffer, &nalu_count); - if (status < 0) - return status; - - sei_nalu_size = sizeof(start_code) + 1 + - get_sei_msg_bytes( - sei, - SEI_TYPE_USER_DATA_REGISTERED_ITU_T_T35) + 1; - out_buf_size = header_size + in_buf_size + sei_nalu_size + - nalu_count * ((int)sizeof(start_code) - - (int)length_code_size); - - status = ff_get_encode_buffer(avctx, pkt, out_buf_size, 0); - if (status < 0) - return status; - - if (add_header) { - status = copy_param_sets(avctx, vid_fmt, pkt->data, - header_size); - if (status < 0) - goto fail_packet; - } - - status = copy_replace_length_codes( - avctx, length_code_size, sample_buffer, sei, - pkt->data + header_size, pkt->size - header_size); - if (status < 0) { - av_log(avctx, AV_LOG_ERROR, - "Error copying packet data: %d\n", status); - goto fail_packet; - } +#if CONFIG_H264_VIDEOTOOLBOX_ENCODER } +#endif } else #endif { diff --git a/libavcodec/videotoolboxenc_rewrite.c b/libavcodec/videotoolboxenc_rewrite.c index e23792df3f..0e7eb90ad3 100644 --- a/libavcodec/videotoolboxenc_rewrite.c +++ b/libavcodec/videotoolboxenc_rewrite.c @@ -23,12 +23,15 @@ #include "libavutil/error.h" +#include "h264.h" +#include "sei.h" #include "videotoolboxenc_rewrite.h" typedef struct ByteWriter { uint8_t *dst; size_t capacity; size_t position; + unsigned zero_count; } ByteWriter; static int writer_reserve(const ByteWriter *writer, size_t size) @@ -57,6 +60,97 @@ static int writer_bytes(ByteWriter *writer, const uint8_t *src, size_t size) return 0; } +static int writer_rbsp_byte(ByteWriter *writer, uint8_t byte) +{ + static const uint8_t emulation_prevention = 3; + int ret; + + if (writer->zero_count == 2 && byte <= 3) { + ret = writer_bytes(writer, &emulation_prevention, 1); + if (ret < 0) + return ret; + writer->zero_count = 0; + } + ret = writer_bytes(writer, &byte, 1); + if (ret < 0) + return ret; + + if (!byte) { + writer->zero_count++; + } else { + writer->zero_count = 0; + } + return 0; +} + +static int writer_ff_field(ByteWriter *writer, size_t value) +{ + int ret; + + while (value >= 255) { + ret = writer_rbsp_byte(writer, 255); + if (ret < 0) + return ret; + value -= 255; + } + + return writer_rbsp_byte(writer, (uint8_t)value); +} + +static int writer_a53_sei(ByteWriter *writer, const uint8_t *data, size_t size) +{ + static const uint8_t start_code[] = { 0, 0, 0, 1 }; + static const uint8_t nal_header = H264_NAL_SEI; + size_t i; + int ret; + + ret = writer_bytes(writer, start_code, sizeof(start_code)); + if (ret < 0) + return ret; + ret = writer_bytes(writer, &nal_header, 1); + if (ret < 0) + return ret; + + writer->zero_count = 0; + ret = writer_rbsp_byte(writer, + SEI_TYPE_USER_DATA_REGISTERED_ITU_T_T35); + if (ret < 0) + return ret; + ret = writer_ff_field(writer, size); + if (ret < 0) + return ret; + + for (i = 0; i < size; i++) { + ret = writer_rbsp_byte(writer, data[i]); + if (ret < 0) + return ret; + } + + return writer_rbsp_byte(writer, 0x80); +} + +static int is_leading_nal(unsigned type) +{ + return type == H264_NAL_SEI || type == H264_NAL_SPS || + type == H264_NAL_PPS || type == H264_NAL_AUD || + type == H264_NAL_SPS_EXT || type == H264_NAL_SUB_SPS || + type == H264_NAL_DPS || type == H264_NAL_RESERVED17 || + type == H264_NAL_RESERVED18; +} + +static int is_vcl_nal(unsigned type) +{ + /* VideoToolbox's public H.264 profiles produce Annex A VCL NALs. */ + return type >= H264_NAL_SLICE && type <= H264_NAL_IDR_SLICE; +} + +static int starts_new_access_unit(unsigned type) +{ + return type == H264_NAL_SEI || type == H264_NAL_SPS || + type == H264_NAL_PPS || type == H264_NAL_AUD || + (type >= H264_NAL_PREFIX && type <= H264_NAL_RESERVED18); +} + static int copy_source(ByteWriter *writer, FFVTByteReader read, void *opaque, size_t offset, size_t size) { @@ -80,7 +174,7 @@ int ff_vt_rewrite_length_prefixes(FFVTByteReader read, void *opaque, size_t *written) { static const uint8_t start_code[] = { 0, 0, 0, 1 }; - ByteWriter writer = { dst, dst_capacity, 0 }; + ByteWriter writer = { dst, dst_capacity, 0, 0 }; size_t src_offset = 0; if (!read || !written) @@ -121,3 +215,73 @@ int ff_vt_rewrite_length_prefixes(FFVTByteReader read, void *opaque, *written = writer.position; return 0; } + +int ff_vt_h264_rewrite_annexb(FFVTByteReader read, void *opaque, + size_t src_size, size_t length_code_size, + const uint8_t *a53_data, size_t a53_size, + uint8_t *dst, size_t dst_capacity, + size_t *written) +{ + static const uint8_t start_code[] = { 0, 0, 0, 1 }; + ByteWriter writer = { dst, dst_capacity, 0, 0 }; + size_t src_offset = 0; + int inserted = 0; + int saw_vcl = 0; + + if (!read || !written || !a53_data || !a53_size) + return AVERROR(EINVAL); + if (length_code_size != 1 && length_code_size != 2 && + length_code_size != 4) + return AVERROR_INVALIDDATA; + + while (src_offset < src_size) { + uint8_t length_bytes[4]; + uint8_t nal_header; + size_t nal_size = 0; + size_t remaining = src_size - src_offset; + size_t i; + unsigned nal_type; + int ret; + + if (length_code_size > remaining) + return AVERROR_INVALIDDATA; + ret = read(opaque, src_offset, length_bytes, length_code_size); + if (ret < 0) + return ret; + src_offset += length_code_size; + remaining -= length_code_size; + + for (i = 0; i < length_code_size; i++) + nal_size = (nal_size << 8) | length_bytes[i]; + if (!nal_size || nal_size > remaining) + return AVERROR_INVALIDDATA; + + ret = read(opaque, src_offset, &nal_header, 1); + if (ret < 0) + return ret; + nal_type = nal_header & 0x1f; + if (saw_vcl && starts_new_access_unit(nal_type)) + return AVERROR_INVALIDDATA; + saw_vcl |= is_vcl_nal(nal_type); + + if (!inserted && !is_leading_nal(nal_type)) { + ret = writer_a53_sei(&writer, a53_data, a53_size); + if (ret < 0) + return ret; + inserted = 1; + } + ret = writer_bytes(&writer, start_code, sizeof(start_code)); + if (ret < 0) + return ret; + ret = copy_source(&writer, read, opaque, src_offset, nal_size); + if (ret < 0) + return ret; + src_offset += nal_size; + } + + if (!inserted || !saw_vcl) + return AVERROR_INVALIDDATA; + + *written = writer.position; + return 0; +} diff --git a/libavcodec/videotoolboxenc_rewrite.h b/libavcodec/videotoolboxenc_rewrite.h index 7db9bc221b..8e100780dd 100644 --- a/libavcodec/videotoolboxenc_rewrite.h +++ b/libavcodec/videotoolboxenc_rewrite.h @@ -36,4 +36,19 @@ int ff_vt_rewrite_length_prefixes(FFVTByteReader read, void *opaque, uint8_t *dst, size_t dst_capacity, size_t *written); +/* + * Rewrite one VideoToolbox-produced length-prefixed H.264 compressed frame as + * Annex B and insert a standalone A53 closed-caption SEI NAL after leading + * AUD, SEI and parameter-set NALs. In particular, the SEI is written before + * any prefix NAL associated with the first Annex A VCL NAL. Existing NAL + * bodies are copied without parsing or re-encoding them. Passing a null dst + * performs a checked count-only pass. Samples without an Annex A VCL NAL, or + * with a definite new-access-unit boundary after VCL, fail closed. + */ +int ff_vt_h264_rewrite_annexb(FFVTByteReader read, void *opaque, + size_t src_size, size_t length_code_size, + const uint8_t *a53_data, size_t a53_size, + uint8_t *dst, size_t dst_capacity, + size_t *written); + #endif /* AVCODEC_VIDEOTOOLBOXENC_REWRITE_H */ -- 2.52.0

1 0

[PR] avcodec/amfenc: Replace hardcoded lookahead maximum with runtime query with workaround for AMF SDK header bug (PR #24055)
by Edison Ling 09 Aug '26

09 Aug '26

PR #24055 opened by Edison Ling (edisonling) URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24055 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24055.patch # Summary of changes This replaces the hardcoded lookahead maximum `#define MAX_LOOKAHEAD_DEPTH 41` with a runtime query, and sizes `async_depth` from the value AMF actually applied rather than the raw option value. The second is what keeps the first safe, so they ship together. The runtime query has to work around an ABI bug in the unfixed AMF SDK headers, where the C bindings do not match the shipped runtime. That handling is transitional and comes out once the minimum AMF version carries the fix. ## Runtime maximum instead of the macro Replace the hardcoded `MAX_LOOKAHEAD_DEPTH 41` bound on `pa_lookahead_buffer_depth` and `async_depth` with the maximum AMF reports at runtime, so the bound tracks what the runtime accepts instead of a constant duplicated in FFmpeg. Values above that maximum are clamped with a warning instead of being rejected by the option parser. `async_depth` is now sized from the lookahead depth read back from the encoder rather than the raw option value. The read-back is the value AMF actually applied, which fixes a latent under-size that could stall the encoder: adaptive mini-GOP forces a lookahead the raw option value does not reflect, leaving `async_depth` at 1 against a lookahead of 1. The two changes cannot be split: raising the option ceiling makes a value like 1000 parse successfully, and without the read-back that would size `async_depth` to 1001, replacing a hard parse error with a throttle that never engages. `async_depth` is no longer bounded by the option parser. The old ceiling capped it at MAX_LOOKAHEAD_DEPTH + 1 at parse time, even with preanalysis off where no lookahead is running. The cap now happens at encoder init, and the maximum is only queryable when preanalysis is on, so with preanalysis off nothing caps it. This is fine because `async_depth` only limits how many frames can be in flight and sizes no allocation. The AMF runtime also applies its own input backpressure by returning AMF_INPUT_FULL, which FFmpeg handles by draining output and retrying. ## Workaround for the AMF SDK header bug The maximum comes from `GetPropertyInfo`. With unfixed AMF SDK headers its C mapping only matches the runtime on non-MSVC ABIs, so the correct call is target-dependent. `configure` probes for the `reserved` field in `AMFPropertyInfo` to detect headers where this is already corrected, and the getter selects one of three arms at compile time: - fixed headers: call `GetPropertyInfo` directly, on every target - unfixed headers on Windows: the MSVC-built AMF runtime places the by-name overload in the by-index slot, so it is called through `GetPropertyInfoAt`. Gated on `_WIN32` rather than the compiler, because the runtime's layout is what matters. MinGW builds need this too, since the AMF runtime is still MSVC-built. - unfixed headers elsewhere: declaration order already matches, so the direct call is correct Unfixed headers also shift `AMFPropertyInfo` in the C mapping. The C++ type has a virtual destructor, so its objects start with a vtable pointer that the C definition leaves out, and every field is offset by the size of one pointer. The getter skips past it before reading. Unlike the call selection above, this is gated on the probe alone and not `_WIN32`, because that vtable pointer is present under both Linux and Windows ABIs. The type check after the skip validates the arithmetic. ## Removing the workaround The ABI handling is transitional. Once the minimum AMF version includes the header fix, all of the following can be deleted in a single commit: - the `configure` probe and its `TYPES_LIST` entry - the `#elif defined(_WIN32)` and `#else` arms of the getter - the `#if !HAVE_AMFPROPERTYINFO_RESERVED` realign and its type check - comments in this commit related to unfixed/fixed headers After removing the workaround, only a single direct `GetPropertyInfo` call should remain. The clamp and async_depth helpers are unaffected; they are not part of the workaround. # Testing Tested on Radeon RX 9070 (Navi 48) and RX 7700 XT / 7800 XT (Navi 32), covering RDNA4 and RDNA3. Both `configure` probe states were built and exercised on every platform: `HAVE_AMFPROPERTYINFO_RESERVED` at 0 (unfixed headers, workaround arms) and at 1 (fixed headers, direct call). | Platform | Toolchain | Backend | |---|---|---| | Linux | GCC | Vulkan | | Windows | MSVC | D3D11 | | Windows | MinGW GCC | D3D11 | Headers and runtime were also mismatched during testing, since users typically build against the public 1.5.2 SDK while running whatever runtime their driver ships. A `--disable-amf` build was checked to confirm the new probe is skipped cleanly. All cases used the same base command, varying the preanalysis and async options: `ffmpeg -i input.mp4 -vf format=nv12 -frames:v 60 -an -c:v h264_amf -preanalysis 1 -pa_lookahead_buffer_depth 10 -async_depth 4 -f null -` On each platform, I ran a test matrix (~200 tests) over `h264_amf`, `hevc_amf` and `av1_amf`. The query returned 41 on every GPU and runtime tested, confirming it reports the same bound the removed macro hardcoded. Within the previously accepted range the only behavioural change is in how `async_depth` is sized. Relevant subset of test results (`h264_amf`, max = 41): | Options | Result | |---|---| | `-pa_lookahead_buffer_depth 40 -async_depth 1` | async 1 -> 41 | | `-pa_lookahead_buffer_depth 41 -async_depth 1` | async 1 -> 42 | | `-pa_lookahead_buffer_depth 42 -async_depth 1` | lookahead 42 -> 41, async 1 -> 42 | | `-pa_lookahead_buffer_depth 1000 -async_depth 16` | lookahead 1000 -> 41, async 16 -> 42 | | `-pa_lookahead_buffer_depth 2147483647` | lookahead -> 41, async 16 -> 42 | | `-pa_lookahead_buffer_depth 16 -async_depth 16` | async 16 -> 17 | | `-pa_lookahead_buffer_depth 10 -async_depth 42` | unchanged | | `-pa_lookahead_buffer_depth 10 -async_depth 43` | async 43 -> 42 | | `-pa_lookahead_buffer_depth 10 -async_depth 1000` | async 1000 -> 42 | | `-pa_adaptive_mini_gop 1 -async_depth 1` | async 1 -> 2 | | `-pa_lookahead_buffer_depth 20 -async_depth 4` (PA off) | unchanged | | `-async_depth 1000` (PA off) | unchanged, no cap | | `-pa_lookahead_buffer_depth -2` | rejected at parse | | `-async_depth 0` | rejected at parse | `hevc_amf` and `av1_amf` match `h264_amf` above except for adaptive mini-GOP: `hevc_amf` has no such option, and `av1_amf`'s effective lookahead is unchanged there, so no bump applies. Upstream and patched builds were also compared directly on the same hardware and toolchain. For all six of my cases within the previously accepted range, including those where `async_depth` now differs, the encoded bitstreams were byte-identical, so the change does not affect encoding decisions. Output equivalence: each container case was hashed with `-c copy -f md5`. Each configuration produces the same bitstream across both header states, and on Windows across both toolchains. Confirmed with synthetic input and with a real 1080p H.264 clip. _Note: I was unable to properly test on the RX 9060 XT. Preanalysis encodes produced no output frames for me. It doesn't appear related to this change since an unpatched master build fails the same way from the same command line._ `ffmpeg -f lavfi -i testsrc2=size=1280x720:rate=30 -vf format=nv12 -frames:v 60 -an -c:v h264_amf -preanalysis 1 -f null -` _The encode reports frame=0 and produces no output. The exit code can still be 0, so the frame count is the important part._ From fb21753a9fc8f2c5dd7c084cdb07e046c1f79419 Mon Sep 17 00:00:00 2001 From: "Ling, Edison" <Edison.Ling(a)amd.com> Date: Fri, 7 Aug 2026 14:53:46 -0400 Subject: [PATCH] avcodec/amfenc: Replace hardcoded lookahead maximum with runtime query Replace the hardcoded `MAX_LOOKAHEAD_DEPTH 41` bound on `pa_lookahead_buffer_depth` and `async_depth` with the maximum AMF reports at runtime, so the bound tracks what the runtime accepts instead of a constant duplicated in FFmpeg. Values above that maximum are clamped with a warning instead of being rejected by the option parser. `async_depth` is now sized from the lookahead depth read back from the encoder rather than the raw option value. The read-back is the value AMF actually applied, which fixes a latent under-size when adaptive mini-GOP raises the effective lookahead. The two changes cannot be split: raising the option ceiling makes a value like 1000 parse successfully, and without the read-back that would size `async_depth` to 1001, replacing a hard parse error with a silent oversized allocation. The maximum comes from `GetPropertyInfo`. With unfixed AMF SDK headers its C mapping only matches the runtime on non-MSVC ABIs, so the correct call is target-dependent. `configure` probes for the `reserved` field in `AMFPropertyInfo` to detect headers where this is already corrected, and the getter selects one of three calls at compile time: - fixed headers: call `GetPropertyInfo` directly, on every target - unfixed headers on Windows: the MSVC-built AMF runtime places the by-name overload in the by-index slot, so it is called through `GetPropertyInfoAt`. Gated on `_WIN32` rather than the compiler, because the runtime's layout is what matters -- MinGW builds need this too - unfixed headers elsewhere: declaration order already matches, so the direct call is correct Unfixed headers also shift `AMFPropertyInfo` by one pointer in the C mapping. That realign is unconditional and the type check after it validates the arithmetic, so both are scoped to that branch only. --- configure | 4 +- libavcodec/amfenc.c | 108 +++++++++++++++++++++++++++++++++++---- libavcodec/amfenc.h | 24 ++++++++- libavcodec/amfenc_av1.c | 11 ++-- libavcodec/amfenc_h264.c | 11 ++-- libavcodec/amfenc_hevc.c | 11 ++-- 6 files changed, 142 insertions(+), 27 deletions(-) diff --git a/configure b/configure index e09bbd7f04..0f645503c8 100755 --- a/configure +++ b/configure @@ -2679,6 +2679,7 @@ TYPES_LIST=" kCVImageBufferTransferFunction_ITU_R_2020 kCVImageBufferTransferFunction_SMPTE_ST_428_1 kVTQPModulationLevel_Default + AMFPropertyInfo_reserved SecPkgContext_KeyingMaterialInfo socklen_t struct_addrinfo @@ -7927,7 +7928,8 @@ fi enabled amf && check_cpp_condition amf "AMF/core/Version.h" \ - "(AMF_VERSION_MAJOR << 48 | AMF_VERSION_MINOR << 32 | AMF_VERSION_RELEASE << 16 | AMF_VERSION_BUILD_NUM) >= 0x1000500020000" + "(AMF_VERSION_MAJOR << 48 | AMF_VERSION_MINOR << 32 | AMF_VERSION_RELEASE << 16 | AMF_VERSION_BUILD_NUM) >= 0x1000500020000" && + check_struct "AMF/core/PropertyStorageEx.h" "AMFPropertyInfo" reserved # Funny iconv installations are not unusual, so check it after all flags have been set if enabled libc_iconv; then diff --git a/libavcodec/amfenc.c b/libavcodec/amfenc.c index 3ffca750d8..4c052b47a2 100644 --- a/libavcodec/amfenc.c +++ b/libavcodec/amfenc.c @@ -36,6 +36,8 @@ #include "amfenc.h" #include "encode.h" +#include <AMF/components/PreAnalysis.h> + #define AMF_AV_FRAME_REF L"av_frame_ref" #define PTS_PROP L"PtsProp" @@ -166,6 +168,103 @@ int av_cold ff_amf_encode_close(AVCodecContext *avctx) return 0; } +int ff_amf_get_lookahead_depth_max(AVCodecContext *avctx, AMFComponent *encoder, amf_int64 *out_max) +{ + // Query the maximum accepted value of AMF_PA_LOOKAHEAD_BUFFER_DEPTH. + // With unfixed AMF SDK headers the C vtable mapping only matches the runtime on + // non-MSVC ABIs, so the correct call is target-dependent (see each arm below). + // HAVE_AMFPROPERTYINFO_RESERVED detects whether the header fix is present. + // TODO: once the minimum AMF version includes the fix, the configure probe and + // legacy branches can be dropped entirely. + const wchar_t *name = AMF_PA_LOOKAHEAD_BUFFER_DEPTH; + const AMFPropertyInfo *prop_info = NULL; + AMF_RESULT res; + + if (!encoder || !out_max) + return AVERROR(EINVAL); + +#if HAVE_AMFPROPERTYINFO_RESERVED + // Fixed SDK ABI: the C *Vtbl declaration order matches the runtime on every + // target (Win-MSVC, Win-MinGW, Linux), so pVtbl->GetPropertyInfo is the + // by-name overload. Call it directly. + res = encoder->pVtbl->GetPropertyInfo(encoder, name, &prop_info); + av_log(avctx, AV_LOG_DEBUG, "GetPropertyInfo() returned %d\n", res); +#elif defined(_WIN32) + // Unfixed SDK ABI on Windows: amfrt64.dll is MSVC-built, and MSVC lays overloaded + // virtuals in reverse declaration order, so the by-name GetPropertyInfo sits in + // the by-index GetPropertyInfoAt slot. Gated on _WIN32, not the compiler: the + // runtime's layout is what matters, so MinGW builds need this too. + typedef AMF_RESULT (AMF_STD_CALL *GetPropInfoByNameFn)(AMFComponent *, const wchar_t *, const AMFPropertyInfo **); + GetPropInfoByNameFn get_by_name = (GetPropInfoByNameFn)(void *)encoder->pVtbl->GetPropertyInfoAt; + res = get_by_name(encoder, name, &prop_info); + av_log(avctx, AV_LOG_DEBUG, "GetPropertyInfoAt() returned %d\n", res); +#else + // Unfixed SDK ABI, non-MSVC runtime (GCC/Clang on Linux; Itanium C++ ABI): + // overloaded virtuals keep declaration order, so the declared GetPropertyInfo + // slot is already by-name. Call it directly; swapping would hit by-index. + res = encoder->pVtbl->GetPropertyInfo(encoder, name, &prop_info); + av_log(avctx, AV_LOG_DEBUG, "GetPropertyInfo() returned %d\n", res); +#endif + if (res != AMF_OK || !prop_info) + return AVERROR_EXTERNAL; + +#if !HAVE_AMFPROPERTYINFO_RESERVED + // Unfixed SDK ABI: AMFPropertyInfo starts with an extra pointer the C mapping + // omits (the C++ destructor's vtable pointer), so skip it to realign before reading. + prop_info = (const AMFPropertyInfo *)((const uint8_t *)prop_info + sizeof(void *)); + if (prop_info->type != AMF_VARIANT_INT64) + return AVERROR_EXTERNAL; +#endif + + *out_max = prop_info->maxValue.int64Value; + return 0; +} + +void ff_amf_clamp_lookahead_depth(AVCodecContext *avctx, AMFEncoderContext *ctx) +{ + amf_int64 hw_max; + AMF_RESULT res; + + if (ff_amf_get_lookahead_depth_max(avctx, ctx->encoder, &hw_max) == 0 && + ctx->pa_lookahead_buffer_depth > hw_max) { + av_log(avctx, AV_LOG_WARNING, + "pa_lookahead_buffer_depth %d exceeds supported maximum %lld, clamping\n", + ctx->pa_lookahead_buffer_depth, (long long)hw_max); + ctx->pa_lookahead_buffer_depth = (int)hw_max; + } + + AMF_ASSIGN_PROPERTY_INT64(res, ctx->encoder, AMF_PA_LOOKAHEAD_BUFFER_DEPTH, ctx->pa_lookahead_buffer_depth); + if (res != AMF_OK) + av_log(avctx, AV_LOG_WARNING, "Setting lookahead buffer depth failed with error %d\n", res); +} + +void ff_amf_fit_async_depth(AVCodecContext *avctx, AMFEncoderContext *ctx) +{ + // Size async_depth to the lookahead window in two independent steps: + // - bump: widen async_depth to fit the lookahead depth read back from the + // encoder (the applied value; only present when preanalysis is enabled). + // - cap: limit async_depth to the reported maximum + 1 when it can be queried. + amf_int64 hw_max; + AMFVariantStruct var = { 0 }; + AMF_RESULT res; + + res = ctx->encoder->pVtbl->GetProperty(ctx->encoder, AMF_PA_LOOKAHEAD_BUFFER_DEPTH, &var); + if (res == AMF_OK && var.int64Value >= ctx->hwsurfaces_in_queue_max) { + av_log(avctx, AV_LOG_WARNING, + "async_depth (%d) too small for lookahead (%lld), increasing to (%lld)\n", + ctx->hwsurfaces_in_queue_max, (long long)var.int64Value, (long long)(var.int64Value + 1)); + ctx->hwsurfaces_in_queue_max = (int)(var.int64Value + 1); + } + + if (ff_amf_get_lookahead_depth_max(avctx, ctx->encoder, &hw_max) == 0 && + ctx->hwsurfaces_in_queue_max > hw_max + 1) { + av_log(avctx, AV_LOG_WARNING, + "async_depth %d exceeds supported range, clamping to %lld\n", + ctx->hwsurfaces_in_queue_max, (long long)(hw_max + 1)); + ctx->hwsurfaces_in_queue_max = (int)(hw_max + 1); + } +} + static int amf_copy_surface(AVCodecContext *avctx, const AVFrame *frame, AMFSurface* surface) { @@ -326,15 +425,6 @@ int ff_amf_encode_init(AVCodecContext *avctx) AMF_RETURN_IF_FALSE(ctx, ret == 0, ret, "Failed to create hardware device context (AMF) : %s\n", av_err2str(ret)); } - if (ctx->pa_lookahead_buffer_depth >= ctx->hwsurfaces_in_queue_max) { - av_log(avctx, AV_LOG_WARNING, - "async_depth (%d) too small for lookahead (%d), increasing to (%d)\n", - ctx->hwsurfaces_in_queue_max, - ctx->pa_lookahead_buffer_depth, - ctx->pa_lookahead_buffer_depth + 1); - ctx->hwsurfaces_in_queue_max = ctx->pa_lookahead_buffer_depth + 1; - } - if ((ret = amf_init_encoder(avctx)) == 0) { return 0; } diff --git a/libavcodec/amfenc.h b/libavcodec/amfenc.h index 1571541b9b..e68f039fdf 100644 --- a/libavcodec/amfenc.h +++ b/libavcodec/amfenc.h @@ -31,8 +31,6 @@ #include "avcodec.h" #include "hwconfig.h" -#define MAX_LOOKAHEAD_DEPTH 41 - /** * AMF encoder context */ @@ -158,6 +156,28 @@ int ff_amf_encode_close(AVCodecContext *avctx); */ int ff_amf_receive_packet(AVCodecContext *avctx, AVPacket *avpkt); +/** +* Query the maximum PA lookahead buffer depth (AMF_PA_LOOKAHEAD_BUFFER_DEPTH). +* Only available once preanalysis is enabled on the encoder. +* @return 0 and sets *out_max on success; a negative AVERROR on failure. +*/ +int ff_amf_get_lookahead_depth_max(AVCodecContext *avctx, AMFComponent *encoder, amf_int64 *out_max); + +/** +* Clamp pa_lookahead_buffer_depth to the AMF-reported maximum and assign it to +* the encoder. Call from within the preanalysis configuration block, before +* encoder Init(). +*/ +void ff_amf_clamp_lookahead_depth(AVCodecContext *avctx, AMFEncoderContext *ctx); + +/** +* Size async_depth (hwsurfaces_in_queue_max) around the lookahead window: widen +* it to hold the lookahead depth read back from the encoder, then cap it at the +* AMF-reported lookahead maximum + 1. Call before encoder Init(), after all +* lookahead/B-frame properties have been set. +*/ +void ff_amf_fit_async_depth(AVCodecContext *avctx, AMFEncoderContext *ctx); + /** * Supported formats */ diff --git a/libavcodec/amfenc_av1.c b/libavcodec/amfenc_av1.c index af20b5d04d..00966c23a8 100644 --- a/libavcodec/amfenc_av1.c +++ b/libavcodec/amfenc_av1.c @@ -101,7 +101,7 @@ static const AVOption options[] = { { "gop", "", 0, AV_OPT_TYPE_CONST, {.i64 = AMF_VIDEO_ENCODER_AV1_HEADER_INSERTION_MODE_GOP_ALIGNED }, 0, 0, VE, .unit = "hdrmode" }, { "frame", "", 0, AV_OPT_TYPE_CONST, {.i64 = AMF_VIDEO_ENCODER_AV1_HEADER_INSERTION_MODE_KEY_FRAME_ALIGNED }, 0, 0, VE, .unit = "hdrmode" }, - { "async_depth", "Set maximum encoding parallelism. Higher values increase output latency.", OFFSET(hwsurfaces_in_queue_max), AV_OPT_TYPE_INT, {.i64 = 16 }, 1, MAX_LOOKAHEAD_DEPTH + 1, VE }, + { "async_depth", "Set maximum encoding parallelism. Higher values increase output latency.", OFFSET(hwsurfaces_in_queue_max), AV_OPT_TYPE_INT, {.i64 = 16 }, 1, INT_MAX, VE }, { "preencode", "Enable preencode", OFFSET(preencode), AV_OPT_TYPE_BOOL, {.i64 = -1 }, -1, 1, VE}, { "enforce_hrd", "Enforce HRD", OFFSET(enforce_hrd), AV_OPT_TYPE_BOOL, {.i64 = -1 }, -1, 1, VE}, @@ -169,7 +169,7 @@ static const AVOption options[] = { { "pa_frame_sad_enable", "Enable Frame SAD algorithm", OFFSET(pa_frame_sad), AV_OPT_TYPE_BOOL, {.i64 = -1 }, -1, 1, VE }, { "pa_ltr_enable", "Enable long term reference frame management", OFFSET(pa_ltr), AV_OPT_TYPE_BOOL, {.i64 = -1 }, -1, 1, VE }, - { "pa_lookahead_buffer_depth", "Sets the PA lookahead buffer size", OFFSET(pa_lookahead_buffer_depth), AV_OPT_TYPE_INT, {.i64 = -1 }, -1, MAX_LOOKAHEAD_DEPTH, VE }, + { "pa_lookahead_buffer_depth", "Sets the PA lookahead buffer size", OFFSET(pa_lookahead_buffer_depth), AV_OPT_TYPE_INT, {.i64 = -1 }, -1, INT_MAX, VE }, { "pa_paq_mode", "Sets the perceptual adaptive quantization mode", OFFSET(pa_paq_mode), AV_OPT_TYPE_INT, {.i64 = -1 }, -1, AMF_PA_PAQ_MODE_CAQ, VE , .unit = "paq_mode" }, { "none", "no perceptual adaptive quantization", 0, AV_OPT_TYPE_CONST, {.i64 = AMF_PA_PAQ_MODE_NONE }, 0, 0, VE, .unit = "paq_mode" }, @@ -504,9 +504,8 @@ static av_cold int amf_encode_init_av1(AVCodecContext* avctx) if (ctx->pa_ltr != -1) { AMF_ASSIGN_PROPERTY_BOOL(res, ctx->encoder, AMF_PA_LTR_ENABLE, ((ctx->pa_ltr == 0) ? false : true)); } - if (ctx->pa_lookahead_buffer_depth != -1) { - AMF_ASSIGN_PROPERTY_INT64(res, ctx->encoder, AMF_PA_LOOKAHEAD_BUFFER_DEPTH, ctx->pa_lookahead_buffer_depth); - } + if (ctx->pa_lookahead_buffer_depth != -1) + ff_amf_clamp_lookahead_depth(avctx, ctx); if (ctx->pa_high_motion_quality_boost_mode != -1) { AMF_ASSIGN_PROPERTY_INT64(res, ctx->encoder, AMF_PA_HIGH_MOTION_QUALITY_BOOST_MODE, ctx->pa_high_motion_quality_boost_mode); } @@ -573,6 +572,8 @@ static av_cold int amf_encode_init_av1(AVCodecContext* avctx) res = ctx->encoder->pVtbl->GetProperty(ctx->encoder, AMF_VIDEO_ENCODER_AV1_QUERY_TIMEOUT, &var); ctx->query_timeout_supported = res == AMF_OK && var.int64Value; + ff_amf_fit_async_depth(avctx, ctx); + // init encoder res = ctx->encoder->pVtbl->Init(ctx->encoder, ctx->format, avctx->width, avctx->height); AMF_RETURN_IF_FALSE(ctx, res == AMF_OK, AVERROR_BUG, "encoder->Init() failed with error %d\n", res); diff --git a/libavcodec/amfenc_h264.c b/libavcodec/amfenc_h264.c index 650a9bc7e9..994d1ec25f 100644 --- a/libavcodec/amfenc_h264.c +++ b/libavcodec/amfenc_h264.c @@ -114,7 +114,7 @@ static const AVOption options[] = { { "header_spacing", "Header Insertion Spacing", OFFSET(header_spacing), AV_OPT_TYPE_INT, { .i64 = -1 }, -1, 1000, VE }, /// Maximum queued frames - { "async_depth", "Set maximum encoding parallelism. Higher values increase output latency.", OFFSET(hwsurfaces_in_queue_max), AV_OPT_TYPE_INT, {.i64 = 16 }, 1, MAX_LOOKAHEAD_DEPTH + 1, VE }, + { "async_depth", "Set maximum encoding parallelism. Higher values increase output latency.", OFFSET(hwsurfaces_in_queue_max), AV_OPT_TYPE_INT, {.i64 = 16 }, 1, INT_MAX, VE }, /// B-Frames // BPicturesPattern=bf @@ -175,7 +175,7 @@ static const AVOption options[] = { { "pa_frame_sad_enable", "Enable Frame SAD algorithm", OFFSET(pa_frame_sad), AV_OPT_TYPE_BOOL, {.i64 = -1 }, -1, 1, VE }, { "pa_ltr_enable", "Enable long term reference frame management", OFFSET(pa_ltr), AV_OPT_TYPE_BOOL, {.i64 = -1 }, -1, 1, VE }, - { "pa_lookahead_buffer_depth", "Sets the PA lookahead buffer size", OFFSET(pa_lookahead_buffer_depth), AV_OPT_TYPE_INT, {.i64 = -1 }, -1, MAX_LOOKAHEAD_DEPTH, VE }, + { "pa_lookahead_buffer_depth", "Sets the PA lookahead buffer size", OFFSET(pa_lookahead_buffer_depth), AV_OPT_TYPE_INT, {.i64 = -1 }, -1, INT_MAX, VE }, { "pa_paq_mode", "Sets the perceptual adaptive quantization mode", OFFSET(pa_paq_mode), AV_OPT_TYPE_INT, {.i64 = -1 }, -1, AMF_PA_PAQ_MODE_CAQ, VE , .unit = "paq_mode" }, { "none", "no perceptual adaptive quantization", 0, AV_OPT_TYPE_CONST, {.i64 = AMF_PA_PAQ_MODE_NONE }, 0, 0, VE, .unit = "paq_mode" }, @@ -465,9 +465,8 @@ static av_cold int amf_encode_init_h264(AVCodecContext *avctx) if (ctx->pa_ltr != -1) { AMF_ASSIGN_PROPERTY_BOOL(res, ctx->encoder, AMF_PA_LTR_ENABLE, ((ctx->pa_ltr == 0) ? false : true)); } - if (ctx->pa_lookahead_buffer_depth != -1) { - AMF_ASSIGN_PROPERTY_INT64(res, ctx->encoder, AMF_PA_LOOKAHEAD_BUFFER_DEPTH, ctx->pa_lookahead_buffer_depth); - } + if (ctx->pa_lookahead_buffer_depth != -1) + ff_amf_clamp_lookahead_depth(avctx, ctx); if (ctx->pa_high_motion_quality_boost_mode != -1) { AMF_ASSIGN_PROPERTY_INT64(res, ctx->encoder, AMF_PA_HIGH_MOTION_QUALITY_BOOST_MODE, ctx->pa_high_motion_quality_boost_mode); } @@ -551,6 +550,8 @@ static av_cold int amf_encode_init_h264(AVCodecContext *avctx) res = ctx->encoder->pVtbl->GetProperty(ctx->encoder, AMF_VIDEO_ENCODER_QUERY_TIMEOUT, &var); ctx->query_timeout_supported = res == AMF_OK && var.int64Value; + ff_amf_fit_async_depth(avctx, ctx); + // Initialize Encoder res = ctx->encoder->pVtbl->Init(ctx->encoder, ctx->format, avctx->width, avctx->height); AMF_RETURN_IF_FALSE(ctx, res == AMF_OK, AVERROR_BUG, "encoder->Init() failed with error %d\n", res); diff --git a/libavcodec/amfenc_hevc.c b/libavcodec/amfenc_hevc.c index 6fa20d172e..e66ba3f7a8 100644 --- a/libavcodec/amfenc_hevc.c +++ b/libavcodec/amfenc_hevc.c @@ -91,7 +91,7 @@ static const AVOption options[] = { { "gop", "", 0, AV_OPT_TYPE_CONST, { .i64 = AMF_VIDEO_ENCODER_HEVC_HEADER_INSERTION_MODE_GOP_ALIGNED }, 0, 0, VE, .unit = "hdrmode" }, { "idr", "", 0, AV_OPT_TYPE_CONST, { .i64 = AMF_VIDEO_ENCODER_HEVC_HEADER_INSERTION_MODE_IDR_ALIGNED }, 0, 0, VE, .unit = "hdrmode" }, - { "async_depth", "Set maximum encoding parallelism. Higher values increase output latency.", OFFSET(hwsurfaces_in_queue_max), AV_OPT_TYPE_INT, {.i64 = 16 }, 1, MAX_LOOKAHEAD_DEPTH + 1, VE }, + { "async_depth", "Set maximum encoding parallelism. Higher values increase output latency.", OFFSET(hwsurfaces_in_queue_max), AV_OPT_TYPE_INT, {.i64 = 16 }, 1, INT_MAX, VE }, { "high_motion_quality_boost_enable", "Enable High motion quality boost mode", OFFSET(hw_high_motion_quality_boost), AV_OPT_TYPE_BOOL, {.i64 = -1 }, -1, 1, VE }, { "gops_per_idr", "GOPs per IDR 0-no IDR will be inserted", OFFSET(gops_per_idr), AV_OPT_TYPE_INT, { .i64 = 1 }, 0, INT_MAX, VE }, @@ -146,7 +146,7 @@ static const AVOption options[] = { { "pa_frame_sad_enable", "Enable Frame SAD algorithm", OFFSET(pa_frame_sad), AV_OPT_TYPE_BOOL, {.i64 = -1 }, -1, 1, VE }, { "pa_ltr_enable", "Enable long term reference frame management", OFFSET(pa_ltr), AV_OPT_TYPE_BOOL, {.i64 = -1 }, -1, 1, VE }, - { "pa_lookahead_buffer_depth", "Sets the PA lookahead buffer size", OFFSET(pa_lookahead_buffer_depth), AV_OPT_TYPE_INT, {.i64 = -1 }, -1, MAX_LOOKAHEAD_DEPTH, VE }, + { "pa_lookahead_buffer_depth", "Sets the PA lookahead buffer size", OFFSET(pa_lookahead_buffer_depth), AV_OPT_TYPE_INT, {.i64 = -1 }, -1, INT_MAX, VE }, { "pa_paq_mode", "Sets the perceptual adaptive quantization mode", OFFSET(pa_paq_mode), AV_OPT_TYPE_INT, {.i64 = -1 }, -1, AMF_PA_PAQ_MODE_CAQ, VE , .unit = "paq_mode" }, { "none", "no perceptual adaptive quantization", 0, AV_OPT_TYPE_CONST, {.i64 = AMF_PA_PAQ_MODE_NONE }, 0, 0, VE, .unit = "paq_mode" }, @@ -470,9 +470,8 @@ static av_cold int amf_encode_init_hevc(AVCodecContext *avctx) if (ctx->pa_ltr != -1) { AMF_ASSIGN_PROPERTY_BOOL(res, ctx->encoder, AMF_PA_LTR_ENABLE, ((ctx->pa_ltr == 0) ? false : true)); } - if (ctx->pa_lookahead_buffer_depth != -1) { - AMF_ASSIGN_PROPERTY_INT64(res, ctx->encoder, AMF_PA_LOOKAHEAD_BUFFER_DEPTH, ctx->pa_lookahead_buffer_depth); - } + if (ctx->pa_lookahead_buffer_depth != -1) + ff_amf_clamp_lookahead_depth(avctx, ctx); if (ctx->pa_high_motion_quality_boost_mode != -1) { AMF_ASSIGN_PROPERTY_INT64(res, ctx->encoder, AMF_PA_HIGH_MOTION_QUALITY_BOOST_MODE, ctx->pa_high_motion_quality_boost_mode); } @@ -483,6 +482,8 @@ static av_cold int amf_encode_init_hevc(AVCodecContext *avctx) res = ctx->encoder->pVtbl->GetProperty(ctx->encoder, AMF_VIDEO_ENCODER_HEVC_QUERY_TIMEOUT, &var); ctx->query_timeout_supported = res == AMF_OK && var.int64Value; + ff_amf_fit_async_depth(avctx, ctx); + // init encoder res = ctx->encoder->pVtbl->Init(ctx->encoder, ctx->format, avctx->width, avctx->height); AMF_RETURN_IF_FALSE(ctx, res == AMF_OK, AVERROR_BUG, "encoder->Init() failed with error %d\n", res); -- 2.52.0

1 0

[PR] [release/9.0] avfilter/vf_bwdif: fix line boundary checks for >8 bits content (PR #24054)
by ffmpeg-devel 08 Aug '26

08 Aug '26

PR #24054 opened by ffmpeg-devel URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24054 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24054.patch **Backport:** https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/23921 The mrefs/prefs boundary conditions scaled the y limits by the number of bytes per sample, even though the ref offsets are already expressed in elements. For content deeper than 8 bits this clamped perfectly valid in-bounds line taps. Found by validating the libplacebo GPU port against this filter, which matches exactly on 16-bit content everywhere except these lines. This changes the output for >8 bit content on the affected lines. From dcf567a048fae062d527b63b94eb9f1bbf03472f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kacper=20Michaj=C5=82ow?= <kasper93(a)gmail.com> Date: Sun, 26 Jul 2026 19:12:40 +0200 Subject: [PATCH 1/2] avfilter/vf_bwdif: fix line boundary checks for >8 bits content MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The mrefs/prefs boundary conditions scaled the y limits by the number of bytes per sample, even though the ref offsets are already expressed in elements. For content deeper than 8 bits this clamped perfectly valid in-bounds line taps. Found by validating the libplacebo GPU port against this filter, which matches exactly on 16-bit content everywhere except these lines. This changes the output for >8 bit content on the affected lines. Signed-off-by: Kacper Michajłow <kasper93(a)gmail.com> (cherry picked from commit 7d96c87a7382fed8d571748d6ed5c1869c192c31) --- libavfilter/vf_bwdif.c | 16 +++++----- tests/ref/fate/filter-bwdif10 | 60 +++++++++++++++++------------------ 2 files changed, 38 insertions(+), 38 deletions(-) diff --git a/libavfilter/vf_bwdif.c b/libavfilter/vf_bwdif.c index dbe35a029b..a34f1906dd 100644 --- a/libavfilter/vf_bwdif.c +++ b/libavfilter/vf_bwdif.c @@ -79,22 +79,22 @@ static int filter_slice(AVFilterContext *ctx, void *arg, int jobnr, int nb_jobs) if (yadif->current_field == YADIF_FIELD_END) { if ((y < 3) || ((y + 3) >= td->h)) { s->dsp.filter_edge(dst, prev, cur, next, td->w, - (y + df) < td->h ? refs : -refs, - y > (df - 1) ? -refs : refs, + (y + 1) < td->h ? refs : -refs, + y > 0 ? -refs : refs, refs << 1, -(refs << 1), td->parity ^ td->tff, clip_max, (y < 2) || ((y + 3) > td->h) ? 0 : 1); } else { - s->dsp.filter_intra(dst, cur, td->w, (y + df) < td->h ? refs : -refs, - y > (df - 1) ? -refs : refs, - (y + 3*df) < td->h ? 3 * refs : -refs, - y > (3*df - 1) ? -3 * refs : refs, + s->dsp.filter_intra(dst, cur, td->w, (y + 1) < td->h ? refs : -refs, + y > 0 ? -refs : refs, + (y + 3) < td->h ? 3 * refs : -refs, + y > 2 ? -3 * refs : refs, td->parity ^ td->tff, clip_max); } } else if ((y < 4) || ((y + 5) > td->h)) { s->dsp.filter_edge(dst, prev, cur, next, td->w, - (y + df) < td->h ? refs : -refs, - y > (df - 1) ? -refs : refs, + (y + 1) < td->h ? refs : -refs, + y > 0 ? -refs : refs, refs << 1, -(refs << 1), td->parity ^ td->tff, clip_max, (y < 2) || ((y + 3) > td->h) ? 0 : 1); diff --git a/tests/ref/fate/filter-bwdif10 b/tests/ref/fate/filter-bwdif10 index d97acea991..70a7f9d3ed 100644 --- a/tests/ref/fate/filter-bwdif10 +++ b/tests/ref/fate/filter-bwdif10 @@ -3,33 +3,33 @@ #codec_id 0: rawvideo #dimensions 0: 720x576 #sar 0: 16/15 -0, 9, 9, 1, 1244160, 0x4f0e6e1c -0, 10, 10, 1, 1244160, 0x57152296 -0, 11, 11, 1, 1244160, 0x0074598b -0, 12, 12, 1, 1244160, 0x44537bb8 -0, 13, 13, 1, 1244160, 0x6f27ab9c -0, 14, 14, 1, 1244160, 0x1f195e10 -0, 15, 15, 1, 1244160, 0xc5644298 -0, 16, 16, 1, 1244160, 0x24c1230e -0, 17, 17, 1, 1244160, 0x2651b213 -0, 18, 18, 1, 1244160, 0x88efe863 -0, 19, 19, 1, 1244160, 0x8bf0ada1 -0, 20, 20, 1, 1244160, 0x8b6c013a -0, 21, 21, 1, 1244160, 0x69ebc579 -0, 22, 22, 1, 1244160, 0xa45d0f0c -0, 23, 23, 1, 1244160, 0x272162e4 -0, 24, 24, 1, 1244160, 0x8c456817 -0, 25, 25, 1, 1244160, 0x872c58ee -0, 26, 26, 1, 1244160, 0x9318cb95 -0, 27, 27, 1, 1244160, 0xcaec95f1 -0, 28, 28, 1, 1244160, 0x8b01a105 -0, 29, 29, 1, 1244160, 0x956de85e -0, 30, 30, 1, 1244160, 0xa2998ab9 -0, 31, 31, 1, 1244160, 0xa56b5621 -0, 32, 32, 1, 1244160, 0x7bd6b161 -0, 33, 33, 1, 1244160, 0xbcb75d7a -0, 34, 34, 1, 1244160, 0xe5e94727 -0, 35, 35, 1, 1244160, 0xf3145c67 -0, 36, 36, 1, 1244160, 0xcf63df5a -0, 37, 37, 1, 1244160, 0x77d08ec8 -0, 38, 38, 1, 1244160, 0xc99d2eb2 +0, 9, 9, 1, 1244160, 0xee7d128d +0, 10, 10, 1, 1244160, 0x6c25ebe8 +0, 11, 11, 1, 1244160, 0xae8e297d +0, 12, 12, 1, 1244160, 0x20284ad0 +0, 13, 13, 1, 1244160, 0x95c57238 +0, 14, 14, 1, 1244160, 0x5ae32569 +0, 15, 15, 1, 1244160, 0x92401342 +0, 16, 16, 1, 1244160, 0x95ed0c85 +0, 17, 17, 1, 1244160, 0xb79398df +0, 18, 18, 1, 1244160, 0x66a0dda3 +0, 19, 19, 1, 1244160, 0x906596c3 +0, 20, 20, 1, 1244160, 0x26f1da33 +0, 21, 21, 1, 1244160, 0xe317ad64 +0, 22, 22, 1, 1244160, 0x2f5fe488 +0, 23, 23, 1, 1244160, 0x7bd64c7b +0, 24, 24, 1, 1244160, 0x7470479e +0, 25, 25, 1, 1244160, 0x61653bee +0, 26, 26, 1, 1244160, 0xb249a6cc +0, 27, 27, 1, 1244160, 0x2cce6f74 +0, 28, 28, 1, 1244160, 0x711d7631 +0, 29, 29, 1, 1244160, 0xa304c119 +0, 30, 30, 1, 1244160, 0x3fb66044 +0, 31, 31, 1, 1244160, 0x97c11ba5 +0, 32, 32, 1, 1244160, 0xd8f37fed +0, 33, 33, 1, 1244160, 0x33d541e7 +0, 34, 34, 1, 1244160, 0x2e2e2535 +0, 35, 35, 1, 1244160, 0xe26132bd +0, 36, 36, 1, 1244160, 0x13b9b37b +0, 37, 37, 1, 1244160, 0x05825e47 +0, 38, 38, 1, 1244160, 0x44f6f505 -- 2.52.0 From 03d9533176e98bb9fbf569c1f34968e73e948dd9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Kacper=20Michaj=C5=82ow?= <kasper93(a)gmail.com> Date: Sun, 26 Jul 2026 19:56:11 +0200 Subject: [PATCH 2/2] Revert "lavfi/bwdif: fix heap-buffer-overflow with small height videos" MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This reverts commit 795bccdaf57772b1803914dee2f32d52776518e2. The heap-buffer-overflow was a symptom of the >8 bit boundary conditions bug fixed in the previous commit. This workaround is no longer needed. This restores spatial-only interpolation of the whole frame for the first and last fields, instead of running the temporal edge filter against cloned reference frames on the outermost lines. Because of that first frame references are updated to pre-fix values. Signed-off-by: Kacper Michajłow <kasper93(a)gmail.com> (cherry picked from commit 3cd408c2a6e5a7de8b6d641dedd12d523b94014c) --- libavfilter/vf_bwdif.c | 19 +++++-------------- tests/ref/fate/filter-bwdif-mode0 | 2 +- tests/ref/fate/filter-bwdif-mode1 | 2 +- tests/ref/fate/filter-bwdif10 | 2 +- 4 files changed, 8 insertions(+), 17 deletions(-) diff --git a/libavfilter/vf_bwdif.c b/libavfilter/vf_bwdif.c index a34f1906dd..5fba0e7a14 100644 --- a/libavfilter/vf_bwdif.c +++ b/libavfilter/vf_bwdif.c @@ -77,20 +77,11 @@ static int filter_slice(AVFilterContext *ctx, void *arg, int jobnr, int nb_jobs) uint8_t *next = &yadif->next->data[td->plane][y * linesize]; uint8_t *dst = &td->frame->data[td->plane][y * td->frame->linesize[td->plane]]; if (yadif->current_field == YADIF_FIELD_END) { - if ((y < 3) || ((y + 3) >= td->h)) { - s->dsp.filter_edge(dst, prev, cur, next, td->w, - (y + 1) < td->h ? refs : -refs, - y > 0 ? -refs : refs, - refs << 1, -(refs << 1), - td->parity ^ td->tff, clip_max, - (y < 2) || ((y + 3) > td->h) ? 0 : 1); - } else { - s->dsp.filter_intra(dst, cur, td->w, (y + 1) < td->h ? refs : -refs, - y > 0 ? -refs : refs, - (y + 3) < td->h ? 3 * refs : -refs, - y > 2 ? -3 * refs : refs, - td->parity ^ td->tff, clip_max); - } + s->dsp.filter_intra(dst, cur, td->w, (y + 1) < td->h ? refs : -refs, + y > 0 ? -refs : refs, + (y + 3) < td->h ? 3 * refs : -refs, + y > 2 ? -3 * refs : refs, + td->parity ^ td->tff, clip_max); } else if ((y < 4) || ((y + 5) > td->h)) { s->dsp.filter_edge(dst, prev, cur, next, td->w, (y + 1) < td->h ? refs : -refs, diff --git a/tests/ref/fate/filter-bwdif-mode0 b/tests/ref/fate/filter-bwdif-mode0 index 91b47dbe70..23dcaee900 100644 --- a/tests/ref/fate/filter-bwdif-mode0 +++ b/tests/ref/fate/filter-bwdif-mode0 @@ -3,7 +3,7 @@ #codec_id 0: rawvideo #dimensions 0: 720x576 #sar 0: 16/15 -0, 9, 9, 1, 622080, 0x3f25bfc2 +0, 9, 9, 1, 622080, 0xd435648a 0, 10, 10, 1, 622080, 0x62085455 0, 11, 11, 1, 622080, 0x60f943a0 0, 12, 12, 1, 622080, 0x5396f14a diff --git a/tests/ref/fate/filter-bwdif-mode1 b/tests/ref/fate/filter-bwdif-mode1 index 1e604646e7..e8db88c932 100644 --- a/tests/ref/fate/filter-bwdif-mode1 +++ b/tests/ref/fate/filter-bwdif-mode1 @@ -3,7 +3,7 @@ #codec_id 0: rawvideo #dimensions 0: 720x576 #sar 0: 16/15 -0, 18, 18, 1, 622080, 0x3f25bfc2 +0, 18, 18, 1, 622080, 0xd435648a 0, 19, 19, 1, 622080, 0xef4617cc 0, 20, 20, 1, 622080, 0x62085455 0, 21, 21, 1, 622080, 0x5b5ae735 diff --git a/tests/ref/fate/filter-bwdif10 b/tests/ref/fate/filter-bwdif10 index 70a7f9d3ed..76fcb8882e 100644 --- a/tests/ref/fate/filter-bwdif10 +++ b/tests/ref/fate/filter-bwdif10 @@ -3,7 +3,7 @@ #codec_id 0: rawvideo #dimensions 0: 720x576 #sar 0: 16/15 -0, 9, 9, 1, 1244160, 0xee7d128d +0, 9, 9, 1, 1244160, 0x71ca561b 0, 10, 10, 1, 1244160, 0x6c25ebe8 0, 11, 11, 1, 1244160, 0xae8e297d 0, 12, 12, 1, 1244160, 0x20284ad0 -- 2.52.0

1 0

[PATCH 0/2] VVC: AVX2 planar intra prediction
by Gudikandula Samith 08 Aug '26

08 Aug '26

This series adds an AVX2 implementation of the VVC planar intra prediction kernel, which currently has no SIMD implementation on any architecture. Only 8-bit is implemented here; the 10 and 12-bit paths still use the C reference and can be added in a follow-up. Patch 1 adds a checkasm test so the second patch can be verified. The test iterates 8, 10 and 12 bit, so it will also cover the higher depths once SIMD for them is added. Patch 2 adds the 8-bit kernel and wires up x86 dispatch. Tested on AMD Ryzen 7 5825U: - checkasm passes: the 8-bit AVX2 kernel matches the C reference - fate-vvc passes, unchanged from before the series - framemd5 output is identical with and without -cpuflags 0 Gudikandula Samith (2): tests/checkasm: add vvc_intra test for pred_planar lavc/vvc: add AVX2 planar intra prediction libavcodec/x86/vvc/Makefile | 1 + libavcodec/x86/vvc/dsp_init.c | 13 ++ libavcodec/x86/vvc/intra.asm | 224 ++++++++++++++++++++++++++++++++++ tests/checkasm/Makefile | 2 +- tests/checkasm/checkasm.c | 7 +- tests/checkasm/checkasm.h | 1 + tests/checkasm/vvc_intra.c | 87 +++++++++++++ 7 files changed, 331 insertions(+), 4 deletions(-) create mode 100644 libavcodec/x86/vvc/intra.asm create mode 100644 tests/checkasm/vvc_intra.c -- 2.53.0

3 6

[PR] avcodec/x86/hevc: add SSSE3 pred_planar (PR #24052)
by Marcos Ashton 08 Aug '26

08 Aug '26

PR #24052 opened by Marcos Ashton (MarcosAsh) URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24052 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24052.patch Adds an x86 SIMD implementation of HEVC planar intra prediction (previously C-only), for 8 bit, as part of #23022. x86 has no HEVC intra prediction at all today; aarch64 NEON already implements the planar, DC and angular groups at 8 bit. Everything in the planar expression except (size-1-x)*left[y] is linear in y, so for a fixed group of eight columns the remaining terms collapse into a single running accumulator whose increment left[size]-top[x] is loop invariant. Iterating columns outermost and rows innermost keeps that accumulator and its delta in registers, so a row costs one pmullw and two paddw with no reload of top[]. For 16x16 and 32x32 two column groups run together, which shares the left[y] broadcast and lets a row be written with a single 16 byte store. Accumulation is in words throughout since the largest value the expression can take is 31*255 + 32*255 + 31*255 + 32*255 + 32 = 32162. SSSE3 is the baseline rather than SSE2 because pshufb does both the byte broadcasts and the byte to word widening, which keeps the row loop short and avoids a dedicated zero register. 4x4 keeps a separate fully unrolled path that does two rows per register. checkasm -t hevc_pred passes bit-exact. Benchmarks (Core Ultra 7 155H, medians of five runs): ``` hevc_pred_planar_4x4_8_c: 18.4 hevc_pred_planar_4x4_8_ssse3: 5.5 hevc_pred_planar_8x8_8_c: 83.1 hevc_pred_planar_8x8_8_ssse3: 16.3 hevc_pred_planar_16x16_8_c: 224.8 hevc_pred_planar_16x16_8_ssse3: 47.3 hevc_pred_planar_32x32_8_c: 629.5 hevc_pred_planar_32x32_8_ssse3: 196.5 ``` From 47d10c66b9c9b35fd5f2dd6b1289a0183488a0f2 Mon Sep 17 00:00:00 2001 From: Marcos Ashton Iglesias <marcosashiglesias(a)gmail.com> Date: Fri, 7 Aug 2026 20:42:41 +0100 Subject: [PATCH] avcodec/x86/hevc: add SSSE3 pred_planar Implements planar intra prediction for 8 bit. x86 has no HEVC intra prediction at all; aarch64 NEON already covers planar, DC and angular. Every term except (size-1-x)*left[y] is linear in y, so for a group of eight columns they collapse into one running accumulator whose increment left[size]-top[x] is loop invariant. Running columns outermost keeps that accumulator and its delta in registers, so a row costs one pmullw and two paddw with no reload of top[]. 16x16 and 32x32 run two groups together, sharing the left[y] broadcast and writing a row with one 16 byte store. Words suffice throughout: the largest value is 32162. SSSE3 rather than SSE2 since pshufb does the broadcasts and the byte to word widening. checkasm --bench on a Core Ultra 7 155H: hevc_pred_planar_4x4_8_c: 18.4 hevc_pred_planar_4x4_8_ssse3: 5.5 ( 3.35x) hevc_pred_planar_8x8_8_c: 83.1 hevc_pred_planar_8x8_8_ssse3: 16.3 ( 5.10x) hevc_pred_planar_16x16_8_c: 224.8 hevc_pred_planar_16x16_8_ssse3: 47.3 ( 4.75x) hevc_pred_planar_32x32_8_c: 629.5 hevc_pred_planar_32x32_8_ssse3: 196.5 ( 3.20x) Part of #23022. --- libavcodec/hevc/pred.c | 3 + libavcodec/hevc/pred.h | 1 + libavcodec/x86/hevc/Makefile | 2 + libavcodec/x86/hevc/pred.asm | 205 ++++++++++++++++++++++++++++++++ libavcodec/x86/hevc/pred_init.c | 51 ++++++++ 5 files changed, 262 insertions(+) create mode 100644 libavcodec/x86/hevc/pred.asm create mode 100644 libavcodec/x86/hevc/pred_init.c diff --git a/libavcodec/hevc/pred.c b/libavcodec/hevc/pred.c index 037cbc413f..673c02d856 100644 --- a/libavcodec/hevc/pred.c +++ b/libavcodec/hevc/pred.c @@ -85,4 +85,7 @@ void ff_hevc_pred_init(HEVCPredContext *hpc, int bit_depth) #if ARCH_MIPS ff_hevc_pred_init_mips(hpc, bit_depth); #endif +#if ARCH_X86 && HAVE_X86ASM + ff_hevc_pred_init_x86(hpc, bit_depth); +#endif } diff --git a/libavcodec/hevc/pred.h b/libavcodec/hevc/pred.h index 849806fefb..9c0cf17349 100644 --- a/libavcodec/hevc/pred.h +++ b/libavcodec/hevc/pred.h @@ -51,6 +51,7 @@ typedef struct HEVCPredContext { void ff_hevc_pred_init(HEVCPredContext *hpc, int bit_depth); void ff_hevc_pred_init_mips(HEVCPredContext *hpc, int bit_depth); void ff_hevc_pred_init_aarch64(HEVCPredContext *hpc, int bit_depth); +void ff_hevc_pred_init_x86(HEVCPredContext *hpc, int bit_depth); /* C angular prediction fallbacks (non-static for arch-specific partial override) */ #define HEVC_PRED_ANGULAR_DECL(depth) \ diff --git a/libavcodec/x86/hevc/Makefile b/libavcodec/x86/hevc/Makefile index d09c613a19..0ac5287bdc 100644 --- a/libavcodec/x86/hevc/Makefile +++ b/libavcodec/x86/hevc/Makefile @@ -7,6 +7,8 @@ X86ASM-OBJS-$(CONFIG_HEVC_DECODER) += x86/hevc/dsp_init.o \ x86/hevc/dequant.o \ x86/hevc/idct.o \ x86/hevc/mc.o \ + x86/hevc/pred.o \ + x86/hevc/pred_init.o \ x86/hevc/sao.o \ x86/hevc/sao_10bit.o \ x86/h26x/h2656dsp.o \ diff --git a/libavcodec/x86/hevc/pred.asm b/libavcodec/x86/hevc/pred.asm new file mode 100644 index 0000000000..724a385a0c --- /dev/null +++ b/libavcodec/x86/hevc/pred.asm @@ -0,0 +1,205 @@ +;****************************************************************************** +;* SIMD-optimized HEVC intra prediction +;* +;* This file is part of FFmpeg. +;* +;* FFmpeg is free software; you can redistribute it and/or +;* modify it under the terms of the GNU Lesser General Public +;* License as published by the Free Software Foundation; either +;* version 2.1 of the License, or (at your option) any later version. +;* +;* FFmpeg is distributed in the hope that it will be useful, +;* but WITHOUT ANY WARRANTY; without even the implied warranty of +;* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +;* Lesser General Public License for more details. +;* +;* You should have received a copy of the GNU Lesser General Public +;* License along with FFmpeg; if not, write to the Free Software +;* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +;****************************************************************************** + +%include "libavutil/x86/x86util.asm" + +SECTION_RODATA 16 + +; planar weights: (size - 1 - x) is a suffix of pw_desc, (x + 1) a prefix of pw_asc +pw_desc: dw 31, 30, 29, 28, 27, 26, 25, 24, 23, 22, 21, 20, 19, 18, 17, 16 + dw 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0 +pw_asc: dw 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16 + dw 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32 + +; pshufb masks: zero-extend bytes to words without a dedicated zero register +pb_bcastw0: times 8 db 0, -1 ; byte 0 -> all eight words +pb_widen: db 0, -1, 1, -1, 2, -1, 3, -1, 4, -1, 5, -1, 6, -1, 7, -1 + +; 4x4-specific: two rows per register +pb_top4dup: db 0, -1, 1, -1, 2, -1, 3, -1, 0, -1, 1, -1, 2, -1, 3, -1 +pb_left01: db 0, -1, 0, -1, 0, -1, 0, -1, 1, -1, 1, -1, 1, -1, 1, -1 +pb_left23: db 2, -1, 2, -1, 2, -1, 2, -1, 3, -1, 3, -1, 3, -1, 3, -1 +pw_planar4_a: dw 3, 2, 1, 0, 3, 2, 1, 0 ; size - 1 - x +pw_planar4_b: dw 1, 2, 3, 4, 1, 2, 3, 4 ; x + 1 +pw_planar4_vy0: dw 3, 3, 3, 3, 2, 2, 2, 2 ; size - 1 - y, rows 0-1 +pw_planar4_vy1: dw 1, 1, 1, 1, 0, 0, 0, 0 ; rows 2-3 +pw_planar4_yb0: dw 1, 1, 1, 1, 2, 2, 2, 2 ; y + 1, rows 0-1 +pw_planar4_yb1: dw 3, 3, 3, 3, 4, 4, 4, 4 ; rows 2-3 + +cextern pw_4 +cextern pw_8 +cextern pw_16 +cextern pw_32 + +SECTION .text + +; void ff_hevc_pred_planar_<idx>_8_<opt>(uint8_t *src, const uint8_t *top, +; const uint8_t *left, ptrdiff_t stride) +; +; src[y*stride + x] = ((size-1-x)*left[y] + (x+1)*top[size] + +; (size-1-y)*top[x] + (y+1)*left[size] + size) >> (log2 + 1) +; +; Everything except (size-1-x)*left[y] depends on y only through two terms that +; are both linear in y, so for a fixed group of eight columns they collapse into +; a single running accumulator +; +; sum(y) = (x+1)*top[size] + (y+1)*left[size] + size + (size-1-y)*top[x] +; sum(y+1) - sum(y) = left[size] - top[x] +; +; which is loop-invariant. Iterating columns outermost and rows innermost keeps +; sum and that delta in registers, so a row costs one pmullw and two paddw and +; no reload of top[]. All intermediates stay in words: the largest value the +; expression can take is 31*255 + 32*255 + 31*255 + 32*255 + 32 = 32162 < 2^15. + +; Set up the running accumulator for the eight columns starting at %6. +; (size-1)*top[x] is formed as (top[x] << log2) - top[x] so that no vector of +; size-1 has to be materialised. +; %1 = sum, %2 = delta, %3 = temp, %4 = size, %5 = index, %6 = first column +%macro PLANAR_INIT 6 + movd %1, [topq + %4] + pshufb %1, [pb_bcastw0] ; top[size] + pmullw %1, [pw_asc + (%6) * 2] ; (x+1)*top[size] + movd %2, [leftq + %4] + pshufb %2, [pb_bcastw0] ; left[size] + paddw %1, %2 + paddw %1, [pw_ %+ %4] ; + left[size] + size + movq %3, [topq + (%6)] + pshufb %3, [pb_widen] ; top[x] + psubw %2, %3 ; delta = left[size] - top[x] + psubw %1, %3 + psllw %3, (%5) + 2 + paddw %1, %3 ; + (size-1)*top[x] = sum(0) +%endmacro + +; One group of eight columns, for the block size that is only eight wide. +; %1 = block size, %2 = index, %3 = first column +%macro PLANAR_CHUNK 3 + PLANAR_INIT m0, m1, m2, %1, %2, %3 + lea srcq, [baseq + %3] + xor yd, yd +%%loop: + movd m2, [leftq + yq] + pshufb m2, [pb_bcastw0] ; left[y] + pmullw m2, [pw_desc + (32 - %1 + %3) * 2] ; (size-1-x)*left[y] + paddw m2, m0 + psrlw m2, %2 + 3 + packuswb m2, m2 + movq [srcq], m2 + add srcq, strideq + paddw m0, m1 + inc yd + cmp yd, %1 + jl %%loop +%endmacro + +; Two adjacent groups at once, so that a row is written with a single 16 byte +; store and the left[y] broadcast is shared between them. +; %1 = block size, %2 = index, %3 = first column +%macro PLANAR_CHUNK2 3 + PLANAR_INIT m0, m1, m4, %1, %2, %3 + PLANAR_INIT m2, m3, m4, %1, %2, %3 + 8 + lea srcq, [baseq + %3] + xor yd, yd +%%loop: + movd m4, [leftq + yq] + pshufb m4, [pb_bcastw0] ; left[y] + mova m5, m4 + pmullw m4, [pw_desc + (32 - %1 + %3) * 2] + paddw m4, m0 + psrlw m4, %2 + 3 + pmullw m5, [pw_desc + (32 - %1 + %3 + 8) * 2] + paddw m5, m2 + psrlw m5, %2 + 3 + packuswb m4, m5 + movu [srcq], m4 + add srcq, strideq + paddw m0, m1 + paddw m2, m3 + inc yd + cmp yd, %1 + jl %%loop +%endmacro + +; %1 = block size, %2 = pred_planar index (log2(size) - 2) +%macro PRED_PLANAR 2 +cglobal hevc_pred_planar_%2_8, 4, 6, 6, src, top, left, stride, base, y + mov baseq, srcq +%if %1 == 8 + PLANAR_CHUNK %1, %2, 0 +%else +%assign %%off 0 +%rep %1 / 16 + PLANAR_CHUNK2 %1, %2, %%off +%assign %%off %%off + 16 +%endrep +%endif + RET +%endmacro + +; 4x4: fully unrolled, two rows per register +%macro PRED_PLANAR4 0 +cglobal hevc_pred_planar_0_8, 4, 4, 5, src, top, left, stride + movd m0, [topq + 4] + pshufb m0, [pb_bcastw0] ; top[4] + pmullw m0, [pw_planar4_b] ; (x+1)*top[4] + mova m1, m0 + movd m2, [leftq + 4] + pshufb m2, [pb_bcastw0] ; left[4] + mova m3, m2 + pmullw m3, [pw_planar4_yb0] + paddw m0, m3 + paddw m0, [pw_4] ; row-invariant terms, rows 0-1 + pmullw m2, [pw_planar4_yb1] + paddw m1, m2 + paddw m1, [pw_4] ; rows 2-3 + movd m2, [topq] + pshufb m2, [pb_top4dup] ; top[0..3], both halves + movd m3, [leftq] + mova m4, m3 + pshufb m3, [pb_left01] ; left[0] x4 | left[1] x4 + pshufb m4, [pb_left23] ; left[2] x4 | left[3] x4 + pmullw m3, [pw_planar4_a] + paddw m3, m0 + mova m0, m2 + pmullw m0, [pw_planar4_vy0] + paddw m3, m0 + psrlw m3, 3 + pmullw m4, [pw_planar4_a] + paddw m4, m1 + pmullw m2, [pw_planar4_vy1] + paddw m4, m2 + psrlw m4, 3 + packuswb m3, m4 + movd [srcq], m3 + psrldq m3, 4 + movd [srcq + strideq], m3 + lea srcq, [srcq + strideq * 2] + psrldq m3, 4 + movd [srcq], m3 + psrldq m3, 4 + movd [srcq + strideq], m3 + RET +%endmacro + +INIT_XMM ssse3 +PRED_PLANAR4 +PRED_PLANAR 8, 1 +PRED_PLANAR 16, 2 +PRED_PLANAR 32, 3 diff --git a/libavcodec/x86/hevc/pred_init.c b/libavcodec/x86/hevc/pred_init.c new file mode 100644 index 0000000000..980e28b861 --- /dev/null +++ b/libavcodec/x86/hevc/pred_init.c @@ -0,0 +1,51 @@ +/* + * SIMD-optimized HEVC intra prediction + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "config.h" + +#include "libavutil/attributes.h" +#include "libavutil/cpu.h" +#include "libavutil/x86/cpu.h" +#include "libavcodec/hevc/pred.h" + +#define PRED_PLANAR_FUNC(idx, opt) \ +void ff_hevc_pred_planar_ ## idx ## _8_ ## opt(uint8_t *src, \ + const uint8_t *top, \ + const uint8_t *left, \ + ptrdiff_t stride); + +PRED_PLANAR_FUNC(0, ssse3) +PRED_PLANAR_FUNC(1, ssse3) +PRED_PLANAR_FUNC(2, ssse3) +PRED_PLANAR_FUNC(3, ssse3) + +av_cold void ff_hevc_pred_init_x86(HEVCPredContext *hpc, int bit_depth) +{ + int cpu_flags = av_get_cpu_flags(); + + if (bit_depth == 8) { + if (EXTERNAL_SSSE3(cpu_flags)) { + hpc->pred_planar[0] = ff_hevc_pred_planar_0_8_ssse3; + hpc->pred_planar[1] = ff_hevc_pred_planar_1_8_ssse3; + hpc->pred_planar[2] = ff_hevc_pred_planar_2_8_ssse3; + hpc->pred_planar[3] = ff_hevc_pred_planar_3_8_ssse3; + } + } +} -- 2.52.0

1 0

[PR] avcodec/h263dsp,mpeg4videodsp: Use ptrdiff_t for stride (PR #24050)
by mkver 08 Aug '26

08 Aug '26

PR #24050 opened by mkver URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24050 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24050.patch Also avoid a shuffle in the x86 h263 loopfilter. From 4e838cd35d071724e75ec3903a8b05e939472732 Mon Sep 17 00:00:00 2001 From: Andreas Rheinhardt <andreas.rheinhardt(a)outlook.com> Date: Fri, 7 Aug 2026 19:19:19 +0200 Subject: [PATCH 1/3] avcodec/h263dsp: Use ptrdiff_t for stride Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt(a)outlook.com> --- libavcodec/h263.c | 4 ++-- libavcodec/h263dsp.c | 4 ++-- libavcodec/h263dsp.h | 5 +++-- libavcodec/mips/h263dsp_mips.h | 4 ++-- libavcodec/mips/h263dsp_msa.c | 8 ++++---- libavcodec/riscv/h263dsp_init.c | 4 ++-- libavcodec/x86/h263_loopfilter.asm | 6 ++---- libavcodec/x86/h263dsp_init.c | 5 +++-- tests/checkasm/h263dsp.c | 4 ++-- 9 files changed, 22 insertions(+), 22 deletions(-) diff --git a/libavcodec/h263.c b/libavcodec/h263.c index 6d2de715ab..61a8fa96b4 100644 --- a/libavcodec/h263.c +++ b/libavcodec/h263.c @@ -96,8 +96,8 @@ void ff_h263_update_motion_val(MpegEncContext * s){ void ff_h263_loop_filter(MpegEncContext * s){ int qp_c; - const int linesize = s->linesize; - const int uvlinesize= s->uvlinesize; + const ptrdiff_t linesize = s->linesize; + const ptrdiff_t uvlinesize = s->uvlinesize; const int xy = s->mb_y * s->mb_stride + s->mb_x; uint8_t *dest_y = s->dest[0]; uint8_t *dest_cb= s->dest[1]; diff --git a/libavcodec/h263dsp.c b/libavcodec/h263dsp.c index 165174a499..bc54e69c22 100644 --- a/libavcodec/h263dsp.c +++ b/libavcodec/h263dsp.c @@ -28,7 +28,7 @@ const uint8_t ff_h263_loop_filter_strength[32] = { 7, 8, 8, 8, 9, 9, 9, 10, 10, 10, 11, 11, 11, 12, 12, 12 }; -static void h263_h_loop_filter_c(uint8_t *src, int stride, int qscale) +static void h263_h_loop_filter_c(uint8_t *src, ptrdiff_t stride, int qscale) { int y; const int strength = ff_h263_loop_filter_strength[qscale]; @@ -71,7 +71,7 @@ static void h263_h_loop_filter_c(uint8_t *src, int stride, int qscale) } } -static void h263_v_loop_filter_c(uint8_t *src, int stride, int qscale) +static void h263_v_loop_filter_c(uint8_t *src, ptrdiff_t stride, int qscale) { int x; const int strength = ff_h263_loop_filter_strength[qscale]; diff --git a/libavcodec/h263dsp.h b/libavcodec/h263dsp.h index 2dccd23392..d94358d07c 100644 --- a/libavcodec/h263dsp.h +++ b/libavcodec/h263dsp.h @@ -19,13 +19,14 @@ #ifndef AVCODEC_H263DSP_H #define AVCODEC_H263DSP_H +#include <stddef.h> #include <stdint.h> extern const uint8_t ff_h263_loop_filter_strength[32]; typedef struct H263DSPContext { - void (*h263_h_loop_filter)(uint8_t *src, int stride, int qscale); - void (*h263_v_loop_filter)(uint8_t *src, int stride, int qscale); + void (*h263_h_loop_filter)(uint8_t *src, ptrdiff_t stride, int qscale); + void (*h263_v_loop_filter)(uint8_t *src, ptrdiff_t stride, int qscale); } H263DSPContext; void ff_h263dsp_init(H263DSPContext *ctx); diff --git a/libavcodec/mips/h263dsp_mips.h b/libavcodec/mips/h263dsp_mips.h index 5ea9fcbb88..7ae896e11c 100644 --- a/libavcodec/mips/h263dsp_mips.h +++ b/libavcodec/mips/h263dsp_mips.h @@ -23,8 +23,8 @@ #include "libavcodec/mpegvideo.h" -void ff_h263_h_loop_filter_msa(uint8_t *src, int stride, int q_scale); -void ff_h263_v_loop_filter_msa(uint8_t *src, int stride, int q_scale); +void ff_h263_h_loop_filter_msa(uint8_t *src, ptrdiff_t stride, int q_scale); +void ff_h263_v_loop_filter_msa(uint8_t *src, ptrdiff_t stride, int q_scale); void ff_dct_unquantize_mpeg2_inter_msa(const MPVContext *s, int16_t *block, int32_t index, int32_t q_scale); void ff_dct_unquantize_h263_inter_msa(const MPVContext *s, int16_t *block, diff --git a/libavcodec/mips/h263dsp_msa.c b/libavcodec/mips/h263dsp_msa.c index 2e1ca0183d..b1f3a2a8a1 100644 --- a/libavcodec/mips/h263dsp_msa.c +++ b/libavcodec/mips/h263dsp_msa.c @@ -26,7 +26,7 @@ static const uint8_t h263_loop_filter_strength_msa[32] = { 7, 8, 8, 8, 9, 9, 9, 10, 10, 10, 11, 11, 11, 12, 12, 12 }; -static void h263_h_loop_filter_msa(uint8_t *src, int32_t stride, int32_t qscale) +static void h263_h_loop_filter_msa(uint8_t *src, ptrdiff_t stride, int32_t qscale) { int32_t strength = h263_loop_filter_strength_msa[qscale]; v16u8 in0, in1, in2, in3, in4, in5, in6, in7; @@ -89,7 +89,7 @@ static void h263_h_loop_filter_msa(uint8_t *src, int32_t stride, int32_t qscale) ST_W8(in0, in3, 0, 1, 2, 3, 0, 1, 2, 3, src, stride); } -static void h263_v_loop_filter_msa(uint8_t *src, int32_t stride, int32_t qscale) +static void h263_v_loop_filter_msa(uint8_t *src, ptrdiff_t stride, int32_t qscale) { int32_t strength = h263_loop_filter_strength_msa[qscale]; uint64_t res0, res1, res2, res3; @@ -150,12 +150,12 @@ static void h263_v_loop_filter_msa(uint8_t *src, int32_t stride, int32_t qscale) SD4(res0, res1, res2, res3, src, stride); } -void ff_h263_h_loop_filter_msa(uint8_t *src, int32_t stride, int32_t q_scale) +void ff_h263_h_loop_filter_msa(uint8_t *src, ptrdiff_t stride, int32_t q_scale) { h263_h_loop_filter_msa(src, stride, q_scale); } -void ff_h263_v_loop_filter_msa(uint8_t *src, int32_t stride, int32_t q_scale) +void ff_h263_v_loop_filter_msa(uint8_t *src, ptrdiff_t stride, int32_t q_scale) { h263_v_loop_filter_msa(src, stride, q_scale); } diff --git a/libavcodec/riscv/h263dsp_init.c b/libavcodec/riscv/h263dsp_init.c index 21b536366c..fa4877364f 100644 --- a/libavcodec/riscv/h263dsp_init.c +++ b/libavcodec/riscv/h263dsp_init.c @@ -25,8 +25,8 @@ #include "libavutil/riscv/cpu.h" #include "libavcodec/h263dsp.h" -void ff_h263_h_loop_filter_rvv(uint8_t *src, int stride, int q); -void ff_h263_v_loop_filter_rvv(uint8_t *src, int stride, int q); +void ff_h263_h_loop_filter_rvv(uint8_t *src, ptrdiff_t stride, int q); +void ff_h263_v_loop_filter_rvv(uint8_t *src, ptrdiff_t stride, int q); av_cold void ff_h263dsp_init_riscv(H263DSPContext *c) { diff --git a/libavcodec/x86/h263_loopfilter.asm b/libavcodec/x86/h263_loopfilter.asm index 083537154b..868f843275 100644 --- a/libavcodec/x86/h263_loopfilter.asm +++ b/libavcodec/x86/h263_loopfilter.asm @@ -86,9 +86,8 @@ SECTION .text %endmacro INIT_XMM sse2 -; void ff_h263_v_loop_filter_sse2(uint8_t *src, int stride, int qscale) +; void ff_h263_v_loop_filter_sse2(uint8_t *src, ptrdiff_t stride, int qscale) cglobal h263_v_loop_filter, 3,5,8 - movsxdifnidn r1, r1d movsxdifnidn r2, r2d lea r3, [h263_loop_filter_strength] @@ -118,10 +117,9 @@ cglobal h263_v_loop_filter, 3,5,8 %endmacro -; void ff_h263_h_loop_filter_sse2(uint8_t *src, int stride, int qscale) +; void ff_h263_h_loop_filter_sse2(uint8_t *src, ptrdiff_t stride, int qscale) INIT_XMM sse2 cglobal h263_h_loop_filter, 3,5,8,32 - movsxdifnidn r1, r1d movsxdifnidn r2, r2d lea r4, [h263_loop_filter_strength] diff --git a/libavcodec/x86/h263dsp_init.c b/libavcodec/x86/h263dsp_init.c index 3dd5d132e5..e0ec75fa7b 100644 --- a/libavcodec/x86/h263dsp_init.c +++ b/libavcodec/x86/h263dsp_init.c @@ -18,6 +18,7 @@ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA */ +#include <stddef.h> #include <stdint.h> #include "libavutil/attributes.h" @@ -25,8 +26,8 @@ #include "libavutil/x86/cpu.h" #include "libavcodec/h263dsp.h" -void ff_h263_h_loop_filter_sse2(uint8_t *src, int stride, int qscale); -void ff_h263_v_loop_filter_sse2(uint8_t *src, int stride, int qscale); +void ff_h263_h_loop_filter_sse2(uint8_t *src, ptrdiff_t stride, int qscale); +void ff_h263_v_loop_filter_sse2(uint8_t *src, ptrdiff_t stride, int qscale); av_cold void ff_h263dsp_init_x86(H263DSPContext *c) { diff --git a/tests/checkasm/h263dsp.c b/tests/checkasm/h263dsp.c index f99d376adc..5fafa24a9a 100644 --- a/tests/checkasm/h263dsp.c +++ b/tests/checkasm/h263dsp.c @@ -26,7 +26,7 @@ #include "libavutil/mem.h" #include "libavutil/mem_internal.h" -typedef void (*filter)(uint8_t *src, int stride, int qscale); +typedef void (*filter)(uint8_t *src, ptrdiff_t stride, int qscale); static void check_loop_filter(char dim, filter func) { @@ -34,7 +34,7 @@ static void check_loop_filter(char dim, filter func) LOCAL_ALIGNED_16(uint8_t, buf1, [32 * 32]); int qscale = rnd() % 32; - declare_func(void, uint8_t *, int, int); + declare_func(void, uint8_t *, ptrdiff_t, int); for (size_t y = 0; y < 32; y++) for (size_t x = 0; x < 32; x++) -- 2.52.0 From 77c491b814852c0dc1fe448cdcf90261e1ad6fb3 Mon Sep 17 00:00:00 2001 From: Andreas Rheinhardt <andreas.rheinhardt(a)outlook.com> Date: Sat, 8 Aug 2026 01:36:44 +0200 Subject: [PATCH 2/3] avcodec/x86/h263dsp_loopfilter: Avoid shuffle Also start splatting a bit earlier. Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt(a)outlook.com> --- libavcodec/x86/h263_loopfilter.asm | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/libavcodec/x86/h263_loopfilter.asm b/libavcodec/x86/h263_loopfilter.asm index 868f843275..3515246de0 100644 --- a/libavcodec/x86/h263_loopfilter.asm +++ b/libavcodec/x86/h263_loopfilter.asm @@ -45,18 +45,17 @@ SECTION .text punpcklbw m1, m7 psubw m1, m2 psllw m1, 2 + movd m2, %5 paddw m1, m0 pxor m6, m6 pcmpgtw m6, m1 + punpcklbw m2, m2 pxor m1, m6 psubw m1, m6 psrlw m1, 3 packuswb m1, m7 + pshuflw m2, m2, 0 packsswb m6, m7 - movd m2, %5 - punpcklbw m2, m2 - punpcklbw m2, m2 - punpcklbw m2, m2 psubusb m2, m1 mova m7, m2 psubusb m7, m1 -- 2.52.0 From 43e2698a278991898da7efac45f4d26ed1c54395 Mon Sep 17 00:00:00 2001 From: Andreas Rheinhardt <andreas.rheinhardt(a)outlook.com> Date: Sat, 8 Aug 2026 02:25:34 +0200 Subject: [PATCH 3/3] avcodec/mpeg4videodsp: Use ptrdiff_t for stride Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt(a)outlook.com> --- libavcodec/mpeg4videodsp.c | 4 ++-- libavcodec/mpeg4videodsp.h | 7 ++++--- libavcodec/ppc/mpeg4videodsp.c | 2 +- libavcodec/x86/mpeg4videodsp.c | 2 +- tests/checkasm/mpeg4videodsp.c | 2 +- 5 files changed, 9 insertions(+), 8 deletions(-) diff --git a/libavcodec/mpeg4videodsp.c b/libavcodec/mpeg4videodsp.c index 1c5661a076..7067d8e885 100644 --- a/libavcodec/mpeg4videodsp.c +++ b/libavcodec/mpeg4videodsp.c @@ -21,7 +21,7 @@ #include "libavutil/common.h" #include "mpeg4videodsp.h" -static void gmc1_c(uint8_t *dst, const uint8_t *src, int stride, int h, +static void gmc1_c(uint8_t *dst, const uint8_t *src, ptrdiff_t stride, int h, int x16, int y16, int rounder) { const int A = (16 - x16) * (16 - y16); @@ -44,7 +44,7 @@ static void gmc1_c(uint8_t *dst, const uint8_t *src, int stride, int h, } } -void ff_gmc_c(uint8_t *dst, const uint8_t *src, int stride, int h, int ox, int oy, +void ff_gmc_c(uint8_t *dst, const uint8_t *src, ptrdiff_t stride, int h, int ox, int oy, int dxx, int dxy, int dyx, int dyy, int shift, int r, int width, int height) { diff --git a/libavcodec/mpeg4videodsp.h b/libavcodec/mpeg4videodsp.h index e1ccb71ce9..05f1ccc8a9 100644 --- a/libavcodec/mpeg4videodsp.h +++ b/libavcodec/mpeg4videodsp.h @@ -19,9 +19,10 @@ #ifndef AVCODEC_MPEG4VIDEODSP_H #define AVCODEC_MPEG4VIDEODSP_H +#include <stddef.h> #include <stdint.h> -void ff_gmc_c(uint8_t *dst, const uint8_t *src, int stride, int h, int ox, int oy, +void ff_gmc_c(uint8_t *dst, const uint8_t *src, ptrdiff_t stride, int h, int ox, int oy, int dxx, int dxy, int dyx, int dyy, int shift, int r, int width, int height); @@ -30,12 +31,12 @@ typedef struct Mpeg4VideoDSPContext { * translational global motion compensation. */ void (*gmc1)(uint8_t *dst /* align 8 */, const uint8_t *src /* align 1 */, - int srcStride, int h, int x16, int y16, int rounder); + ptrdiff_t srcStride, int h, int x16, int y16, int rounder); /** * global motion compensation. */ void (*gmc)(uint8_t *dst /* align 8 */, const uint8_t *src /* align 1 */, - int stride, int h, int ox, int oy, + ptrdiff_t stride, int h, int ox, int oy, int dxx, int dxy, int dyx, int dyy, int shift, int r, int width, int height); } Mpeg4VideoDSPContext; diff --git a/libavcodec/ppc/mpeg4videodsp.c b/libavcodec/ppc/mpeg4videodsp.c index 8b30af4258..79e19ab6eb 100644 --- a/libavcodec/ppc/mpeg4videodsp.c +++ b/libavcodec/ppc/mpeg4videodsp.c @@ -32,7 +32,7 @@ /* AltiVec-enhanced gmc1. ATM this code assumes stride is a multiple of 8 * to preserve proper dst alignment. */ static void gmc1_altivec(uint8_t *dst /* align 8 */, const uint8_t *src /* align1 */, - int stride, int h, int x16, int y16, int rounder) + ptrdiff_t stride, int h, int x16, int y16, int rounder) { int i; const DECLARE_ALIGNED(16, unsigned short, rounder_a) = rounder; diff --git a/libavcodec/x86/mpeg4videodsp.c b/libavcodec/x86/mpeg4videodsp.c index d326327f13..337a4b32d7 100644 --- a/libavcodec/x86/mpeg4videodsp.c +++ b/libavcodec/x86/mpeg4videodsp.c @@ -37,7 +37,7 @@ typedef struct { DECLARE_ASM_CONST(16, xmm_u16, pw_0to7) = { { 0, 1, 2, 3, 4, 5, 6, 7 } }; static void gmc_ssse3(uint8_t *dst, const uint8_t *src, - int stride, int h, int ox, int oy, + ptrdiff_t stride, int h, int ox, int oy, int dxx, int dxy, int dyx, int dyy, int shift, int r, int width, int height) { diff --git a/tests/checkasm/mpeg4videodsp.c b/tests/checkasm/mpeg4videodsp.c index 49557224f5..a24ef480d8 100644 --- a/tests/checkasm/mpeg4videodsp.c +++ b/tests/checkasm/mpeg4videodsp.c @@ -81,7 +81,7 @@ static void checkasm_check_gmc(const Mpeg4VideoDSPContext *const mdsp) DECLARE_ALIGNED_4(uint8_t, srcbuf)[MAX_STRIDE * MAX_HEIGHT]; declare_func(void, uint8_t *dst, const uint8_t *src, - int stride, int h, int ox, int oy, + ptrdiff_t stride, int h, int ox, int oy, int dxx, int dxy, int dyx, int dyy, int shift, int r, int width, int height); -- 2.52.0

1 0

[PR] avfilter/dnn_backend_onnx: implement DFT_ANALYTICS_CLASSIFY for dnn_classify (PR #24049)
by Jake 08 Aug '26

08 Aug '26

PR #24049 opened by Jake (jakefineman) URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24049 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24049.patch The ONNX backend (`dnn_backend_onnx.c`) switches on `model.func_type` in both `fill_model_input_onnx()` and `infer_completion_callback()`, but only handles `DFT_PROCESS_FRAME` (plus `DFT_ANALYTICS_DETECT` on the input side). `DFT_ANALYTICS_CLASSIFY` falls through to the default `avpriv_report_missing_feature()` branch unconditionally, so `dnn_classify` cannot use the ONNX backend at all: ``` model function type 3 is not implemented ``` ## 1/2 — implement `DFT_ANALYTICS_CLASSIFY` Adds the classify case to both switches, mirroring the pattern the OpenVINO backend already uses (`dnn_backend_openvino.c`): the input side calls `ff_frame_to_dnn_classify()` with the bbox index carried on the task item; the output side invokes the filter's `classify_post_proc` callback (set by `vf_dnn_classify.c`) with the same bbox index. The completion callback ignores `classify_post_proc`'s return value. That is deliberate — it matches the OpenVINO backend's own precedent exactly, which calls `classify_post_proc(...)` as a bare statement at `dnn_backend_openvino.c:473` (and `detect_post_proc` likewise at :463). Changing that convention is out of scope here; if maintainers want the return propagated it should be done for both backends in one change. ## 2/2 — register the `onnx` named constant for `dnn_classify` `dnn_classify` registered only `tensorflow`/`openvino` named `AVOption` constants for `dnn_backend`, so after 1/2 the ONNX backend was reachable only by numeric value (`dnn_backend=8`). This registers the `onnx` constant behind `CONFIG_LIBONNXRUNTIME` — same macro and same position in the per-backend `#if` chain as `vf_dnn_processing.c` — and adds `DNN_ONNX` to `dnn_classify`'s `AVFILTER_DNN_DEFINE_CLASS` backend mask so `-h filter=dnn_classify` lists the ONNX sub-options. **`dnn_detect` is deliberately not included.** An earlier revision of this series also registered the constant for `dnn_detect`, added `DNN_ONNX` to its backend mask, and allowlisted `DNN_ONNX` in its `check_output_nb()`. That was wrong and has been dropped: `infer_completion_callback()` has no `DFT_ANALYTICS_DETECT` case, so the ONNX backend cannot produce bounding boxes. Admitting `dnn_detect` past its init-time rejection would only move the failure later, from a clear "does not support current backend" at filter init to "model function type 2 is not implemented" at inference time. Detect support belongs in a separate change that implements the output-side dispatch first. This series leaves `vf_dnn_detect.c` byte-identical to master. ## Repro Build: FFmpeg `n9.0` (pinned commit `d32b387`), debian trixie, ONNX Runtime 1.28.0, `--enable-libonnxruntime`, native linux/arm64. Before 1/2 — bare code-path check, no model needed: ``` $ ffmpeg -f lavfi -i testsrc2=size=64x64:rate=1:duration=1 \ -vf 'dnn_classify=dnn_backend=8:model=/tmp/x.onnx' -frames:v 1 -f null - [dnn_backend_onnx] model function type 3 is not implemented ``` After both patches — the named constant parses and the filter reaches ONNX session creation, erroring only on the missing model file: ``` $ ffmpeg -f lavfi -i testsrc2=size=64x64:rate=1:duration=1 \ -vf 'dnn_classify=dnn_backend=onnx:model=/tmp/nonexistent.onnx' -frames:v 1 -f null - [dnn_base] Using CPU execution provider [dnn_base] Failed to create ONNX session: Load model from /tmp/nonexistent.onnx failed. File doesn't exist [Parsed_dnn_classify_0] could not load DNN model ``` `-h filter=dnn_classify` now lists it, and `dnn_detect` is unchanged: ``` $ ffmpeg -h filter=dnn_classify | grep onnx onnx 8 ..FV....... onnx backend flag $ ffmpeg -h filter=dnn_detect | grep onnx # no output — unchanged by this series ``` `dnn_detect` with the ONNX backend selected numerically still fails closed at init, as it does on master: ``` [dnn_detect] Dnn detect filter does not support current backend ``` With a real model (MobileNetV2, `onnx/models` zoo, Apache-2.0, sha256 `c1c513582d56afceff8516c73804e484c81c6a830712ab6d682253f4a3cd042f`) an ONNX Runtime session loads and the classify path is entered. To be precise about what that does and does not prove: session creation is reached, but a standalone `dnn_classify` run then stops at the pre-existing `av_assert0(sd)` in `ff_frame_to_dnn_classify()` (`dnn_io_proc.c:355`), because `dnn_classify` expects `AV_FRAME_DATA_DETECTION_BBOXES` side data — it is architecturally designed to run chained after `dnn_detect`, and a bare `testsrc2` frame has none. So this is a code-path-entry receipt, not an end-to-end inference receipt. That assertion is pre-existing, backend-agnostic code that this series does not touch: `ff_frame_to_dnn_classify()` is called identically from `dnn_backend_openvino.c:315`, so a standalone `dnn_classify` behaves the same way on the OpenVINO backend today. This series makes ONNX match the established behaviour rather than introducing a new failure mode. Whether that bare-frame case deserves a clean error instead of an assert is a real question, but it applies to every backend and belongs in its own patch — happy to send one if maintainers agree. ## Checks Both patches apply clean with `git am` on `n9.0` and on current `master`, build with `--enable-libonnxruntime`, and pass `tools/patcheck` with only the expected "missing changelog entry" advisory. No existing trac ticket or patchwork series was found for this gap (searched `dnn_classify`, `dnn_backend_onnx classify`, and the in-flight 20-part `dnn_classify`-as-multimedia-filter series on patchwork — that series targets the Torch backend's zero-shot-classification path and does not overlap this fix). From 33fb19c89f88fb158b876337a1ed6090d5147f35 Mon Sep 17 00:00:00 2001 From: Jake Fineman <jake(a)wave.online> Date: Fri, 7 Aug 2026 17:55:34 -0400 Subject: [PATCH 1/2] avfilter/dnn_backend_onnx: implement DFT_ANALYTICS_CLASSIFY The ONNX backend switch()es on model.func_type in both the input-prep path (fill_model_input_onnx) and the output-extraction path (infer_completion_callback), but only DFT_PROCESS_FRAME and (on the input side) DFT_ANALYTICS_DETECT were handled; DFT_ANALYTICS_CLASSIFY fell through to the default avpriv_report_missing_feature() branch unconditionally, regardless of model shape ("model function type 3 is not implemented"). Add the classify case to both switches, mirroring the pattern already used by the OpenVINO backend (dnn_backend_openvino.c): the input path calls ff_frame_to_dnn_classify() with the per-bounding-box index carried on the LastLevelTaskItem, and the output path invokes the filters classify_post_proc callback (set by vf_dnn_classify.c) with that same bbox index. Signed-off-by: Jake Fineman <jake(a)wave.online> --- libavfilter/dnn/dnn_backend_onnx.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/libavfilter/dnn/dnn_backend_onnx.c b/libavfilter/dnn/dnn_backend_onnx.c index 6c75d6eb24..1bdd4ea1da 100644 --- a/libavfilter/dnn/dnn_backend_onnx.c +++ b/libavfilter/dnn/dnn_backend_onnx.c @@ -383,6 +383,9 @@ static int fill_model_input_onnx(ONNXModel *onnx_model, ONNXRequestItem *request case DFT_ANALYTICS_DETECT: ff_frame_to_dnn_detect(task->in_frame, &input, ctx); break; + case DFT_ANALYTICS_CLASSIFY: + ff_frame_to_dnn_classify(task->in_frame, &input, lltask->bbox_index, ctx); + break; default: avpriv_report_missing_feature(ctx, "model function type %d", onnx_model->model.func_type); ret = AVERROR(ENOSYS); @@ -599,6 +602,17 @@ static void infer_completion_callback(void *args) task->out_frame->height = outputs.dims[dnn_get_height_idx_by_layout(outputs.layout)]; } break; + case DFT_ANALYTICS_CLASSIFY: + if (!onnx_model->model.classify_post_proc) { + av_log(ctx, AV_LOG_ERROR, "classify filter needs to provide post proc\n"); + av_free(dims); + g_ort->ReleaseTensorTypeAndShapeInfo(tensor_info); + goto err; + } + onnx_model->model.classify_post_proc(task->in_frame, &outputs, + lltask->bbox_index, + onnx_model->model.filter_ctx); + break; default: avpriv_report_missing_feature(ctx, "model function type %d", onnx_model->model.func_type); av_free(dims); -- 2.52.0 From 3c615716f60a33aa2a1cf4d750deb42f1164224a Mon Sep 17 00:00:00 2001 From: Jake Fineman <jake(a)wave.online> Date: Fri, 7 Aug 2026 17:55:34 -0400 Subject: [PATCH 2/2] avfilter/vf_dnn_classify: register named onnx backend constant dnn_classify only registered "tensorflow"/"openvino" named AVOption constants for dnn_backend, even though the ONNX execution provider is reachable by numeric value (dnn_backend=8, DNN_ONNX). This left "onnx" undefined as a symbolic option value ("Undefined constant or missing (in onnx") and excluded DNN_ONNX from the backend mask that gates the filter's child-class option iteration, so -h filter=dnn_classify never listed the ONNX-backend sub-options. Register the "onnx" named constant behind CONFIG_LIBONNXRUNTIME, matching the existing pattern in vf_dnn_processing.c, and add DNN_ONNX to dnn_classify's AVFILTER_DNN_DEFINE_CLASS backend mask. dnn_detect is deliberately left alone. The ONNX backend's infer_completion_callback() has no DFT_ANALYTICS_DETECT case, so it cannot produce bounding boxes: advertising the backend there would admit dnn_detect past its init-time check_output_nb() rejection only to fail later at inference with "model function type 2 is not implemented", replacing a clear early error with a late one. Detect support belongs in a separate change that implements the output-side dispatch first. Signed-off-by: Jake Fineman <jake(a)wave.online> --- libavfilter/vf_dnn_classify.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/libavfilter/vf_dnn_classify.c b/libavfilter/vf_dnn_classify.c index f92c41ab76..8faf6685ee 100644 --- a/libavfilter/vf_dnn_classify.c +++ b/libavfilter/vf_dnn_classify.c @@ -48,6 +48,9 @@ static const AVOption dnn_classify_options[] = { { "dnn_backend", "DNN backend", OFFSET(backend_type), AV_OPT_TYPE_INT, { .i64 = DNN_OV }, INT_MIN, INT_MAX, FLAGS, .unit = "backend" }, #if (CONFIG_LIBOPENVINO == 1) { "openvino", "openvino backend flag", 0, AV_OPT_TYPE_CONST, { .i64 = DNN_OV }, 0, 0, FLAGS, .unit = "backend" }, +#endif +#if (CONFIG_LIBONNXRUNTIME == 1) + { "onnx", "onnx backend flag", 0, AV_OPT_TYPE_CONST, { .i64 = DNN_ONNX }, 0, 0, FLAGS, .unit = "backend" }, #endif { "confidence", "threshold of confidence", OFFSET2(confidence), AV_OPT_TYPE_FLOAT, { .dbl = 0.5 }, 0, 1, FLAGS}, { "labels", "path to labels file", OFFSET2(labels_filename), AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS }, @@ -55,7 +58,7 @@ static const AVOption dnn_classify_options[] = { { NULL } }; -AVFILTER_DNN_DEFINE_CLASS(dnn_classify, DNN_OV); +AVFILTER_DNN_DEFINE_CLASS(dnn_classify, DNN_OV | DNN_ONNX); static int dnn_classify_post_proc(AVFrame *frame, DNNData *output, uint32_t bbox_index, AVFilterContext *filter_ctx) { -- 2.52.0

1 0

[PR] avfilter/dnn: reject channel-mismatched model output in dnn_processing (PR #24048)
by Jake 08 Aug '26

08 Aug '26

PR #24048 opened by Jake (jakefineman) URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24048 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24048.patch The `AV_PIX_FMT_RGB24`/`AV_PIX_FMT_BGR24` output case in `ff_proc_from_dnn_to_frame()` (`libavfilter/dnn/dnn_io_proc.c`) hardcodes the `sws_getContext`/`sws_scale` width arguments and strides to `frame->width * 3`, unconditional on the model's real output channel count. For any model whose output has fewer than 3 channels — e.g. a single-channel alpha-matte output from an RGB-in background-removal model such as MODNet — that mismatch is memory-unsafe on both layouts: - The first `sws_scale` runs for **both** `DL_NCHW` and `DL_NHWC` output, before the planar-to-packed branch, and reads `frame->width * 3 * src_datatype_size * frame->height` bytes from `output->data`; a 1-channel tensor holds a third of that: a ~3x source-side over-read. - On `DL_NCHW`, `middle_data` is correctly sized for the real channel count (`plane_size * output->dims[1]`), so the same hardcoded width also makes `sws_scale` write `frame->width * 3 * frame->height` bytes into a buffer sized for only `frame->width * frame->height * output->dims[1]` bytes: a heap write 3x past the allocation for `dims[1] == 1`. glibc's allocator catches this on a subsequent `free()`/malloc consistency check, producing `SIGABRT`, not a clean FFmpeg error return. ## Fix Reject channel-mismatched model output at the top of the RGB24/BGR24 case, layout-agnostically: read the channel dim via `dnn_get_channel_idx_by_layout(output->layout)` (the same helper this file already uses on the input side, `dnn_io_proc.c:370`), and if `output->dims[channel_idx] != 3`, log a clear error and return `AVERROR(ENOSYS)` before any of the mismatched-stride arithmetic runs. One check covers all three hazards: the source-side over-read (both layouts), the `DL_NCHW` destination over-write, and the `DL_NCHW` planar-to-packed GBRP repack a few lines further down, which likewise unconditionally assumes 3 planes and is never reached once channel count is checked. ## Repro Build: FFmpeg `n9.0` (pinned commit `d32b387`), debian trixie, ONNX Runtime 1.28.0, `--enable-libonnxruntime`, native linux/arm64. Model: `onnx-community/modnet-webnn` `onnx/model.onnx`, sha256 `07c308cf0fc7e6e8b2065a12ed7fc07e1de8febb7dc7839d7b7f15dd66584df9` (output `[1,1,H,W]`, i.e. `DL_NCHW` with `dims[1] == 1`). **Before** — identical build with this patch omitted: ``` $ ffmpeg -f lavfi -i testsrc2=size=320x320:rate=1:duration=1 \ -vf 'format=rgb24,dnn_processing=dnn_backend=onnx:model=modnet.onnx:input=input:output=output' \ -frames:v 1 -f null - [dnn_base] Using CPU execution provider free(): invalid pointer $ echo $? 134 ``` **After** — same command, same model, patched build: ``` [dnn_base] dnn_processing to a rgb24 frame requires a 3-channel model output, got 1 channels; channel-reducing/expanding models (e.g. single-channel matte output) are not supported by this filter $ echo $? 0 ``` Scope note on the receipt: the runtime repro above exercises the `DL_NCHW` destination over-write, which is the path that aborts. The `DL_NHWC` source-side over-read is identified from source rather than separately reproduced — I did not have a channel-reducing NHWC model to hand — but it runs through the same hardcoded-stride `sws_scale` call above the layout branch, which is why the guard is keyed on the layout-resolved channel index rather than on `dims[1]`. ## Deliberate scope This patch rejects channel-reducing model output cleanly; it does **not** add support for actually rendering a 1-channel (or 2-channel) model output through this filter. Real support for that would need a new pixel-format/plane-count target on the output side (e.g. gray/ya8 for a matte, or a caller-supplied channel mapping) — a bigger design decision than a minimal heap-safety fix should make unilaterally, and out of scope here. I'm sending the minimal safety fix first because the crash is the more urgent problem: a SIGABRT driven by model-controlled output shape is a hardening bug independent of whether anyone wants channel-reducing rendering. Happy to follow up with a rendering-support patch, or take direction on it, if a maintainer prefers that be done in the same series instead of split. One further caveat, so the receipt does not overclaim: `vf_dnn_processing.c`'s own caller does not hard-abort the filter pipeline on this specific `AVERROR(ENOSYS)` return (pre-existing behaviour of that file, untouched here) — for a real (non-`-f null`) pipeline the practical effect is an unfiltered/passthrough frame rather than a full pipeline failure. ## Checks Applies clean with `git am` on both `n9.0` and current `master` (`master`'s `dnn_io_proc.c` still hardcodes `frame->width * 3` unconditionally as of this writing — not yet fixed upstream). Builds with `--enable-libonnxruntime`. `tools/patcheck` reports only the expected "missing changelog entry" advisory for a one-function bugfix. No existing trac ticket or patchwork series found for this bug in a bounded search (`dnn_processing sws_scale`, `dnn_io_proc heap`, `background removal matte crash dnn`). From b3f3be7585bd7ac299c044316843acfd047c8626 Mon Sep 17 00:00:00 2001 From: Jake Fineman <jake(a)wave.online> Date: Fri, 7 Aug 2026 17:57:03 -0400 Subject: [PATCH] avfilter/dnn: reject channel-mismatched model output in dnn_processing ff_proc_from_dnn_to_frame()'s AV_PIX_FMT_RGB24/BGR24 case assumes the model's output has exactly 3 channels, but never checks it. The first sws_scale() in that case reads from output->data with a hardcoded source stride of frame->width * 3 * src_datatype_size over frame->height rows, i.e. frame->width * 3 * frame->height * src_datatype_size bytes. For a model whose output has fewer channels than that -- e.g. the single-channel alpha matte an RGB-in background-removal model produces -- the real tensor only holds frame->width * frame->height * channels * src_datatype_size bytes, so the call reads roughly 3x past the end of it. This affects both DL_NCHW and DL_NHWC output, since it happens before the layout-specific planar-to-packed branch. For DL_NCHW there is a second, independent overflow on the write side: middle_data is allocated plane_size * output->dims[1] bytes, but linesize[0] is set to frame->width * 3 unconditionally, so the same sws_scale() also writes past that allocation. This reproduces as a SIGABRT (glibc "free(): invalid pointer", or a malloc.c assertion, depending on allocator state) inside dnn_processing, after inference has already completed successfully in the backend. The input side is guarded -- check_modelinput_inlink() in vf_dnn_processing.c -- but the output side is not. dnn_processing only supports channel-preserving models for these pixel formats by design, so reject a mismatched channel count with a clear error instead of reading and writing out of bounds. The channel dimension is resolved via dnn_get_channel_idx_by_layout() so the check covers both layouts. This does not add support for rendering a 1- or 2-channel model output through this filter; that would need a different pixel-format target and is left as future work. Signed-off-by: Jake Fineman <jake(a)wave.online> --- libavfilter/dnn/dnn_io_proc.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/libavfilter/dnn/dnn_io_proc.c b/libavfilter/dnn/dnn_io_proc.c index 826110dab0..55981c64da 100644 --- a/libavfilter/dnn/dnn_io_proc.c +++ b/libavfilter/dnn/dnn_io_proc.c @@ -50,6 +50,7 @@ int ff_proc_from_dnn_to_frame(AVFrame *frame, DNNData *output, void *log_ctx) int plane_size = frame->width * frame->height * sizeof(uint8_t); enum AVPixelFormat src_fmt = AV_PIX_FMT_NONE; int src_datatype_size = get_datatype_size(output->dt); + int channel_idx; int bytewidth = av_image_get_linesize(frame->format, frame->width, 0); if (bytewidth < 0) { @@ -83,6 +84,17 @@ int ff_proc_from_dnn_to_frame(AVFrame *frame, DNNData *output, void *log_ctx) switch (frame->format) { case AV_PIX_FMT_RGB24: case AV_PIX_FMT_BGR24: + channel_idx = dnn_get_channel_idx_by_layout(output->layout); + if (output->dims[channel_idx] != 3) { + av_log(log_ctx, AV_LOG_ERROR, + "dnn_processing to a %s frame requires a 3-channel model " + "output, got %d channels; channel-reducing/expanding " + "models (e.g. single-channel matte output) are not " + "supported by this filter\n", + av_get_pix_fmt_name(frame->format), output->dims[channel_idx]); + ret = AVERROR(ENOSYS); + goto err; + } sws_ctx = sws_getContext(frame->width * 3, frame->height, src_fmt, -- 2.52.0

1 0

[PR] WIP: avcodec/ac3enc: add short transform support (PR #24047)
by AYOUB NABIL BOUBAGRAT 07 Aug '26

07 Aug '26

PR #24047 opened by AYOUB NABIL BOUBAGRAT (ayoubnabil) URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24047 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24047.patch i'm starting work on block switching for the AC-3 encoder as part of #23317. this first commit adds short MDCT support for both the float and fixed-point encoders. i'll add the remaining block switching support in follow-up commits. this only targets AC-3 for now. From 79ba410f0e7b9363e60a3fb47eeb5a5d824cb6bd Mon Sep 17 00:00:00 2001 From: Ayoub Nabil Boubagrat <237098474+ayoubnabil(a)users.noreply.github.com> Date: Fri, 7 Aug 2026 23:00:00 +0200 Subject: [PATCH] avcodec/ac3enc: add short transform support Signed-off-by: Ayoub Nabil Boubagrat <237098474+ayoubnabil(a)users.noreply.github.com> --- libavcodec/ac3enc.c | 1 + libavcodec/ac3enc.h | 3 ++ libavcodec/ac3enc_fixed.c | 10 ++++-- libavcodec/ac3enc_float.c | 16 ++++++++-- libavcodec/ac3enc_template.c | 60 ++++++++++++++++++++++++++++++++++-- 5 files changed, 83 insertions(+), 7 deletions(-) diff --git a/libavcodec/ac3enc.c b/libavcodec/ac3enc.c index 0482d9ee58..9dcb173784 100644 --- a/libavcodec/ac3enc.c +++ b/libavcodec/ac3enc.c @@ -2184,6 +2184,7 @@ av_cold int ff_ac3_encode_close(AVCodecContext *avctx) ff_af_queue_close(&s->afq); av_tx_uninit(&s->tx); + av_tx_uninit(&s->tx_short); return 0; } diff --git a/libavcodec/ac3enc.h b/libavcodec/ac3enc.h index 3e92af17f0..20d199b800 100644 --- a/libavcodec/ac3enc.h +++ b/libavcodec/ac3enc.h @@ -137,6 +137,7 @@ typedef struct AC3Block { uint16_t *qmant[AC3_MAX_CHANNELS]; ///< quantized mantissas uint8_t *cpl_coord_exp[AC3_MAX_CHANNELS]; ///< coupling coord exponents (cplcoexp) uint8_t *cpl_coord_mant[AC3_MAX_CHANNELS]; ///< coupling coord mantissas (cplcomant) + uint8_t block_switch[AC3_MAX_CHANNELS]; ///< block switch flags (blksw) uint8_t new_rematrixing_strategy; ///< send new rematrixing flags in this block int num_rematrixing_bands; ///< number of rematrixing bands uint8_t rematrixing_flags[4]; ///< rematrixing flags @@ -170,6 +171,8 @@ typedef struct AC3EncodeContext { AC3DSPContext ac3dsp; ///< AC-3 optimized functions AVTXContext *tx; ///< FFT context for MDCT calculation av_tx_fn tx_fn; + AVTXContext *tx_short; ///< FFT context for short MDCT calculation + av_tx_fn tx_fn_short; AC3Block blocks[AC3_MAX_BLOCKS]; ///< per-block info diff --git a/libavcodec/ac3enc_fixed.c b/libavcodec/ac3enc_fixed.c index 876bb6bbc6..c2880f2c43 100644 --- a/libavcodec/ac3enc_fixed.c +++ b/libavcodec/ac3enc_fixed.c @@ -79,6 +79,7 @@ static av_cold int ac3_fixed_mdct_init(AVCodecContext *avctx, AC3EncodeContext * const float scale = -1.0f; int32_t *iwin = s->mdct_window_fixed; + int ret; ff_kbd_window_init(fwin, 5.0, AC3_BLOCK_SIZE); for (int i = 0; i < AC3_BLOCK_SIZE; i++) @@ -88,8 +89,13 @@ static av_cold int ac3_fixed_mdct_init(AVCodecContext *avctx, AC3EncodeContext * if (!s->fdsp) return AVERROR(ENOMEM); - return av_tx_init(&s->tx, &s->tx_fn, AV_TX_INT32_MDCT, 0, - AC3_BLOCK_SIZE, &scale, 0); + ret = av_tx_init(&s->tx, &s->tx_fn, AV_TX_INT32_MDCT, 0, + AC3_BLOCK_SIZE, &scale, 0); + if (ret < 0) + return ret; + + return av_tx_init(&s->tx_short, &s->tx_fn_short, AV_TX_INT32_MDCT, 0, + AC3_BLOCK_SIZE / 2, &scale, 0); } diff --git a/libavcodec/ac3enc_float.c b/libavcodec/ac3enc_float.c index 974c38ac25..990bfed878 100644 --- a/libavcodec/ac3enc_float.c +++ b/libavcodec/ac3enc_float.c @@ -85,12 +85,22 @@ static void sum_square_butterfly(AC3EncodeContext *s, float sum[4], */ static av_cold int ac3_float_mdct_init(AC3EncodeContext *s) { - const float scale = -2.0 / AC3_WINDOW_SIZE; + const float scale = -2.0 / AC3_WINDOW_SIZE; + /* A/52 uses -2/N with N=256 for each short transform. */ + const float short_scale = -4.0 / AC3_WINDOW_SIZE; + int ret; ff_kbd_window_init(s->mdct_window_float, 5.0, AC3_BLOCK_SIZE); - return av_tx_init(&s->tx, &s->tx_fn, AV_TX_FLOAT_MDCT, 0, - AC3_BLOCK_SIZE, &scale, 0); + ret = av_tx_init(&s->tx, &s->tx_fn, AV_TX_FLOAT_MDCT, 0, + AC3_BLOCK_SIZE, &scale, 0); + if (ret < 0) + return ret; + if (s->eac3) + return 0; + + return av_tx_init(&s->tx_short, &s->tx_fn_short, AV_TX_FLOAT_MDCT, 0, + AC3_BLOCK_SIZE / 2, &short_scale, 0); } diff --git a/libavcodec/ac3enc_template.c b/libavcodec/ac3enc_template.c index 5331b45cb9..d7a0fc1055 100644 --- a/libavcodec/ac3enc_template.c +++ b/libavcodec/ac3enc_template.c @@ -44,6 +44,59 @@ #define RENAME(element) element ## _fixed #endif +/* + * Apply the two short MDCTs and interleave their coefficients. + */ +static void apply_short_mdct(AC3EncodeContext *s, CoefType *coef, + const SampleType *samples) +{ + const int quarter = AC3_BLOCK_SIZE / 4; + const int half = AC3_BLOCK_SIZE / 2; + const int three_quarters = 3 * AC3_BLOCK_SIZE / 4; + LOCAL_ALIGNED_32(SampleType, input, [AC3_BLOCK_SIZE]); + LOCAL_ALIGNED_32(CoefType, output, [AC3_MAX_COEFS]); + + /* AVTX implements the alpha=0 MDCT. The alpha=-1 transform in A/52 + * Section 8.2.3.2 is equivalent to [x[N/4..N-1], -x[0..N/4-1]], + * while the alpha=+1 transform starts at the last quarter of the + * 2N-sample window. */ +#if AC3ENC_FLOAT + memcpy(input, samples + quarter, three_quarters * sizeof(*input)); + for (int i = 0; i < quarter; i++) + input[three_quarters + i] = -samples[i]; +#else + /* The fixed MDCT scale cannot exceed one, so apply its factor of two to + * the input. FixedDSP scales the product of S32 input and the Q22 window + * by 2^-31, limiting the windowed samples to 2^22. Doubling them here + * therefore remains within 2^23. */ + for (int i = 0; i < three_quarters; i++) + input[i] = 2 * samples[quarter + i]; + for (int i = 0; i < quarter; i++) + input[three_quarters + i] = -2 * samples[i]; +#endif + + s->tx_fn_short(s->tx_short, output, input, sizeof(*input)); + +#if AC3ENC_FLOAT + for (int i = 0; i < quarter; i++) + input[i] = -samples[AC3_WINDOW_SIZE - quarter + i]; + memcpy(input + quarter, samples + AC3_BLOCK_SIZE, + three_quarters * sizeof(*input)); +#else + for (int i = 0; i < quarter; i++) + input[i] = -2 * samples[AC3_WINDOW_SIZE - quarter + i]; + for (int i = 0; i < three_quarters; i++) + input[quarter + i] = 2 * samples[AC3_BLOCK_SIZE + i]; +#endif + + s->tx_fn_short(s->tx_short, output + half, input, sizeof(*input)); + + for (int i = 0; i < half; i++) { + coef[2 * i ] = output[i]; + coef[2 * i + 1] = output[half + i]; + } +} + /* * Apply the MDCT to input samples to generate frequency coefficients. * This applies the KBD window and normalizes the input to reduce precision @@ -69,8 +122,11 @@ static void apply_mdct(AC3EncodeContext *s, uint8_t * const *samples) input_samples1, s->RENAME(mdct_window), AC3_BLOCK_SIZE); - s->tx_fn(s->tx, block->mdct_coef[ch+1], - windowed_samples, sizeof(*windowed_samples)); + if (block->block_switch[ch + 1]) + apply_short_mdct(s, block->mdct_coef[ch + 1], windowed_samples); + else + s->tx_fn(s->tx, block->mdct_coef[ch + 1], + windowed_samples, sizeof(*windowed_samples)); input_samples0 = input_samples1; input_samples1 += AC3_BLOCK_SIZE; } while (++blk < s->num_blocks); -- 2.52.0

1 0

[PR] avutil/hwcontext_d3d11va: Set BindFlags for AMD GPUs (PR #24046)
by softworkz 07 Aug '26

07 Aug '26

PR #24046 opened by softworkz URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24046 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24046.patch - AMD drivers require the shader_resource flag for all processing and encoding. - When bind flags are zero, texture creation fails, so set the most compatible bind_decoder flag which works in all cases. ### Why this is needed (and hasn't before) It's a prerequisite for upcoming PRs which will enable mapping hw frames - from D3D11(AMD) to OpenCL(AMD) - from OpenCL(AMD) back to D3D11(AMD) - from AMF to D3D11 - from D3D11 to AMF From ec6aa7e356f93ad03d8cefcfe275b582df1477d8 Mon Sep 17 00:00:00 2001 From: softworkz <softworkz(a)hotmail.com> Date: Wed, 29 Jul 2026 04:19:24 +0200 Subject: [PATCH] avutil/hwcontext_d3d11va: Set BindFlags for AMD GPUs AMD drivers require the shader_resource flag for all processing and encoding. When bind flags are zero, texture creation fails, so set the most compatible bind_decoder flag which works in all cases. Signed-off-by: softworkz <softworkz(a)hotmail.com> --- libavutil/hwcontext_d3d11va.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/libavutil/hwcontext_d3d11va.c b/libavutil/hwcontext_d3d11va.c index 834c2ce3dd..faa9ce52c9 100644 --- a/libavutil/hwcontext_d3d11va.c +++ b/libavutil/hwcontext_d3d11va.c @@ -116,6 +116,31 @@ static const struct { { DXGI_FORMAT_420_OPAQUE, AV_PIX_FMT_YUV420P }, }; +static int d3d11va_is_amd(const AVD3D11VADeviceContext *device_hwctx) +{ + IDXGIDevice *dxgi_device = NULL; + IDXGIAdapter *adapter = NULL; + DXGI_ADAPTER_DESC desc; + HRESULT hr; + int is_amd = 0; + + hr = ID3D11Device_QueryInterface(device_hwctx->device, &IID_IDXGIDevice, (void **)&dxgi_device); + if (FAILED(hr)) + return 0; + + hr = IDXGIDevice_GetAdapter(dxgi_device, &adapter); + if (SUCCEEDED(hr)) { + hr = IDXGIAdapter_GetDesc(adapter, &desc); + if (SUCCEEDED(hr)) + is_amd = desc.VendorId == 0x1002; // AMD + IDXGIAdapter_Release(adapter); + } + + IDXGIDevice_Release(dxgi_device); + + return is_amd; +} + static void d3d11va_default_lock(void *ctx) { WaitForSingleObjectEx(ctx, INFINITE, FALSE); @@ -294,6 +319,14 @@ static int d3d11va_frames_init(AVHWFramesContext *ctx) hwctx->BindFlags |= device_hwctx->BindFlags; hwctx->MiscFlags |= device_hwctx->MiscFlags; + // AMD drivers require the shader_resource flag for all processing and encoding and + // texture creation fails when BindFlags is zero + if (d3d11va_is_amd(device_hwctx)) { + if (hwctx->BindFlags == 0) + hwctx->BindFlags = D3D11_BIND_DECODER; + hwctx->BindFlags |= D3D11_BIND_SHADER_RESOURCE; + } + ctx->initial_pool_size = FFMIN(ctx->initial_pool_size, MAX_ARRAY_SIZE); texDesc = (D3D11_TEXTURE2D_DESC){ -- 2.52.0

1 0