February 2024 - ffmpeg-devel - lists.ffmpeg.org

[PATCH 0/1] fate-aac-encode-pred UBsan fix
by Sean McGovern 22 Oct '24

22 Oct '24

From: Sean McGovern <sean(a)elysia.seanmcgovern.ca> Hi FFmpeg-devel, I've started looking into the results posted by the UBsan FATE node -- http://fate.ffmpeg.org/history.cgi?slot=x86_64-archlinux-gcc-ubsan Here is the error reported by FATE (snipped for brevity) for 'fate-aac-encode-pred': [aist#0:0/pcm_s16le @ 0x77fee40] Guessed Channel Layout: stereo Input #0, wav, from '/srv/VIDEO/fate-suite/audio-reference/luckynight_2ch_44kHz_s16.wav': Duration: 00:00:09.50, bitrate: 1411 kb/s Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 44100 Hz, stereo, s16, 1411 kb/s Stream mapping: Stream #0:0 -> #0:0 (pcm_s16le (native) -> aac (native)) Output #0, adts, to '/home/sean/build/ffmpeg-ubsan/tests/data/fate/aac-pred-encode.adts': Stream #0:0: Audio: aac (Main), 44100 Hz, stereo, fltp, 128 kb/s Metadata: encoder : Lavc aac src/libavcodec/aacenc_pred.c:169:48: runtime error: index 41 out of bounds for type 'uint8_t [41]' threads=8 Ignore the thread count, this fails the same when threads=1 Here is the loop it fails in -- for this test 'sfb == 49': libavcodec/aacenc_pred.c:165: for (w = 0; w < sce0->ics.num_windows; w += sce0->ics.group_len[w]) { libavcodec/aacenc_pred.c:166: start = 0; libavcodec/aacenc_pred.c:167: for (g = 0; g < sce0->ics.num_swb; g++) { libavcodec/aacenc_pred.c:168: int sfb = w*16+g; libavcodec/aacenc_pred.c:169: int sum = sce0->ics.prediction_used[sfb] + sce1->ics.prediction_used[sfb]; ... Here is the array definition for prediction_used: libavcodec/aac.h:188: uint8_t prediction_used[41]; Here is the line num_swb is set on: libavcodec/aacenc.c:899: ics->num_swb = tag == TYPE_LFE ? ics->num_swb : s->psy.num_bands[ics->num_windows == 8]; (side note: interesting, I've never seen a boolean condition used an an array index before...) My attached patch just uses FFMIN to ensure we don't navigate past the end of ics.prediction_used[] It corrects the undefined behaviour, but it feels naïve. Any thoughts? Sean McGovern (1): aacenc_pred: prevent UB in ff_aac_adjust_common_pred() libavcodec/aacenc_pred.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) -- 2.39.2

4 10

[PATCH 1/3] avcodec/ffv1enc: Slice combination is unsupported
by Michael Niedermayer 01 Oct '24

01 Oct '24

We always write minimal slices, the size calculation is wrong in some corner cases but as its always 1x1 (minus1) we can for now just hard-code it This helps with ticket 5548 Signed-off-by: Michael Niedermayer <michael(a)niedermayer.cc> --- libavcodec/ffv1enc.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/libavcodec/ffv1enc.c b/libavcodec/ffv1enc.c index 2778c63012e..4d0ddc167ae 100644 --- a/libavcodec/ffv1enc.c +++ b/libavcodec/ffv1enc.c @@ -914,8 +914,8 @@ static void encode_slice_header(FFV1Context *f, FFV1Context *fs) put_symbol(c, state, (fs->slice_x +1)*f->num_h_slices / f->width , 0); put_symbol(c, state, (fs->slice_y +1)*f->num_v_slices / f->height , 0); - put_symbol(c, state, (fs->slice_width +1)*f->num_h_slices / f->width -1, 0); - put_symbol(c, state, (fs->slice_height+1)*f->num_v_slices / f->height-1, 0); + put_symbol(c, state, 0, 0); + put_symbol(c, state, 0, 0); for (j=0; j<f->plane_count; j++) { put_symbol(c, state, f->plane[j].quant_table_index, 0); av_assert0(f->plane[j].quant_table_index == f->context_model); -- 2.17.1

2 5

[PATCH] avformat/mov: add support for multiple decryption keys
by Adrien Guinet 01 Aug '24

01 Aug '24

This commit introduces new options to support more than one decryption keys: * add a decryption_keys option to MOV, that supports a dictionnary of KID=>key (in hex), using AV_OPT_TYPE_DICT * add the corresponding cenc_decryption_keys option to DASH --- doc/demuxers.texi | 12 +++- libavformat/dashdec.c | 6 +- libavformat/isom.h | 5 +- libavformat/mov.c | 80 ++++++++++++++++++---- tests/fate/mov.mak | 12 ++++ tests/ref/fate/mov-3elist-encrypted-kid | 57 +++++++++++++++ tests/ref/fate/mov-frag-encrypted-kid | 57 +++++++++++++++ tests/ref/fate/mov-tenc-only-encrypted-kid | 57 +++++++++++++++ 8 files changed, 269 insertions(+), 17 deletions(-) create mode 100644 tests/ref/fate/mov-3elist-encrypted-kid create mode 100644 tests/ref/fate/mov-frag-encrypted-kid create mode 100644 tests/ref/fate/mov-tenc-only-encrypted-kid diff --git a/doc/demuxers.texi b/doc/demuxers.texi index ca1563abb0..9d0ab6025c 100644 --- a/doc/demuxers.texi +++ b/doc/demuxers.texi @@ -281,7 +281,11 @@ This demuxer accepts the following option: @table @option @item cenc_decryption_key -16-byte key, in hex, to decrypt files encrypted using ISO Common Encryption (CENC/AES-128 CTR; ISO/IEC 23001-7). +Default 16-byte key, in hex, to decrypt files encrypted using ISO Common Encryption (CENC/AES-128 CTR; ISO/IEC 23001-7). + +@item cenc_decryption_keys +Dictionary of 16-byte key ID => 16-byte key, both in hex, to decrypt files encrypted using ISO Common Encryption +(CENC/AES-128 CTR; ISO/IEC 23001-7). @end table @@ -769,7 +773,11 @@ Fixed key used for handling Audible AAX/AAX+ files. It has been pre-set so shoul specify. @item decryption_key -16-byte key, in hex, to decrypt files encrypted using ISO Common Encryption (CENC/AES-128 CTR; ISO/IEC 23001-7). +Default 16-byte key, in hex, to decrypt files encrypted using ISO Common Encryption (CENC/AES-128 CTR; ISO/IEC 23001-7). + +@item decryption_keys +Dictionary of 16-byte key ID => 16-byte key, both in hex, to decrypt files encrypted using ISO Common Encryption +(CENC/AES-128 CTR; ISO/IEC 23001-7). @item max_stts_delta Very high sample deltas written in a trak's stts box may occasionally be intended but usually they are written in diff --git a/libavformat/dashdec.c b/libavformat/dashdec.c index 1215407f3c..091deac854 100644 --- a/libavformat/dashdec.c +++ b/libavformat/dashdec.c @@ -153,6 +153,7 @@ typedef struct DASHContext { AVDictionary *avio_opts; int max_url_size; char *cenc_decryption_key; + char *cenc_decryption_keys; /* Flags for init section*/ int is_init_section_common_video; @@ -1903,6 +1904,8 @@ static int reopen_demux_for_component(AVFormatContext *s, struct representation if (c->cenc_decryption_key) av_dict_set(&in_fmt_opts, "decryption_key", c->cenc_decryption_key, 0); + if (c->cenc_decryption_keys) + av_dict_set(&in_fmt_opts, "decryption_keys", c->cenc_decryption_keys, 0); // provide additional information from mpd if available ret = avformat_open_input(&pls->ctx, "", in_fmt, &in_fmt_opts); //pls->init_section->url @@ -2344,7 +2347,8 @@ static const AVOption dash_options[] = { OFFSET(allowed_extensions), AV_OPT_TYPE_STRING, {.str = "aac,m4a,m4s,m4v,mov,mp4,webm,ts"}, INT_MIN, INT_MAX, FLAGS}, - { "cenc_decryption_key", "Media decryption key (hex)", OFFSET(cenc_decryption_key), AV_OPT_TYPE_STRING, {.str = NULL}, INT_MIN, INT_MAX, .flags = FLAGS }, + { "cenc_decryption_key", "Media default decryption key (hex)", OFFSET(cenc_decryption_key), AV_OPT_TYPE_STRING, {.str = NULL}, INT_MIN, INT_MAX, .flags = FLAGS }, + { "cenc_decryption_keys", "Media decryption keys by KID (hex)", OFFSET(cenc_decryption_keys), AV_OPT_TYPE_STRING, {.str = NULL}, INT_MIN, INT_MAX, .flags = FLAGS }, {NULL} }; diff --git a/libavformat/isom.h b/libavformat/isom.h index 3d375d7a46..83bacb96a1 100644 --- a/libavformat/isom.h +++ b/libavformat/isom.h @@ -312,8 +312,8 @@ typedef struct MOVContext { void *audible_iv; int audible_iv_size; struct AVAES *aes_decrypt; - uint8_t *decryption_key; - int decryption_key_len; + uint8_t *decryption_default_key; + int decryption_default_key_len; int enable_drefs; int32_t movie_display_matrix[3][3]; ///< display matrix from mvhd int have_read_mfra_size; @@ -328,6 +328,7 @@ typedef struct MOVContext { } *avif_info; int avif_info_size; int interleaved_read; + AVDictionary* decryption_keys; } MOVContext; int ff_mp4_read_descr_len(AVIOContext *pb); diff --git a/libavformat/mov.c b/libavformat/mov.c index 34ca8095c2..550bce86f5 100644 --- a/libavformat/mov.c +++ b/libavformat/mov.c @@ -7118,19 +7118,62 @@ static int mov_read_dfla(MOVContext *c, AVIOContext *pb, MOVAtom atom) return 0; } +static int get_key_from_kid(uint8_t* out, int len, MOVContext *c, AVEncryptionInfo *sample) { + AVDictionaryEntry *key_entry_hex; + char kid_hex[16*2+1]; + + if (c->decryption_default_key && c->decryption_default_key_len != len) { + av_log(c->fc, AV_LOG_ERROR, "invalid default decryption key length: got %d, expected %d\n", c->decryption_default_key_len, len); + return -1; + } + + if (!c->decryption_keys) { + av_assert0(c->decryption_default_key); + memcpy(out, c->decryption_default_key, len); + return 0; + } + + if (sample->key_id_size != 16) { + av_log(c->fc, AV_LOG_ERROR, "invalid key ID size: got %u, expected 16\n", sample->key_id_size); + return -1; + } + + ff_data_to_hex(kid_hex, sample->key_id, 16, 1); + key_entry_hex = av_dict_get(c->decryption_keys, kid_hex, NULL, AV_DICT_DONT_STRDUP_KEY|AV_DICT_DONT_STRDUP_VAL); + if (!key_entry_hex) { + if (!c->decryption_default_key) { + av_log(c->fc, AV_LOG_ERROR, "unable to find KID %s\n", kid_hex); + return -1; + } + memcpy(out, c->decryption_default_key, len); + return 0; + } + if (strlen(key_entry_hex->value) != len*2) { + return -1; + } + ff_hex_to_data(out, key_entry_hex->value); + return 0; +} + static int cenc_scheme_decrypt(MOVContext *c, MOVStreamContext *sc, AVEncryptionInfo *sample, uint8_t *input, int size) { int i, ret; int bytes_of_protected_data; + uint8_t decryption_key[AES_CTR_KEY_SIZE]; if (!sc->cenc.aes_ctr) { + ret = get_key_from_kid(decryption_key, sizeof(decryption_key), c, sample); + if (ret < 0) { + return ret; + } + /* initialize the cipher */ sc->cenc.aes_ctr = av_aes_ctr_alloc(); if (!sc->cenc.aes_ctr) { return AVERROR(ENOMEM); } - ret = av_aes_ctr_init(sc->cenc.aes_ctr, c->decryption_key); + ret = av_aes_ctr_init(sc->cenc.aes_ctr, decryption_key); if (ret < 0) { return ret; } @@ -7176,15 +7219,21 @@ static int cbc1_scheme_decrypt(MOVContext *c, MOVStreamContext *sc, AVEncryption int i, ret; int num_of_encrypted_blocks; uint8_t iv[16]; + uint8_t decryption_key[16]; if (!sc->cenc.aes_ctx) { + ret = get_key_from_kid(decryption_key, sizeof(decryption_key), c, sample); + if (ret < 0) { + return ret; + } + /* initialize the cipher */ sc->cenc.aes_ctx = av_aes_alloc(); if (!sc->cenc.aes_ctx) { return AVERROR(ENOMEM); } - ret = av_aes_init(sc->cenc.aes_ctx, c->decryption_key, 16 * 8, 1); + ret = av_aes_init(sc->cenc.aes_ctx, decryption_key, 16 * 8, 1); if (ret < 0) { return ret; } @@ -7235,15 +7284,21 @@ static int cens_scheme_decrypt(MOVContext *c, MOVStreamContext *sc, AVEncryption { int i, ret, rem_bytes; uint8_t *data; + uint8_t decryption_key[AES_CTR_KEY_SIZE]; if (!sc->cenc.aes_ctr) { + ret = get_key_from_kid(decryption_key, sizeof(decryption_key), c, sample); + if (ret < 0) { + return ret; + } + /* initialize the cipher */ sc->cenc.aes_ctr = av_aes_ctr_alloc(); if (!sc->cenc.aes_ctr) { return AVERROR(ENOMEM); } - ret = av_aes_ctr_init(sc->cenc.aes_ctr, c->decryption_key); + ret = av_aes_ctr_init(sc->cenc.aes_ctr, decryption_key); if (ret < 0) { return ret; } @@ -7301,15 +7356,21 @@ static int cbcs_scheme_decrypt(MOVContext *c, MOVStreamContext *sc, AVEncryption int i, ret, rem_bytes; uint8_t iv[16]; uint8_t *data; + uint8_t decryption_key[16]; if (!sc->cenc.aes_ctx) { + ret = get_key_from_kid(decryption_key, sizeof(decryption_key), c, sample); + if (ret < 0) { + return ret; + } + /* initialize the cipher */ sc->cenc.aes_ctx = av_aes_alloc(); if (!sc->cenc.aes_ctx) { return AVERROR(ENOMEM); } - ret = av_aes_init(sc->cenc.aes_ctx, c->decryption_key, 16 * 8, 1); + ret = av_aes_init(sc->cenc.aes_ctx, decryption_key, 16 * 8, 1); if (ret < 0) { return ret; } @@ -7452,7 +7513,7 @@ static int cenc_filter(MOVContext *mov, AVStream* st, MOVStreamContext *sc, AVPa return AVERROR_INVALIDDATA; } - if (mov->decryption_key) { + if (mov->decryption_keys || mov->decryption_default_key) { return cenc_decrypt(mov, sc, encrypted_sample, pkt->data, pkt->size); } else { size_t size; @@ -8601,12 +8662,6 @@ static int mov_read_header(AVFormatContext *s) MOVAtom atom = { AV_RL32("root") }; int i; - if (mov->decryption_key_len != 0 && mov->decryption_key_len != AES_CTR_KEY_SIZE) { - av_log(s, AV_LOG_ERROR, "Invalid decryption key len %d expected %d\n", - mov->decryption_key_len, AES_CTR_KEY_SIZE); - return AVERROR(EINVAL); - } - mov->fc = s; mov->trak_index = -1; /* .mov and .mp4 aren't streamable anyway (only progressive download if moov is before mdat) */ @@ -9305,7 +9360,8 @@ static const AVOption mov_options[] = { "Fixed key used for handling Audible AAX files", OFFSET(audible_fixed_key), AV_OPT_TYPE_BINARY, {.str="77214d4b196a87cd520045fd20a51d67"}, .flags = AV_OPT_FLAG_DECODING_PARAM }, - { "decryption_key", "The media decryption key (hex)", OFFSET(decryption_key), AV_OPT_TYPE_BINARY, .flags = AV_OPT_FLAG_DECODING_PARAM }, + { "decryption_key", "The default media decryption key (hex)", OFFSET(decryption_default_key), AV_OPT_TYPE_BINARY, .flags = AV_OPT_FLAG_DECODING_PARAM }, + { "decryption_keys", "The media decryption keys by KID (hex)", OFFSET(decryption_keys), AV_OPT_TYPE_DICT, .flags = AV_OPT_FLAG_DECODING_PARAM }, { "enable_drefs", "Enable external track support.", OFFSET(enable_drefs), AV_OPT_TYPE_BOOL, {.i64 = 0}, 0, 1, FLAGS }, { "max_stts_delta", "treat offsets above this value as invalid", OFFSET(max_stts_delta), AV_OPT_TYPE_INT, {.i64 = UINT_MAX-48000*10 }, 0, UINT_MAX, .flags = AV_OPT_FLAG_DECODING_PARAM }, diff --git a/tests/fate/mov.mak b/tests/fate/mov.mak index 6cb493ceab..c420d8ab10 100644 --- a/tests/fate/mov.mak +++ b/tests/fate/mov.mak @@ -8,6 +8,9 @@ FATE_MOV = fate-mov-3elist \ fate-mov-3elist-encrypted \ fate-mov-frag-encrypted \ fate-mov-tenc-only-encrypted \ + fate-mov-3elist-encrypted-kid \ + fate-mov-frag-encrypted-kid \ + fate-mov-tenc-only-encrypted-kid \ fate-mov-invalid-elst-entry-count \ fate-mov-gpmf-remux \ fate-mov-440hz-10ms \ @@ -57,6 +60,15 @@ fate-mov-frag-encrypted: CMD = framemd5 -decryption_key 123456789012345678901234 # Full-sample encryption and constant IV using only tenc atom (no senc/saio/saiz). fate-mov-tenc-only-encrypted: CMD = framemd5 -decryption_key 12345678901234567890123456789012 -i $(TARGET_SAMPLES)/mov/mov-tenc-only-encrypted.mp4 +# Edit list with encryption, using the decryption_keys option. +fate-mov-3elist-encrypted-kid: CMD = framemd5 -decryption_keys 12345678901234567890123456789012=12345678901234567890123456789012 -i $(TARGET_SAMPLES)/mov/mov-3elist-encrypted.mov + +# Fragmented encryption with senc boxes in movie fragments, using the decryption_keys option. +fate-mov-frag-encrypted-kid: CMD = framemd5 -decryption_keys abba271e8bcf552bbd2e86a434a9a5d9=12345678901234567890123456789012 -i $(TARGET_SAMPLES)/mov/mov-frag-encrypted.mp4 + +# Full-sample encryption and constant IV using only tenc atom (no senc/saio/saiz), using the decryption_keys option. +fate-mov-tenc-only-encrypted-kid: CMD = framemd5 -decryption_keys abba271e8bcf552bbd2e86a434a9a5d9=12345678901234567890123456789012 -i $(TARGET_SAMPLES)/mov/mov-tenc-only-encrypted.mp4 + # Makes sure that the CTTS is also modified when we fix avindex in mov.c while parsing edit lists. fate-mov-elist-starts-ctts-2ndsample: CMD = framemd5 -i $(TARGET_SAMPLES)/mov/mov-elist-starts-ctts-2ndsample.mov diff --git a/tests/ref/fate/mov-3elist-encrypted-kid b/tests/ref/fate/mov-3elist-encrypted-kid new file mode 100644 index 0000000000..daf4927e0f --- /dev/null +++ b/tests/ref/fate/mov-3elist-encrypted-kid @@ -0,0 +1,57 @@ +#format: frame checksums +#version: 2 +#hash: MD5 +#tb 0: 1/24 +#media_type 0: video +#codec_id 0: rawvideo +#dimensions 0: 640x480 +#sar 0: 0/1 +#stream#, dts, pts, duration, size, hash +0, 0, 0, 1, 460800, 80fbbdec589e15e6c493b44d243f92a9 +0, 1, 1, 1, 460800, f4b23293bb2ecf69cc3570853d8c56a1 +0, 2, 2, 1, 460800, 0c03ce2c1c6ec405d7455465ecd559a3 +0, 3, 3, 1, 460800, 7921791695537fba2c3c123da4834cb9 +0, 4, 4, 1, 460800, 30c8e2903a561b84d4cbaf95c668d236 +0, 5, 5, 1, 460800, 7ff42e998217c17592ddf6b584f26cef +0, 6, 6, 1, 460800, 5e402c48bf097db2d31b82bb4194a382 +0, 7, 7, 1, 460800, 824c49e92c8ae6d99a0207b514dd756c +0, 8, 8, 1, 460800, 24f189216a1d9cf2313b2d6dbe3dbdd3 +0, 9, 9, 1, 460800, 519179a8e74275d26b183374637e003f +0, 10, 10, 1, 460800, f18331ddcef0adf5b069bfa98baf8db4 +0, 11, 11, 1, 460800, 081f61688690d47dbdddd5384e5d5a70 +0, 12, 12, 1, 460800, 90dbf019b9035433371a8df41a9268b7 +0, 13, 13, 1, 460800, bb5adfb9c66732898b34186eca1667ba +0, 14, 14, 1, 460800, cc08cfd64f37783ecddaf143f6ad78bc +0, 15, 15, 1, 460800, b8ae21d024fe4df903d56f4521993c72 +0, 16, 16, 1, 460800, b45a99907f045dcadf0a2befc11555e3 +0, 17, 17, 1, 460800, 603ba935845e65ab6cccbbec88bbf60d +0, 18, 18, 1, 460800, df80c8d3e6a77258a306903f17995a18 +0, 19, 19, 1, 460800, 4b7e90c0a5fd0e0cd958d47f0afac636 +0, 20, 20, 1, 460800, 9feb6e36182f1745be6387edea240eb6 +0, 21, 21, 1, 460800, 86e6de4bd0a5ff7558f4cf6c1ec3930d +0, 22, 22, 1, 460800, 726b69df77edbe7b503d4698656d1320 +0, 23, 23, 1, 460800, d282fb7a953ac205b0a43d00c2d60a33 +0, 24, 24, 1, 460800, eece3daa70cc20208dd75d91ac84c8fd +0, 25, 25, 1, 460800, c86d23e73bcce351fc315fb1f13348da +0, 26, 26, 1, 460800, 93497b4f7c5ad9d61212239b7c9d2770 +0, 27, 27, 1, 460800, eb217d2c12de67903835a8c58f620488 +0, 28, 28, 1, 460800, d966480867bb54c8cd044f18388ed486 +0, 29, 29, 1, 460800, 3ea6207942b3181fdd8e8aa6cae1062a +0, 30, 30, 1, 460800, 2620df54aca086ec0fb9527c6e6f5135 +0, 31, 31, 1, 460800, 43bb7320f0bb583188dc965ddbfade90 +0, 32, 32, 1, 460800, 0cddaa04645f804e02f65b0836412113 +0, 33, 33, 1, 460800, 83b2dc95807289d7f4a4632bf18c2e97 +0, 34, 34, 1, 460800, 98134d0e41e6dd12827049ccf33b4669 +0, 35, 35, 1, 460800, 56f55631731fa39c7acbab0afeb2eb1b +0, 36, 36, 1, 460800, 379c1105be09d836a515dc909455ddf4 +0, 37, 37, 1, 460800, 1df87c47e9d98731faf1c3885b77e5da +0, 38, 38, 1, 460800, 9a8734bcbfdb4d97e530683b8b556a26 +0, 39, 39, 1, 460800, c7a7990d0cddc5adfbe27da7a42e025e +0, 40, 40, 1, 460800, 0c81e46011e03be410feaf056207fd55 +0, 41, 41, 1, 460800, ca76e4e63016ff29d8aeeb9cb053bb6c +0, 42, 42, 1, 460800, cebfbe299c17c1f8fc1e6b189555c3c2 +0, 43, 43, 1, 460800, 4f002c5feca5e75f07089e0df47507dd +0, 44, 44, 1, 460800, c5fd83fc4a745abee9b3d9a6eec9dd3e +0, 45, 45, 1, 460800, 57d9bad9b45aa2746de5d8bdc2c24969 +0, 46, 46, 1, 460800, 9831673ad7dec167af4a959f64258949 +0, 47, 47, 1, 460800, 77a1cb208f70f51bcb01e28d8cba73b4 diff --git a/tests/ref/fate/mov-frag-encrypted-kid b/tests/ref/fate/mov-frag-encrypted-kid new file mode 100644 index 0000000000..e6c109b566 --- /dev/null +++ b/tests/ref/fate/mov-frag-encrypted-kid @@ -0,0 +1,57 @@ +#format: frame checksums +#version: 2 +#hash: MD5 +#tb 0: 1/24 +#media_type 0: video +#codec_id 0: rawvideo +#dimensions 0: 120x52 +#sar 0: 544/545 +#stream#, dts, pts, duration, size, hash +0, 0, 0, 1, 9360, 920bdc277a6a31c1daed9aca44b10caf +0, 1, 1, 1, 9360, f1c0b61fef593de57cb97be7fa846569 +0, 2, 2, 1, 9360, 6ef32d9d4398355aebf6d3fb11d51d3f +0, 3, 3, 1, 9360, d38fd3ef1e5a92fc109b8dd9eb6dadeb +0, 4, 4, 1, 9360, 54cc0c8a25d2f14f32663837d5e646f1 +0, 5, 5, 1, 9360, b4b6829726dc3decb8b80ba0c35bcf30 +0, 6, 6, 1, 9360, fca3f941e60a2f0a4ce30d5e0efbec3c +0, 7, 7, 1, 9360, cda6e26b6c1039ff3d229b262c9210c3 +0, 8, 8, 1, 9360, f0d69255e3a27a8b4ae8a4b7b210929d +0, 9, 9, 1, 9360, 12cb23dd4e32af9c3b35f943714e3fdd +0, 10, 10, 1, 9360, 082aaf3216124ddcecb422fe5c832e82 +0, 11, 11, 1, 9360, ff37bb8cd6bd0412a3b3cb45db54afc9 +0, 12, 12, 1, 9360, dfb9085441575732844b6c2f05d5f542 +0, 13, 13, 1, 9360, 0017100feaaa9fc7eacd2447d50d7542 +0, 14, 14, 1, 9360, 4e2f1b8c4e04c59934c2f58541e62613 +0, 15, 15, 1, 9360, 27a44dfea7cd2d30e488194c34ab473c +0, 16, 16, 1, 9360, fc7b56bd95e990a33cf575d1ef820902 +0, 17, 17, 1, 9360, fa2d1609e69714dffc410e65f3c8b755 +0, 18, 18, 1, 9360, 705d7429f447cb13febe202d567795f2 +0, 19, 19, 1, 9360, 234802ce86e868faaf2cd40a286846ea +0, 20, 20, 1, 9360, 2f0354b40d211d0a4ade4568bea4f85e +0, 21, 21, 1, 9360, e96af3b6c0cc931463ca77d6be0f1148 +0, 22, 22, 1, 9360, 04a904d798361959971361401879c7e4 +0, 23, 23, 1, 9360, 2f119642340df6d25362b5590ded46b7 +0, 24, 24, 1, 9360, 5993fca2e60050706f857ac76e48f386 +0, 25, 25, 1, 9360, 2ff3b5775fed3d527bfbbeea786787fe +0, 26, 26, 1, 9360, 42024dbe23d3fb5b0d8987ae1ce390a8 +0, 27, 27, 1, 9360, d804204f0bd9db5f6a758e2c934d9e38 +0, 28, 28, 1, 9360, e322712e6e34c58ec1a2ab5e2c1e3bfe +0, 29, 29, 1, 9360, 3975bd1a5f6a6b6260276777f9de611e +0, 30, 30, 1, 9360, 4388f0412efc6310706a7cdedc859ea9 +0, 31, 31, 1, 9360, b4b9a11b0b86635267345a569640e8d4 +0, 32, 32, 1, 9360, 31879c7b8d6b67a4209ffde786bb8cb4 +0, 33, 33, 1, 9360, 4b6dc02d7c889fe4abd4e013b25f585a +0, 34, 34, 1, 9360, dc73aae82bd39a1220d1106c8d3e8252 +0, 35, 35, 1, 9360, 54c7dfbd49f312806f6c1a89f7c2c36f +0, 36, 36, 1, 9360, 150abc64f8994d444a521ea90570443c +0, 37, 37, 1, 9360, d277cdc7dcadbe0016f2e950459e7ebf +0, 38, 38, 1, 9360, 2196bf338ead90ea54687b85c73c8229 +0, 39, 39, 1, 9360, 53ce5da5365abc0bd3217dd98e7c465d +0, 40, 40, 1, 9360, 34ee9832aea55c0c4e6f4381c413c10e +0, 41, 41, 1, 9360, 1769c7b5849e4681119067a06ac29a4f +0, 42, 42, 1, 9360, 71f53df739ef283a5184c91ef4b158e8 +0, 43, 43, 1, 9360, d2d394739e9a59c06f0354c16843cb63 +0, 44, 44, 1, 9360, d8e458e92ae29344505a24a3059fc584 +0, 45, 45, 1, 9360, 0f1b11a09911851b798df2ef76253a7f +0, 46, 46, 1, 9360, 5c4a9f22baecf4e749c0d5c65a4f1007 +0, 47, 47, 1, 9360, 3e2b7e7262fdca08d9d1ef6070125c4b diff --git a/tests/ref/fate/mov-tenc-only-encrypted-kid b/tests/ref/fate/mov-tenc-only-encrypted-kid new file mode 100644 index 0000000000..1d57aa6a80 --- /dev/null +++ b/tests/ref/fate/mov-tenc-only-encrypted-kid @@ -0,0 +1,57 @@ +#format: frame checksums +#version: 2 +#hash: MD5 +#tb 0: 1/24 +#media_type 0: video +#codec_id 0: rawvideo +#dimensions 0: 1024x436 +#sar 0: 1/1 +#stream#, dts, pts, duration, size, hash +0, 0, 0, 1, 669696, f48f296a85eda5ba069dc851a3228bef +0, 1, 1, 1, 669696, a50c5f69bfa3387d49b5bdf738e6529c +0, 2, 2, 1, 669696, 05061299003760f6a4795b408f72aa31 +0, 3, 3, 1, 669696, 2572119f0b0cdd83f8a7e06252cecd3b +0, 4, 4, 1, 669696, 29fe6a6bdb4a69018e318886a297f07e +0, 5, 5, 1, 669696, e8233c7fbaecfbff965c7dfdd3982b1b +0, 6, 6, 1, 669696, d9259df9880ff5d4a4b38282e67f407b +0, 7, 7, 1, 669696, 3e8d795195038993503ea9ab6984c915 +0, 8, 8, 1, 669696, bc4e2d253b715a34f85aae1b080e3460 +0, 9, 9, 1, 669696, 09aba8b3a96f53f9268e7420a10bfab6 +0, 10, 10, 1, 669696, 179447977dd580da8b35fb5310a809ca +0, 11, 11, 1, 669696, 7a0eea9d54577990345f5705ab9882be +0, 12, 12, 1, 669696, 5bb96eb76f461825740e5938456df759 +0, 13, 13, 1, 669696, bd4ac4a760ead774b9422a27dc071964 +0, 14, 14, 1, 669696, 1cc05f760a9b751fc89e77f2bcc97259 +0, 15, 15, 1, 669696, 825d0dee6f0174ba7102892c7de30b4d +0, 16, 16, 1, 669696, d26a2ef5267f6bb03c4e1d8514eee0df +0, 17, 17, 1, 669696, c916ffdeadca76596a8f7fd47914b5ef +0, 18, 18, 1, 669696, 6e085acfa7fee0658ea0ae6188274c17 +0, 19, 19, 1, 669696, 1e95fa5b3561283f05bf0bd44cb91721 +0, 20, 20, 1, 669696, 37e3d135aba9dfb8b87e441753115374 +0, 21, 21, 1, 669696, 9c398310e8564491de624393c16265ce +0, 22, 22, 1, 669696, c87209e4d2617bc2ab40a75f455f09da +0, 23, 23, 1, 669696, 2679c2f8d1d1af21982e245945c1ee60 +0, 24, 24, 1, 669696, 6151ab4781f31c5beb66b356ad547122 +0, 25, 25, 1, 669696, f7ef6293bfb3a6a329061cb6a5ed5a38 +0, 26, 26, 1, 669696, 2f6e666d14dfc407ca0c0f347b13eb08 +0, 27, 27, 1, 669696, 3454fa1730d79b1aa8dbbc865dc150f4 +0, 28, 28, 1, 669696, e93dc683e2453419a0419ab9af0f8f95 +0, 29, 29, 1, 669696, 031eb3154f7f83cf86d42bee66be9cf7 +0, 30, 30, 1, 669696, 1205c36723e88811206c68892d3aaed6 +0, 31, 31, 1, 669696, 7dd7a8a19dcd73b31ddc6a6d0c597a42 +0, 32, 32, 1, 669696, 7c91115368ea2531262a1197468bc3f4 +0, 33, 33, 1, 669696, 3cf6d9ba385e0fff76da33299ed5380c +0, 34, 34, 1, 669696, 859fc8c3ef049e3c1175a85fb0a90a3d +0, 35, 35, 1, 669696, 1d09ce6c7027103d99a4d5799f6e72ab +0, 36, 36, 1, 669696, 3dcb8357408ac88abd734128d8f5dd6f +0, 37, 37, 1, 669696, 4dafce137a0a5178f6efaec878e64d36 +0, 38, 38, 1, 669696, 44c478f29a1399ed03275a7357f57d48 +0, 39, 39, 1, 669696, 6e9edaac7414c0e14591ac3d4d0b1ac4 +0, 40, 40, 1, 669696, 522e4aaeea0825da27f631a9e690d654 +0, 41, 41, 1, 669696, 85f2502a718440834c40051d30f8a65e +0, 42, 42, 1, 669696, ae8816f7bd4645ef1a17ee6d09b4c8d2 +0, 43, 43, 1, 669696, 914b006fa92f1eb3e590245749f6810d +0, 44, 44, 1, 669696, 9406901542e94c429dff46108782ed69 +0, 45, 45, 1, 669696, 324c13641c39eef5c476023e358c0391 +0, 46, 46, 1, 669696, 4058e886e17c22e4eb9da1dd0d6ad891 +0, 47, 47, 1, 669696, 9edf9cd15eea985b42fd1f5035b1d693 -- 2.42.0

3 2

[PATCH] libavcodec: fix -Wint-conversion in vulkan
by Sam James 30 Jul '24

30 Jul '24

FIx warnings (soon to be errors in GCC 14, already so in Clang 15): ``` src/libavcodec/vulkan_av1.c: In function ‘vk_av1_create_params’: src/libavcodec/vulkan_av1.c:183:43: error: initialization of ‘long long unsigned int’ from ‘void *’ makes integer from pointer without a cast [-Wint-conversion] 183 | .videoSessionParametersTemplate = NULL, | ^~~~ src/libavcodec/vulkan_av1.c:183:43: note: (near initialization for ‘(anonymous).videoSessionParametersTemplate’) ``` Use Vulkan's VK_NULL_HANDLE instead of bare NULL. Fix Trac ticket #10724. Was reported downstream in Gentoo at https://bugs.gentoo.org/919067. Signed-off-by: Sam James <sam(a)gentoo.org> --- libavcodec/vulkan_av1.c | 2 +- libavcodec/vulkan_decode.c | 6 +++--- libavcodec/vulkan_h264.c | 2 +- libavcodec/vulkan_hevc.c | 2 +- libavcodec/vulkan_video.c | 2 +- 5 files changed, 7 insertions(+), 7 deletions(-) diff --git a/libavcodec/vulkan_av1.c b/libavcodec/vulkan_av1.c index 4998bf7ebc..9730e4b08d 100644 --- a/libavcodec/vulkan_av1.c +++ b/libavcodec/vulkan_av1.c @@ -180,7 +180,7 @@ static int vk_av1_create_params(AVCodecContext *avctx, AVBufferRef **buf) .sType = VK_STRUCTURE_TYPE_VIDEO_SESSION_PARAMETERS_CREATE_INFO_KHR, .pNext = &av1_params, .videoSession = ctx->common.session, - .videoSessionParametersTemplate = NULL, + .videoSessionParametersTemplate = VK_NULL_HANDLE, }; err = ff_vk_decode_create_params(buf, avctx, ctx, &session_params_create); diff --git a/libavcodec/vulkan_decode.c b/libavcodec/vulkan_decode.c index a89d84fcaa..fdbcbb450a 100644 --- a/libavcodec/vulkan_decode.c +++ b/libavcodec/vulkan_decode.c @@ -188,9 +188,9 @@ int ff_vk_decode_prepare_frame(FFVulkanDecodeContext *dec, AVFrame *pic, return 0; vkpic->dpb_frame = NULL; - vkpic->img_view_ref = NULL; - vkpic->img_view_out = NULL; - vkpic->img_view_dest = NULL; + vkpic->img_view_ref = VK_NULL_HANDLE; + vkpic->img_view_out = VK_NULL_HANDLE; + vkpic->img_view_dest = VK_NULL_HANDLE; vkpic->destroy_image_view = vk->DestroyImageView; vkpic->wait_semaphores = vk->WaitSemaphores; diff --git a/libavcodec/vulkan_h264.c b/libavcodec/vulkan_h264.c index e727aafb16..39c123ddca 100644 --- a/libavcodec/vulkan_h264.c +++ b/libavcodec/vulkan_h264.c @@ -315,7 +315,7 @@ static int vk_h264_create_params(AVCodecContext *avctx, AVBufferRef **buf) .sType = VK_STRUCTURE_TYPE_VIDEO_SESSION_PARAMETERS_CREATE_INFO_KHR, .pNext = &h264_params, .videoSession = ctx->common.session, - .videoSessionParametersTemplate = NULL, + .videoSessionParametersTemplate = VK_NULL_HANDLE, }; /* SPS list */ diff --git a/libavcodec/vulkan_hevc.c b/libavcodec/vulkan_hevc.c index 99fdcf3b45..033172cbd6 100644 --- a/libavcodec/vulkan_hevc.c +++ b/libavcodec/vulkan_hevc.c @@ -653,7 +653,7 @@ static int vk_hevc_create_params(AVCodecContext *avctx, AVBufferRef **buf) .sType = VK_STRUCTURE_TYPE_VIDEO_SESSION_PARAMETERS_CREATE_INFO_KHR, .pNext = &h265_params, .videoSession = ctx->common.session, - .videoSessionParametersTemplate = NULL, + .videoSessionParametersTemplate = VK_NULL_HANDLE, }; HEVCHeaderSet *hdr; diff --git a/libavcodec/vulkan_video.c b/libavcodec/vulkan_video.c index 5fa8292b28..fb20315db4 100644 --- a/libavcodec/vulkan_video.c +++ b/libavcodec/vulkan_video.c @@ -287,7 +287,7 @@ av_cold void ff_vk_video_common_uninit(FFVulkanContext *s, if (common->session) { vk->DestroyVideoSessionKHR(s->hwctx->act_dev, common->session, s->hwctx->alloc); - common->session = NULL; + common->session = VK_NULL_HANDLE; } if (common->nb_mem && common->mem) -- 2.43.0

3 4

[PATCH] cbs_av1: Reject thirty-two zero bits in uvlc code
by Mark Thompson 21 Jul '24

21 Jul '24

The spec allows at least thirty-two zero bits followed by a one to mean 2^32-1, with no constraint on the number of zeroes. The libaom reference decoder does not match this, instead reading thirty-two zeroes but not the following one to mean 2^32-1. These two interpretations are incompatible and other implementations may follow one or the other. Therefore reject thirty-two zeroes because the intended behaviour is not clear. --- libaom, dav1d and SVT-AV1 all have the same nonstandard behaviour of stopping at thirty-two zeroes and not reading the one. gav1 just rejects thirty-two zeroes. This is also a source of arbitrarily large single syntax elements to hit <https://lists.ffmpeg.org/pipermail/ffmpeg-devel/2023-October/315973.html>. libavcodec/cbs_av1.c | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/libavcodec/cbs_av1.c b/libavcodec/cbs_av1.c index 1d9ac5ab44..13c749a25b 100644 --- a/libavcodec/cbs_av1.c +++ b/libavcodec/cbs_av1.c @@ -36,7 +36,7 @@ static int cbs_av1_read_uvlc(CodedBitstreamContext *ctx, GetBitContext *gbc, CBS_TRACE_READ_START(); zeroes = 0; - while (1) { + while (zeroes < 32) { if (get_bits_left(gbc) < 1) { av_log(ctx->log_ctx, AV_LOG_ERROR, "Invalid uvlc code at " "%s: bitstream ended.\n", name); @@ -49,10 +49,18 @@ static int cbs_av1_read_uvlc(CodedBitstreamContext *ctx, GetBitContext *gbc, } if (zeroes >= 32) { - // Note that the spec allows an arbitrarily large number of - // zero bits followed by a one bit in this case, but the - // libaom implementation does not support it. - value = MAX_UINT_BITS(32); + // The spec allows at least thirty-two zero bits followed by a + // one to mean 2^32-1, with no constraint on the number of + // zeroes. The libaom reference decoder does not match this, + // instead reading thirty-two zeroes but not the following one + // to mean 2^32-1. These two interpretations are incompatible + // and other implementations may follow one or the other. + // Therefore we reject thirty-two zeroes because the intended + // behaviour is not clear. + av_log(ctx->log_ctx, AV_LOG_ERROR, "Thirty-two zero bits in " + "%s uvlc code: considered invalid due to conflicting " + "standard and reference decoder behaviour.\n", name); + return AVERROR_INVALIDDATA; } else { if (get_bits_left(gbc) < zeroes) { av_log(ctx->log_ctx, AV_LOG_ERROR, "Invalid uvlc code at " -- 2.39.2

2 4

RISC-V vector DSP functions: Motivation for commit 446b009
by Michael Platzer 06 Jul '24

06 Jul '24

Hi, Commit 446b0090cbb66ee614dcf6ca79c78dc8eb7f0e37 by Remi Denis-Courmont has replaced RISC-V vector loads and stores with negative stride with vrgather (generalized permutation within vector registers) instructions in order to reverse the elements in a vector register. The commit message explains that this change was done, but it does not explain why. I fail to see what could possibly have motivated this change. The RISC-V vector loads and stores support negative stride values for use cases such as this one. Using vrgather instead replaces the more specific operation with a more generic one, which is likely to be less performant on most HW architectures. In addition, it requires to setup an index vector, thus raising dynamic instruction count. Could someone familiar with this change (perhaps Remi himself) please explain the motivation for this change? Thanks, Michael

2 4

[PATCH] avfilter: add v360_vulkan filter
by Paul B Mahol 05 Jul '24

05 Jul '24

Signed-off-by: Paul B Mahol <onemda(a)gmail.com> --- Please test it and reports benchmark results on non toyish GPUs. --- configure | 1 + libavfilter/Makefile | 1 + libavfilter/allfilters.c | 1 + libavfilter/vf_v360_vulkan.c | 668 +++++++++++++++++++++++++++++++++++ 4 files changed, 671 insertions(+) create mode 100644 libavfilter/vf_v360_vulkan.c diff --git a/configure b/configure index 7495f35faa..2a3ac563db 100755 --- a/configure +++ b/configure @@ -3622,6 +3622,7 @@ transpose_opencl_filter_deps="opencl" transpose_vaapi_filter_deps="vaapi VAProcPipelineCaps_rotation_flags" unsharp_opencl_filter_deps="opencl" uspp_filter_deps="gpl avcodec" +v360_vulkan_filter_deps="vulkan libglslang" vaguedenoiser_filter_deps="gpl" vidstabdetect_filter_deps="libvidstab" vidstabtransform_filter_deps="libvidstab" diff --git a/libavfilter/Makefile b/libavfilter/Makefile index 191826a622..6440794bcf 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -441,6 +441,7 @@ OBJS-$(CONFIG_UNSHARP_OPENCL_FILTER) += vf_unsharp_opencl.o opencl.o \ OBJS-$(CONFIG_UNTILE_FILTER) += vf_untile.o OBJS-$(CONFIG_USPP_FILTER) += vf_uspp.o OBJS-$(CONFIG_V360_FILTER) += vf_v360.o +OBJS-$(CONFIG_V360_VULKAN_FILTER) += vf_v360_vulkan.o vulkan.o OBJS-$(CONFIG_VAGUEDENOISER_FILTER) += vf_vaguedenoiser.o OBJS-$(CONFIG_VECTORSCOPE_FILTER) += vf_vectorscope.o OBJS-$(CONFIG_VFLIP_FILTER) += vf_vflip.o diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index de5884529c..65669c4422 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -420,6 +420,7 @@ extern AVFilter ff_vf_unsharp_opencl; extern AVFilter ff_vf_untile; extern AVFilter ff_vf_uspp; extern AVFilter ff_vf_v360; +extern AVFilter ff_vf_v360_vulkan; extern AVFilter ff_vf_vaguedenoiser; extern AVFilter ff_vf_vectorscope; extern AVFilter ff_vf_vflip; diff --git a/libavfilter/vf_v360_vulkan.c b/libavfilter/vf_v360_vulkan.c new file mode 100644 index 0000000000..7db200f1c6 --- /dev/null +++ b/libavfilter/vf_v360_vulkan.c @@ -0,0 +1,668 @@ +/* + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include "libavutil/random_seed.h" +#include "libavutil/opt.h" +#include "vulkan.h" +#include "internal.h" + +enum RotationOrder { + YAW, + PITCH, + ROLL, + NB_RORDERS, +}; + +enum Projections { + EQUIRECTANGULAR, + CUBEMAP_3_2, + CUBEMAP_6_1, + EQUIANGULAR, + FLAT, + DUAL_FISHEYE, + BARREL, + CUBEMAP_1_6, + STEREOGRAPHIC, + MERCATOR, + BALL, + HAMMER, + SINUSOIDAL, + FISHEYE, + PANNINI, + CYLINDRICAL, + PERSPECTIVE, + TETRAHEDRON, + BARREL_SPLIT, + TSPYRAMID, + HEQUIRECTANGULAR, + NB_PROJECTIONS, +}; + +#define CGROUPS (int [3]){ 32, 32, 1 } + +typedef struct V360VulkanContext { + VulkanFilterContext vkctx; + + int initialized; + FFVkExecContext *exec; + VulkanPipeline *pl; + + /* Shader updators, must be in the main filter struct */ + VkDescriptorImageInfo input_images[3]; + VkDescriptorImageInfo output_images[3]; + + int planewidth[4], planeheight[4]; + int inplanewidth[4], inplaneheight[4]; + int in, out; + int width, height; + float h_fov, v_fov; + float ih_fov, iv_fov; + float yaw, pitch, roll; + char *rorder; + int rotation_order[3]; + + /* Push constants / options */ + struct { + float flat_range[2]; + float iflat_range[2]; + float rot_mat[4][4]; + } opts; +} V360VulkanContext; + +static const char flat_to_xyz[] = { + C(0, void out_transform(out vec3 v, in ivec2 out_size, in ivec2 pos) ) + C(0, { ) + C(1, vec2 fpos = vec2(pos) + vec2(0.5f, 0.5f); ) + C(1, vec2 p = ((fpos / vec2(out_size)) - 0.5f)*2.0f; ) + C(1, v = vec3(p[0], p[1], 1.f) * vec3(flat_range, 1.f); ) + C(1, v = normalize(v); ) + C(0, } ) +}; + +static const char xyz_to_flat[] = { + C(0, void in_transform(int idx, in vec3 v, in ivec2 pos, in ivec2 in_size) ) + C(0, { ) + C(1, const float r = tan(acos(v[2])); ) + C(1, const float rr = abs(r) < 1e+6f ? r : length(in_size); ) + C(1, const float h = length(vec2(v[0], v[1])); ) + C(1, const float c = h <= 1e-6f ? 1.f : rr / h; ) + C(1, vec2 p = vec2(v[0], v[1]) / iflat_range * c; ) + C(1, p = IS_WITHIN(abs(p), vec2(1.f)) ? (p/2.0f)+0.5f:vec2(0.f); ) + C(1, p = v[2] >= 0.f ? p : vec2(0.f); ) + C(1, vec4 res = texture(input_img[idx], p); ) + C(1, imageStore(output_img[idx], pos, res); ) + C(0, } ) +}; + +static const char equirect_to_xyz[] = { + C(0, void out_transform(out vec3 v, in ivec2 out_size, in ivec2 pos) ) + C(0, { ) + C(1, vec2 fpos = 2.f * vec2(pos) + 0.5f; ) + C(1, vec2 p = fpos / vec2(out_size) - 1.f; ) + C(1, p = vec2(p[0] * PI, p[1] * PI_2); ) + C(1, v = vec3(cos(p[1]) * sin(p[0]), sin(p[1]), cos(p[1])*cos(p[0])); ) + C(0, } ) +}; + +static const char xyz_to_equirect[] = { + C(0, void in_transform(int idx, in vec3 v, in ivec2 pos, in ivec2 in_size) ) + C(0, { ) + C(1, vec2 p = vec2(atan(v[0], v[2]) / PI, asin(v[1]) / PI_2); ) + C(1, vec4 res = texture(input_img[idx], (p/2.0f) + 0.5f); ) + C(1, imageStore(output_img[idx], pos, res); ) + C(0, } ) +}; + +static const char stereographic_to_xyz[] = { + C(0, void out_transform(out vec3 v, in ivec2 out_size, in ivec2 pos) ) + C(0, { ) + C(1, vec2 fpos = vec2(pos) + vec2(0.5f, 0.5f); ) + C(1, vec2 p = (fpos / vec2(out_size) - 0.5f) * 2.0f * flat_range; ) + C(1, const float r = length(p); ) + C(1, const float theta = atan(r) * 2.0f; ) + C(1, v = vec3(p[0] / r*sin(theta), p[1] / r*sin(theta), cos(theta)); ) + C(1, v = normalize(v); ) + C(0, } ) +}; + +static const char xyz_to_stereographic[] = { + C(0, void in_transform(int idx, in vec3 v, in ivec2 pos, in ivec2 in_size) ) + C(0, { ) + C(1, const float theta = acos(v[2]); ) + C(1, const float r = tan(theta * 0.5f); ) + C(1, const vec2 c = (r / length(vec2(v[0], v[1]))) / iflat_range; ) + C(1, vec2 p = vec2(v[0], v[1]) * c; ) + C(1, p = IS_WITHIN(abs(p), vec2(1.f)) ? (p/2.0f)+0.5f:vec2(0.f); ) + C(1, vec4 res = texture(input_img[idx], p); ) + C(1, imageStore(output_img[idx], pos, res); ) + C(0, } ) +}; + +static const char fisheye_to_xyz[] = { + C(0, void out_transform(out vec3 v, in ivec2 out_size, in ivec2 pos) ) + C(0, { ) + C(1, vec2 fpos = vec2(pos) + vec2(0.5f, 0.5f); ) + C(1, vec2 p = (fpos / vec2(out_size) - 0.5f) * 2.0f * flat_range; ) + C(1, const float r = length(p); ) + C(1, const float phi = atan(p[1], p[0]); ) + C(1, const float theta = (1.f - r) * PI_2; ) + C(1, v = vec3(cos(theta)*cos(phi), cos(theta)*sin(phi), sin(theta)); ) + C(1, v = normalize(v); ) + C(0, } ) +}; + +static const char xyz_to_fisheye[] = { + C(0, void in_transform(int idx, in vec3 v, in ivec2 pos, in ivec2 in_size) ) + C(0, { ) + C(1, const float h = length(vec2(v[0], v[1])); ) + C(1, const float lh = h > 0.f ? h / 2.f : 1.f; ) + C(1, const float phi = atan(h, v[2]) / PI; ) + C(1, vec2 p = vec2(v[0], v[1]) * phi / lh / iflat_range; ) + C(1, p = (length(p) <= 1.f) ? (p/2.0f)+0.5f:vec2(0.f); ) + C(1, vec4 res = texture(input_img[idx], p); ) + C(1, imageStore(output_img[idx], pos, res); ) + C(0, } ) +}; + +static const char dfisheye_to_xyz[] = { + C(0, void out_transform(out vec3 v, in ivec2 out_size, in ivec2 pos) ) + C(0, { ) + C(1, const float m = pos[0] >= out_size[0] / 2 ? 1.f : -1.f; ) + C(1, vec2 npos = m == 1.f ? vec2(out_size[0] / 2, 0.f) : vec2(0.f); ) + C(1, vec2 fpos = vec2(pos) - npos + vec2(0.5f, 0.5f); ) + C(1, vec2 osize = vec2(out_size) * vec2(0.5f, 1.f); ) + C(1, vec2 p = (fpos / osize - 0.5f) * 2.0f * flat_range; ) + C(1, const float h = length(p); ) + C(1, const float lh = h > 0.f ? h : 1.f; ) + C(1, const float theta = m * PI_2 * (1.f - h); ) + C(1, p = p / lh; ) + C(1, v = vec3(cos(theta)*m*p[0], cos(theta)*p[1], sin(theta)); ) + C(1, v = normalize(v); ) + C(0, } ) +}; + +static const char xyz_to_dfisheye[] = { + C(0, void in_transform(int idx, in vec3 v, in ivec2 pos, in ivec2 in_size) ) + C(0, { ) + C(1, const float h = length(vec2(v[0], v[1])); ) + C(1, const float lh = h > 0.f ? h : 1.f; ) + C(1, const float theta = acos(abs(v[2])) / PI; ) + C(1, vec2 p = (vec2(v[0], v[1]) * theta)/lh/iflat_range + 0.5f; ) + C(1, p = p * vec2(0.5f, 1.f); ) + C(1, p = v[2] >= 0.f ? vec2(p[0]+0.5f, p[1]) : vec2(0.5f-p[0], p[1]); ) + C(1, vec4 res = texture(input_img[idx], p); ) + C(1, imageStore(output_img[idx], pos, res); ) + C(0, } ) +}; + +static void multiply_matrix(float c[4][4], const float a[4][4], const float b[4][4]) +{ + for (int i = 0; i < 3; i++) { + for (int j = 0; j < 3; j++) { + float sum = 0.f; + + for (int k = 0; k < 3; k++) + sum += a[i][k] * b[k][j]; + + c[i][j] = sum; + } + } +} + +static inline void calculate_rotation_matrix(float yaw, float pitch, float roll, + float rot_mat[4][4], + const int rotation_order[3]) +{ + const float yaw_rad = yaw * M_PI / 180.f; + const float pitch_rad = pitch * M_PI / 180.f; + const float roll_rad = roll * M_PI / 180.f; + + const float sin_yaw = sinf(yaw_rad); + const float cos_yaw = cosf(yaw_rad); + const float sin_pitch = sinf(pitch_rad); + const float cos_pitch = cosf(pitch_rad); + const float sin_roll = sinf(roll_rad); + const float cos_roll = cosf(roll_rad); + + float m[3][4][4]; + float temp[4][4]; + + m[0][0][0] = cos_yaw; m[0][0][1] = 0; m[0][0][2] = sin_yaw; + m[0][1][0] = 0; m[0][1][1] = 1; m[0][1][2] = 0; + m[0][2][0] = -sin_yaw; m[0][2][1] = 0; m[0][2][2] = cos_yaw; + + m[1][0][0] = 1; m[1][0][1] = 0; m[1][0][2] = 0; + m[1][1][0] = 0; m[1][1][1] = cos_pitch; m[1][1][2] = -sin_pitch; + m[1][2][0] = 0; m[1][2][1] = sin_pitch; m[1][2][2] = cos_pitch; + + m[2][0][0] = cos_roll; m[2][0][1] = -sin_roll; m[2][0][2] = 0; + m[2][1][0] = sin_roll; m[2][1][1] = cos_roll; m[2][1][2] = 0; + m[2][2][0] = 0; m[2][2][1] = 0; m[2][2][2] = 1; + + multiply_matrix(temp, m[rotation_order[0]], m[rotation_order[1]]); + multiply_matrix(rot_mat, temp, m[rotation_order[2]]); +} + +static void set_dimensions(int *outw, int *outh, int w, int h, const AVPixFmtDescriptor *desc) +{ + outw[1] = outw[2] = FF_CEIL_RSHIFT(w, desc->log2_chroma_w); + outw[0] = outw[3] = w; + outh[1] = outh[2] = FF_CEIL_RSHIFT(h, desc->log2_chroma_h); + outh[0] = outh[3] = h; +} + +static av_cold int init_filter(AVFilterContext *ctx, AVFrame *in) +{ + AVFilterLink *outlink = ctx->outputs[0]; + V360VulkanContext *s = ctx->priv; + int err; + + /* Create a sampler */ + VkSampler *sampler = ff_vk_init_sampler(ctx, 0, VK_FILTER_LINEAR); + if (!sampler) + return AVERROR_EXTERNAL; + + s->vkctx.queue_family_idx = s->vkctx.hwctx->queue_family_comp_index; + s->vkctx.queue_count = GET_QUEUE_COUNT(s->vkctx.hwctx, 0, 1, 0); + s->vkctx.cur_queue_idx = av_get_random_seed() % s->vkctx.queue_count; + + s->pl = ff_vk_create_pipeline(ctx); + if (!s->pl) + return AVERROR(ENOMEM); + + switch (s->out) { + case FLAT: + s->opts.flat_range[0] = tanf(0.5f * s->h_fov * M_PI / 180.f); + s->opts.flat_range[1] = tanf(0.5f * s->v_fov * M_PI / 180.f); + break; + case STEREOGRAPHIC: + s->opts.flat_range[0] = tanf(FFMIN(s->h_fov, 359.f) * M_PI / 720.f); + s->opts.flat_range[1] = tanf(FFMIN(s->v_fov, 359.f) * M_PI / 720.f); + break; + case DUAL_FISHEYE: + case FISHEYE: + s->opts.flat_range[0] = s->h_fov / 180.f; + s->opts.flat_range[1] = s->v_fov / 180.f; + break; + } + + switch (s->in) { + case FLAT: + s->opts.iflat_range[0] = tanf(0.5f * s->ih_fov * M_PI / 180.f); + s->opts.iflat_range[1] = tanf(0.5f * s->iv_fov * M_PI / 180.f); + break; + case STEREOGRAPHIC: + s->opts.iflat_range[0] = tanf(FFMIN(s->ih_fov, 359.f) * M_PI / 720.f); + s->opts.iflat_range[1] = tanf(FFMIN(s->iv_fov, 359.f) * M_PI / 720.f); + break; + case DUAL_FISHEYE: + case FISHEYE: + s->opts.iflat_range[0] = s->ih_fov / 180.f; + s->opts.iflat_range[1] = s->iv_fov / 180.f; + break; + } + + s->rotation_order[0] = YAW; + s->rotation_order[1] = PITCH; + s->rotation_order[2] = ROLL; + + { /* Create the shader */ + const int planes = av_pix_fmt_count_planes(s->vkctx.output_format); + const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(s->vkctx.output_format); + + set_dimensions(s->inplanewidth, s->inplaneheight, in->width, in->height, desc); + set_dimensions(s->planewidth, s->planeheight, outlink->w, outlink->h, desc); + calculate_rotation_matrix(s->yaw, s->pitch, s->roll, s->opts.rot_mat, s->rotation_order); + + VulkanDescriptorSetBinding desc_i[2] = { + { + .name = "input_img", + .type = VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER, + .dimensions = 2, + .elems = planes, + .stages = VK_SHADER_STAGE_COMPUTE_BIT, + .updater = s->input_images, + .samplers = DUP_SAMPLER_ARRAY4(*sampler), + }, + { + .name = "output_img", + .type = VK_DESCRIPTOR_TYPE_STORAGE_IMAGE, + .mem_layout = ff_vk_shader_rep_fmt(s->vkctx.output_format), + .mem_quali = "writeonly", + .dimensions = 2, + .elems = planes, + .stages = VK_SHADER_STAGE_COMPUTE_BIT, + .updater = s->output_images, + }, + }; + + SPIRVShader *shd = ff_vk_init_shader(ctx, s->pl, "v360_compute", + VK_SHADER_STAGE_COMPUTE_BIT); + if (!shd) + return AVERROR(ENOMEM); + + ff_vk_set_compute_shader_sizes(ctx, shd, CGROUPS); + + GLSLC(0, layout(push_constant, std430) uniform pushConstants { ); + GLSLC(1, vec2 flat_range; ); + GLSLC(1, vec2 iflat_range; ); + GLSLC(1, mat4 rot_mat; ); + GLSLC(0, }; ); + GLSLC(0, ); + + ff_vk_add_push_constant(ctx, s->pl, 0, sizeof(s->opts), + VK_SHADER_STAGE_COMPUTE_BIT); + + RET(ff_vk_add_descriptor_set(ctx, s->pl, shd, desc_i, 2, 0)); /* set 0 */ + + GLSLF(0, #define PI (%f) ,M_PI); + GLSLF(0, #define PI_2 (%f) ,M_PI_2); + GLSLF(0, #define SQRT2 (%f) ,M_SQRT2); + GLSLF(0, #define inplanewidth ivec4(%i, %i, %i, %i), s->inplanewidth[0], + s->inplanewidth[1], + s->inplanewidth[2], + s->inplanewidth[3]); + GLSLF(0, #define inplaneheight ivec4(%i, %i, %i, %i), s->inplaneheight[0], + s->inplaneheight[1], + s->inplaneheight[2], + s->inplaneheight[3]); + + switch (s->out) { + case FLAT: + GLSLD(flat_to_xyz); + break; + case EQUIRECTANGULAR: + GLSLD(equirect_to_xyz); + break; + case STEREOGRAPHIC: + GLSLD(stereographic_to_xyz); + break; + case FISHEYE: + GLSLD(fisheye_to_xyz); + break; + case DUAL_FISHEYE: + GLSLD(dfisheye_to_xyz); + break; + } + + switch (s->in) { + case FLAT: + GLSLD(xyz_to_flat); + break; + case EQUIRECTANGULAR: + GLSLD(xyz_to_equirect); + break; + case STEREOGRAPHIC: + GLSLD(xyz_to_stereographic); + break; + case FISHEYE: + GLSLD(xyz_to_fisheye); + break; + case DUAL_FISHEYE: + GLSLD(xyz_to_dfisheye); + break; + } + + GLSLC(0, void main() ); + GLSLC(0, { ); + GLSLC(1, ivec2 pos = ivec2(gl_GlobalInvocationID.xy); ); + GLSLC(1, vec3 vector; ); + GLSLF(1, int planes = %i; ,planes); + GLSLC(1, for (int i = 0; i < planes; i++) { ); + GLSLC(2, ivec2 out_size = imageSize(output_img[i]); ); + GLSLC(2, ivec2 in_size = ivec2(inplanewidth[i],inplaneheight[i]);); + GLSLC(2, out_transform(vector, out_size, pos); ); + GLSLC(2, vector = normalize((rot_mat * vec4(vector, 1.f)).xyz); ); + GLSLC(2, in_transform(i, vector, pos, in_size); ); + GLSLC(1, } ); + GLSLC(0, } ); + + RET(ff_vk_compile_shader(ctx, shd, "main")); + } + + RET(ff_vk_init_pipeline_layout(ctx, s->pl)); + RET(ff_vk_init_compute_pipeline(ctx, s->pl)); + + /* Execution context */ + RET(ff_vk_create_exec_ctx(ctx, &s->exec)); + + s->initialized = 1; + + return 0; + +fail: + return err; +} + +static int process_frames(AVFilterContext *avctx, AVFrame *out_f, AVFrame *in_f) +{ + int err = 0; + VkCommandBuffer cmd_buf; + V360VulkanContext *s = avctx->priv; + AVVkFrame *in = (AVVkFrame *)in_f->data[0]; + AVVkFrame *out = (AVVkFrame *)out_f->data[0]; + int planes = av_pix_fmt_count_planes(s->vkctx.output_format); + + /* Update descriptors and init the exec context */ + ff_vk_start_exec_recording(avctx, s->exec); + cmd_buf = ff_vk_get_exec_buf(avctx, s->exec); + + for (int i = 0; i < planes; i++) { + RET(ff_vk_create_imageview(avctx, s->exec, &s->input_images[i].imageView, in->img[i], + av_vkfmt_from_pixfmt(s->vkctx.input_format)[i], + ff_comp_identity_map)); + + RET(ff_vk_create_imageview(avctx, s->exec, &s->output_images[i].imageView, out->img[i], + av_vkfmt_from_pixfmt(s->vkctx.output_format)[i], + ff_comp_identity_map)); + + s->input_images[i].imageLayout = VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL; + s->output_images[i].imageLayout = VK_IMAGE_LAYOUT_GENERAL; + } + + ff_vk_update_descriptor_set(avctx, s->pl, 0); + + for (int i = 0; i < planes; i++) { + VkImageMemoryBarrier bar[2] = { + { + .sType = VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER, + .srcAccessMask = 0, + .dstAccessMask = VK_ACCESS_SHADER_READ_BIT, + .oldLayout = in->layout[i], + .newLayout = s->input_images[i].imageLayout, + .srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, + .dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, + .image = in->img[i], + .subresourceRange.aspectMask = VK_IMAGE_ASPECT_COLOR_BIT, + .subresourceRange.levelCount = 1, + .subresourceRange.layerCount = 1, + }, + { + .sType = VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER, + .srcAccessMask = 0, + .dstAccessMask = VK_ACCESS_SHADER_WRITE_BIT, + .oldLayout = out->layout[i], + .newLayout = s->output_images[i].imageLayout, + .srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, + .dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED, + .image = out->img[i], + .subresourceRange.aspectMask = VK_IMAGE_ASPECT_COLOR_BIT, + .subresourceRange.levelCount = 1, + .subresourceRange.layerCount = 1, + }, + }; + + vkCmdPipelineBarrier(cmd_buf, VK_PIPELINE_STAGE_TOP_OF_PIPE_BIT, + VK_PIPELINE_STAGE_COMPUTE_SHADER_BIT, 0, + 0, NULL, 0, NULL, FF_ARRAY_ELEMS(bar), bar); + + in->layout[i] = bar[0].newLayout; + in->access[i] = bar[0].dstAccessMask; + + out->layout[i] = bar[1].newLayout; + out->access[i] = bar[1].dstAccessMask; + } + + ff_vk_bind_pipeline_exec(avctx, s->exec, s->pl); + + ff_vk_update_push_exec(avctx, s->exec, VK_SHADER_STAGE_COMPUTE_BIT, + 0, sizeof(s->opts), &s->opts); + + vkCmdDispatch(cmd_buf, + FFALIGN(s->vkctx.output_width, CGROUPS[0])/CGROUPS[0], + FFALIGN(s->vkctx.output_height, CGROUPS[1])/CGROUPS[1], 1); + + ff_vk_add_exec_dep(avctx, s->exec, in_f, VK_PIPELINE_STAGE_TOP_OF_PIPE_BIT); + ff_vk_add_exec_dep(avctx, s->exec, out_f, VK_PIPELINE_STAGE_TOP_OF_PIPE_BIT); + + err = ff_vk_submit_exec_queue(avctx, s->exec); + if (err) + return err; + +fail: + ff_vk_discard_exec_deps(avctx, s->exec); + return err; +} + +static int v360_vulkan_filter_frame(AVFilterLink *link, AVFrame *in) +{ + int err; + AVFilterContext *ctx = link->dst; + V360VulkanContext *s = ctx->priv; + AVFilterLink *outlink = ctx->outputs[0]; + + AVFrame *out = ff_get_video_buffer(outlink, outlink->w, outlink->h); + if (!out) { + err = AVERROR(ENOMEM); + goto fail; + } + + if (!s->initialized) + RET(init_filter(ctx, in)); + + RET(process_frames(ctx, out, in)); + + err = av_frame_copy_props(out, in); + if (err < 0) + goto fail; + + av_frame_free(&in); + + return ff_filter_frame(outlink, out); + +fail: + av_frame_free(&in); + av_frame_free(&out); + return err; +} + +static int v360_vulkan_config_output(AVFilterLink *outlink) +{ + AVFilterContext *avctx = outlink->src; + V360VulkanContext *s = avctx->priv; + AVFilterLink *inlink = avctx->inputs[0]; + int err; + + if (s->width > 0 && s->height > 0) { + s->vkctx.output_width = s->width; + s->vkctx.output_height = s->height; + } + + s->vkctx.output_format = s->vkctx.input_format; + + err = ff_vk_filter_config_output(outlink); + if (err < 0) + return err; + + outlink->sample_aspect_ratio = inlink->sample_aspect_ratio; + + return 0; +} + +static void v360_vulkan_uninit(AVFilterContext *avctx) +{ + V360VulkanContext *s = avctx->priv; + + ff_vk_filter_uninit(avctx); + + s->initialized = 0; +} + +#define OFFSET(x) offsetof(V360VulkanContext, x) +#define FLAGS (AV_OPT_FLAG_FILTERING_PARAM | AV_OPT_FLAG_VIDEO_PARAM) +static const AVOption v360_vulkan_options[] = { + { "input", "set input projection", OFFSET(in), AV_OPT_TYPE_INT, {.i64=EQUIRECTANGULAR}, 0, NB_PROJECTIONS-1, FLAGS, "in" }, + { "e", "equirectangular", 0, AV_OPT_TYPE_CONST, {.i64=EQUIRECTANGULAR}, 0, 0, FLAGS, "in" }, + { "equirect", "equirectangular", 0, AV_OPT_TYPE_CONST, {.i64=EQUIRECTANGULAR}, 0, 0, FLAGS, "in" }, + { "flat", "regular video", 0, AV_OPT_TYPE_CONST, {.i64=FLAT}, 0, 0, FLAGS, "in" }, + { "dfisheye", "dual fisheye", 0, AV_OPT_TYPE_CONST, {.i64=DUAL_FISHEYE}, 0, 0, FLAGS, "in" }, + { "sg", "stereographic", 0, AV_OPT_TYPE_CONST, {.i64=STEREOGRAPHIC}, 0, 0, FLAGS, "in" }, + { "fisheye", "fisheye", 0, AV_OPT_TYPE_CONST, {.i64=FISHEYE} , 0, 0, FLAGS, "in" }, + { "output", "set output projection", OFFSET(out), AV_OPT_TYPE_INT, {.i64=FLAT}, 0, NB_PROJECTIONS-1, FLAGS, "out" }, + { "e", "equirectangular", 0, AV_OPT_TYPE_CONST, {.i64=EQUIRECTANGULAR}, 0, 0, FLAGS, "out" }, + { "equirect", "equirectangular", 0, AV_OPT_TYPE_CONST, {.i64=EQUIRECTANGULAR}, 0, 0, FLAGS, "out" }, + { "flat", "regular video", 0, AV_OPT_TYPE_CONST, {.i64=FLAT}, 0, 0, FLAGS, "out" }, + { "dfisheye", "dual fisheye", 0, AV_OPT_TYPE_CONST, {.i64=DUAL_FISHEYE}, 0, 0, FLAGS, "out" }, + { "sg", "stereographic", 0, AV_OPT_TYPE_CONST, {.i64=STEREOGRAPHIC}, 0, 0, FLAGS, "out" }, + { "fisheye", "fisheye", 0, AV_OPT_TYPE_CONST, {.i64=FISHEYE} , 0, 0, FLAGS, "out" }, + { "w", "output width", OFFSET(width), AV_OPT_TYPE_INT, {.i64=0}, 0, INT16_MAX, FLAGS, "w"}, + { "h", "output height", OFFSET(height), AV_OPT_TYPE_INT, {.i64=0}, 0, INT16_MAX, FLAGS, "h"}, + { "yaw", "yaw rotation", OFFSET(yaw), AV_OPT_TYPE_FLOAT, {.dbl=0.f}, -180.f, 180.f, FLAGS, "yaw"}, + { "pitch", "pitch rotation", OFFSET(pitch), AV_OPT_TYPE_FLOAT, {.dbl=0.f}, -180.f, 180.f, FLAGS, "pitch"}, + { "roll", "roll rotation", OFFSET(roll), AV_OPT_TYPE_FLOAT, {.dbl=0.f}, -180.f, 180.f, FLAGS, "roll"}, + { "rorder", "rotation order", OFFSET(rorder), AV_OPT_TYPE_STRING, {.str="ypr"}, 0, 0, FLAGS, "rorder"}, + { "h_fov", "set output horizontal FOV angle", OFFSET(h_fov), AV_OPT_TYPE_FLOAT, {.dbl = 90.0f}, 0.00001f, 360.0f, .flags = FLAGS }, + { "v_fov", "set output vertical FOV angle", OFFSET(v_fov), AV_OPT_TYPE_FLOAT, {.dbl = 45.0f}, 0.00001f, 360.0f, .flags = FLAGS }, + { "ih_fov", "set input horizontal FOV angle", OFFSET(ih_fov), AV_OPT_TYPE_FLOAT, {.dbl = 90.0f}, 0.00001f, 360.0f, .flags = FLAGS }, + { "iv_fov", "set input vertical FOV angle", OFFSET(iv_fov), AV_OPT_TYPE_FLOAT, {.dbl = 45.0f}, 0.00001f, 360.0f, .flags = FLAGS }, + { NULL }, +}; + +AVFILTER_DEFINE_CLASS(v360_vulkan); + +static const AVFilterPad v360_vulkan_inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .filter_frame = &v360_vulkan_filter_frame, + .config_props = &ff_vk_filter_config_input, + }, + { NULL } +}; + +static const AVFilterPad v360_vulkan_outputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .config_props = &v360_vulkan_config_output, + }, + { NULL } +}; + +AVFilter ff_vf_v360_vulkan = { + .name = "v360_vulkan", + .description = NULL_IF_CONFIG_SMALL("Convert 360 projection of video."), + .priv_size = sizeof(V360VulkanContext), + .init = &ff_vk_filter_init, + .uninit = &v360_vulkan_uninit, + .query_formats = &ff_vk_filter_query_formats, + .inputs = v360_vulkan_inputs, + .outputs = v360_vulkan_outputs, + .priv_class = &v360_vulkan_class, + .flags_internal = FF_FILTER_FLAG_HWFRAME_AWARE, +}; -- 2.17.1

5 18

[PATCH] libavutil/timestamp.h: Fix loss of precision in timestamps for silencedetect on long files
by Allan Cady 29 Jun '24

29 Jun '24

When the silencedetect audio filter is run against long files, the output timestamps gradually lose precision as the scan proceeds further into the file. This is because the output format specifier ("%.6g" in libavutil/timestamp.h) limits the total field width to six significant digits. As the offset into the file increases, digits drop off the end, until eventually, for offsets greater than 100000 seconds (about 28 hours), fractions of a second disappear altogether, and the timestamps are logged as whole seconds. This patch changes the format to "%.6f" for silencedetect, which will give microsecond precision for all timestamps regardless of offset. libavutil/timestamp.h exposes a macro, av_ts2timestr, as the public interface. This macro was used by silencedetect.c, as well as other source files. In order to fix the issue for silencedetect without affecting other files and tests, I have added a new macro, av_ts2timestr_fixed_precision, which uses the new format specifier. The original av_ts_make_time_string remains, with the original behavior. timestamp.h also exposes a function, av_ts_make_time_string, which was called only from av_ts2timestr. After this patch, both of the macros now use a new function, av_ts_make_time_string_format, which takes a format specifier as an argument, so the function av_ts_make_time_string is no longer used. I've left it in place, but flagged it for deprecation with FF_API_AV_MAKETIMESTRING. The test reference file filter-metadata-silencedetect has been updated to match the new functionality. Signed-off-by: Allan Cady <allancady(a)yahoo.com> --- libavfilter/af_silencedetect.c | 12 +++---- libavutil/timestamp.h | 34 +++++++++++++++++--- libavutil/version.h | 1 + tests/ref/fate/filter-metadata-silencedetect | 2 +- 4 files changed, 38 insertions(+), 11 deletions(-) diff --git a/libavfilter/af_silencedetect.c b/libavfilter/af_silencedetect.c index 845c65bfed..f1a8096540 100644 --- a/libavfilter/af_silencedetect.c +++ b/libavfilter/af_silencedetect.c @@ -86,11 +86,11 @@ static av_always_inline void update(SilenceDetectContext *s, AVFrame *insamples, s->start[channel] = insamples->pts + av_rescale_q(current_sample / s->channels + 1 - nb_samples_notify * s->independent_channels / s->channels, (AVRational){ 1, s->last_sample_rate }, time_base); set_meta(insamples, s->mono ? channel + 1 : 0, "silence_start", - av_ts2timestr(s->start[channel], &time_base)); + av_ts2timestr_fixed_precision(s->start[channel], &time_base)); if (s->mono) av_log(s, AV_LOG_INFO, "channel: %d | ", channel); av_log(s, AV_LOG_INFO, "silence_start: %s\n", - av_ts2timestr(s->start[channel], &time_base)); + av_ts2timestr_fixed_precision(s->start[channel], &time_base)); } } } else { @@ -101,15 +101,15 @@ static av_always_inline void update(SilenceDetectContext *s, AVFrame *insamples, int64_t duration_ts = end_pts - s->start[channel]; if (insamples) { set_meta(insamples, s->mono ? channel + 1 : 0, "silence_end", - av_ts2timestr(end_pts, &time_base)); + av_ts2timestr_fixed_precision(end_pts, &time_base)); set_meta(insamples, s->mono ? channel + 1 : 0, "silence_duration", - av_ts2timestr(duration_ts, &time_base)); + av_ts2timestr_fixed_precision(duration_ts, &time_base)); } if (s->mono) av_log(s, AV_LOG_INFO, "channel: %d | ", channel); av_log(s, AV_LOG_INFO, "silence_end: %s | silence_duration: %s\n", - av_ts2timestr(end_pts, &time_base), - av_ts2timestr(duration_ts, &time_base)); + av_ts2timestr_fixed_precision(end_pts, &time_base), + av_ts2timestr_fixed_precision(duration_ts, &time_base)); } s->nb_null_samples[channel] = 0; s->start[channel] = INT64_MIN; diff --git a/libavutil/timestamp.h b/libavutil/timestamp.h index 9ae64da8a1..b483b5e12d 100644 --- a/libavutil/timestamp.h +++ b/libavutil/timestamp.h @@ -31,6 +31,8 @@ #endif #define AV_TS_MAX_STRING_SIZE 32 +#define AV_TS_FMT_DEFAULT "%.6g" +#define AV_TS_FMT_FIXED_PRECISION_6 "%.6f" /** * Fill the provided buffer with a string containing a timestamp @@ -53,9 +55,14 @@ static inline char *av_ts_make_string(char *buf, int64_t ts) */ #define av_ts2str(ts) av_ts_make_string((char[AV_TS_MAX_STRING_SIZE]){0}, ts) +#if FF_API_AV_MAKETIMESTRING /** + * This function is probably deprecated. It was originally called by + * av_ts_make_time_string defined below, which now uses av_ts_make_time_string_format. + * instead. Nothing external references this function directly. + * * Fill the provided buffer with a string containing a timestamp time - * representation. + * representation, using the default format AV_TS_FMT_DEFAULT. * * @param buf a buffer with size in bytes of at least AV_TS_MAX_STRING_SIZE * @param ts the timestamp to represent @@ -65,14 +72,33 @@ static inline char *av_ts_make_string(char *buf, int64_t ts) static inline char *av_ts_make_time_string(char *buf, int64_t ts, AVRational *tb) { if (ts == AV_NOPTS_VALUE) snprintf(buf, AV_TS_MAX_STRING_SIZE, "NOPTS"); - else snprintf(buf, AV_TS_MAX_STRING_SIZE, "%.6g", av_q2d(*tb) * ts); + else snprintf(buf, AV_TS_MAX_STRING_SIZE, AV_TS_FMT_DEFAULT, av_q2d(*tb) * ts); return buf; } +#endif /** - * Convenience macro, the return value should be used only directly in + * Fill the provided buffer with a string containing a timestamp time + * representation. + * + * @param buf a buffer with size in bytes of at least AV_TS_MAX_STRING_SIZE + * @param ts the timestamp to represent + * @param tb the timebase of the timestamp + * @param format format string for timestamp output, e.g. AV_TS_FMT_DEFAULT. + * @return the buffer in input + */ +static inline char *av_ts_make_time_string_format(char *buf, int64_t ts, AVRational *tb, const char *format) +{ + if (ts == AV_NOPTS_VALUE) snprintf(buf, AV_TS_MAX_STRING_SIZE, "NOPTS"); + else snprintf(buf, AV_TS_MAX_STRING_SIZE, format, av_q2d(*tb) * ts); + return buf; +} + +/** + * Convenience macros, the return values should be used only directly in * function arguments but never stand-alone. */ -#define av_ts2timestr(ts, tb) av_ts_make_time_string((char[AV_TS_MAX_STRING_SIZE]){0}, ts, tb) +#define av_ts2timestr(ts, tb) av_ts_make_time_string_format((char[AV_TS_MAX_STRING_SIZE]){0}, ts, tb, AV_TS_FMT_DEFAULT) +#define av_ts2timestr_fixed_precision(ts, tb) av_ts_make_time_string_format((char[AV_TS_MAX_STRING_SIZE]){0}, ts, tb, AV_TS_FMT_FIXED_PRECISION_6) #endif /* AVUTIL_TIMESTAMP_H */ diff --git a/libavutil/version.h b/libavutil/version.h index 9f45af93df..a2daeda772 100644 --- a/libavutil/version.h +++ b/libavutil/version.h @@ -119,6 +119,7 @@ #define FF_API_FRAME_KEY (LIBAVUTIL_VERSION_MAJOR < 59) #define FF_API_PALETTE_HAS_CHANGED (LIBAVUTIL_VERSION_MAJOR < 59) #define FF_API_VULKAN_CONTIGUOUS_MEMORY (LIBAVUTIL_VERSION_MAJOR < 59) +#define FF_API_AV_MAKETIMESTRING (LIBAVUTIL_VERSION_MAJOR < 59) /** * @} diff --git a/tests/ref/fate/filter-metadata-silencedetect b/tests/ref/fate/filter-metadata-silencedetect index bc53fea047..e66ffe5fdd 100644 --- a/tests/ref/fate/filter-metadata-silencedetect +++ b/tests/ref/fate/filter-metadata-silencedetect @@ -1,5 +1,5 @@ pts=0|tag:lavfi.silence_duration=0.523107|tag:lavfi.silence_end=0.690023|tag:lavfi.silence_start=0.736417 -pts=46080|tag:lavfi.silence_start=1.27626|tag:lavfi.silence_end=1.80751|tag:lavfi.silence_duration=0.531247 +pts=46080|tag:lavfi.silence_start=1.276259|tag:lavfi.silence_end=1.807506|tag:lavfi.silence_duration=0.531247 pts=92160 pts=138240 pts=184320 -- 2.25.1

6 36

[PATCH] tools/target_dec_fuzzer: Adjust threshold for MV30
by Michael Niedermayer 25 Jun '24

25 Jun '24

Fixes: 60867/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_MV30_fuzzer-6381933108527104 Fixes: Timeout Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg Signed-off-by: Michael Niedermayer <michael(a)niedermayer.cc> --- tools/target_dec_fuzzer.c | 1 + 1 file changed, 1 insertion(+) diff --git a/tools/target_dec_fuzzer.c b/tools/target_dec_fuzzer.c index 62085cf0803..1c8dde3cf94 100644 --- a/tools/target_dec_fuzzer.c +++ b/tools/target_dec_fuzzer.c @@ -261,6 +261,7 @@ int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) { case AV_CODEC_ID_MSS2: maxpixels /= 16384; break; case AV_CODEC_ID_MSZH: maxpixels /= 128; break; case AV_CODEC_ID_MTS2: maxpixels /= 4096; break; + case AV_CODEC_ID_MV30: maxpixels /= 128; break; case AV_CODEC_ID_MVC2: maxpixels /= 128; break; case AV_CODEC_ID_MVHA: maxpixels /= 16384; break; case AV_CODEC_ID_MVDV: maxpixels /= 1024; break; -- 2.17.1

1 1

[PATCH 1/2] avformat/avr: Check sample rate
by Michael Niedermayer 25 Jun '24

25 Jun '24

Fixes: 54979/clusterfuzz-testcase-minimized-ffmpeg_dem_AVR_fuzzer-6681035461230592 Fixes: Timeout Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg Signed-off-by: Michael Niedermayer <michael(a)niedermayer.cc> --- libavformat/avr.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/libavformat/avr.c b/libavformat/avr.c index 1cc4d56bfb..3fe8614b25 100644 --- a/libavformat/avr.c +++ b/libavformat/avr.c @@ -75,6 +75,9 @@ static int avr_read_header(AVFormatContext *s) avio_skip(s->pb, 20); avio_skip(s->pb, 64); + if (st->codecpar->sample_rate == 0) + return AVERROR_INVALIDDATA; + st->codecpar->codec_id = ff_get_pcm_codec_id(bps, 0, 1, sign); if (st->codecpar->codec_id == AV_CODEC_ID_NONE) { avpriv_request_sample(s, "Bps %d and sign %d", bps, sign); -- 2.17.1

2 4