[FFmpeg-devel] avformat_seek_file in H265 video seeks beyond the max_ts passed in?

[A]

I apologize for the spam but I realized that this ML is mostly about
patches. So I created a ticket here:

https://trac.ffmpeg.org/ticket/11137

Errata for the previous email: the seek timestamps should not be doubles
but int64_t so the correct max_ts is 5120, not 0.5.

On Mon, Aug 12, 2024 at 4:24 PM A <ahmadsharif at gmail.com> wrote:

> Hi,
>
> My understanding is that avformat_seek_file() with these parameters:
>
> avformat_seek_file(format_context, 0, INT64_MIN, timestamp, timestamp)
>
> should seek in the video to an I-Frame that is strictly <= timestamp
> (because ts=timestamp and max_ts=timestamp).
>
> However, the observed behavior that I see is that for certain H265 videos,
> FFMPEG seeks beyond the timestamp passed in. To repro this behavior I ran
> these commands:
>
> # Create a clean conda environment for testing purposes
> conda create --name test
> conda activate test
> conda install -c conda-forge x265
>
> # Install some build pre-requisites
> conda install pkg-config
>
> # Build ffmpeg from source with x265 enabled
> git clone https://github.com/FFmpeg/FFmpeg.git
> ./configure --enable-nonfree --enable-gpl --prefix=$(readlink -f ../bin)
> --enable-libx265  --enable-rpath
> --extra-ldflags=-Wl,-rpath=$CONDA_PREFIX/lib --enable-filter=drawtext
> --enable-libfontconfig --enable-libfreetype --enable-libharfbuzz
> make -j install
>
> # Now generate a video with just frame numbers in the text per frame:
> ffmpeg -f lavfi -i color=size=128x128:duration=1:rate=10:color=blue -vf
> "drawtext=fontsize=30:fontcolor=white:x=(w-text_w)/2:y=(h-text_h)/2:text='Frame
> %{frame_num}'" -vcodec libx265 -pix_fmt yuv420p -g 2 -crf 10 test.mp4 -y
>
> Note that this video has 10 frames. ffprobe shows the following:
>
> ffprobe -v error -select_streams v:0 -show_entries
> frame=pts,pts_time,duration,pkt_pts_time,pkt_duration,key_frame -of csv
> test.mp4
> frame,1,0,0.000000,1024,1024,
> frame,0,1024,0.100000,1024,1024
> frame,1,2048,0.200000,1024,1024
> frame,0,3072,0.300000,1024,1024
> frame,1,4096,0.400000,1024,1024
> frame,0,5120,0.500000,1024,1024
> frame,1,6144,0.600000,1024,1024
> frame,0,7168,0.700000,1024,1024
> frame,1,8192,0.800000,1024,1024
> frame,0,9216,0.900000,1024,1024
>
> Now, when I open this video using FFMPEG as a library, I get an
> AVFormatContext. I want to decode the frame with pts=0.5. So I call
> avformat_seek_file with min_ts=-INT64_MAX, ts=0.5 and max_ts=0.5.
>
> I expect that FFMPEG will seek to the frame with pts=0.4 so I can then
> decode forward and eventually get frame with pts=0.5 with
> avcodec_receive_frame(), but it seems like the first frame that I get from
> avcodec_receive_frame() is the one with pts=0.6.
>
> More context:
> I am writing a library that wraps FFMPEG and returns frames at arbitrary
> timestamps. The full source code of the library is here:
> https://github.com/pytorch/torchcodec. The pull-request that reproduces
> this exact scenario is here:
> https://github.com/pytorch/torchcodec/pull/178.
>
> It would be nice if FFMPEG always seeked to a frame with pts <= the
> max_pts passed into avformat_seek_file. This normally does work with other
> codecs. Am I calling the library wrong? Should I be calling
> avformat_seek_file() with other flags? The documentation of
> avformat_seek_file is here:
>
>
> https://ffmpeg.org/doxygen/7.0/group__lavf__decoding.html#ga3b40fc8d2fda6992ae6ea2567d71ba30
>
> Here is the seek call in my code:
>
>
> https://github.com/pytorch/torchcodec/blob/dbfef1223522639d2b036a185b444eecf7748466/src/torchcodec/decoders/_core/VideoDecoder.cpp#L735
>
> I would be happy to file a ticket as well, if that helps. The full repro
> instructions are in this email for reference.
>