HEVC decoding of first frame
Hello everyone, I've got a question regarding the decoding process for HEVC-coded sequences. When timing the process I found that the decoding of the first frame always takes twice as long as the decoding of all the other frames. This seems to be true independent from the resolution. It is also not related to inter coding, as all frames are intra coded with the same QP in the example given below. Other decoders (e.g. HM or libde265) do not show this behavior for the same input sequences. Here is an examplary output for a single frame and for two frames. It is timed using the linux time-function, furthermore single core processing is forced and priority is set to the highest level. System: Ubuntu 12.04, Pandaboard with OMAP4430 SoC Command line for single frame: time -p taskset -c 0 nice -n-30 /root/bin/ffmpeg -i PeopleOnStreet_intra_1frames_QP10.bin -f rawvideo -y /dev/null Output: ffmpeg version 2.1.git Copyright (c) 2000-2014 the FFmpeg developers built on Feb 20 2014 16:27:13 with gcc 4.6 (Ubuntu/Linaro 4.6.3-1ubuntu5) configuration: --prefix=/root/ffmpeg_build --extra-cflags=-I/root/ffmpeg_build/include --extra-ldflags=-L/root/ffmpeg_build/lib --bindir=/root/bin --extra-libs=-ldl --enable-gpl --enable-libass --enable-nonfree libavutil 52. 64.100 / 52. 64.100 libavcodec 55. 52.102 / 55. 52.102 libavformat 55. 33.100 / 55. 33.100 libavdevice 55. 10.100 / 55. 10.100 libavfilter 4. 1.103 / 4. 1.103 libswscale 2. 5.101 / 2. 5.101 libswresample 0. 17.104 / 0. 17.104 libpostproc 52. 3.100 / 52. 3.100 Input #0, hevc, from 'PeopleOnStreet_intra_1frames_QP10.bin': Duration: N/A, bitrate: N/A Stream #0:0: Video: hevc (Main), yuv420p(tv), 2560x1600, 25 tbr, 1200k tbn, 25 tbc Output #0, rawvideo, to '/dev/null': Metadata: encoder : Lavf55.33.100 Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 2560x1600, q=2-31, 200 kb/s, 90k tbn, 25 tbc Stream mapping: Stream #0:0 -> #0:0 (hevc -> rawvideo) Press [q] to stop, [?] for help frame= 1 fps=0.3 q=0.0 size= 6000kB time=00:00:00.04 bitrate=1228800.0kbitframe= 1 fps=0.3 q=0.0 Lsize= 6000kB time=00:00:00.04 bitrate=1228800.0kbits/s video:6000kB audio:0kB subtitle:0 data:0 global headers:0kB muxing overhead 0.000000% real 5.93 user 5.69 sys 0.22 Command line two frames: time -p taskset -c 0 nice -n-30 /root/bin/ffmpeg -i PeopleOnStreet_intra_2frames_QP10.bin -f rawvideo -y /dev/null Output: ffmpeg version 2.1.git Copyright (c) 2000-2014 the FFmpeg developers built on Feb 20 2014 16:27:13 with gcc 4.6 (Ubuntu/Linaro 4.6.3-1ubuntu5) configuration: --prefix=/root/ffmpeg_build --extra-cflags=-I/root/ffmpeg_build/include --extra-ldflags=-L/root/ffmpeg_build/lib --bindir=/root/bin --extra-libs=-ldl --enable-gpl --enable-libass --enable-nonfree libavutil 52. 64.100 / 52. 64.100 libavcodec 55. 52.102 / 55. 52.102 libavformat 55. 33.100 / 55. 33.100 libavdevice 55. 10.100 / 55. 10.100 libavfilter 4. 1.103 / 4. 1.103 libswscale 2. 5.101 / 2. 5.101 libswresample 0. 17.104 / 0. 17.104 libpostproc 52. 3.100 / 52. 3.100 Input #0, hevc, from 'PeopleOnStreet_intra_2frames_QP10.bin': Duration: N/A, bitrate: N/A Stream #0:0: Video: hevc (Main), yuv420p(tv), 2560x1600, 25 tbr, 1200k tbn, 25 tbc Output #0, rawvideo, to '/dev/null': Metadata: encoder : Lavf55.33.100 Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 2560x1600, q=2-31, 200 kb/s, 90k tbn, 25 tbc Stream mapping: Stream #0:0 -> #0:0 (hevc -> rawvideo) Press [q] to stop, [?] for help frame= 2 fps=0.7 q=0.0 size= 12000kB time=00:00:00.08 bitrate=1228800.0kbitframe= 2 fps=0.3 q=0.0 size= 12000kB time=00:00:00.08 bitrate=1228800.0kbitframe= 2 fps=0.3 q=0.0 Lsize= 12000kB time=00:00:00.08 bitrate=1228800.0kbits/s video:12000kB audio:0kB subtitle:0 data:0 global headers:0kB muxing overhead 0.000000% real 8.88 user 8.58 sys 0.28 Apparently, decoding of a single frame takes about 6s, decoding of two frames about 9s. Is it possible that the first frame is always decoded twice in this software? Thanks for your help! Christian Herglotz -- Dipl.-Ing. Dipl.-Wirt.Ing. Christian Herglotz Chair of Multimedia Communications and Signal Processing Friedrich-Alexander University Erlangen-Nürnberg Cauerstr. 7, D-91058 Erlangen, Germany
Hallo Christian, On Thu, Sep 17, 2015 at 10:48:16 +0200, Christian Herglotz wrote:
I've got a question regarding the decoding process for HEVC-coded sequences. When timing the process I found that the decoding of the first frame always takes twice as long as the decoding of all the other frames.
While I'm happy to try to reproduce your problem and ask smart questions, let me please point out:
ffmpeg version 2.1.git Copyright (c) 2000-2014 the FFmpeg developers built on Feb 20 2014 16:27:13 with gcc 4.6 (Ubuntu/Linaro 4.6.3-1ubuntu5)
This version of ffmpeg is extremely old, considering the amount of development happening. Especially with such a "new" codec as HEVC, you can expect many improvements over 18 months. And if ffmpeg happens/happened to have a bug, it may very well have been fixed in the meantime. Nobody here will help trying to identify a bug in such an old version. Please repeat your tests with a very recent version of ffmpeg, best is if you build if from git HEAD. Thanks, Gruß, Moritz
Hello, thanks for the quick answer! I upgraded my ffmpeg to version 2.8 and still observe the same problem as stated before: When timing the process I found that the decoding of the first frame always takes twice as long as the decoding of all the other frames. Output of stream 1: root@herglotz-panda:/ramcache# time -p taskset -c 0 ./ffmpeg -i PeopleOnStreet_intra_1frames_QP10.bin -f rawvideo -y /dev/null ffmpeg version 2.8 Copyright (c) 2000-2015 the FFmpeg developers built with gcc 4.6 (Ubuntu/Linaro 4.6.3-1ubuntu5) configuration: libavutil 54. 31.100 / 54. 31.100 libavcodec 56. 60.100 / 56. 60.100 libavformat 56. 40.101 / 56. 40.101 libavdevice 56. 4.100 / 56. 4.100 libavfilter 5. 40.101 / 5. 40.101 libswscale 3. 1.101 / 3. 1.101 libswresample 1. 2.101 / 1. 2.101 Input #0, hevc, from 'PeopleOnStreet_intra_1frames_QP10.bin': Duration: N/A, bitrate: N/A Stream #0:0: Video: hevc (Main), yuv420p(tv), 2560x1600, 25 tbr, 1200k tbn, 25 tbc Output #0, rawvideo, to '/dev/null': Metadata: encoder : Lavf56.40.101 Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 2560x1600, q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc Metadata: encoder : Lavc56.60.100 rawvideo Stream mapping: Stream #0:0 -> #0:0 (hevc (native) -> rawvideo (native)) Press [q] to stop, [?] for help frame= 1 fps=0.5 q=-0.0 size= 6000kB time=00:00:00.04 bitrate=1228800.0kbiframe= 1 fps=0.5 q=-0.0 Lsize= 6000kB time=00:00:00.04 bitrate=1228800.0kbits/s video:6000kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000% real 4.39 user 4.28 sys 0.10 Output of stream 2: root@herglotz-panda:/ramcache# time -p taskset -c 0 ./ffmpeg -i PeopleOnStreet_intra_2frames_QP10.bin -f rawvideo -y /dev/null ffmpeg version 2.8 Copyright (c) 2000-2015 the FFmpeg developers built with gcc 4.6 (Ubuntu/Linaro 4.6.3-1ubuntu5) configuration: libavutil 54. 31.100 / 54. 31.100 libavcodec 56. 60.100 / 56. 60.100 libavformat 56. 40.101 / 56. 40.101 libavdevice 56. 4.100 / 56. 4.100 libavfilter 5. 40.101 / 5. 40.101 libswscale 3. 1.101 / 3. 1.101 libswresample 1. 2.101 / 1. 2.101 Input #0, hevc, from 'PeopleOnStreet_intra_2frames_QP10.bin': Duration: N/A, bitrate: N/A Stream #0:0: Video: hevc (Main), yuv420p(tv), 2560x1600, 25 tbr, 1200k tbn, 25 tbc Output #0, rawvideo, to '/dev/null': Metadata: encoder : Lavf56.40.101 Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 2560x1600, q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc Metadata: encoder : Lavc56.60.100 rawvideo Stream mapping: Stream #0:0 -> #0:0 (hevc (native) -> rawvideo (native)) Press [q] to stop, [?] for help frame= 2 fps=0.9 q=-0.0 size= 12000kB time=00:00:00.08 bitrate=1228800.0kbiframe= 2 fps=0.5 q=-0.0 size= 12000kB time=00:00:00.08 bitrate=1228800.0kbiframe= 2 fps=0.5 q=-0.0 Lsize= 12000kB time=00:00:00.08 bitrate=1228800.0kbits/s video:12000kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000% real 6.62 user 6.40 sys 0.21 This observation is also holds for sequences with a lower resolution. Thanks for your help! Christian Am 18.09.2015 um 09:39 schrieb Moritz Barsnick:
Hallo Christian,
On Thu, Sep 17, 2015 at 10:48:16 +0200, Christian Herglotz wrote:
I've got a question regarding the decoding process for HEVC-coded sequences. When timing the process I found that the decoding of the first frame always takes twice as long as the decoding of all the other frames. While I'm happy to try to reproduce your problem and ask smart questions, let me please point out:
ffmpeg version 2.1.git Copyright (c) 2000-2014 the FFmpeg developers built on Feb 20 2014 16:27:13 with gcc 4.6 (Ubuntu/Linaro 4.6.3-1ubuntu5) This version of ffmpeg is extremely old, considering the amount of development happening. Especially with such a "new" codec as HEVC, you can expect many improvements over 18 months. And if ffmpeg happens/happened to have a bug, it may very well have been fixed in the meantime. Nobody here will help trying to identify a bug in such an old version.
Please repeat your tests with a very recent version of ffmpeg, best is if you build if from git HEAD.
Thanks, Gruß, Moritz _______________________________________________ ffmpeg-user mailing list ffmpeg-user@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-user
-- Dipl.-Ing. Dipl.-Wirt.Ing. Christian Herglotz Chair of Multimedia Communications and Signal Processing Friedrich-Alexander University Erlangen-Nürnberg Cauerstr. 7, D-91058 Erlangen, Germany
On Thu, Oct 15, 2015 at 08:22:35 +0200, Christian Herglotz wrote:
When timing the process I found that the decoding of the first frame always takes twice as long as the decoding of all the other frames.
I'm not sure I consider the test you provide as valid. a) Initializing a decoder has (some) overhead. That aligns your numbers a bit more. b) HEVC does not only consist of I-frames (unless you claim the opposite of your content). Who says P- or B-frames are easier/faster to decode than I-frames? That's just the theory from me. Someone who knows the code could help more, or someone who is willing to add the benchmarking macros into the code and rebuild. :-) Cheers, Gruß nach Franken, Moritz
Hello, a) I know that there is some overhead. So I also tested this for sequences with different resolutions where I could observe the same behavior: Mean decoding times for FFmpeg HM 460*240 1 I-frame: 0.15s 0.18s 460*240 2 I-frames: 0.23s 0.36s 1920*1080 1 I-frame: 2.26s 2.89s 1920*1080 2 I-frames: 3.43s 5.8s 2560*1600 1 I-frame: 4.25s 5.46s 2560*1600 2 I-frames: 6.43s 11s A constant offset would not scale like this. Furthermore, I'm giving the corresponding decoding times for the HM-software as a reference where obviously the decoding time doubles for all cases. b) The bit streams I am talking about are all-intra coded. Thanks for your help! Grüße zurück aus Franken! Christian Am 16.10.2015 um 11:37 schrieb Moritz Barsnick:
On Thu, Oct 15, 2015 at 08:22:35 +0200, Christian Herglotz wrote:
When timing the process I found that the decoding of the first frame always takes twice as long as the decoding of all the other frames. I'm not sure I consider the test you provide as valid.
a) Initializing a decoder has (some) overhead. That aligns your numbers a bit more.
b) HEVC does not only consist of I-frames (unless you claim the opposite of your content). Who says P- or B-frames are easier/faster to decode than I-frames?
That's just the theory from me. Someone who knows the code could help more, or someone who is willing to add the benchmarking macros into the code and rebuild. :-)
Cheers, Gruß nach Franken, Moritz _______________________________________________ ffmpeg-user mailing list ffmpeg-user@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-user
-- Dipl.-Ing. Dipl.-Wirt.Ing. Christian Herglotz Chair of Multimedia Communications and Signal Processing Friedrich-Alexander University Erlangen-Nürnberg Cauerstr. 7, D-91058 Erlangen, Germany
Christian Herglotz <christian.herglotz <at> fau.de> writes:
Mean decoding times for FFmpeg HM 460*240 1 I-frame: 0.15s 0.18s 460*240 2 I-frames: 0.23s 0.36s
I may absolutely misunderstand what you are trying to show but please understand that afaik, the two decoders cannot be compared. Please provide command line and complete, uncut console output, add -threads 1 if you really want to measure performance decoding one frame and please test current FFmpeg git head, nothing else is supported on this mailing list. Please don't top-post here, Carl Eugen
On Fri, Oct 16, 2015 at 12:23:14 +0200, Christian Herglotz wrote:
Hello,
a) I know that there is some overhead. So I also tested this for sequences with different resolutions where I could observe the same behavior:
Okay, point taken. There's a new thread on ffmpeg-devel which points out that certain formats decode the first image twice (once for format detection, once for actual decoding). The described behavior may be identical to yours: http://ffmpeg.org/pipermail/ffmpeg-devel/2015-October/181529.html Moritz
participants (3)
-
Carl Eugen Hoyos -
Christian Herglotz -
Moritz Barsnick