Low latency H.264 streaming with ffmpeg
I would like to hear from anyone who has experience with low latency video streaming and ffmpeg. In my application I take video from a PCIe capture device and stream live over network. The end to end latency must be below 200ms otherwise it is not usable. Let us assume that the network latency is near zero. This question is about encoder+decoder+buffering latency. Very low latency can be achieved when using the x264 encoder by using settings like tune zerolatency. From my own testing with x264 (without ffmpeg) I can confirm that this works. I have measured approx 90 ms (3 frames) of latency added by encoder+decoder combined. Awesome. Sadly I have been unable to get anywhere near those numbers when using ffmpeg and libx264. As player I use either ffplay or a dot-net player built on the ffmpeg libraries, with about equal results. The lowest end to end latency I have seen is 400-500ms. That includes capture latency also. Yes, there are many unknowns here. To help pin point the problem I used mjpeg streaming for comparison. Same everything except I change ffmpeg vcodec to mjpeg and format to mjpeg. I change nothing with ffplay. With mjpeg I now get approx 150ms end to end latency. Observation: x264 codec + mpegts format compared to mjpeg codec + mjpeg format adds about 300 ms (10 frames) latency. I have tried every ffplay-, ffmpeg- and libx264 setting that I believe relevant. Most have not made any noticable difference in latency. I will not list a lot of settings here. Here are the basic settings and commands that I have used. I can post more details on request. x264-preset: vcodec=libx264 thread_type=slice slices=1 # x264 profile=baseline level=32 preset=superfast tune=zerolatency intra-refresh=1 crf=15 x264-params=vbv-maxrate=5000:vbv-bufsize=1:slice-max-size=1500:keyint=60 $ ffmpeg -r 30 -f dshow -i video="devicename" -pix_fmt yuv420p -an -vpre x264-preset -f mpegts udp://127.0.0.1:8888 $ ffplay -fflags nobuffer -flags low_delay -framedrop -strict experimental udp://127.0.0.1:8888 Everything is run on Windows 7 64-bit, with 32-bit executables. ffmpeg version N-76539-g480bad7 Copyright (c) 2000-2015 the FFmpeg developers built with gcc 5.2.0 (GCC) configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libdcadec --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-lzma --enable-decklink --enable-zlib libavutil 55. 5.100 / 55. 5.100 libavcodec 57. 15.100 / 57. 15.100 libavformat 57. 14.100 / 57. 14.100 libavdevice 57. 0.100 / 57. 0.100 libavfilter 6. 15.100 / 6. 15.100 libswscale 4. 0.100 / 4. 0.100 libswresample 2. 0.101 / 2. 0.101 libpostproc 54. 0.100 / 54. 0.100 cheers -- Greger Burman
On 1/12/16, Greger Burman <gb@moroco.se> wrote:
I would like to hear from anyone who has experience with low latency video streaming and ffmpeg.
In my application I take video from a PCIe capture device and stream live over network. The end to end latency must be below 200ms otherwise it is not usable. Let us assume that the network latency is near zero. This question is about encoder+decoder+buffering latency.
Very low latency can be achieved when using the x264 encoder by using settings like tune zerolatency. From my own testing with x264 (without ffmpeg) I can confirm that this works. I have measured approx 90 ms (3 frames) of latency added by encoder+decoder combined. Awesome.
Sadly I have been unable to get anywhere near those numbers when using ffmpeg and libx264. As player I use either ffplay or a dot-net player built on the ffmpeg libraries, with about equal results. The lowest end to end latency I have seen is 400-500ms. That includes capture latency also.
mplayer -benchmark has been the only client I've found with reasonable latency, though I'm sure there are others/more. I've gotten pretty low latency before with ultrafast+zerolatency Basically my advice is "be wary of ffplay (and vlc)" I'm not sure if its latency is low enough... https://trac.ffmpeg.org/wiki/StreamingGuide
yes, we’re having the same issue now. but -benchmark also really degrades the stream. Is there any options possible for -benchmark (moderate, strict)? Ken
mplayer -benchmark has been the only client I've found with reasonable latency, though I'm sure there are others/more. I've gotten pretty low latency before with ultrafast+zerolatency
Basically my advice is "be wary of ffplay (and vlc)" I'm not sure if its latency is low enough...
On Jan 13, 2016, at 2:30 AM, Roger Pack <rogerdpack2@gmail.com> wrote:
On 1/12/16, Greger Burman <gb@moroco.se <mailto:gb@moroco.se>> wrote:
I would like to hear from anyone who has experience with low latency video streaming and ffmpeg.
In my application I take video from a PCIe capture device and stream live over network. The end to end latency must be below 200ms otherwise it is not usable. Let us assume that the network latency is near zero. This question is about encoder+decoder+buffering latency.
Very low latency can be achieved when using the x264 encoder by using settings like tune zerolatency. From my own testing with x264 (without ffmpeg) I can confirm that this works. I have measured approx 90 ms (3 frames) of latency added by encoder+decoder combined. Awesome.
Sadly I have been unable to get anywhere near those numbers when using ffmpeg and libx264. As player I use either ffplay or a dot-net player built on the ffmpeg libraries, with about equal results. The lowest end to end latency I have seen is 400-500ms. That includes capture latency also.
https://trac.ffmpeg.org/wiki/StreamingGuide <https://trac.ffmpeg.org/wiki/StreamingGuide> _______________________________________________ ffmpeg-user mailing list ffmpeg-user@ffmpeg.org <mailto:ffmpeg-user@ffmpeg.org> http://ffmpeg.org/mailman/listinfo/ffmpeg-user <http://ffmpeg.org/mailman/listinfo/ffmpeg-user>
On 1/12/16, Kenneth Fields <ken.fields@gmail.com> wrote:
yes, we’re having the same issue now. but -benchmark also really degrades the stream. Is there any options possible for -benchmark (moderate, strict)?
what type of degredation? are you using udp? are you including audio?
Ok, so the ffmpeg based player is buffering frames somewhere even though I'm telling it not to. This happens with h264 video but not with mjpeg. That suggests to me that the latency is caused by either the h264 decoder or the mpegts demuxer. I have also tried m4v instead of mpegts, with about the same result. Anyone who knows enough about the h264 decoding process to confirm/deny? Is there a way to affect this behaviour without diving deep into the code? cheers //Greger
On 1/13/16, Greger Burman <gb@moroco.se> wrote:
Ok, so the ffmpeg based player is buffering frames somewhere even though I'm telling it not to. This happens with h264 video but not with mjpeg. That suggests to me that the latency is caused by either the h264 decoder or the mpegts demuxer. I have also tried m4v instead of mpegts, with about the same result.
Anyone who knows enough about the h264 decoding process to confirm/deny? Is there a way to affect this behaviour without diving deep into the code?
I don't know much about the h264 decoder. If you put I-frames in more often it might help. I've heard rumors of this flag -fflags nobuffer but really no little of any internal workings.
Greger Burman wrote
Observation: x264 codec + mpegts format compared to mjpeg codec + mjpeg format adds about 300 ms (10 frames) latency.
Hi, I just started to transcode the 1-minute input video(mjpeg) from the USB camera with the below command. time ~/bin/ffmpeg -benchmark -threads 4 -t 00:01:00.000000 -f video4linux2 -input_format mjpeg -framerate 30 -i /dev/video0 -an -c:v libx264 -preset ultrafast -tune zerolatency -thread_type 1 -r:v 30 -f null - Output: ffmpeg version 3.3 Copyright (c) 2000-2017 the FFmpeg developers built with gcc 6.2.0 (Ubuntu 6.2.0-5ubuntu12) 20161005 configuration: --prefix=/usr --pkg-config-flags=--static --extra-cflags=-I/usr/include --extra-ldflags=-L/usr/lib --bindir=/root/bin --enable-gpl --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libtheora --enable-libvorbis --enable-libx264 --enable-libx265 --enable-nonfree --enable-vaapi --enable-libopus --enable-libvpx --enable-hwaccel=h263_vaapi --enable-hwaccel=h264_vaapi --enable-hwaccel=mpeg2_vaapi --enable-hwaccel=mpeg4_vaapi --enable-hwaccel=vc1_vaapi --enable-hwaccel=vp8_vaapi --enable-hwaccel=vp9_vaapi --enable-hwaccel=wmv3_vaapi --enable-decoder='vp8,vp9' libavutil 55. 58.100 / 55. 58.100 libavcodec 57. 89.100 / 57. 89.100 libavformat 57. 71.100 / 57. 71.100 libavdevice 57. 6.100 / 57. 6.100 libavfilter 6. 82.100 / 6. 82.100 libswscale 4. 6.100 / 4. 6.100 libswresample 2. 7.100 / 2. 7.100 libpostproc 54. 5.100 / 54. 5.100 Input #0, video4linux2,v4l2, from '/dev/video0': Duration: N/A, start: 5864.558330, bitrate: N/A Stream #0:0: Video: mjpeg, yuvj422p(pc, bt470bg/unknown/unknown), 640x480, 30 fps, 30 tbr, 1000k tbn, 1000k tbc Stream mapping: Stream #0:0 -> #0:0 (mjpeg (native) -> h264 (libx264)) Press [q] to stop, [?] for help No pixel format specified, yuvj422p for H.264 encoding chosen. Use -pix_fmt yuv420p for compatibility with outdated media players. [libx264 @ 0x555b636ea220] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 AVX2 LZCNT BMI2 [libx264 @ 0x555b636ea220] profile High 4:2:2, level 3.0, 4:2:2 8-bit Output #0, null, to 'pipe:': Metadata: encoder : Lavf57.71.100 Stream #0:0: Video: h264 (libx264), yuvj422p(pc), 640x480, q=-1--1, 30 fps, 30 tbn, 30 tbc Metadata: encoder : Lavc57.89.100 libx264 Side data: cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1 frame= 1800 fps= 30 q=-1.0 Lsize=N/A time=00:01:00.00 bitrate=N/A speed= 1x video:8981kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown bench: utime=5.746s bench: maxrss=57952kB [libx264 @ 0x555b636ea220] frame I:8 Avg QP:19.12 size: 62426 [libx264 @ 0x555b636ea220] frame P:1792 Avg QP:22.09 size: 4853 [libx264 @ 0x555b636ea220] mb I I16..4: 100.0% 0.0% 0.0% [libx264 @ 0x555b636ea220] mb P I16..4: 1.5% 0.0% 0.0% P16..4: 58.3% 0.0% 0.0% 0.0% 0.0% skip:40.2% [libx264 @ 0x555b636ea220] coded y,uvDC,uvAC intra: 29.4% 66.3% 13.1% inter: 14.5% 42.2% 8.9% [libx264 @ 0x555b636ea220] i16 v,h,dc,p: 32% 38% 18% 12% [libx264 @ 0x555b636ea220] i8c dc,h,v,p: 36% 23% 31% 10% [libx264 @ 0x555b636ea220] kb/s:1226.19 real 1m1.359s user 0m5.772s sys 0m0.115s May I know the values (bench: utime=5.746s or user 0m5.772s or sys 0m0.115s) I need to consider to measure the end-to-end transcoding latency? I am interested in measuring the combined encode and decode latency per video frame. Please suggest me. -- Sent from: http://www.ffmpeg-archive.org/
participants (4)
-
Greger Burman -
Kenneth Fields -
rameshperumal -
Roger Pack