On 11/22/2012 10:42 AM, Nicolas George wrote:

...

Does -probesize improve the situation? (I am not sure if it only affects initial latency or also while playback.) AFAIK, ffplay and MPlayer will try to respect the frame rate they find in

Le duodi 2 frimaire, an CCXXI, Carl Eugen Hoyos a écrit : the stream (or the default they assume if it is not present). Therefore, they will not be able to "catch up" with the stream.

ffmpeg ... -f sdl -

should not have that feature, and therefore should be able to catch up.

See also the low_delay flag.

Regards,

Nicolas, Thanks for the tip. However, I'm not using ffmpeg to encode. I'm not sure if you are suggesting options for the encoding or the decoding process. My camera encoders to mpeg4 itself and has very low delay. I'm just trying to play with ffplay, also with low delay. It looks like (from the man page) that low_delay is only an encoding option for motion encoding or something - but I could be wrong. Can you elaborate, perchance? Thanks & have a great day, ~Jesse

On 11/22/2012 11:01 AM, Nicolas George wrote:

Le duodi 2 frimaire, an CCXXI, Jesse Gordon a écrit :

...

...

ffmpeg ... -f sdl - ..... It looks like (from the man page) that low_delay is only an encoding option for motion encoding or something - but I could be wrong. The flags for the "low_delay" option seem to indicate it is relevant both for encoding and decoding, and I see a lot of decoders actually checking it.

Again, trying it to see if it makes a difference is the most efficient course of action.

Regards,

Like this? ffmpeg -i test.flv -f sdl - (Didn't seem to work -- see below.) (This is the static build from Burek.) Actually, you've got me to thinking: if I can make ffmpeg (or ffplay) output a PNM image stream (or other raw image format) I can pipe it into my existing XVIDEO raw image stream player. Or can I make ffplay or ffmpeg spit out YUV frames directly? (I have a simple C program that reads a PNM image stream in stdin, converts it to YUV and displays it with XVIDEO. It's reasonably efficient and low-latency. However it seems more simple to just tell ffmpeg to "play as fast as possible." I guess if it doesn't already have a -benchmark type option it would be easy enough to add one. I looked through ffplay.c to try to find where I could comment out a call to the frame delay routine but couldn't find it. Anyone have a function name I could look for? I did try -vf "setpts=(PTS*0.9)" which make sit play a bit faster -- which does let the playing catch up with the streaming -- but then it still seems to stop and buffer in order to try to maintain the desired frame rate. Thanks, ~Jesse root@gatewaylaptop:/big# ffmpeg -i test.flv -f sdl - ffmpeg version N-46821-g8ad9b48 Copyright (c) 2000-2012 the FFmpeg developers built on Nov 17 2012 05:11:19 with gcc 4.6 (Debian 4.6.3-1) configuration: --prefix=/root/ffmpeg-static/32bit --arch=x86_32 --extra-cflags='-m32 -I/root/ffmpeg-static/32bit/include -static' --extra-ldflags='-m32 -L/root/ffmpeg-static/32bit/lib -static' --extra-libs='-lxml2 -lexpat -lfreetype' --enable-static --disable-shared --disable-ffserver --disable-doc --enable-bzlib --enable-zlib --enable-postproc --enable-runtime-cpudetect --enable-libx264 --enable-gpl --enable-libtheora --enable-libvorbis --enable-libmp3lame --enable-gray --enable-libass --enable-libfreetype --enable-libopenjpeg --enable-libspeex --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-version3 --enable-libvpx libavutil 52. 7.100 / 52. 7.100 libavcodec 54. 71.100 / 54. 71.100 libavformat 54. 36.100 / 54. 36.100 libavdevice 54. 3.100 / 54. 3.100 libavfilter 3. 23.100 / 3. 23.100 libswscale 2. 1.102 / 2. 1.102 libswresample 0. 16.100 / 0. 16.100 libpostproc 52. 1.100 / 52. 1.100 Input #0, flv, from 'test.flv': Metadata: major_brand : mp42 minor_version : 1 compatible_brands: mp42avc1 creation_time : 2012-07-29 15:04:46 comment : CAMCORDER BW10 comment-eng : CAMCORDER BW10 encoder : Lavf52.100.1 Duration: 00:01:02.25, start: 0.000000, bitrate: 7918 kb/s Stream #0:0: Video: flv1, yuv420p, 1280x720, 7680 kb/s, 29.97 tbr, 1k tbn, 1k tbc Stream #0:1: Audio: mp3, 44100 Hz, stereo, s16, 256 kb/s [NULL @ 0x969e160] Requested output format 'sdl' is not a suitable output format pipe:: Invalid argument root@gatewaylaptop:/big#

On 11/23/2012 11:57 AM, ved kpl wrote:

did you check if you are getting similar issue with other application? like an app based on gstreamer?

The network camera has a built in webpage which has a plugin that loads if you use a microsoft web browser, and that plugin plays the mpeg4 video with extremely low latency. So I know that the camera is sending the data out quickly. (ha ha it probably doesn't have enough memory to do it any other way) I have not tried a gstreamer app yet. Do I need to? Any app that is trying to play a smooth video is going to buffer enough to not run out of frames. That's why I want to modify ffmpeg to play each frame as soon as it arrives. Thanks, ~Jesse

On 11/23/2012 12:51 PM, Nicolas George wrote:

Le tridi 3 frimaire, an CCXXI, Jesse Gordon a écrit :

...

ffmpeg -i test.flv -f sdl - Yes, exactly.

...

(Didn't seem to work -- see below.) [NULL @ 0x969e160] Requested output format 'sdl' is not a suitable output format This build of ffmpeg is not linked with SDL. You should try to build your own, that is not difficult and usually more efficient in the long run.

Yes, I've built ffmpeg many many times -- but my internet's a bit slow so git takes a little while, and then compile takes a little while, so sometimes I get lazy and go grab a static build when I want to know what the latest version is like :-) Anyway, so I tried that command above with a fresh git from a couple or few days ago and it does play the video stream from the HTTP cam nicely but the delay is still about a quarter of a second, or maybe only a tenth of a second longer than mplayer -benchmark -- and mplayer adds about half a second of delay. So I guess my quest for fast decoding is still on! I guess my next question is how to remove the delay out of ffmpeg so it plays as fast/soon as it can. There should be a call to a function I can comment out. I have a medium skill with C so if someone can point me in the right direction I should be able to figure it out. Or is this the wrong mailing list to ask about source code specifics? Thank you all very much, Jesse

On 11/24/2012 01:08 PM, Carl Eugen Hoyos wrote:

Jesse Gordon <tojesseg <at> gmail.com> writes:

...

Anyway, so I tried that command above with a fresh git from a couple or few days ago and it does play the video stream from the HTTP cam nicely but the delay is still about a quarter of a second, or maybe only a tenth of a second longer than mplayer -benchmark Sorry to insist, but could you test again that -probesize makes no difference? That seems unlikely to me.

Carl Eugen

I just went and tried -probesize again. By the way, the build I tested with is Nov 21, 2012 from git. (It's on my robot control laptop, not my email laptop.) I played around a bit with different probesizes. If I specify less than 32, it complains that it's gotta be bewteen 32 and 2*10^9 or something. If I specify greater than 31 but less than 2048, it fails, saying that it doesn't have enough data to probe the file or some such. (I'm reading from an http stream.) With -probesize 2048, it buffers then pops up the play window in about 1-2 seconds after starting ffplay. With a probesize of 2,048,000 through 20,480,000 it takes about 3-4 seconds to pop up the play window. Theoretically, if I tell it to load 20M to buffer it should wait several minutes before playing because that's how long it takes for 20M to come from the camera since it's a low bit rate camera. But evidently -probesize is just a limit, and ffplay stops probing once it has what it needs, which makes sense I guess. But regardless of whether I use 2K, ~20K, 2M or 20M, the play latency appears to the human eye to be about the same. I didn't video it and count frames to see if there was a frame's difference -- I just waved my hand in front of the camera, watching it on the screen as well, but it seemed to always have almost a second of latency. The man page says that -probesize is the number of bytes used in probing. Do you really think that -probesize will affect how many milliseconds of playbuffer ffplay will try to keep or what frame rate will be used? Thanks again, ~Jesse

On 11/25/2012 12:25 AM, Carl Eugen Hoyos wrote:

Jesse Gordon <tojesseg <at> gmail.com> writes:

...

With -probesize 2048, it buffers then pops up the play window in about 1-2 seconds after starting ffplay. With a probesize of 2,048,000 through 20,480,000 it takes about 3-4 seconds to pop up the play window. (This sounds as if it does make a difference.)

Absolutely! I do appologize for not being more verbose when I said it made no difference. I had not intended to communicate that the -probesize option made no difference on probesize, only that it made no difference on my particular issue of latency, because I'd assumed that was the reason it was suggested. But probesize does definitely influence the maximum probesize used when probing the http stream!

...

But regardless of whether I use 2K, ~20K, 2M or 20M, the play latency appears to the human eye to be about the same. Thank you for testing, sorry that it didn't help.

Carl Eugen

It was worth a try! Do you know where in the code I should look for adjusting the play frame timing, or do I need to sign up for the dev mailing list for that sort of question? Thanks again very much for all of your effort and help! ~Jesse

On 11/25/2012 12:01 PM, Carl Eugen Hoyos wrote:

Jesse Gordon <tojesseg <at> gmail.com> writes:

...

Do you know where in the code I should look for adjusting the play frame timing, Unfortunately not. (And I am not sure "frame timing" is what you are searching for. Perhaps there is a buffer constant that you can try to reduce.)

Yes - there might be a frame fifo as well. I may also be dealing with TCP buffers that are storing data and causing delay.

...

or do I need to sign up for the dev mailing list for that sort of question? Such questions are not exactly welcome on the developers mailing list, try at the risk of being flamed;-)

Carl Eugen

I don't know why the developers would feel the need to flame me just because I want to add a new and useful feature but I need a little help. It's not like I haven't exhausted the user mailing list first! But you still may be right that I'd get flamed :-) Would I be better off requesting a feature? Do features like that get added as a result of user requests? Realtime low latency streaming video is a growing field these days and I think a feature like this would be extremely helpful to a small percentage but significant number of ffmpeg users. It's a feature that mplayer supports, although not nearly as well as it could. Thanks again for all your help! ~Jesse

Roger Pack <rogerdpack2 <at> gmail.com> writes:

You could add a feature request for ffplay to "ignore timestamps and display frames as fast as possible"

What would be the difference between this new feature and "ffmpeg -f sdl" ? The main reason I ask is that we do not need more feature requests, we need more manpower to implement the existing ones;-) Carl Eugen

On 11/27/2012 09:05 AM, Carl Eugen Hoyos wrote:

Roger Pack <rogerdpack2 <at> gmail.com> writes:

...

You could add a feature request for ffplay to "ignore timestamps and display frames as fast as possible" What would be the difference between this new feature and "ffmpeg -f sdl" ?

The differences would be that it would be a feature to do correctly with ffplay what can be done as a hack with ffmpeg, essentially. (No offense, but using the converter to play by converting to the video buffer so to speak is a hack -- exceedingly clever and wonderful, but it's still a hack. I love hacks like that.) The advantages to having a 'benchmark' or 'realtime' option for ffplay would include it being as part of the player, and also the user could full-screen ffplay (doesn't seem to work with ffmpeg -f sdl --) and the user could also close the player by clicking the X (also doesn't work with the SDL window popped up by ffmpeg -f sdl -) -- and the user could also use many of the other player controls and play type options that ffplay has. The difference might also include tuning to the TCP connection to try to keep the buffers cleared out. These optimizations really are all suited to the player rather than the converter.

The main reason I ask is that we do not need more feature requests, we need more manpower to implement the existing ones;-)

Carl Eugen

Indeed - I am well aware that great new features are a lot easier/quicker to think of than to implement. And that's why I had asked about getting help on the developer's list to get me started in the right place adding the feature I wanted. But alas, I was informed such a request would probably get me flamed ;-) (which I don't understand..) Such is life. I'll try to look through ffplay.c from time to time and see if I can figure it out and see if I can make it do what I like. Or maybe it'd be more to the point to use ffmpeg libraries and write my own simple player. That way it could be built from the ground up with low latency in mind. Actually I could link ffmpeg and sdl into the source for wget ha ha. wplay. Thank you all very much for all your effort on this issue! I really do appreciate it. Jesse Gordon

On 11/27/2012 06:53 AM, Roger Pack wrote:

...

Would I be better off requesting a feature? Do features like that get added as a result of user requests? Realtime low latency streaming video is a growing field these days and I think a feature like this would be extremely helpful to a small percentage but significant number of ffmpeg users. You could add a feature request for ffplay to "ignore timestamps and display frames as fast as possible" though..I'm not entirely convinced that this is your problem here...maybe it's some latency in the mpeg4 decoder itself?

What about ffplay/ffmpeg with -loglevel debug, does it show it receiving frames quickly? Did you try the other tricks mentioned on the website? -loglevel debug showed a few green lines on startup but nothing

I suspect TCP is the issue. Basically, tcp has buffers to make things run smoothly which is great for file transfers, but not great for things where a fraction of a second for a small group of bytes matters. TCP does have a "PUSH" flag that one end can send to ask the other to send whatever it's got buffered, so it may be that the windows client (at only about 5 frames of latency) is tweaking the tcp stream parameters to get keep the pipe flushed, as well as keeping its own buffers short. The tcp handling in ffmpeg is probably just regular, and not optimized for realtime video. pertaining to the frames. I did try the other tricks that were mentioned on the website which pertained to playing a video. However, much of them were specific to encoding a video with low latency -- but my encoding is done in the camera and I have no control over it (and besides, with windows/mpeg4 or mjpeg on firefox in linux, it gets those frames sent out and displayed within 4 or 5 frames, so I know the camera can do it.) I've given up for now on getting ffmpeg/ffplay to play with low latency. Perhaps as I learn more about ffmpeg I can eventually begin to understand the source code enough to start trying things to optimize for my application, but I'll cross that bridge when I'm there. Thanks, Jesse Gordon

Le quintidi 5 frimaire, an CCXXI, Jesse Gordon a écrit :

I wonder if I'm losing time in the TCP/HTTP buffers or process or maybe it has to do with tcp window size.

I believe the problem is in the ASF demuxer. For example, compare this: f=asf; \ ffmpeg -re -f lavfi -i testsrc=r=10:n=1:s=160x120,crop=80:60:80:30 \ -f $f - -pix_fmt yuv420p -f sdl - | \ ./ffmpeg_g -probesize 2k -flags +low_delay -f $f -i - \ -pix_fmt yuv420p -f sdl - and the same with f=m4v instead of f=asf: the second (m4v) is smooth and has almost no delay, the first is completely choppy and has a visible latency. Regards, -- Nicolas George

On 11/22/2012 10:10 AM, Carl Eugen Hoyos wrote:

Jesse Gordon <tojesseg <at> gmail.com> writes:

...

The problem I'm having is that ffplay seems to introduce a second or two of latency Does -probesize improve the situation? (I am not sure if it only affects initial latency or also while playback.)

Carl Eugen

-probesize doesn't seem to improve the situation at all. Is there a way I can tell ffplay to play at 20 fps even though my source is 15 or whatever -- so that it will always be running out of frames? (and keeping its buffer empty? Ahh, I just thought of this -- I could go into the ffmpeg source if I were smart enough and comment out all the timing code that enforced the fps..! thanks, Jesse

On 11/26/2012 11:02 AM, Roger Pack wrote:

...

With mplayer I was using the -benchmark option which makes it play as fast as possible, but even that has just under a second of latency. what about -nocache? same?

I got mplayer to run once with -nocache but after that I couldn't get it to play my stream with -nocache enabled.

...

I know that the video camera is getting the data out the network port quickly because when I use the windows web client, it has a plugin that streams/plays the mpeg4 and it has very low latency -- so I know the camera is not the source of the latency. Are you sure the web client is reading the "mp4" version? the mp4 version may be introducing latency by forcing a client to wait for a "first i-frame" before outputting.

I do know that the windows webclient is reading the mp4 version because I ran wireshark and captured the HTTP exchange. The windows client can play either the mpeg4 or the mjpeg -- and the difference in quality is clearly visible.

http://ffmpeg.org/trac/ffmpeg/wiki/StreamingGuide#Latency might also be interesting.

And I finally did an objective test where I video taped the latency of the whole video system and looked counted the frames. "Frames" are NTSC frames - 29.97 or whatever per second. Note that frame readings could be off by one in either direction. Windows client, Mpeg4: 5 frames of latency WIndows client, MJPEG: 9 frames of latency Linux Firefox, MJPEG: 4 frames of latency Linux, Mplayer, -benchmark mpeg4: 9 frames of latency Linux, Mpeg4, ffplay, no options: 13 frames of latency Linux, mpeg4, ffmpeg ... -f sdl -: 12 frames of latency Linux, mpeg4, ffplay -fflags nobuffer -vf "setpts=(PTS*0.95)": 13 frames of latency Linux, mpeg4, ffplay -vf "setpts=(PTS*0.95)": 12 frames of latency I really should have also tried playing the mjpeg via ffplay to see what the latency was like, but I'm out of time for tonight. Perhaps I'll try that another day. I know the obvious solution is to use mjpeg -- but I cannot because that uses a massively larger quantity of network bandwidth which I can't do in most situations. I suppose it could also have to do with operating system network buffer sizes. If (for example) the tcp socket has a 4K buffer, it might hold 13 frames of video at the low bitrate of the mpeg4 stream. Maybe windows has smaller buffers. The mjpeg of course is much higher bitrate, so it wouldn't take any time for it to fill up the 4k buffer or whatever. I'll have to see if my camera supports any UDP mode since UDP packets wouldn't be queued up and stored by the operating system - at least not like the buffer on a TCP stream. I really wish for the mpeg4 to work because it gives me realtime low-latency video at 30 fps (640x480) over a low enough bitrate that it's managable for most internet connections and wireless routers. I know it sounds silly to be worried about a few frames -- but if you'd tried driving something with a half-second delay you'd realize why I'm trying to address the issue! I'm probably at a dead end on this topic, but if anyone has any bright ideas I'll eagerly check them out! Thanks very much, Jesse Gordon PS There is one possibility which I have not explored: It is possible that the network camera detects the browser client strings and changes the encoding latency based on that, however I really doubt that such is the case. But I do know that ffplay is fetching the exact same URL for the same video.asf "file" on the camera as the windows active X plugin fetches.