abstractable threading api
As a note, Some code, for instance libavformat/udp.c, basically require pthread support for some functionality (circular buffer). I see that there is some work to providing that "natively" in the compat/w32* files, but it seems to require HAVE_PTHREAD_CANCEL ? Does this mean that the compat file isn't complete, or is it somehow not being applied (does configure by default use, or does it need a different precompiler directive check? Anyway, on a related note, it might be nice to abstract the threading primitives slightly, kind of like how libx264 has x264_thread_create or what not, so that you're aware you're using a compatibility layer, and not bound to a particular api (in this case, pthreads, though I suppose it's common enough it may not matter)). Thanks. Feedback welcome. -roger-
On Mon, Dec 16, 2013 at 8:03 PM, Roger Pack <rogerdpack2@gmail.com> wrote:
As a note, Some code, for instance libavformat/udp.c, basically require pthread support for some functionality (circular buffer). I see that there is some work to providing that "natively" in the compat/w32* files, but it seems to require HAVE_PTHREAD_CANCEL ? Does this mean that the compat file isn't complete, or is it somehow not being applied (does configure by default use, or does it need a different precompiler directive check?
The compat isn't complete, it just contains the functions needed by avcodec et al, well short of pthread_cancel. However, providing pthread_cancel with win32 threading api is also non-trivial, if not impossible to find a "drop in" compat functionality. Canceling threads from the outside is a no-no in the win32 API. Most blocking functions usually have the option to give it a unblock event that you can trigger to release it, and safely shut down the thread from within, however as you can imagine thats not something that ports well over different APIs. The best course of action in this case would probably be to find a way to re-write this worker thread without pthread_cancel, other solutions seem rather ugly. - Hendrik
Le sextidi 26 frimaire, an CCXXII, Hendrik Leppkes a écrit :
The compat isn't complete, it just contains the functions needed by avcodec et al, well short of pthread_cancel. However, providing pthread_cancel with win32 threading api is also non-trivial, if not impossible to find a "drop in" compat functionality.
Canceling threads from the outside is a no-no in the win32 API. Most blocking functions usually have the option to give it a unblock event that you can trigger to release it, and safely shut down the thread from within, however as you can imagine thats not something that ports well over different APIs.
The best course of action in this case would probably be to find a way to re-write this worker thread without pthread_cancel, other solutions seem rather ugly.
Regarding the pthread_cancel thing, I am the one responsible for this bit of code, I can explain how it is needed. The basic idea is that we have a thread blocked inside a system call: read(udp_sock_fd, buffer, size); and we want to terminate it, because reading on the socket is finished. Note that no packet may arrive there for a long time, so waiting for the system call to return by itself is not an option. First, a small discussion: in this particular case, we want to stop the whole thread. If we had a way of interrupting just the system call, stopping the thread is just a matter of checking a flag afterwards. If the situation were inverted: we knew how to cancel the thread but want to interrupt just the system call, then we could start a new thread just for the system call: more work, but that does the trick. In short: whether we cancel the system call or the whole thread, we are good. Now, let us see the options: * Signals Using a signal to interrupt a system call is a long tradition, but handling signals reliably is a portability nightmare, and mixing threads with signals is that squared. Let us avoid it. * pthread_cancel() With true POSIX threads, it is perfect: pthread_cancel() only interrupts at specific points of the code, so we do not need to care about atomicity, and reading on a socket is guaranteed to be one. * Closing the socket ... does not work. * The asynchronous I/O API aio_* It could work, but I am afraid it does not. This is a seldom used API, and thus it will have a lot of pitfalls. On GNU/Linux, aio_* is implemented with hidden threads; furthermore, a blocking operation is not really cancelled. * Non-blocking I/O + usleep() + retry loop / poll() with periodic timeout The "poll() with periodic timeout" solution was used before I changed it to pthread_cancel(). Both solution work, but they are universally frowned upon. * poll() + self-pipe The idea is to poll() on both the socket and a pipe internal to the program. Writing on the internal pipe allows to interrupt the poll() call immediately. IMHO, the best solutions for this case, are, in decreasing order: interrupting the system call (drawback: portability nightmare); cancelling the thread (drawback: pthreads-specific); poll+self-pipe (drawback: two system calls per read). If someone (Hendrik?) knows what the non-Unix APIs allows, especially the windows ones, please share it here. In a next mail, I will discuss why we want this thread for UDP sockets, and what other options are available. Regards, -- Nicolas George
On 12/19/13, Nicolas George <george@nsup.org> wrote:
Le sextidi 26 frimaire, an CCXXII, Hendrik Leppkes a écrit :
The compat isn't complete, it just contains the functions needed by avcodec et al, well short of pthread_cancel. However, providing pthread_cancel with win32 threading api is also non-trivial, if not impossible to find a "drop in" compat functionality.
Canceling threads from the outside is a no-no in the win32 API. Most blocking functions usually have the option to give it a unblock event that you can trigger to release it, and safely shut down the thread from within, however as you can imagine thats not something that ports well over different APIs.
The best course of action in this case would probably be to find a way to re-write this worker thread without pthread_cancel, other solutions seem rather ugly.
Regarding the pthread_cancel thing, I am the one responsible for this bit of code, I can explain how it is needed.
The basic idea is that we have a thread blocked inside a system call:
read(udp_sock_fd, buffer, size);
and we want to terminate it, because reading on the socket is finished. Note that no packet may arrive there for a long time, so waiting for the system call to return by itself is not an option.
First, a small discussion: in this particular case, we want to stop the whole thread. If we had a way of interrupting just the system call, stopping the thread is just a matter of checking a flag afterwards. If the situation were inverted: we knew how to cancel the thread but want to interrupt just the system call, then we could start a new thread just for the system call: more work, but that does the trick. In short: whether we cancel the system call or the whole thread, we are good.
Now, let us see the options:
* Signals
Using a signal to interrupt a system call is a long tradition, but handling signals reliably is a portability nightmare, and mixing threads with signals is that squared. Let us avoid it.
* pthread_cancel()
With true POSIX threads, it is perfect: pthread_cancel() only interrupts at specific points of the code, so we do not need to care about atomicity, and reading on a socket is guaranteed to be one.
Yeah, unfortunately I'm not even sure if win32-pthreads is able to emulate this (though it says it does) without relying on 3rd party QueueUserAPCEx library which feels a bit violent...
* Closing the socket
... does not work.
Do you have an example where this doesn't work? For fun I tried this (ruby) code: https://gist.github.com/rdp/8058776 on OS X, windows, and Linux and (at least with ruby 1.8.x) closing the socket seemed to work ok. However, this may have been because at first glance it seems to do a select before doing the actual recv... I think this one is the reason that today's code works in windows with a crippled win32-pthreads (it closes the socket, allowing the thread to terminate). Though I suppose you might want to mutex the closings so you don't accidentally double close (the second close could accidentally close a newly opened socket on the same fd number).) Anyway the good news here is that it works today on windows as is, because AFAICT it's using the "closing the socket to interrupt the thread" trick, FWIW.
* Non-blocking I/O + usleep() + retry loop / poll() with periodic timeout
The "poll() with periodic timeout" solution was used before I changed it to pthread_cancel(). Both solution work, but they are universally frowned upon.
(I assume poll() is the equivalent of using select in windows?) The disadvantage to this and the reason it is frowned upon is because it requires a timeout worth of time to cancel the thread?
* poll() + self-pipe
The idea is to poll() on both the socket and a pipe internal to the program. Writing on the internal pipe allows to interrupt the poll() call immediately.
This would seem to overcome the problem of the timeout (if it's possible to write it portably--in windows I've seen people use a UDP socket for this purpose before [1] or maybe a TCP socket would work?). So I guess my order of preference would be "closing the socket" (or using current pthread cancel code, which happens to do this) then "poll+self-pipe" Cheers. -roger- [1] https://github.com/eventmachine/eventmachine/blob/master/ext/em.cpp#L318
On 12/21/2013 3:24 AM, Roger Pack wrote:
So I guess my order of preference would be "closing the socket" (or using current pthread cancel code, which happens to do this) then "poll+self-pipe"
POSIX requires fd reusing. I think closing a socket from another thread is inherently racy.
Le decadi 30 frimaire, an CCXXII, Roger Pack a écrit :
Do you have an example where this doesn't work?
It takes five minutes to write one.
(ruby) code: https://gist.github.com/rdp/8058776 on OS X, windows, and Linux and (at least with ruby 1.8.x) closing the socket seemed to work ok. However, this may have been because at first glance it seems to do a select before doing the actual recv...
Possible. A few more tests show that shutdown(s, SHUT_RDWR) interrupts the read on Linux. But that is not specified anywhere. Systems where sockets are implemented in userland on top of other APIs, like Solaris, are likely to behave differently.
Anyway the good news here is that it works today on windows as is, because AFAICT it's using the "closing the socket to interrupt the thread" trick, FWIW.
I do not think this is a good news.
(I assume poll() is the equivalent of using select in windows?) The disadvantage to this and the reason it is frowned upon is because it requires a timeout worth of time to cancel the thread?
There is that, and there is the other side of the coin: it causes frequent useless wake-ups. For small embedded devices, it can ruin the power savings.
So I guess my order of preference would be "closing the socket" (or using current pthread cancel code, which happens to do this) then "poll+self-pipe"
Having an order of preference is way too premature before discussing the direction we want to take this receiving thread (see the mail I am currently writing) and having someone who actually knows the windows API explain the possibilities. Note that there is absolutely no obligation to have the same implementation for both OSes. Regarding the windows API, I asked around, and here is what I got (translated from French by me): # The cleanest way is to use asynchronous IO (overlapped IO in Microsoft # slang). There are several variants, but in this case I would advise either # to create an event object signaled when the read is finished and another # to request the interruption and to do a WaitForMultipleObjects, or to use # a completion port object (which is, in my opinion, the most elegant # version of asynchronous IO for windows). # # For a quick-and-dirty gizmo, CancelSynchronousIo may do the trick. # [...] It seems to me there is a race condition that does not exist with # the clean solution. Furthermore, the API is less flexible and I suspect it # will give code that is harder to maintain. Now it just needs someone to implement that. Regards, -- Nicolas George
On 12/22/13, Nicolas George <george@nsup.org> wrote:
Le decadi 30 frimaire, an CCXXII, Roger Pack a écrit :
Do you have an example where this doesn't work?
It takes five minutes to write one.
The code I did write and test worked--the ruby code. I doubt I would be able to create a failing test case in C land for the OS's I have available to me, feel free to prove me wrong :)
(ruby) code: https://gist.github.com/rdp/8058776 on OS X, windows, and Linux and (at least with ruby 1.8.x) closing the socket seemed to work ok. However, this may have been because at first glance it seems to do a select before doing the actual recv...
Possible. A few more tests show that shutdown(s, SHUT_RDWR) interrupts the read on Linux. But that is not specified anywhere. Systems where sockets are implemented in userland on top of other APIs, like Solaris, are likely to behave differently.
Solaris has pthreads so could use that option? I suppose one option is to use the "cancel to interrupt" option only for windows.
Anyway the good news here is that it works today on windows as is, because AFAICT it's using the "closing the socket to interrupt the thread" trick, FWIW. I do not think this is a good news.
I think the reason it "accidentally" happens to work in windows today is because the socket is closed before sending the pthread cancel message. Which may or may not be a bug depending on how you see it. I think what it was *meant* to do was to send pthread cancel *then* close it, but maybe in windows it can be made to still close it early.
(I assume poll() is the equivalent of using select in windows?) The disadvantage to this and the reason it is frowned upon is because it requires a timeout worth of time to cancel the thread?
There is that, and there is the other side of the coin: it causes frequent useless wake-ups. For small embedded devices, it can ruin the power savings.
Thanks for the clarification.
Regarding the windows API, I asked around, and here is what I got (translated from French by me):
# The cleanest way is to use asynchronous IO (overlapped IO in Microsoft # slang). There are several variants, but in this case I would advise either # to create an event object signaled when the read is finished and another # to request the interruption and to do a WaitForMultipleObjects, or to use # a completion port object (which is, in my opinion, the most elegant # version of asynchronous IO for windows). # # For a quick-and-dirty gizmo, CancelSynchronousIo may do the trick. # [...] It seems to me there is a race condition that does not exist with # the clean solution. Furthermore, the API is less flexible and I suspect it # will give code that is harder to maintain.
Plus CancelSynchronousIo is only available on vista+ FWIW. One thought I had was to use the poll+selfpipe option for windows, pthread cancel for linux, and single threaded for everything else, but I'll also respond to your other email. Thanks! -roger-
Le nonidi 29 frimaire, an CCXXII, Nicolas George a écrit :
In a next mail, I will discuss why we want this thread for UDP sockets, and what other options are available.
So, why do we need a thread to receive UDP packets? When data arrives for a socket, the kernel stores it in a buffer until the userland application reads it. For UDP, if the buffer is full, packets are discarded, and can not be recovered since UDP does not have flow control nor retransmission. The maximum size of the buffer seems to be around 300k, at least on Linux (it could be higher with the help of root privileges, but we can not count on it). That is about half a second of digital TV, unless I am mistaken. A video processing application can easily be busy for that amount of time, even if it is on average fast enough to work in real time. Therefore, we need something whose task is to read from the socket as fast as possible, storing the data into a larger buffer. Doing much more work in this thread would clearly defeat the purpose. But still, there may be room to make a little more work in it. A few ideas. A thread for TCP too TCP has the same kind of problem: the kernel buffer is the same size, and the same treatment happens inside FFmpeg. The difference is that TCP has flow control, therefore when the buffer is full, the data is not discarded. But it can lead to other kinds of unpleasantness. The least of it would be to achieve a lower throughput than actually possible. At worst, the flow control could reach the sending application and cause it to lose some of its input. A thread for devices too Some devices have the same kind of problem. ALSA may do buffer overruns, x11grab may skip frames, etc. Unlike the protocols, the devices are at the demuxer level, so it is not possible to handle everything together, but the solution will likely be similar. Also, note that ffmpeg itself implements threads for the demuxers. An API for that in the library, with clean pause and shutdown, would be useful. A single thread for all sockets Having a whole thread for each socket may seem expensive, but the big 5 megabytes circular buffer kind of dwarfs it. Still, it may be beneficial under heavy load on single-core systems, it would bear some benchmarking. Integration in a generic event loop Some projects may need to avoid uncontrollable threads and to integrate the network IO in their own event loop. Making sure this is possible, at least for well known event loop APIs (libev2 for example) would be a good idea. That would probably require making ffurl_get_file_handle() public again (but probably with some extensions), and adding a public API to tell a protocol "do the shortest non-blocking operation you can". Input-driven processing If we have a thread, then we are notified as soon as data arrive on the socket. We could add an API to register a callback to forward the notification to the application immediately; the callback would have to be fast, of course. This would be especially beneficial for protocols that use several underlying sockets: they all implement their own poll()-based stuff. It is an actual miracle that they work more or less reliably with that poll() and the UDP thread at the same time. Input-driven processing with a callback would simplify greatly all that, and probably make it more efficient too. (There is the small issue of getting it to work without threads too, but I believe it can be managed.) Regards, -- Nicolas George
On Mon, Dec 23, 2013 at 01:04:01AM +0100, Nicolas George wrote:
Le nonidi 29 frimaire, an CCXXII, Nicolas George a écrit : [...] Input-driven processing
If we have a thread, then we are notified as soon as data arrive on the socket. We could add an API to register a callback to forward the notification to the application immediately; the callback would have to be fast, of course.
This would be especially beneficial for protocols that use several underlying sockets: they all implement their own poll()-based stuff. It is an actual miracle that they work more or less reliably with that poll() and the UDP thread at the same time.
at least for build_udp_url() in rtpproto.c we disable the udp thread because it does not work together
Input-driven processing with a callback would simplify greatly all that, and probably make it more efficient too.
(There is the small issue of getting it to work without threads too, but I believe it can be managed.)
Regards,
-- Nicolas George
_______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
-- Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB You can kill me, but you cannot change the truth.
The maximum size of the buffer seems to be around 300k, at least on Linux (it could be higher with the help of root privileges, but we can not count on it). That is about half a second of digital TV, unless I am mistaken. A video processing application can easily be busy for that amount of time, even if it is on average fast enough to work in real time. Therefore, we need something whose task is to read from the socket as fast as possible, storing the data into a larger buffer.
I'd be happy to come up with a patch that increased the thread priority (for windows) for the udp thread, if it had a chance of being accepted and would be helpful :)
A thread for TCP too
TCP has the same kind of problem: the kernel buffer is the same size, and the same treatment happens inside FFmpeg. The difference is that TCP has flow control, therefore when the buffer is full, the data is not discarded. But it can lead to other kinds of unpleasantness. The least of it would be to achieve a lower throughput than actually possible. At worst, the flow control could reach the sending application and cause it to lose some of its input.
Agree, to me it seems like it wouldn't be as useful for TCP [?] though simplification of the API might be useful. My question was for "a thread for reading from file handles" so that it can be cacheing/buffering future data from files before it is requested (if it doesn't do that already). Might result in a nice speedup. Same with writing to them too, though that is a different story.
A thread for devices too
Some devices have the same kind of problem. ALSA may do buffer overruns, x11grab may skip frames, etc. Unlike the protocols, the devices are at the demuxer level, so it is not possible to handle everything together, but the solution will likely be similar.
dshow does something similar (the current code discards frames if it can't buffer them, but in general directshow, upstream discards frames if they're not received in time, at least AFAICT).
Integration in a generic event loop
Some projects may need to avoid uncontrollable threads and to integrate the network IO in their own event loop. Making sure this is possible, at least for well known event loop APIs (libev2 for example) would be a good idea. That would probably require making ffurl_get_file_handle() public again (but probably with some extensions), and adding a public API to tell a protocol "do the shortest non-blocking operation you can".
libev doesn't have awesome windows support (supports only select, not IOCP), but might be an option. Combining lots of sockets for evented processing might slow down the event loop though (where today a single thread services a single port, trying to avoid any lost UDP packets) since it has to execute the callback of each packet before being able to recv() again. Anyway thanks for your help on this. -roger-
On 12/23/13, Roger Pack <rogerdpack2@gmail.com> wrote:
The maximum size of the buffer seems to be around 300k, at least on Linux (it could be higher with the help of root privileges, but we can not count on it). That is about half a second of digital TV, unless I am mistaken. A video processing application can easily be busy for that amount of time, even if it is on average fast enough to work in real time. Therefore, we need something whose task is to read from the socket as fast as possible, storing the data into a larger buffer.
Interestingly, I looked into the numbers for different OS's SO_RECVBUF. In OS X currently the code by default sets it to 65K from a 42K default. In windows it sets it to 64K from an 8K default. In linux it attempts to set it to 64K from about 100K default (it actually sets it *down*) so...for at least those OS's, unless you specify manually a high "buffer_size" then it is prone to packet loss, especially when i-frames come "all at once" in several UDP packets, etc. Maybe it should set it higher by default, and this is expected? It appears all the OS's mentioned can handle 1MB SO_RECVBUF at least... # The cleanest way is to use asynchronous IO (overlapped IO in Microsoft # slang). There are several variants, but in this case I would advise either # to create an event object signaled when the read is finished and another # to request the interruption and to do a WaitForMultipleObjects, or to use # a completion port object (which is, in my opinion, the most elegant # version of asynchronous IO for windows). That would probably work well (WaitForMultipleObjects), though I don't have the expertise to quite do it myself... Thank you. -roger-
On Mon, Jan 06, 2014 at 12:36:26PM -0700, Roger Pack wrote:
On 12/23/13, Roger Pack <rogerdpack2@gmail.com> wrote:
The maximum size of the buffer seems to be around 300k, at least on Linux (it could be higher with the help of root privileges, but we can not count on it). That is about half a second of digital TV, unless I am mistaken. A video processing application can easily be busy for that amount of time, even if it is on average fast enough to work in real time. Therefore, we need something whose task is to read from the socket as fast as possible, storing the data into a larger buffer.
Interestingly, I looked into the numbers for different OS's SO_RECVBUF. In OS X currently the code by default sets it to 65K from a 42K default. In windows it sets it to 64K from an 8K default. In linux it attempts to set it to 64K from about 100K default (it actually sets it *down*) so...for at least those OS's, unless you specify manually a high "buffer_size" then it is prone to packet loss, especially when i-frames come "all at once" in several UDP packets, etc.
Maybe it should set it higher by default, and this is expected? It appears all the OS's mentioned can handle 1MB SO_RECVBUF at least...
1mb is a much more reasonable default than a few kb for video when theres no thread around to pick up data quickly [...] -- Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB Concerning the gods, I have no means of knowing whether they exist or not or of what sort they may be, because of the obscurity of the subject, and the brevity of human life -- Protagoras
On 06.01.2014, at 21:34, Michael Niedermayer <michaelni@gmx.at> wrote:
On Mon, Jan 06, 2014 at 12:36:26PM -0700, Roger Pack wrote:
On 12/23/13, Roger Pack <rogerdpack2@gmail.com> wrote:
The maximum size of the buffer seems to be around 300k, at least on Linux (it could be higher with the help of root privileges, but we can not count on it). That is about half a second of digital TV, unless I am mistaken. A video processing application can easily be busy for that amount of time, even if it is on average fast enough to work in real time. Therefore, we need something whose task is to read from the socket as fast as possible, storing the data into a larger buffer.
Interestingly, I looked into the numbers for different OS's SO_RECVBUF. In OS X currently the code by default sets it to 65K from a 42K default. In windows it sets it to 64K from an 8K default. In linux it attempts to set it to 64K from about 100K default (it actually sets it *down*) so...for at least those OS's, unless you specify manually a high "buffer_size" then it is prone to packet loss, especially when i-frames come "all at once" in several UDP packets, etc.
Maybe it should set it higher by default, and this is expected? It appears all the OS's mentioned can handle 1MB SO_RECVBUF at least...
No, they can't, not unless someone with root privileges changes the max size, in which case they could probably almost as well change the default size. Also if I remember right trying to set a larger size can make things even worse on some OSs. Maybe I misremember, but at least read the commit log _very carefully_ before changing this, this was tried before and failed.
1mb is a much more reasonable default than a few kb for video when theres no thread around to pick up data quickly
I am fairly certain we tried that before deciding to require threads. It might still be a reasonable fallback thing to try if threads are not available I guess...
On 06.01.2014 22:34, Michael Niedermayer wrote:
On Mon, Jan 06, 2014 at 12:36:26PM -0700, Roger Pack wrote:
Interestingly, I looked into the numbers for different OS's SO_RECVBUF. In OS X currently the code by default sets it to 65K from a 42K default. In windows it sets it to 64K from an 8K default. In linux it attempts to set it to 64K from about 100K default (it actually sets it *down*) so...for at least those OS's, unless you specify manually a high "buffer_size" then it is prone to packet loss, especially when i-frames come "all at once" in several UDP packets, etc.
Maybe it should set it higher by default, and this is expected? It appears all the OS's mentioned can handle 1MB SO_RECVBUF at least...
1mb is a much more reasonable default than a few kb for video when theres no thread around to pick up data quickly
Yep but default sysctl settings in linux allow no more than ~200 KB, see /proc/sys/net/core/rmem_max (on my stations now i have 196608 and 212992 set). With such default settings, UDP loss happens anyway, and sometimes changing sysctl settings is not possible. So changing default kernal buffer size doesn't fix everything automatically. But still, raising ffmpeg default value is worthwhile, because AFAIK as a result of corresponding setsockopt() the kernal buffer size is raised to available maximum. Although growing kernal buffer should be taken seriously, because it affects available system resources. If there is a lot of ffmpeg UDP processing instances, kernal buffers can take quite big part of RAM. -- Andrey Utkin
On 1/6/14, Andrey Utkin <me@andrey-utkin.pp.ua> wrote:
On 06.01.2014 22:34, Michael Niedermayer wrote:
On Mon, Jan 06, 2014 at 12:36:26PM -0700, Roger Pack wrote:
Interestingly, I looked into the numbers for different OS's SO_RECVBUF. In OS X currently the code by default sets it to 65K from a 42K default. In windows it sets it to 64K from an 8K default. In linux it attempts to set it to 64K from about 100K default (it actually sets it *down*) so...for at least those OS's, unless you specify manually a high "buffer_size" then it is prone to packet loss, especially when i-frames come "all at once" in several UDP packets, etc.
Maybe it should set it higher by default, and this is expected? It appears all the OS's mentioned can handle 1MB SO_RECVBUF at least...
1mb is a much more reasonable default than a few kb for video when theres no thread around to pick up data quickly
Yep but default sysctl settings in linux allow no more than ~200 KB, see /proc/sys/net/core/rmem_max (on my stations now i have 196608 and 212992 set). With such default settings, UDP loss happens anyway, and sometimes changing sysctl settings is not possible. So changing default kernal buffer size doesn't fix everything automatically. But still, raising ffmpeg default value is worthwhile, because AFAIK as a result of corresponding setsockopt() the kernal buffer size is raised to available maximum. Although growing kernal buffer should be taken seriously, because it affects available system resources. If there is a lot of ffmpeg UDP processing instances, kernal buffers can take quite big part of RAM.
Appears you are correct. In OS X you can set it to (at least) 1MB, in windows 10MB (or more?) and in linux, I didn't notice but seemingly has max 262142 (which is still higher than the default setting of 212992 on my side). So...I guess that pthread cancel thread might be more useful/necessary in Linux (setting it above the max 262142 does seem to set it to the max allowable, which is an improvement though, as you mentioned, attached patch might be useful/helpful there to allow users to notice they are unable to set it higher--or should be be INFO not DEBUG?). Thank you. -roger-
On Tue, Jan 07, 2014 at 02:22:06PM -0700, Roger Pack wrote:
On 1/6/14, Andrey Utkin <me@andrey-utkin.pp.ua> wrote:
On 06.01.2014 22:34, Michael Niedermayer wrote:
On Mon, Jan 06, 2014 at 12:36:26PM -0700, Roger Pack wrote:
Interestingly, I looked into the numbers for different OS's SO_RECVBUF. In OS X currently the code by default sets it to 65K from a 42K default. In windows it sets it to 64K from an 8K default. In linux it attempts to set it to 64K from about 100K default (it actually sets it *down*) so...for at least those OS's, unless you specify manually a high "buffer_size" then it is prone to packet loss, especially when i-frames come "all at once" in several UDP packets, etc.
Maybe it should set it higher by default, and this is expected? It appears all the OS's mentioned can handle 1MB SO_RECVBUF at least...
1mb is a much more reasonable default than a few kb for video when theres no thread around to pick up data quickly
Yep but default sysctl settings in linux allow no more than ~200 KB, see /proc/sys/net/core/rmem_max (on my stations now i have 196608 and 212992 set). With such default settings, UDP loss happens anyway, and sometimes changing sysctl settings is not possible. So changing default kernal buffer size doesn't fix everything automatically. But still, raising ffmpeg default value is worthwhile, because AFAIK as a result of corresponding setsockopt() the kernal buffer size is raised to available maximum. Although growing kernal buffer should be taken seriously, because it affects available system resources. If there is a lot of ffmpeg UDP processing instances, kernal buffers can take quite big part of RAM.
Appears you are correct. In OS X you can set it to (at least) 1MB, in windows 10MB (or more?) and in linux, I didn't notice but seemingly has max 262142 (which is still higher than the default setting of 212992 on my side). So...I guess that pthread cancel thread might be more useful/necessary in Linux (setting it above the max 262142 does seem to set it to the max allowable, which is an improvement though, as you mentioned, attached patch might be useful/helpful there to allow users to notice they are unable to set it higher--or should be be INFO not DEBUG?). Thank you. -roger-
udp.c | 9 +++++++++ 1 file changed, 9 insertions(+) de134ef36275227a2aa172ffa3d3f4a051829e53 0001-debug-log-udp-socket-size.patch From 1a86525dd466c1a0fef517536e83e12026c7c849 Mon Sep 17 00:00:00 2001 From: rogerdpack <rogerpack2005@gmail.com> Date: Mon, 6 Jan 2014 12:57:52 -0700 Subject: [PATCH] debug log udp socket size
Signed-off-by: rogerdpack <rogerpack2005@gmail.com>
applied thanks [...] -- Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB Frequently ignored answer#1 FFmpeg bugs should be sent to our bugtracker. User questions about the command line tools should be sent to the ffmpeg-user ML. And questions about how to use libav* should be sent to the libav-user ML.
On 12/19/13, Nicolas George <george@nsup.org> wrote:
Le sextidi 26 frimaire, an CCXXII, Hendrik Leppkes a écrit :
The compat isn't complete, it just contains the functions needed by avcodec et al, well short of pthread_cancel. However, providing pthread_cancel with win32 threading api is also non-trivial, if not impossible to find a "drop in" compat functionality.
Canceling threads from the outside is a no-no in the win32 API. Most blocking functions usually have the option to give it a unblock event that you can trigger to release it, and safely shut down the thread from within, however as you can imagine thats not something that ports well over different APIs.
The best course of action in this case would probably be to find a way to re-write this worker thread without pthread_cancel, other solutions seem rather ugly.
Regarding the pthread_cancel thing, I am the one responsible for this bit of code, I can explain how it is needed.
The basic idea is that we have a thread blocked inside a system call:
read(udp_sock_fd, buffer, size);
and we want to terminate it, because reading on the socket is finished. Note that no packet may arrive there for a long time, so waiting for the system call to return by itself is not an option.
First, a small discussion: in this particular case, we want to stop the whole thread. If we had a way of interrupting just the system call, stopping the thread is just a matter of checking a flag afterwards. If the situation were inverted: we knew how to cancel the thread but want to interrupt just the system call, then we could start a new thread just for the system call: more work, but that does the trick. In short: whether we cancel the system call or the whole thread, we are good.
Now, let us see the options:
* Signals
Using a signal to interrupt a system call is a long tradition, but handling signals reliably is a portability nightmare, and mixing threads with signals is that squared. Let us avoid it.
* pthread_cancel()
With true POSIX threads, it is perfect: pthread_cancel() only interrupts at specific points of the code, so we do not need to care about atomicity, and reading on a socket is guaranteed to be one.
* Closing the socket
... does not work.
* The asynchronous I/O API aio_*
It could work, but I am afraid it does not. This is a seldom used API, and thus it will have a lot of pitfalls. On GNU/Linux, aio_* is implemented with hidden threads; furthermore, a blocking operation is not really cancelled.
* Non-blocking I/O + usleep() + retry loop / poll() with periodic timeout
The "poll() with periodic timeout" solution was used before I changed it to pthread_cancel(). Both solution work, but they are universally frowned upon.
I did notice that at least these people: http://forum.xda-developers.com/showthread.php?t=2654753 Basically added a call to select in their code, to avoid having to cancel the thread. FWIW :)
On Mon, 2014-04-07 at 14:56 -0600, Roger Pack wrote:
On 12/19/13, Nicolas George <george@nsup.org> wrote:
Le sextidi 26 frimaire, an CCXXII, Hendrik Leppkes a écrit :
The compat isn't complete, it just contains the functions needed by avcodec et al, well short of pthread_cancel. However, providing pthread_cancel with win32 threading api is also non-trivial, if not impossible to find a "drop in" compat functionality.
Canceling threads from the outside is a no-no in the win32 API. Most blocking functions usually have the option to give it a unblock event that you can trigger to release it, and safely shut down the thread from within, however as you can imagine thats not something that ports well over different APIs.
The best course of action in this case would probably be to find a way to re-write this worker thread without pthread_cancel, other solutions seem rather ugly.
Regarding the pthread_cancel thing, I am the one responsible for this bit of code, I can explain how it is needed.
The basic idea is that we have a thread blocked inside a system call:
read(udp_sock_fd, buffer, size);
and we want to terminate it, because reading on the socket is finished. Note that no packet may arrive there for a long time, so waiting for the system call to return by itself is not an option.
First, a small discussion: in this particular case, we want to stop the whole thread. If we had a way of interrupting just the system call, stopping the thread is just a matter of checking a flag afterwards. If the situation were inverted: we knew how to cancel the thread but want to interrupt just the system call, then we could start a new thread just for the system call: more work, but that does the trick. In short: whether we cancel the system call or the whole thread, we are good.
Now, let us see the options:
* Signals
Using a signal to interrupt a system call is a long tradition, but handling signals reliably is a portability nightmare, and mixing threads with signals is that squared. Let us avoid it.
* pthread_cancel()
With true POSIX threads, it is perfect: pthread_cancel() only interrupts at specific points of the code, so we do not need to care about atomicity, and reading on a socket is guaranteed to be one.
* Closing the socket
... does not work.
* The asynchronous I/O API aio_*
It could work, but I am afraid it does not. This is a seldom used API, and thus it will have a lot of pitfalls. On GNU/Linux, aio_* is implemented with hidden threads; furthermore, a blocking operation is not really cancelled.
* Non-blocking I/O + usleep() + retry loop / poll() with periodic timeout
The "poll() with periodic timeout" solution was used before I changed it to pthread_cancel(). Both solution work, but they are universally frowned upon.
Hi, This thread has caught my attention and I went to see how pthread_cancel() was used in libavformat/udp.c and I might have spotted a race condition in it. cancellation points are implemented inside glibc not in the kernel. That means that once the cancellation point check is done and the thread is blocked inside the kernel, cancelling the thread will have no effect. In order to have the code behave as designed, pthread_setcanceltype(PTHREAD_CANCEL_ASYNCHRONOUS, NULL); should be used. asynchronous cancels are implemented with RT signals. For that reason only async cancels can interrupt and cancel a thread that is already inside a system call. https://github.com/lano1106/glcs/blob/master/src/glc/common/signal.c is an example where I have implemented a timed_waitpid() by using a thread and pthread_cancel(). if there is interest, I can point you to source files in glibc to look to understand how pthread_cancel is implemented in glibc. Just let me know, Greetings, Olivier
This thread has caught my attention and I went to see how pthread_cancel() was used in libavformat/udp.c and I might have spotted a race condition in it.
cancellation points are implemented inside glibc not in the kernel. That means that once the cancellation point check is done and the thread is blocked inside the kernel, cancelling the thread will have no effect.
In order to have the code behave as designed,
pthread_setcanceltype(PTHREAD_CANCEL_ASYNCHRONOUS, NULL);
should be used. asynchronous cancels are implemented with RT signals. For that reason only async cancels can interrupt and cancel a thread that is already inside a system call.
My mistake. Ignore this whole post. Half of it is true. The other half is that inside cancellation point functions, async cancel type if flipped on/off around the syscall. ie: int oldstate = LIBC_CANCEL_ASYNC (); r = INTERNAL_SYSCALL (clock_nanosleep, err, 4, clock_id, flags, req, rem); LIBC_CANCEL_RESET (oldstate); ________________________________ CONFIDENTIALITY : This e-mail and any attachments are confidential and may be privileged. If you are not a named recipient, please notify the sender immediately and do not disclose the contents to another person, use it for any purpose or store or copy the information in any medium.
Le nonidi 19 germinal, an CCXXII, LANGLOIS Olivier PIS -EXT a écrit :
This thread has caught my attention and I went to see how pthread_cancel() was used in libavformat/udp.c and I might have spotted a race condition in it.
cancellation points are implemented inside glibc not in the kernel. That means that once the cancellation point check is done and the thread is blocked inside the kernel, cancelling the thread will have no effect.
In order to have the code behave as designed,
pthread_setcanceltype(PTHREAD_CANCEL_ASYNCHRONOUS, NULL);
should be used. asynchronous cancels are implemented with RT signals. For that reason only async cancels can interrupt and cancel a thread that is already inside a system call.
My mistake. Ignore this whole post. Half of it is true. The other half is that inside cancellation point functions, async cancel type if flipped on/off around the syscall. ie:
int oldstate = LIBC_CANCEL_ASYNC ();
r = INTERNAL_SYSCALL (clock_nanosleep, err, 4, clock_id, flags, req, rem);
LIBC_CANCEL_RESET (oldstate);
Thanks for clarifying. The idea of "cancellation points" would have been pretty useless if it did not reach the actual blocking system calls, but that would not have been the first time an unusable API found its way into Unix. Regards, -- Nicolas George
On Thu, 10 Apr 2014 16:49:37 +0200 Nicolas George <george@nsup.org> wrote:
Le nonidi 19 germinal, an CCXXII, LANGLOIS Olivier PIS -EXT a écrit :
This thread has caught my attention and I went to see how pthread_cancel() was used in libavformat/udp.c and I might have spotted a race condition in it.
cancellation points are implemented inside glibc not in the kernel. That means that once the cancellation point check is done and the thread is blocked inside the kernel, cancelling the thread will have no effect.
In order to have the code behave as designed,
pthread_setcanceltype(PTHREAD_CANCEL_ASYNCHRONOUS, NULL);
should be used. asynchronous cancels are implemented with RT signals. For that reason only async cancels can interrupt and cancel a thread that is already inside a system call.
My mistake. Ignore this whole post. Half of it is true. The other half is that inside cancellation point functions, async cancel type if flipped on/off around the syscall. ie:
int oldstate = LIBC_CANCEL_ASYNC ();
r = INTERNAL_SYSCALL (clock_nanosleep, err, 4, clock_id, flags, req, rem);
LIBC_CANCEL_RESET (oldstate);
Thanks for clarifying. The idea of "cancellation points" would have been pretty useless if it did not reach the actual blocking system calls, but that would not have been the first time an unusable API found its way into Unix.
I think it's a quality of implementation issue. It is meant to cancel system calls which are specified as cancellation points. Asynchronous cancellation is actually not needed. Though it's possible that glibc and OSX have various bugs (see http://ewontfix.com/2/ ).
Le primidi 21 germinal, an CCXXII, wm4 a écrit :
I think it's a quality of implementation issue. It is meant to cancel system calls which are specified as cancellation points. Asynchronous cancellation is actually not needed. Though it's possible that glibc and OSX have various bugs (see http://ewontfix.com/2/ ).
I do not understand what you are saying. What I was saying is this: If cancellation points worked in the libc but not in the kernel, they would be completely useless. But that is not the case (in implementations we know of), and therefore the issue is moot. Since it is about cancellation points, it is about DEFERRED cancel, not ASYNCHRONOUS cancel. ASYNCHRONOUS cancel is another story entirely, but as you write, it is unneeded here. Regards, -- Nicolas George
On Wed, Apr 16, 2014 at 5:07 PM, Nicolas George <george@nsup.org> wrote:
Le primidi 21 germinal, an CCXXII, wm4 a écrit :
I think it's a quality of implementation issue. It is meant to cancel system calls which are specified as cancellation points. Asynchronous cancellation is actually not needed. Though it's possible that glibc and OSX have various bugs (see http://ewontfix.com/2/ ).
I do not understand what you are saying. What I was saying is this:
If cancellation points worked in the libc but not in the kernel, they would be completely useless. But that is not the case (in implementations we know of), and therefore the issue is moot.
Since it is about cancellation points, it is about DEFERRED cancel, not ASYNCHRONOUS cancel. ASYNCHRONOUS cancel is another story entirely, but as you write, it is unneeded here.
Did we ever get any kind of progress done on this? I've been using pthreads on windows to make use of the UDP buffer, but I found a problem with the interaction of the pthreads-w32 library and interaction with windows threads (it causes a leak of thread handles) - plus after checking the pthreads-w32 code, I'm unsure pthreads_cancel actually works properly, and the UDP code may just happen to close down "by accident" somehow. I originally switched back to pthreads because of a user complaining about UDP problems, and it fixed it nicely, but now facing this again... I'm willing to work on any solution, as long as we can decide on one to pursue. - Hendrik
On 5/8/14, Hendrik Leppkes <h.leppkes@gmail.com> wrote:
On Wed, Apr 16, 2014 at 5:07 PM, Nicolas George <george@nsup.org> wrote:
Le primidi 21 germinal, an CCXXII, wm4 a écrit :
I think it's a quality of implementation issue. It is meant to cancel system calls which are specified as cancellation points. Asynchronous cancellation is actually not needed. Though it's possible that glibc and OSX have various bugs (see http://ewontfix.com/2/ ).
I do not understand what you are saying. What I was saying is this:
If cancellation points worked in the libc but not in the kernel, they would be completely useless. But that is not the case (in implementations we know of), and therefore the issue is moot.
Since it is about cancellation points, it is about DEFERRED cancel, not ASYNCHRONOUS cancel. ASYNCHRONOUS cancel is another story entirely, but as you write, it is unneeded here.
Did we ever get any kind of progress done on this? I've been using pthreads on windows to make use of the UDP buffer, but I found a problem with the interaction of the pthreads-w32 library and interaction with windows threads (it causes a leak of thread handles)
It would be nice to understand how/where it is leaking.
- plus after checking the pthreads-w32 code, I'm unsure pthreads_cancel actually works properly, and the UDP code may just happen to close down "by accident" somehow.
Right I think.
L'octidi 18 germinal, an CCXXII, Roger Pack a écrit :
I did notice that at least these people:
http://forum.xda-developers.com/showthread.php?t=2654753
Basically added a call to select in their code, to avoid having to cancel the thread. FWIW :)
That is almost exactly the code as it was before pthread_cancel() was used. This ugly because the cancellation is passed through a shared flag that is not integrated into the select() call and therefore must be polled at regular interval. That means a long latency to actually cancel the thread, and useless wake-ups. The wake-ups are bad for battery life on embedded devices. The long latency can cause actual problems if the application wants to stop a socket and then start it again on the same port: if the thread is not cancelled yet, it will fail due to the port being still in use. With regular polling, the race condition is very likely to trigger. Regards, -- Nicolas George
participants (10)
-
Andrey Utkin -
Hendrik Leppkes -
LANGLOIS Olivier PIS -EXT -
Michael Niedermayer -
Nicolas George -
Olivier Langlois -
Reimar Döffinger -
Roger Pack -
wm4 -
Yu Xiaolei