[PATCH v2 0/6] WebRTC sub-second live streaming support
This patch series adds support for WHIP and WHEP (WebRTC-HTTP ingestion protocol and WebRTC-HTTP egress protocol). It supersedes the previous patch series. Thank you all for your feedback and suggestions! The WHIP and WHEP are defined in the following draft RFCs: - WHIP: https://datatracker.ietf.org/doc/draft-ietf-wish-whip - WHEP: https://datatracker.ietf.org/doc/draft-murillo-whep/ The implementation builds on FFmpeg's existing RTP, HTTP and SDP support as well as libdatachannel for the rest. This library is much more lightweight than other libraries (e.g. libwebrtc). At the same time, using this library avoids reimplementing parts of WebRTC in FFmpeg. This patch series was tested with WebRTC servers Dolby.io (formerly Millicast) and SRS (https://github.com/ossrs/srs). Using a local server, an end-to-end latency of 3 frames (50 ms at 60 fps) was measured with video-only output, and 12 frames (200 ms at 60 fps) with audio and video output. Using a DeckLink input device, an end-to-end latency of about 4 frames (70 ms at 60 fps) was measured with both, video-only and audio and video output. Using a remote server, only the RTT is added to the end-to-end latency. For example, using a server in Amsterdam (The Netherlands) from a location in Salzburg (Austria), with RTT=18ms, an end-to-end latency of 50+18=68 ms was measured for video-only output. Michael Riedl (6): libavformat/http: expose actual Location header value libavformat/sdp: remove whitespaces in fmtp configure: add libdatachannel as external library libavformat/webrtc: add common code for WebRTC streaming libavformat/webrtc_demux: add WebRTC-HTTP egress protocol (WHEP) demuxer libavformat/webrtc_mux: add WebRTC-HTTP ingestion protocol (WHIP) muxer Changelog | 5 + MAINTAINERS | 1 + configure | 7 + doc/demuxers.texi | 22 ++ doc/muxers.texi | 21 ++ libavformat/Makefile | 2 + libavformat/allformats.c | 2 + libavformat/http.c | 1 + libavformat/sdp.c | 4 +- libavformat/webrtc.c (new) | 410 +++++++++++++++++++++++++++++++ libavformat/webrtc.h (new) | 70 ++++++ libavformat/webrtc_demux.c (new) | 246 +++++++++++++++++++ libavformat/webrtc_mux.c (new) | 273 ++++++++++++++++++++ 13 files changed, 1062 insertions(+), 2 deletions(-) create mode 100644 libavformat/webrtc.c create mode 100644 libavformat/webrtc.h create mode 100644 libavformat/webrtc_demux.c create mode 100644 libavformat/webrtc_mux.c -- 2.39.2
This patchset is missing tests. I know that we for some reason don't really have tests for protocols, but I feel the issue is worthwhile to bring up. I've worked a bit with WebRTC recently and it's fiddly, so it'd be nice to have some automated thing that keeps track of which WebRTC implementations this works with. Or maybe this is better handled with an external project, testing which implementations interoperate? /Tomas
On 11/14/23 11:05, Tomas Härdin wrote:
This patchset is missing tests. I know that we for some reason don't really have tests for protocols, but I feel the issue is worthwhile to bring up. I've worked a bit with WebRTC recently and it's fiddly, so it'd be nice to have some automated thing that keeps track of which WebRTC implementations this works with. Or maybe this is better handled with an external project, testing which implementations interoperate?
I agree that having automated tests would be useful for stability in the future. I tested the patchset with both SRS and Millicast, and it worked well. For automated testing, we could use FATE, but it needs an extra server with protection and someone to keep it updated. Testing with paid services like Millicast is tricky. Another option is an external project for WebRTC testing, but the challenge is keeping it maintained and compatible with changes in implementations. External services might update their software, causing issues with tests. We would need a plan for dealing with that. For paid services like Millicast, we need to figure out who pays for the tests. Understanding the costs is essential if we want to use paid services for testing. I'd like to hear your thoughts on these points and how you would propose to proceed with adding tests for protocols like WebRTC.
tis 2023-11-14 klockan 13:59 +0100 skrev Michael Riedl:
Another option is an external project for WebRTC testing, but the challenge is keeping it maintained and compatible with changes in implementations.
Perhaps there are funds to raise for such an effort? I can imagine Google et al have an interest in this.
External services might update their software, causing issues with tests. We would need a plan for dealing with that.
Just making sure various free software implementations interoperate would be more than enough work, and provides some confidence. Surveying what features are supported by each implementation would also be of interest I imagine. For example a client of mine uses aiortc which is lacking FEC and rate control. A local company that sometimes contributes to FFmpeg is also involved in various streaming stuff (though not WebRTC so far, I think).
For paid services like Millicast, we need to figure out who pays for the tests. Understanding the costs is essential if we want to use paid services for testing.
I'd like to hear your thoughts on these points and how you would propose to proceed with adding tests for protocols like WebRTC.
The way I've been testing this stuff so far is by hand via Firefox and Chromium. Both can be automated, though frontend isn't really my area. As it stands at the moment, we let users be guinea pigs for our protocols. While not great, this at least means we get bug reports. But there's no way to ensure old bugs don't resurface. /Tomas
On Tue, Nov 14, 2023 at 01:59:48PM +0100, Michael Riedl wrote:
On 11/14/23 11:05, Tomas Härdin wrote:
This patchset is missing tests. I know that we for some reason don't really have tests for protocols, but I feel the issue is worthwhile to bring up. I've worked a bit with WebRTC recently and it's fiddly, so it'd be nice to have some automated thing that keeps track of which WebRTC implementations this works with. Or maybe this is better handled with an external project, testing which implementations interoperate?
I agree that having automated tests would be useful for stability in the future. I tested the patchset with both SRS and Millicast, and it worked well.
For automated testing, we could use FATE, but it needs an extra server with protection and someone to keep it updated. Testing with paid services like Millicast is tricky.
Another option is an external project for WebRTC testing, but the challenge is keeping it maintained and compatible with changes in implementations.
External services might update their software, causing issues with tests. We would need a plan for dealing with that.
For paid services like Millicast, we need to figure out who pays for the tests. Understanding the costs is essential if we want to use paid services for testing.
I'd like to hear your thoughts on these points and how you would propose to proceed with adding tests for protocols like WebRTC.
simple, add server support for this to FFmpeg. FATE is run in cases without network access (for example googles ossfuzz setup i belives does not permit the fuzzed code to access external things IIRC) The practice of implementing only one end of a protocol is honestly wrong. And if there is no usable free server, then even more so. thx [...] -- Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB If you drop bombs on a foreign country and kill a hundred thousand innocent people, expect your government to call the consequence "unprovoked inhuman terrorist attacks" and use it to justify dropping more bombs and killing more people. The technology changed, the idea is old.
On 11/15/23 22:45, Michael Niedermayer wrote:
On Tue, Nov 14, 2023 at 01:59:48PM +0100, Michael Riedl wrote:
On 11/14/23 11:05, Tomas Härdin wrote:
This patchset is missing tests. I know that we for some reason don't really have tests for protocols, but I feel the issue is worthwhile to bring up. I've worked a bit with WebRTC recently and it's fiddly, so it'd be nice to have some automated thing that keeps track of which WebRTC implementations this works with. Or maybe this is better handled with an external project, testing which implementations interoperate?
I agree that having automated tests would be useful for stability in the future. I tested the patchset with both SRS and Millicast, and it worked well.
For automated testing, we could use FATE, but it needs an extra server with protection and someone to keep it updated. Testing with paid services like Millicast is tricky.
Another option is an external project for WebRTC testing, but the challenge is keeping it maintained and compatible with changes in implementations.
External services might update their software, causing issues with tests. We would need a plan for dealing with that.
For paid services like Millicast, we need to figure out who pays for the tests. Understanding the costs is essential if we want to use paid services for testing.
I'd like to hear your thoughts on these points and how you would propose to proceed with adding tests for protocols like WebRTC. simple, add server support for this to FFmpeg.
FATE is run in cases without network access (for example googles ossfuzz setup i belives does not permit the fuzzed code to access external things IIRC)
The practice of implementing only one end of a protocol is honestly wrong. And if there is no usable free server, then even more so.
thx
Server support was planned for later, but maybe it's better to do it now. Multiple solutions are possible, but one solution requires adding 2 more (de)muxers for server support to this patch series. It would also be possible to split the patch series into 2 parts, one for WHIP client and server support and one for WHEP client and server support. But I'm not sure which solution is better. My idea is to add 2 more (de)muxers for server support. This would allow us to test the following: - WHIP muxer (client) --> WHIP demuxer (server) - WHEP muxer (server) --> WHEP demuxer (client) Regardless, this will only test the implementation against itself. If that makes sense and sounds reasonable, I will add the server support to this patch series. Please let me know what you think.
Michael Riedl (12023-11-27):
Regardless, this will only test the implementation against itself. If that makes sense and sounds reasonable, I will add the server support to this patch series.
Thanks. It will not be enough in a perfect world, but it still is a very good start and much better than no testing at all. Regards, -- Nicolas George
On Mon, 27 Nov 2023 at 12:48, Nicolas George <george@nsup.org> wrote:
Regardless, this will only test the implementation against itself. If
Michael Riedl (12023-11-27): that makes
sense and sounds reasonable, I will add the server support to this patch series.
Thanks. It will not be enough in a perfect world, but it still is a very good start and much better than no testing at all.
Regards,
-- Nicolas George
An update: The libdatachannel project has already implemented support for AV1 and HEVC codecs in the RTP (de)packetizers, as of version v0.22.6. Updating the patchset to match these capabilities for WHIP & WHEP would be timely.
participants (5)
-
Dennis Mungai -
Michael Niedermayer -
Michael Riedl -
Nicolas George -
Tomas Härdin