[FFmpeg-devel] nut-np (was: Re: [PATCH] Only output necessary NAL units in H.264 extradata in SDP)
Michael Niedermayer
michaelni
Thu Apr 30 22:55:25 CEST 2009
On Thu, Apr 30, 2009 at 04:39:00PM +0200, Luca Abeni wrote:
> Hi Michael,
>
> Michael Niedermayer wrote:
> [...]
>> the idea pretty much was to send nut packets, because there is significant
>> overlap between a container format for files and a network protocol
>> handling the same data. (timestamps, metadata, error recovery, seperating
>> packets, seperating streams, ...)
> Sorry for the delayed reply. No, I did not forget about this... I just
> wanted to think about it for some time before writing, because I realise
> that I am too much RTP-biased, so I did not want to write my immediate
> reactions.
>
> Anyway, I agree about the overlap, and I agree that using the format header
> as a part of the protocol header is a good idea.
>
>> The differences pretty much are AFAICS down to
>> * telling the server to seek somewhere
> So, you propose to mix the "media transport" functionalities with the
> "control" functionalities (instead of separating them in two different
> protocols like RTP and RTSP)... Initially, I was not sure this is a good
> idea. But double thinking about it maybe it is ok.
>
>
>> * acknowledge & retransmitt if the underlaying protocol does not gurantee
>> delivery (UDP)
> As above, I initially did not like this. But now I think this is ok; my
> only
After some thought prior to reading your reply i decided to drop it,
its almost like reimplementing TCP which is silly ...
> concern is that this mechanism should not be mandatory (it should be
> possible
> to somehow disable it)... Think about multicast, or "unidirectional"
> networks.
it was never intended to be mandatory
>
>
>> * numbering the packets so one can refer to them in the protcol
> Do nut packet headers contain some kind of continuity counter? If yes, then
> we just need to number the fragments of nut packets, not all the network
> packets.
i dont think theres anything in nut packets that could easily be used
for that
>
> [...]
>> i was hoping when i started that thread that people would read&review
>> and make suggestions for improvments.
> Well, unless I am wrong it seems that the thread died almost immediately
> (and most of the comments were about unrelated stuff :).
yes, sadly :(
>
> Anyway, two important requirements for me are:
there are 3 below ;)
> 1) the ability to send different related streams to different ports or
> even different multicast groups (for example, video to one multicast
> group and audio to a different one)
> 2) the ability to return some feedback (about network statistics, and so
> on) from the client to the server... Something like the RTCP RR packets.
> 3) the possibility to go over a generic transport protocol (UDP and TCP as
> a minimum - but maybe we can define our own transport protocol...).
is there something in the spec that would cause a problem with a non
UDP/TCP protocol ?
>
> After reading your proposal, I think 1) is already supported (the only
> important thing is to ensure that the two different NUT streams have
> consistent
> timestamps)... Right?
yes
> We need to think a little bit about 2), but I think adding a new message
> type
> would be the way to go... Right? Then, we need to define the format of the
> feedback information.
yes
>
>> For example it surely is too
>> vague about a few things like which port to connect to.
> Regarding the port, I do not think it should be fixed. I think the best
> solution is to use an SDP description (yes, I can see why many people do
> not like RTP, but I see no problems with SDP and I think we should use
> it). I see that people often like "nut://<IP ADDR>:<port>" URL.
> Of course, then we might need some kind of client-server protocol for
> requesting a stream...
hmm
anyway my new nut-np spec is attached, i mostly wrote this prior to your
mail ...
comments very welcome!
[...]
--
Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB
In a rich man's house there is no place to spit but his face.
-- Diogenes of Sinope
-------------- next part --------------
Nut Network Protocol draft 20090420
nut-np packet:
sequence_num v
while(not end){
type v
if(type == 0){ //complete nut packet
data vb
}else if(type == 1){ //fragmented nut packet
fragment_count v
fragment_num v
fragment vb
}else if(type==3){ //FEC
weight v
for(i=0; i<weight; i++){
sequence_num_diff[i] v
}
data vb
}else if(type >=5 && type <=9){ // ok(5)
// debug(6), warning(7)
// error(8) (unsupported or failed command)
// fatal(9) (cant recover / connection closed)
sequence_num_diff v // sequence number this message referes to
sub_seq_num v // which type+data part of the packet this refers to
message vb // human readable message
}else if(type == 10 || type == 11){ // discard stream flag
stream_id v // type=10 -> dont send stream / type=11 send
}else if(type==12){ // keyframe request
stream_id v
}else if(type==13){ // set channel
channel_id v
}else if(type==14){ //server side seek
min t
target t
max t
}else if(type==15){ //transmit rate
transmit_rate s
}
}
sequence_num
unique number identifying this packet.
Each packet has a sequence_num equal to the last packet in the same
direction + 1.
type
The type of the network packet. The server MUST reply to unknown
packets with a error message (type=8 packet)
[FIXME and the client?]
Complete nut frames (type 0) Server -> Client (Mandatory)
---------------------------------------------
data
The nut packet starting with the packet_header or frame_code
Fragmented nut frames (type 1) Server -> Client (Mandatory)
------------------------------------------------
fragment_count
The number of fragments the nut frame is split into
fragment_num
The number of the the current fragment. The server must send
fragments in order starting with fragment_num=0. The client
should be able to handle packets out of order if it claims
support for a protocol that does not gurantee to maintain order.
fragment
Fragment of a nut packet
Bytewise concatenation of these in order of fragment_num will
lead to a complete nut packet identical in meaning to data of
type 0 packets
[FIXME, fragment_num vs byte offset ?]
Forward Error Correction (type 3) Server -> Client (optional)
--------------------------------------------------
type 3 FEC packets use LDPC codes over GF(256)
if other FEC codes will be added then this will be done by using a
packet of different type than 3.
weight
The number of non zero elements in the given row of the parity
check matrix
sequence_num_diff
The number of zero elements in the row between the current point
and the next non zero element
coeff
The value of the non zero element, calculated by using KISS99 PRNG
modulo 255 + 1, KISS99 is seeded using the packets sequence_num
data
the weighted sum of the packets that correspond to elements of the row
Example:
Parity matrix:
9 . 5 1 . . 1 . . .
. 8 . . . 2 8 1 . .
. 2 2 . . 5 . 7 1 .
1 . . 3 9 . . . 2 1
pk0 pk1 pk2 pk3 pk4 pk5 EC0 EC1 EC2 EC3
packet EC1 would store
weight=3, diff=0, coeff=8, diff=0, coeff=2, diff=3, coeff=8
data[i] = 8*pk1[i] + 2*pk5[i] + 8*EC0[i] (over GF(256))
[FIXME clarify KISS99 and try to come up with something more sane than
transmitting the diff values]
Messages (type 5-9) Server<->Client (Mandatory)
-----------------------------------
sequence_num_diff
The difference between the sequence_num of the current packet and
the packet to which the message refers, is 0 if it does not refer
to any packet.
sub_seq_num
The number of the entry in the packet to which it refers, that is
the iteration of the while(not end) loop.
message
human readable message.
Commands (type 10-15)
=========================================
To all the following the server MUST reply with a type 5 (ok)
if the command was successfull or a type=8 (error message) if not.
Disable stream (type 10) Client -> Server (optional)
Enable stream (type 11) Client -> Server (optional)
-----------------------------------------
stream_id
The stream_id to enable or disable
Initially on connection establishment all streams MUST be enabled.
The server MUST NOT send frames for streams that have been disabled.
The server SHOULD NOT send stream headers for streams that have been
disabled. But if the server does not send such stream headers, it MUST
transmit them when such a stream is enabled again.
[FIXME what about info packets?]
Keyframe request (type 12) Client -> Server (optional)
-------------------------------------------
stream_id
The stream_id for which a keyframe is requested
If the server replies with ok, it MUST also (produce and)
transmit a keyframe.
Set channel (type 13) Client -> Server (optional)
--------------------------------------
channel_id
The channel id to choose
[FIXME]
Server side seek (type 14) Client -> Server (mandatory)
-------------------------------------------
min
Lowest timestamp to which seeking is wanted
max
Highest timestamp to which seeking is wanted
target
timestamp to which seeking is wanted
Seeking SHOULD happen to the closest point to target that is possible
which is within min and max.
Seeking MUST be supported for files, it may only fail for invalid
values or for realtime/live content.
Set transmit rate (type 15) Client -> Server (mandatory)
--------------------------------------------
transmit_rate (default=65536)
16.16 fixed point rate, that is 65536 means transmit data as
with a rate matching the wall clock.
131072 would mean transmit data twice as fast.
This command is usefull to adjust the fill state of the clients buffers
if the client is incapable to adjust its clock or the client wishes to
increase the fill state without slowing the rate of playback. Its also
usefull to quickly download a file or for tricks like fast or slow playback
A server MUST support this for non realtime content, that is stuff
like stored files instead of live encodings.
Whichever the value of transmit_rate, each packet MUST be sent close to
when clock matches transmit_ts, if such field is set in the nut file
or the equivalent SCR/PCR in mpeg-ps/ts. Clock here is the wallclock
adjusted dynamically by transmit_rate
fragmentation stuff:
* the decoder is generally able to find the start of slices as they have
startcodes so we dont need to take care of that
* the decoder can not use a slice if any part of that slice prior to it
is missing
->encoders/nut-np servers should fragment packets so that the amount of
data lost if a packet is lost, (that is everything inside that packet +
all parts of slices where a earlier part of that slice has been lost)
is minimized
1. Reliable outer protocol (example TCP)
----------------------------------------
no FEC packets are allowed to be sent by either server or client.
2. Unreliable outer protocol (broadcast / multicast / unicast UDP)
-------------------------------------------------------------
FEC packets are allowed
Why no Acknowledges and retransmits?
------------------------------------
My attempt to support this ended up very similar to TCP, thus I decided
one as well could use TCP and omit a considerable amount of complexity
reimplementing it.
What disadvantages could using TCP have over UDP + our own methods of
selectively retransmitting and acknowledging?
* lost packets and retransmitts could delay the availablity of future
data
-> nothing in TCP makes future data unavailable until lost packets are
filled in. Fix your TCP implementation or write a new one, really
it shouldnt be harder than implementing a NIH-syndrom nut-np that
reimplements TCP on top of UDP.
* wasted bandwidth from packets retransmitted that would be too late
anyway
-> the client could fork its decoder when the next packet is unavailable
so that one continues showing video with error concealment and the
second wait until the retransmit succeeds and then once it caught
up replace the worse one.
This sounds complicated, but selectively refraining from retransmits
in a reinvented TCP protocol is too.
* wasted bandwidth from retransmited B frames
-> noone is useing bframes in low delay environments and if its not low
delay the frames could and should be retransmitted probably
in which outer layer protocol this all is used is outside the scope of this
spec
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20090430/d9978640/attachment.pgp>
More information about the ffmpeg-devel
mailing list