[FFmpeg-devel] nut-np (was: Re: [PATCH] Only output necessary NAL units in H.264 extradata in SDP)

Michael Niedermayer michaelni
Thu Apr 30 22:55:25 CEST 2009

On Thu, Apr 30, 2009 at 04:39:00PM +0200, Luca Abeni wrote:
> Hi Michael,
> Michael Niedermayer wrote:
> [...]
>> the idea pretty much was to send nut packets, because there is significant
>> overlap between a container format for files and a network protocol
>> handling the same data. (timestamps, metadata, error recovery, seperating
>> packets, seperating streams, ...)
> Sorry for the delayed reply. No, I did not forget about this... I just
> wanted to think about it for some time before writing, because I realise
> that I am too much RTP-biased, so I did not want to write my immediate
> reactions.
> Anyway, I agree about the overlap, and I agree that using the format header
> as a part of the protocol header is a good idea.
>> The differences pretty much are AFAICS down to
>> * telling the server to seek somewhere
> So, you propose to mix the "media transport" functionalities with the
> "control" functionalities (instead of separating them in two different
> protocols like RTP and RTSP)... Initially, I was not sure this is a good
> idea. But double thinking about it maybe it is ok.

>> * acknowledge & retransmitt if the underlaying protocol does not gurantee
>>   delivery (UDP)
> As above, I initially did not like this. But now I think this is ok; my 
> only

After some thought prior to reading your reply i decided to drop it,
its almost like reimplementing TCP which is silly ...

> concern is that this mechanism should not be mandatory (it should be 
> possible
> to somehow disable it)... Think about multicast, or "unidirectional" 
> networks.

it was never intended to be mandatory

>> * numbering the packets so one can refer to them in the protcol
> Do nut packet headers contain some kind of continuity counter? If yes, then
> we just need to number the fragments of nut packets, not all the network
> packets.

i dont think theres anything in nut packets that could easily be used
for that

> [...]
>> i was hoping when i started that thread that people would read&review
>> and make suggestions for improvments.
> Well, unless I am wrong it seems that the thread died almost immediately
> (and most of the comments were about unrelated stuff :).

yes, sadly :(

> Anyway, two important requirements for me are:

there are 3 below ;)

> 1) the ability to send different related streams to different ports or
>    even different multicast groups (for example, video to one multicast
>    group and audio to a different one)

> 2) the ability to return some feedback (about network statistics, and so
>    on) from the client to the server... Something like the RTCP RR packets.

> 3) the possibility to go over a generic transport protocol (UDP and TCP as
>    a minimum - but maybe we can define our own transport protocol...).

is there something in the spec that would cause a problem with a non
UDP/TCP protocol ?

> After reading your proposal, I think 1) is already supported (the only
> important thing is to ensure that the two different NUT streams have 
> consistent
> timestamps)... Right?


> We need to think a little bit about 2), but I think adding a new message 
> type
> would be the way to go... Right? Then, we need to define the format of the
> feedback information.


>> For example it surely is too
>> vague about a few things like which port to connect to.
> Regarding the port, I do not think it should be fixed. I think the best
> solution is to use an SDP description (yes, I can see why many people do
> not like RTP, but I see no problems with SDP and I think we should use
> it). I see that people often like "nut://<IP ADDR>:<port>" URL.
> Of course, then we might need some kind of client-server protocol for
> requesting a stream...


anyway my new nut-np spec is attached, i mostly wrote this prior to your
mail ...
comments very welcome!

Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

In a rich man's house there is no place to spit but his face.
-- Diogenes of Sinope
-------------- next part --------------
Nut Network Protocol draft 20090420

nut-np packet:
sequence_num                    v
while(not end){
    type                        v
    if(type == 0){                      //complete nut packet
        data                    vb
    }else if(type == 1){                //fragmented nut packet
        fragment_count          v
        fragment_num            v
        fragment                vb
    }else if(type==3){                  //FEC
        weight                  v
        for(i=0; i<weight; i++){
           sequence_num_diff[i] v
        data                    vb
    }else if(type >=5 && type <=9){     // ok(5)
                                        // debug(6), warning(7)
                                        // error(8) (unsupported or failed command)
                                        // fatal(9) (cant recover / connection closed)
        sequence_num_diff       v       // sequence number this message referes to
        sub_seq_num             v       // which type+data part of the packet this refers to
        message                 vb      // human readable message
    }else if(type == 10 || type == 11){ // discard stream flag
        stream_id               v       // type=10 -> dont send stream / type=11 send
    }else if(type==12){                 // keyframe request
        stream_id               v
    }else if(type==13){                 // set channel
        channel_id              v
    }else if(type==14){                 //server side seek
        min                     t
        target                  t
        max                     t
    }else if(type==15){                 //transmit rate
        transmit_rate           s

    unique number identifying this packet.
    Each packet has a sequence_num equal to the last packet in the same
    direction + 1.

    The type of the network packet. The server MUST reply to unknown
    packets with a error message (type=8 packet)
    [FIXME and the client?]

Complete nut frames (type 0) Server -> Client (Mandatory)
    The nut packet starting with the packet_header or frame_code

Fragmented nut frames (type 1)  Server -> Client (Mandatory)
    The number of fragments the nut frame is split into

    The number of the the current fragment. The server must send
    fragments in order starting with fragment_num=0. The client
    should be able to handle packets out of order if it claims
    support for a protocol that does not gurantee to maintain order.

    Fragment of a nut packet
    Bytewise concatenation of these in order of fragment_num will
    lead to a complete nut packet identical in meaning to data of
    type 0 packets

[FIXME, fragment_num vs byte offset ?]

Forward Error Correction (type 3) Server -> Client (optional)
type 3 FEC packets use LDPC codes over GF(256)
if other FEC codes will be added then this will be done by using a
packet of different type than 3.

    The number of non zero elements in the given row of the parity
    check matrix

    The number of zero elements in the row between the current point
    and the next non zero element

    The value of the non zero element, calculated by using KISS99 PRNG
    modulo 255 + 1, KISS99 is seeded using the packets sequence_num

    the weighted sum of the packets that correspond to elements of the row

Parity matrix:
    9   .   5   1   .   .   1   .   .   .
    .   8   .   .   .   2   8   1   .   .
    .   2   2   .   .   5   .   7   1   .
    1   .   .   3   9   .   .   .   2   1
    pk0 pk1 pk2 pk3 pk4 pk5 EC0 EC1 EC2 EC3

packet EC1 would store
weight=3, diff=0, coeff=8, diff=0, coeff=2, diff=3, coeff=8
data[i] = 8*pk1[i] + 2*pk5[i] + 8*EC0[i] (over GF(256))

[FIXME clarify KISS99 and try to come up with something more sane than
 transmitting the diff values]

Messages (type 5-9) Server<->Client (Mandatory)
    The difference between the sequence_num of the current packet and
    the packet to which the message refers, is 0 if it does not refer
    to any packet.

    The number of the entry in the packet to which it refers, that is
    the iteration of the while(not end) loop.

    human readable message.

Commands (type 10-15)
To all the following the server MUST reply with a type 5 (ok)
if the command was successfull or a type=8 (error message) if not.

Disable stream (type 10) Client -> Server (optional)
Enable  stream (type 11) Client -> Server (optional)
    The stream_id to enable or disable

Initially on connection establishment all streams MUST be enabled.
The server MUST NOT send frames for streams that have been disabled.
The server SHOULD NOT send stream headers for streams that have been
disabled. But if the server does not send such stream headers, it MUST
transmit them when such a stream is enabled again.
[FIXME what about info packets?]

Keyframe request (type 12) Client -> Server (optional)
    The stream_id for which a keyframe is requested

If the server replies with ok, it MUST also (produce and)
transmit a keyframe.

Set channel (type 13) Client -> Server (optional)
    The channel id to choose


Server side seek (type 14) Client -> Server (mandatory)
    Lowest timestamp to which seeking is wanted

    Highest timestamp to which seeking is wanted

    timestamp to which seeking is wanted

Seeking SHOULD happen to the closest point to target that is possible
which is within min and max.
Seeking MUST be supported for files, it may only fail for invalid
values or for realtime/live content.

Set transmit rate (type 15) Client -> Server (mandatory)
transmit_rate (default=65536)
    16.16 fixed point rate, that is 65536 means transmit data as
    with a rate matching the wall clock.
    131072 would mean transmit data twice as fast.

This command is usefull to adjust the fill state of the clients buffers
if the client is incapable to adjust its clock or the client wishes to
increase the fill state without slowing the rate of playback. Its also
usefull to quickly download a file or for tricks like fast or slow playback
A server MUST support this for non realtime content, that is stuff
like stored files instead of live encodings.

Whichever the value of transmit_rate, each packet MUST be sent close to
when clock matches transmit_ts, if such field is set in the nut file
or the equivalent SCR/PCR in mpeg-ps/ts. Clock here is the wallclock
adjusted dynamically by transmit_rate

fragmentation stuff:
* the decoder is generally able to find the start of slices as they have
  startcodes so we dont need to take care of that
* the decoder can not use a slice if any part of that slice prior to it
  is missing
->encoders/nut-np servers should fragment packets so that the amount of
  data lost if a packet is lost, (that is everything inside that packet + 
  all parts of slices where a earlier part of that slice has been lost) 
  is minimized

1. Reliable outer protocol (example TCP)
no FEC packets are allowed to be sent by either server or client.

2. Unreliable outer protocol (broadcast / multicast / unicast UDP)
FEC packets are allowed

Why no Acknowledges and retransmits?
My attempt to support this ended up very similar to TCP, thus I decided
one as well could use TCP and omit a considerable amount of complexity
reimplementing it.
What disadvantages could using TCP have over UDP + our own methods of
selectively retransmitting and acknowledging?
* lost packets and retransmitts could delay the availablity of future
 -> nothing in TCP makes future data unavailable until lost packets are
    filled in. Fix your TCP implementation or write a new one, really
    it shouldnt be harder than implementing a NIH-syndrom nut-np that
    reimplements TCP on top of UDP.
* wasted bandwidth from packets retransmitted that would be too late
 -> the client could fork its decoder when the next packet is unavailable
    so that one continues showing video with error concealment and the
    second wait until the retransmit succeeds and then once it caught
    up replace the worse one.
    This sounds complicated, but selectively refraining from retransmits
    in a reinvented TCP protocol is too.
* wasted bandwidth from retransmited B frames
 -> noone is useing bframes in low delay environments and if its not low
    delay the frames could and should be retransmitted probably

in which outer layer protocol this all is used is outside the scope of this
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20090430/d9978640/attachment.pgp>

More information about the ffmpeg-devel mailing list