[FFmpeg-devel] [PATCH 1/2] avcodec, avformat: deprecate anything related to side data merging

wm4 nfxjfg at googlemail.com
Sat Mar 18 04:55:42 EET 2017

On Fri, 17 Mar 2017 14:23:21 +0100
Michael Niedermayer <michael at niedermayer.cc> wrote:

> On Fri, Mar 17, 2017 at 01:59:40PM +0100, Hendrik Leppkes wrote:
> > On Fri, Mar 17, 2017 at 12:32 AM, Michael Niedermayer
> > <michael at niedermayer.cc> wrote:  
> > > On Thu, Mar 16, 2017 at 09:20:55PM +0100, Nicolas George wrote:  
> > >> Le sextidi 26 ventôse, an CCXXV, Michael Niedermayer a écrit :  
> > >> > Applications which depend on the current default would need  
> > >>
> > >> ... to implement a correct structure to carry the data from their
> > >> demuxer to the lavc decoders.  
> > >
> > > Is this possible for every application using every framework ?
> > >  
> > 
> > If you are using a multimedia framework, then putting non-media data
> > into the media buffers is even worse then doing it internally in
> > avformat->avcodec only.
> > There could be all sorts of other components in play in such a
> > framework which have no idea what this sidedata is, and as someone
> > actually working with such a framework daily (DirectShow), it can and
> > has broken playback when non-avcodec decoders were used and such data
> > was in the packets.
> > 
> > So in short, the "merge" solution has never solved anything for me
> > using a media framework, luckily it was only used very sparingly
> > (until recently when every packet out of the mpegts demuxer suddenly
> > had pointless sidedata and broke all the things).
> > If you use a multimedia framework that does in no way allow you to
> > attach side data to the packets, then the person working with the
> > framework should decide what to do about that, because only they know
> > how their framework may work - merging it into the data has potential
> > for way more breakage.  
> i agree
> i think the open question is if removing the possibility for user
> apps to call the merge / split functions is a good idea or not.
> Removing them makes it obviously impossible for a user app to use them
> Leaving them gives a user application the choice to use or not use them

My argument is that this mechanism is close to worthless.

A media framework which really needs this will generally have no direct
connection between demuxer and decoder. Instead, it must all go through
the framework's abstraction layer. (Otherwise, you could obviously just
pass the AVPacket directly.)

If this is the case, the packet side data will be the least of their
worries. Packet side data is relatively rare, and if it's missing, it
usually causes small problems only, such as missing metadata. After
all, side data is for _exceptions_ - normal data needed by the codec is
usually part of the proper data.

As mentioned in other replies, if the decoder side is actually not
libavcodec (not unlikely if you embed libavformat into a foreign media
framework), the merged side data will make things actively _worse_,
because to other decoders it will look like undecodable garbage data
was added to the packets.

Anyway, the worse problem about interfacing lavf/lavc through a generic
media framework are the codec parameters. Until recently, any field in
AVCodecContext could have been required for correct decoding. How would
you pass this AVCodecContext to the decoder? If you can't do this with
AVPacket, you likely have the same problem with AVCodecContext.

So why is there not a helper for this? As I mentioned in the other
mail, there could have been a mechanism that serializes the
AVCodecContext fields to the first packet, which could be used to
lazily initialize the decoder correctly. (Like in the merged side data
case, this would of course only work for lavf->layers->lavc.)

(Nowadays, the situation is slightly better - only AVCodecParameters
needs to be transported over the layers, which has fewer and simpler
fields, and in particular no private fields.)

All this makes me think that your argument about this isn't as strong
as you think. Those who want to embed lavf/lavc into foreign frameworks
will face a number of challenges. Sure, side data will be one of them.
But as I argued here and in another reply, merged side data will also
be a challenge, because it could upset foreign decoders. And the codec
parameters issue is a worse issue anyway. Users of such frameworks will
have to come up with their own tricks how to best embed FFmpeg into
them. No, we can't provide them with a set of wild hacks. We can't know
their needs, and they can do it better.

Even if we provided such helpers, ways to cleanly serialize such fields
to a separate byte array would probably be more helpful. As I argued
before, the current way how side data is merged/unmerged opens to
potential for attackers to "setup" side data in unexpected/unintended,
and normally impossible ways, which opens a whole other can of worms.

More information about the ffmpeg-devel mailing list