[FFmpeg-devel] [PATCH] [RFC]avformat: introduce AVStreamGroup

[Tomas Härdin]

fre 2023-09-15 klockan 15:10 -0300 skrev James Almer:
> On 9/13/2023 6:34 AM, Tomas Härdin wrote:
> > ons 2023-09-06 klockan 16:16 -0300 skrev James Almer:
> > > On 9/6/2023 2:53 PM, Tomas Härdin wrote:
> > > > ons 2023-09-06 klockan 11:38 -0300 skrev James Almer:
> > > > > Signed-off-by: James Almer <jamrial at gmail.com>
> > > > > ---
> > > > > This is an initial proof of concept for AVStream groups,
> > > > > something
> > > > > that's
> > > > > needed for quite a few existing and upcoming formats that
> > > > > lavf
> > > > > has no
> > > > > way to
> > > > > currently export. Said formats define a single video or audio
> > > > > stream
> > > > > composed
> > > > > by merging several individualy multiplexed streams within a
> > > > > media
> > > > > file.
> > > > > This is the case of HEIF, a format defining a tiled image
> > > > > where
> > > > > each
> > > > > tile is a
> > > > > separate image (either hevc, av1, etc) all of which need to
> > > > > be
> > > > > decoded
> > > > > individualy and then stitched together for presentation using
> > > > > container level
> > > > > information;
> > > > 
> > > > I remember this blocking HEIF as a GSoC project. Honestly the
> > > > way
> > > > that
> > > > format is designed is immensely horrible.
> > > > 
> > > > > MPEG-TS programs, currently exported as
> > > > > AVProgram, which this new general purpose API would replace.
> > > > 
> > > > I can foresee this being a nuisance for users accustomed to
> > > > AVProgram.
> > > > Also this feature borders on NLE territory. Not necessarily a
> > > > bad
> > > > thing, but FFmpeg is overall poorly architectured for NLE
> > > > stuff. I
> > > > believe I raised this issue back when lavfi was proposed, it
> > > > being
> > > > wholly unsuitable for NLE work.
> > > > 
> > > > 
> > > > > +typedef struct AVStreamGroup {
> > > > > +    /**
> > > > > +     * A class for @ref avoptions. Set on stream creation.
> > > > > +     */
> > > > > +    const AVClass *av_class;
> > > > > +
> > > > > +    /**
> > > > > +     * Group index in AVFormatContext.
> > > > > +     */
> > > > > +    int index;
> > > > > +
> > > > > +    /**
> > > > > +     * Format-specific group ID.
> > > > > +     * decoding: set by libavformat
> > > > > +     * encoding: set by the user, replaced by libavformat if
> > > > > left
> > > > > unset
> > > > > +     */
> > > > > +    int id;
> > > > > +
> > > > > +    /**
> > > > > +     * Codec parameters associated with this stream group.
> > > > > Allocated
> > > > > and freed
> > > > > +     * by libavformat in avformat_new_stream_group() and
> > > > > avformat_free_context()
> > > > > +     * respectively.
> > > > > +     *
> > > > > +     * - demuxing: filled by libavformat on stream group
> > > > > creation or
> > > > > in
> > > > > +     *             avformat_find_stream_info()
> > > > > +     * - muxing: filled by the caller before
> > > > > avformat_write_header()
> > > > > +     */
> > > > > +    AVCodecParameters *codecpar;
> > > > > +
> > > > > +    void *priv_data;
> > > > > +
> > > > > +    /**
> > > > > +     * Number of elements in AVStreamGroup.stream_index.
> > > > > +     *
> > > > > +     * Set by av_stream_group_add_stream() and
> > > > > av_stream_group_new_stream(), must not
> > > > > +     * be modified by any other code.
> > > > > +     */
> > > > > +    int nb_stream_indexes;
> > > > > +
> > > > > +    /**
> > > > > +     * A list of indexes of streams in the group. New
> > > > > entries
> > > > > are
> > > > > created with
> > > > > +     * av_stream_group_add_stream() and
> > > > > av_stream_group_new_stream().
> > > > > +     *
> > > > > +     * - demuxing: entries are created by libavformat in
> > > > > avformat_open_input().
> > > > > +     *             If AVFMTCTX_NOHEADER is set in ctx_flags,
> > > > > then
> > > > > new entries may also
> > > > > +     *             appear in av_read_frame().
> > > > > +     * - muxing: entries are created by the user before
> > > > > avformat_write_header().
> > > > > +     *
> > > > > +     * Freed by libavformat in avformat_free_context().
> > > > > +     */
> > > > > +    int *stream_index;
> > > > > +} AVStreamGroup;
> > > > 
> > > > I see no provisions for attaching metadata, for example HEIF
> > > > stitching.
> > > > Putting it in coderpar seems wrong, since it is container-level
> > > > metadata. We could just have an HEIF specific struct as
> > > > container
> > > > metadata.
> > > 
> > > The doxy for AVCodecParameters says "This struct describes the
> > > properties of an encoded stream.", so It's not about container
> > > level
> > > props.
> > 
> > It *is* container level props. The underlying codecs have no
> > concept of
> > this kind of stitching. The closest you're going to get is tiles in
> > JPEG2000, but I doubt HEIF support JPEG2000.
> > 
> > We might say "well the resulting stream group has resolution so
> > it's
> > like a codec" but see below.
> > 
> > > Although codecpar will be used to export the merged/stitched
> > > stream
> > > props like dimensions and channel layout, maybe you're right
> > > about
> > > the
> > > metadata because there would be a clash between actual
> > > HEVC/Opus/AAC/AV1
> > > extradata and the HEIF/IAMF/etc specific info if both use
> > > codecpar.extradata, even if one will be in AVStream and the other
> > > in
> > > AVStreamGroup.
> > 
> > Yes, pretty much. But it's more that codecpar is pressed into
> > service
> > where it probably doesn't belong. It might be more appropriate to
> > call
> > these "essence parameters". I'm going to stick my neck out further
> > and
> > say that picture and sound essence should be handled with different
> > structs, not smushed together into one struct like
> > AVCodecParameters.
> 
> Can you suggest how to approach this then, if not with 
> AVCodecParameters? For the resulting merged/stitched stream, we need
> at 
> the very least dimensions, and for audio we need channel layout, and
> for 
> each different kind of group type (HEIF, IAMF, TS programs) we need 
> specific parameters, like order of tiles for HEIF, mixing parameters
> for 
> IAMF, and these parameters should be in a form easy for the user
> (like 
> our CLI) to feed to lavfi, where the actual merging/stitching would
> take 
> place with new or existing filters for this specific purpose.
> 
> Maybe something like:
> 
> ----
> enum AVStreamGroupParamsType {
>      AV_STREAM_GROUP_PARAMS_NONE,
>      AV_STREAM_GROUP_PARAMS_TS,
>      AV_STREAM_GROUP_PARAMS_HEIF,
>      AV_STREAM_GROUP_PARAMS_IAMF,
> };
> 
> typedef struct AVStreamGroupTSParams {
>      // Basically AVProgram
> } AVStreamGroupTSParams;
> 
> typedef struct AVStreamGroupHEIFParams {
>      // dimensions, tile order, etc
> } AVStreamGroupHEIFParams;
> 
> typedef struct AVStreamGroupIAMFParams {
>      // channel layout, mixing params
> } AVStreamGroupIAMFParams;
> 
> typedef struct AVStreamGroup {
>      [...]
>      enum AVStreamGroupParamsType type;
>      union {
>          AVStreamGroupTSParams ts;
>          AVStreamGroupHEIFParams heif;
>          AVStreamGroupIAMFParams iamf;
>      } essence;
>      [...]
> } AVStreamGroup;

Sorry for not replying to this sooner.

Yes, a typed union like this should work nicely. This way we keep
things related to each type of stream group separate.

/Tomas