[Libav-user] Recomended approach for a timed metadata track within an mp4 (ISOBMFF section

Tom Vercauteren tom.vercauteren at gmail.com
Thu Mar 31 13:15:42 EEST 2022


I am interested in embeding metadata corresponding to each video frame
within an mp4 file. I am however a bit unsure how to best approach this
with ffmpeg / libav. I tried looking for exisiting tutorials / guidelines
but couldn't find it so far and would thus appreciate further pointers.

>From what I read so far, it seems like frame-level metadata, also refered
to as timed metadata, is defined in section of the ISO/IEC base
media file format (ISOBMFF standardised in ISO/IEC 14496-12 – MPEG-4 Part
12) specification. Below is an excerpt for reference.

I have seen reference to such timed metadata tracks in other frameworks:

Reading though previous queries related to frame-level metadata with ffmpeg
/ libav, it seems like creating a stream of type AV_MEDIA_TYPE_DATA that
AV_CODEC_ID_TEXT or AV_CODEC_ID_BINTEXT may be a way forward but I am not
quite sure how to best approach it. Links to some of the most relevant
questions I found so far are provided below:

Is AV_MEDIA_TYPE_DATA indeed the right approach to achieving compliant
timed metadata and if so, what codec id would be appropriate? Also, how
should one set the MIME type format for the track? Any other help would be

Best wishes,

**** ISOBMFF excerpt
12.3 Metadata media
12.3.1 Media handler
Timed metadata media uses the ‘meta’ handler type in the handler box of the
media box [...].
12.3.2 Media header
Metadata tracks use a null media header (‘nmhd’), as defined in subclause
12.3.3 Sample entry Definition
Timed metadata tracks use MetaDataSampleEntry.
[...] Syntax
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://ffmpeg.org/pipermail/libav-user/attachments/20220331/3b6afbd3/attachment.htm>

More information about the Libav-user mailing list