Per frame metadata example?
Hello, I'm investigating on possibility to encode per frame metadata using ffmpeg API. Naive attempts to set a dictionary on "AVFrame.metadata" with some key values failed without any warning or error during encoding, but no values are retrieved on the field during decoding. I'm using ffmpeg 4.1.3 with modern send/receive API usage. Other users on SO seemed to try similar approach without success[1][2]. I've no specific requirements on container format: any format supporting this feature plus streaming would be good. I first tried with flv, mp4 formats and ended up testing nut format noticing some trace of specific support for the feature[3]. Is there any obvious mistake I'm possibly doing? Is the feature actually supported? Regards, Francesco [1] https://stackoverflow.com/questions/51415447/ffmpeg-sidedata-or-metadata-per... [2] https://stackoverflow.com/questions/39330048/ffmpeg-per-frame-metadata-with-... [3] https://ffmpeg.org/pipermail/ffmpeg-devel/2013-March/141270.html
On 7/13/19, Francesco Pretto <ceztko@gmail.com> wrote:
Hello,
I'm investigating on possibility to encode per frame metadata using ffmpeg API. Naive attempts to set a dictionary on "AVFrame.metadata" with some key values failed without any warning or error during encoding, but no values are retrieved on the field during decoding. I'm using ffmpeg 4.1.3 with modern send/receive API usage. Other users on SO seemed to try similar approach without success[1][2]. I've no specific requirements on container format: any format supporting this feature plus streaming would be good. I first tried with flv, mp4 formats and ended up testing nut format noticing some trace of specific support for the feature[3]. Is there any obvious mistake I'm possibly doing? Is the feature actually supported?
What gives you reason this is supported? None of containers mentioned can support it even theoretically.
Regards, Francesco
[1] https://stackoverflow.com/questions/51415447/ffmpeg-sidedata-or-metadata-per... [2] https://stackoverflow.com/questions/39330048/ffmpeg-per-frame-metadata-with-... [3] https://ffmpeg.org/pipermail/ffmpeg-devel/2013-March/141270.html _______________________________________________ ffmpeg-user mailing list ffmpeg-user@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-user
To unsubscribe, visit link above, or email ffmpeg-user-request@ffmpeg.org with subject "unsubscribe".
On Sat, 13 Jul 2019 at 20:21, Paul B Mahol <onemda@gmail.com> wrote:
What gives you reason this is supported? None of containers mentioned can support it even theoretically.
I didn't have knowledge enough to speculate about support of this feature on specific formats but I couldn't either find hints suggesting very limited support so I just tried. As said nut[1] format, which I also mentioned, seems to directly support it but I was not able to make it work by setting AVFrame.metadata and pushing the frame: when decoding the same field is always NULL. Without an example I can't understand if I am doing something wrong or the support for per-frame metadata is broken in nut. Any help is appreciated. Cheers, Francesco [1] https://ffmpeg.org/~michael/nut.txt
On 7/13/19, Francesco Pretto <ceztko@gmail.com> wrote:
On Sat, 13 Jul 2019 at 20:21, Paul B Mahol <onemda@gmail.com> wrote:
What gives you reason this is supported? None of containers mentioned can support it even theoretically.
I didn't have knowledge enough to speculate about support of this feature on specific formats but I couldn't either find hints suggesting very limited support so I just tried. As said nut[1] format, which I also mentioned, seems to directly support it but I was not able to make it work by setting AVFrame.metadata and pushing the frame: when decoding the same field is always NULL. Without an example I can't understand if I am doing something wrong or the support for per-frame metadata is broken in nut. Any help is appreciated.
NUT does not support that. What convinced you it does support it? The only thing that theoretically supports frame metadata is tiff and anything that uses exif. But note that frame metadata is implemented for decoding and for encoding it is simply not there.
Cheers, Francesco
[1] https://ffmpeg.org/~michael/nut.txt _______________________________________________ ffmpeg-user mailing list ffmpeg-user@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-user
To unsubscribe, visit link above, or email ffmpeg-user-request@ffmpeg.org with subject "unsubscribe".
On Sat, 13 Jul 2019 at 22:33, Paul B Mahol <onemda@gmail.com> wrote:
NUT does not support that. What convinced you it does support it?
The NUT specification[1] talks specifically about per-frame metadata support: ---------------------------------------------- sm_data / side_data / meta_data This data structure is used both in frames for per frame side and metadata as well as info tags for metadata covering the whole file, a stream chapter or other. [...] Types of per frame side data: [...] "UserData<identifer here>" User specific side data, the "<identifer here>" should be replaced by a globally unique identifer of the project that uses/creates/understands the side data. For example "UserDataFFmpeg" ---------------------------------------------- The nut encoder[2] and decoder[3] code seems to fully handle this data already at AVPacket level, so either this metadata was (improperly?) named per-frame when it's actually per-packet, and some glue code is missing that mux and demux the metadata between frames <-> packet, or there's a bug somewhere. Also this feature request[4] asks for per-frame metadata in Matroska similarly to what it was done in NUT. This was all the evidence I could find but no full example. Cheers, Francesco [1] http://ffmpeg.org/~michael/nut.txt [2] https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/nutenc.c , write_sm_data() [3] https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/nutdec.c , read_sm_data() [4] https://github.com/cellar-wg/matroska-specification/issues/270
On 7/13/19, Francesco Pretto <ceztko@gmail.com> wrote:
On Sat, 13 Jul 2019 at 22:33, Paul B Mahol <onemda@gmail.com> wrote:
NUT does not support that. What convinced you it does support it?
The NUT specification[1] talks specifically about per-frame metadata support:
That is very vague and unspecific wording to claim such feature. And specification is misleading you. Anyway NUT specification was written much before introduction of frame metadata as we know it. So what you actually pointed out have nothing to do with frame metadata, but with updating stream metadata within new packet side data.
---------------------------------------------- sm_data / side_data / meta_data This data structure is used both in frames for per frame side and metadata as well as info tags for metadata covering the whole file, a stream chapter or other. [...] Types of per frame side data: [...] "UserData<identifer here>" User specific side data, the "<identifer here>" should be replaced by a globally unique identifer of the project that uses/creates/understands the side data. For example "UserDataFFmpeg" ----------------------------------------------
The nut encoder[2] and decoder[3] code seems to fully handle this data already at AVPacket level, so either this metadata was (improperly?) named per-frame when it's actually per-packet, and some glue code is missing that mux and demux the metadata between frames <-> packet, or there's a bug somewhere.
Also this feature request[4] asks for per-frame metadata in Matroska similarly to what it was done in NUT. This was all the evidence I could find but no full example.
Cheers, Francesco
[1] http://ffmpeg.org/~michael/nut.txt [2] https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/nutenc.c , write_sm_data() [3] https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/nutdec.c , read_sm_data() [4] https://github.com/cellar-wg/matroska-specification/issues/270 _______________________________________________ ffmpeg-user mailing list ffmpeg-user@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-user
To unsubscribe, visit link above, or email ffmpeg-user-request@ffmpeg.org with subject "unsubscribe".
On Sat, Jul 13, 2019, 23:54 Paul B Mahol <onemda@gmail.com> wrote: That is very vague and unspecific wording to claim such feature. And specification is misleading you. Anyway NUT specification was written much before introduction of frame metadata as we know it. So what you actually pointed out have nothing to do with frame metadata, but with updating stream metadata within new packet side data. It's of course very likely that I can be mislead by that specification. If so, I would call it a problem of the specification itself to name something "per frame side and metadata" and not refer precisely to further data that has 1:1 relationship with frame. Look at the Matroska proposal because IMO they understood exactly this (referring for example to camera real-time parameters, e.g zoom/iris). I would love to hear a word from Michael about this feature and possible implementation in NUT but in the mean time I will consider it as missing from all formats. Cheers, Francesco
participants (2)
-
Francesco Pretto -
Paul B Mahol