[FFmpeg-user] editing video stream with KLV metadata

Moritz Barsnick barsnick at gmx.net
Mon May 7 15:33:13 EEST 2018


Hi Claire,

On Mon, May 07, 2018 at 10:41:44 +0000, Claire Mantel wrote:
> Yes, KLV is a standard for encoding data (https://en.wikipedia.org/wiki/KLV)
> I believe it's quite commonly used in commercial UAVs (drones) to tag the video with info like timestamp, GPS coordinates and pitch, roll and yaw of the camera.

I see.

> I tried the first command and it works, so thanks a lot!

Very nice, thanks for the feedback.

> So I guess it means ffmpeg is able to tell which KLV data is
> associated with which frame as the editing also edited the KLV stream
> properly.

That means that the data stream has timestamps as well (which seems
necessary, if you want to correlate changing GPS positions with a
recording).

> I'm also trying to output the KLV data in some text file, I tried 
> ffmpeg -i video_out_2018_05_01_14_11_53.ts -f ffmetadata testMetadata.txt
> ffmpeg -i video_out_2018_05_01_14_11_53.ts -map 0:d -f ffmetadata testMetadata.txt
> ffmpeg -i video_out_2018_05_01_14_11_53.ts -c:d copy  -map 0:d -f ffmetadata testMetadata.txt

> but I only get " ;FFMETADATA1 encoder=Lavf58.13.100 [STREAM]"  in my testMetadata.txt file

Again, as far as ffmpeg is concerned, that KLV stream is a binary data
stream with unknown content. While that content may be "metadata", what
ffmpeg usually considers as metadata is additional data which is
"attached" to a stream, not contained. That's why your ffmpeg metadata
commands are to no avail.

> Would you have any input?

You need to access the binary data of the stream. One method is
outlined here:
https://stackoverflow.com/a/29461404/3974309

So you can dump that data to a file. As far as I know, it will lose its
timestamps though. (I *may* be wrong here.) You would need to read the
KLV specification to understand that data, ffmpeg doesn't.

There was once a suggestion for an "fftextdata" muxer (and demuxer)
which might suit your needs. It would dump each received packet with a
timestamp and the base64-encoded hex data payload. Again, you need to
interpret the data on your own.

The patch was never accepted (and perhaps was never meant to be):
https://ffmpeg.org/pipermail/ffmpeg-devel/2016-May/194445.html
I am attaching a rebased version, in case it's something which could
help you. I assume it doesn't help, as you actually want to have the
KLV data interpreted?

Med vennlig hilsen,
Moritz
-------------- next part --------------
diff -urN ffmpeg-snapshot-2018-02-07/doc/demuxers.texi ffmpeg-snapshot-2018-02-07-textdata/doc/demuxers.texi
--- ffmpeg-snapshot-2018-02-07/doc/demuxers.texi	2018-01-14 18:20:06.000000000 +0100
+++ ffmpeg-snapshot-2018-02-07-textdata/doc/demuxers.texi	2018-02-08 17:25:37.000000000 +0100
@@ -471,6 +471,46 @@
 @end example
 @end itemize
 
+ at anchor{fftextdata}
+ at section fftextdata, fftd
+
+FFmpeg text data demuxer.
+
+This special demuxer allows to read serialized data base64-encoded and
+remux it. It is especially useful for injecting opaque data streams.
+
+The fftextdata bytestream consists of a sequence of packets. Each
+packet starts with a timestamps expressed in a format recognized by
+FFmpeg (see
+ at ref{time duration syntax,,the Time duration section in the ffmpeg-utils(1) manual,ffmpeg-utils})
+followed by a sequence of spaces and the base64 encoded data for the
+packet, terminated by ";". The data representation may contain
+interleaved space characters (a space, a tab, or a newline) which are
+ignored.
+
+At the moment a single stream can be represented by an fftextdata
+bytestream.
+
+If an input filename is "fftextdata" or "fftd" then the file format is
+recognized as fftextdata.
+
+ at subsection Options
+ at table @option
+ at item codec_name
+Set the codec name for the packets data.
+ at end table
+
+ at subsection Examples
+
+ at itemize
+ at item
+Inject timed_id3 packed data stored into the data.fftd file into the
+output file.
+ at example
+ffmpeg -i input.mp4 -codec_name timed_id3 -f fftextdata -i data.fftd -y -map 0 -map 1 -c copy output.ts
+ at end example
+ at end itemize
+
 @section libgme
 
 The Game Music Emu library is a collection of video game music file emulators.
diff -urN ffmpeg-snapshot-2018-02-07/doc/muxers.texi ffmpeg-snapshot-2018-02-07-textdata/doc/muxers.texi
--- ffmpeg-snapshot-2018-02-07/doc/muxers.texi	2018-01-24 18:20:05.000000000 +0100
+++ ffmpeg-snapshot-2018-02-07-textdata/doc/muxers.texi	2018-02-08 17:25:37.000000000 +0100
@@ -263,6 +263,37 @@
 When no assignment is defined, this defaults to an AdaptationSet for each stream.
 @end table
 
+ at section fftextdata, fftd
+
+FFmpeg text data muxer.
+
+The fftextdata bytestream consists of a sequence of packets. Each
+packet starts with a timestamps expressed in a format recognized by
+FFmpeg (see
+ at ref{time duration syntax,,the Time duration section in the ffmpeg-utils(1) manual,ffmpeg-utils})
+followed by a sequence of spaces and the base64 encoded data for the
+packet, terminated by ";". The data representation may contain
+interleaved space characters (a space, a tab, or a newline) which are
+ignored.
+
+At the moment only a single stream can be represented by an fftextdata
+bytestream.
+
+This muxer can be used to reinject the stream (e.g. a data stream) in
+a different output, or to provide serialized data of the encoded data.
+
+The output can then be read using the fftextdata demuxer.
+
+ at subsection Examples
+
+ at itemize
+ at item
+Store a data stream to an output file:
+ at example
+ffmpeg -i INPUT -codec copy -map 0 -an -vn data.fftd
+ at end example
+ at end itemize
+
 @anchor{framecrc}
 @section framecrc
 
diff -urN ffmpeg-snapshot-2018-02-07/libavformat/allformats.c ffmpeg-snapshot-2018-02-07-textdata/libavformat/allformats.c
--- ffmpeg-snapshot-2018-02-07/libavformat/allformats.c	2018-02-07 18:20:05.000000000 +0100
+++ ffmpeg-snapshot-2018-02-07-textdata/libavformat/allformats.c	2018-02-08 17:26:22.000000000 +0100
@@ -127,6 +127,8 @@
 extern AVOutputFormat ff_f4v_muxer;
 extern AVInputFormat  ff_ffmetadata_demuxer;
 extern AVOutputFormat ff_ffmetadata_muxer;
+extern AVInputFormat  ff_fftextdata_demuxer;
+extern AVOutputFormat ff_fftextdata_muxer;
 extern AVOutputFormat ff_fifo_muxer;
 extern AVOutputFormat ff_fifo_test_muxer;
 extern AVInputFormat  ff_filmstrip_demuxer;
diff -urN ffmpeg-snapshot-2018-02-07/libavformat/fftextdatadec.c ffmpeg-snapshot-2018-02-07-textdata/libavformat/fftextdatadec.c
--- ffmpeg-snapshot-2018-02-07/libavformat/fftextdatadec.c	1970-01-01 01:00:00.000000000 +0100
+++ ffmpeg-snapshot-2018-02-07-textdata/libavformat/fftextdatadec.c	2018-02-08 17:25:37.000000000 +0100
@@ -0,0 +1,212 @@
+/*
+ * Copyright (c) 2016 Stefano Sabatini
+ *
+ * This file is part of FFmpeg.
+ *
+ * FFmpeg is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * FFmpeg is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with FFmpeg; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+/**
+ * @file
+ * timestamped data virtual demuxer
+ */
+
+#include "libavutil/base64.h"
+#include "libavutil/bprint.h"
+#include "libavutil/opt.h"
+#include "libavutil/parseutils.h"
+#include "avformat.h"
+#include "internal.h"
+
+typedef struct {
+    const AVClass *class;  /**< Class for private options. */
+    int nb_packets;
+    AVBPrint bp;
+    const char *codec_name;
+} FFTextdataContext;
+
+av_cold static int fftextdata_read_close(AVFormatContext *avctx)
+{
+    FFTextdataContext *td = avctx->priv_data;
+
+    av_bprint_finalize(&td->bp, NULL);
+    return 0;
+}
+
+av_cold static int fftextdata_read_header(AVFormatContext *s)
+{
+    FFTextdataContext *td = s->priv_data;
+    AVStream *st;
+    const AVCodecDescriptor *cd;
+
+    st = avformat_new_stream(s, NULL);
+    if (!st)
+        return AVERROR(ENOMEM);
+
+    cd = avcodec_descriptor_get_by_name(td->codec_name);
+    if (!cd) {
+        av_log(s, AV_LOG_ERROR, "Impossible to find a codec with name '%s'\n",
+               td->codec_name);
+        return AVERROR(EINVAL);
+    }
+
+    st->codecpar->codec_type = cd->type;
+    st->codecpar->codec_id = cd->id;
+    avpriv_set_pts_info(st, 64, 1, 1000000);
+
+    av_bprint_init(&(td->bp), 0, 1);
+    td->nb_packets = 0;
+
+    return 0;
+}
+
+static inline int is_space(char c)
+{
+    return c == ' '  || c == '\t' || c == '\r' || c == '\n';
+}
+
+static int read_word(AVIOContext *avio, AVBPrint *bp)
+{
+    int c;
+
+    av_bprint_clear(bp);
+
+    /* skip spaces */
+    do {
+        c = avio_r8(avio);
+        if (!c)
+            goto end;
+    } while (is_space(c));
+
+    /* read word */
+    av_bprint_chars(bp, c, 1);
+    do {
+        c = avio_r8(avio);
+        if (!c)
+            goto end;
+        if (is_space(c)) {
+            avio_skip(avio, -1);
+            goto end;
+        }
+        av_bprint_chars(bp, c, 1);
+    } while (1);
+
+end:
+    return bp->len;
+}
+
+static int read_data(AVIOContext *avio, AVBPrint *bp)
+{
+    int c;
+
+    av_bprint_clear(bp);
+
+    /* skip spaces */
+    do {
+        c = avio_r8(avio);
+        if (!c)
+            goto end;
+    } while (is_space(c));
+
+    /* read data chunk */
+    av_bprint_chars(bp, c, 1);
+    do {
+        c = avio_r8(avio);
+        if (!c || c == ';')
+            goto end;
+        if (is_space(c)) {
+            continue;
+        }
+        av_bprint_chars(bp, c, 1);
+    } while (1);
+
+end:
+    return bp->len;
+}
+
+static int fftextdata_read_packet(AVFormatContext *s, AVPacket *pkt)
+{
+    FFTextdataContext *td = s->priv_data;
+    AVIOContext *avio = s->pb;
+    int ret;
+    AVBPrint *bp = &(td->bp);
+
+    pkt->pos = avio_tell(avio);
+
+    /* read PTS  */
+    ret = read_word(avio, bp);
+    if (ret == 0)
+        return AVERROR_EOF;
+
+    ret = av_parse_time(&pkt->pts, bp->str, 1);
+    if (ret < 0) {
+        av_log(s, AV_LOG_ERROR, "Invalid time specification '%s' for data packet #%d\n",
+               bp->str, td->nb_packets);
+        return ret;
+    }
+
+    ret = read_data(avio, bp);
+    if (ret == 0) {
+        av_log(s, AV_LOG_WARNING, "Incomplete packet #%d with no data at the end of the data stream\n",
+               td->nb_packets);
+        return AVERROR_EOF;
+    }
+
+    pkt->size = AV_BASE64_DECODE_SIZE(ret);
+    pkt->data = av_malloc(pkt->size);
+    if (ret < 0)
+        return ret;
+
+    ret = av_base64_decode(pkt->data, bp->str, pkt->size);
+    if (ret < 0) {
+        av_freep(&pkt->data);
+        return ret;
+    }
+
+    pkt->size = ret;
+    pkt->flags |= AV_PKT_FLAG_KEY;
+    td->nb_packets++;
+
+    return ret;
+}
+
+#define OFFSET(x) offsetof(FFTextdataContext, x)
+
+#define D AV_OPT_FLAG_DECODING_PARAM
+
+#define OFFSET(x) offsetof(FFTextdataContext, x)
+
+static const AVOption options[] = {
+    { "codec_name",  "set output codec name", OFFSET(codec_name), AV_OPT_TYPE_STRING, {.str = "bin_data"}, CHAR_MIN, CHAR_MAX, D },
+    { NULL },
+};
+
+static const AVClass fftextdata_class = {
+    .class_name = "fftexdata demuxer",
+    .item_name  = av_default_item_name,
+    .option     = options,
+    .version    = LIBAVUTIL_VERSION_INT,
+};
+
+AVInputFormat ff_fftextdata_demuxer = {
+    .name           = "fftextdata",
+    .long_name      = NULL_IF_CONFIG_SMALL("Timestamped data virtual demuxer"),
+    .extensions     = "fftextdata,fftd",
+    .priv_data_size = sizeof(FFTextdataContext),
+    .read_header    = fftextdata_read_header,
+    .read_packet    = fftextdata_read_packet,
+    .read_close     = fftextdata_read_close,
+    .priv_class     = &fftextdata_class,
+};
diff -urN ffmpeg-snapshot-2018-02-07/libavformat/fftextdataenc.c ffmpeg-snapshot-2018-02-07-textdata/libavformat/fftextdataenc.c
--- ffmpeg-snapshot-2018-02-07/libavformat/fftextdataenc.c	1970-01-01 01:00:00.000000000 +0100
+++ ffmpeg-snapshot-2018-02-07-textdata/libavformat/fftextdataenc.c	2018-02-08 17:25:37.000000000 +0100
@@ -0,0 +1,103 @@
+/*
+ * Copyright (c) 2016 Stefano Sabatini
+ *
+ * This file is part of FFmpeg.
+ *
+ * FFmpeg is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * FFmpeg is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with FFmpeg; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ */
+
+/**
+ * @file
+ * timestamped data virtual muxer
+ */
+
+#include "avformat.h"
+#include "libavutil/base64.h"
+
+typedef struct {
+    uint8_t *buf;
+    size_t   buf_size;
+} FFTextdataContext;
+
+static int fftextdata_write_header(AVFormatContext *s)
+{
+    FFTextdataContext *td = s->priv_data;
+
+    td->buf = NULL;
+    td->buf_size = 0;
+
+    return 0;
+}
+
+static int fftextdata_write_trailer(AVFormatContext *s)
+{
+    FFTextdataContext *td = s->priv_data;
+
+    av_freep(&td->buf);
+    td->buf_size = 0;
+
+    return 0;
+}
+
+static int fftextdata_write_packet(AVFormatContext *s, AVPacket *pkt)
+{
+    FFTextdataContext *td = s->priv_data;
+    char ts[32];
+    size_t encoded_data_size;
+    AVStream *st = s->streams[pkt->stream_index];
+    int64_t pts = pkt->pts;
+    double secs;
+    int hours, mins;
+
+    if (st->start_time != AV_NOPTS_VALUE)
+        pts += st->start_time;
+
+    secs = (double)pkt->pts * av_q2d(st->time_base);
+    mins  = (int)secs / 60;
+    secs  = secs - mins * 60;
+    hours = mins / 60;
+    mins %= 60;
+    snprintf(ts, sizeof(ts), "%d:%02d:%09.6f", hours, mins, secs);
+    avio_put_str(s->pb, ts);
+    avio_skip(s->pb, -1);
+    avio_w8(s->pb, '\n');
+
+    encoded_data_size = AV_BASE64_SIZE(pkt->size);
+    if (encoded_data_size > td->buf_size) {
+        td->buf = av_realloc_f(td->buf, encoded_data_size, 1);
+        if (!td->buf)
+            return AVERROR(ENOMEM);
+        td->buf_size = encoded_data_size;
+    }
+
+    av_base64_encode(td->buf, td->buf_size, pkt->data, pkt->size);
+    avio_put_str(s->pb, td->buf);
+    avio_skip(s->pb, -1);
+
+    avio_put_str(s->pb, "\n;\n");
+    avio_skip(s->pb, -1);
+
+    return 0;
+}
+
+AVOutputFormat ff_fftextdata_muxer = {
+    .name          = "fftextdata",
+    .long_name     = NULL_IF_CONFIG_SMALL("Timestamped data virtual muxer"),
+    .extensions    = "fftextdata,fftd",
+    .priv_data_size = sizeof(FFTextdataContext),
+    .write_header  = fftextdata_write_header,
+    .write_packet  = fftextdata_write_packet,
+    .write_trailer = fftextdata_write_trailer,
+};
diff -urN ffmpeg-snapshot-2018-02-07/libavformat/Makefile ffmpeg-snapshot-2018-02-07-textdata/libavformat/Makefile
--- ffmpeg-snapshot-2018-02-07/libavformat/Makefile	2018-02-07 18:20:05.000000000 +0100
+++ ffmpeg-snapshot-2018-02-07-textdata/libavformat/Makefile	2018-02-08 17:25:37.000000000 +0100
@@ -163,6 +163,8 @@
 OBJS-$(CONFIG_EPAF_DEMUXER)              += epafdec.o pcm.o
 OBJS-$(CONFIG_FFMETADATA_DEMUXER)        += ffmetadec.o
 OBJS-$(CONFIG_FFMETADATA_MUXER)          += ffmetaenc.o
+OBJS-$(CONFIG_FFTEXTDATA_DEMUXER)        += fftextdatadec.o
+OBJS-$(CONFIG_FFTEXTDATA_MUXER)          += fftextdataenc.o
 OBJS-$(CONFIG_FIFO_MUXER)                += fifo.o
 OBJS-$(CONFIG_FIFO_TEST_MUXER)           += fifo_test.o
 OBJS-$(CONFIG_FILMSTRIP_DEMUXER)         += filmstripdec.o


More information about the ffmpeg-user mailing list