October 2019 - ffmpeg-cvslog - lists.ffmpeg.org

doc/filters: move agraphmonitor to multimedia filters chapter
by Paul B Mahol 30 Oct '19

30 Oct '19

ffmpeg | branch: master | Paul B Mahol <onemda(a)gmail.com> | Wed Oct 30 17:47:05 2019 +0100| [2539fea777a7ebbb211280f7de0bc872be23b1fa] | committer: Paul B Mahol doc/filters: move agraphmonitor to multimedia filters chapter > http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=2539fea777a7ebbb2… --- doc/filters.texi | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/doc/filters.texi b/doc/filters.texi index 11f715319a..8fdcb196b5 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -11202,7 +11202,8 @@ gradfun=radius=8 @end itemize -@section graphmonitor, agraphmonitor +@anchor{graphmonitor} +@section graphmonitor Show various filtergraph stats. With this filter one can debug complete filtergraph. @@ -21497,6 +21498,10 @@ draw channels. Unrecognized or missing colors will be replaced by white color. @end table +@section agraphmonitor + +See @ref{graphmonitor}. + @section ahistogram Convert input audio to a video output, displaying the volume histogram.

1 0

avfilter/vf_sr: correct flags since the filter changes frame w/h
by Guo, Yejun 30 Oct '19

30 Oct '19

ffmpeg | branch: master | Guo, Yejun <yejun.guo(a)intel.com> | Mon Oct 28 21:11:02 2019 +0800| [912ab246f16699b2b2c782ab63884b00f2e39dae] | committer: Pedro Arthur avfilter/vf_sr: correct flags since the filter changes frame w/h If filter changes frame w/h, AVFILTER_FLAG_SUPPORT_TIMELINE_GENERIC cannot be supported. Signed-off-by: Guo, Yejun <yejun.guo(a)intel.com> Signed-off-by: Pedro Arthur <bygrandao(a)gmail.com> > http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=912ab246f16699b2b… --- libavfilter/vf_sr.c | 1 - 1 file changed, 1 deletion(-) diff --git a/libavfilter/vf_sr.c b/libavfilter/vf_sr.c index fff19ea693..562b03069b 100644 --- a/libavfilter/vf_sr.c +++ b/libavfilter/vf_sr.c @@ -317,5 +317,4 @@ AVFilter ff_vf_sr = { .inputs = sr_inputs, .outputs = sr_outputs, .priv_class = &sr_class, - .flags = AVFILTER_FLAG_SUPPORT_TIMELINE_GENERIC, };

1 0

avfilter/dnn: add a new interface to query dnn model's input info
by Guo, Yejun 30 Oct '19

30 Oct '19

ffmpeg | branch: master | Guo, Yejun <yejun.guo(a)intel.com> | Mon Oct 21 20:38:17 2019 +0800| [f4b3c0e55c84434eb897c2a4a1179cb1d202c52c] | committer: Pedro Arthur avfilter/dnn: add a new interface to query dnn model's input info to support dnn networks more general, we need to know the input info of the dnn model. background: The data type of dnn model's input could be float32, uint8 or fp16, etc. And the w/h of input image could be fixed or variable. Signed-off-by: Guo, Yejun <yejun.guo(a)intel.com> Signed-off-by: Pedro Arthur <bygrandao(a)gmail.com> > http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=f4b3c0e55c84434eb… --- libavfilter/dnn/dnn_backend_native.c | 24 +++++++++++++++++++++++- libavfilter/dnn/dnn_backend_tf.c | 32 ++++++++++++++++++++++++++++++++ libavfilter/dnn_interface.h | 3 +++ 3 files changed, 58 insertions(+), 1 deletion(-) diff --git a/libavfilter/dnn/dnn_backend_native.c b/libavfilter/dnn/dnn_backend_native.c index add1db42cf..94634b3065 100644 --- a/libavfilter/dnn/dnn_backend_native.c +++ b/libavfilter/dnn/dnn_backend_native.c @@ -28,6 +28,28 @@ #include "dnn_backend_native_layer_conv2d.h" #include "dnn_backend_native_layers.h" +static DNNReturnType get_input_native(void *model, DNNData *input, const char *input_name) +{ + ConvolutionalNetwork *network = (ConvolutionalNetwork *)model; + + for (int i = 0; i < network->operands_num; ++i) { + DnnOperand *oprd = &network->operands[i]; + if (strcmp(oprd->name, input_name) == 0) { + if (oprd->type != DOT_INPUT) + return DNN_ERROR; + input->dt = oprd->data_type; + av_assert0(oprd->dims[0] == 1); + input->height = oprd->dims[1]; + input->width = oprd->dims[2]; + input->channels = oprd->dims[3]; + return DNN_SUCCESS; + } + } + + // do not find the input operand + return DNN_ERROR; +} + static DNNReturnType set_input_output_native(void *model, DNNData *input, const char *input_name, const char **output_names, uint32_t nb_output) { ConvolutionalNetwork *network = (ConvolutionalNetwork *)model; @@ -37,7 +59,6 @@ static DNNReturnType set_input_output_native(void *model, DNNData *input, const return DNN_ERROR; /* inputs */ - av_assert0(input->dt == DNN_FLOAT); for (int i = 0; i < network->operands_num; ++i) { oprd = &network->operands[i]; if (strcmp(oprd->name, input_name) == 0) { @@ -234,6 +255,7 @@ DNNModel *ff_dnn_load_model_native(const char *model_filename) } model->set_input_output = &set_input_output_native; + model->get_input = &get_input_native; return model; } diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index ed91d0500d..a921667424 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -105,6 +105,37 @@ static TF_Tensor *allocate_input_tensor(const DNNData *input) input_dims[1] * input_dims[2] * input_dims[3] * size); } +static DNNReturnType get_input_tf(void *model, DNNData *input, const char *input_name) +{ + TFModel *tf_model = (TFModel *)model; + TF_Status *status; + int64_t dims[4]; + + TF_Output tf_output; + tf_output.oper = TF_GraphOperationByName(tf_model->graph, input_name); + if (!tf_output.oper) + return DNN_ERROR; + + tf_output.index = 0; + input->dt = TF_OperationOutputType(tf_output); + + status = TF_NewStatus(); + TF_GraphGetTensorShape(tf_model->graph, tf_output, dims, 4, status); + if (TF_GetCode(status) != TF_OK){ + TF_DeleteStatus(status); + return DNN_ERROR; + } + TF_DeleteStatus(status); + + // currently only NHWC is supported + av_assert0(dims[0] == 1); + input->height = dims[1]; + input->width = dims[2]; + input->channels = dims[3]; + + return DNN_SUCCESS; +} + static DNNReturnType set_input_output_tf(void *model, DNNData *input, const char *input_name, const char **output_names, uint32_t nb_output) { TFModel *tf_model = (TFModel *)model; @@ -568,6 +599,7 @@ DNNModel *ff_dnn_load_model_tf(const char *model_filename) model->model = (void *)tf_model; model->set_input_output = &set_input_output_tf; + model->get_input = &get_input_tf; return model; } diff --git a/libavfilter/dnn_interface.h b/libavfilter/dnn_interface.h index fdefcb708b..b20e5c8fab 100644 --- a/libavfilter/dnn_interface.h +++ b/libavfilter/dnn_interface.h @@ -43,6 +43,9 @@ typedef struct DNNData{ typedef struct DNNModel{ // Stores model that can be different for different backends. void *model; + // Gets model input information + // Just reuse struct DNNData here, actually the DNNData.data field is not needed. + DNNReturnType (*get_input)(void *model, DNNData *input, const char *input_name); // Sets model input and output. // Should be called at least once before model execution. DNNReturnType (*set_input_output)(void *model, DNNData *input, const char *input_name, const char **output_names, uint32_t nb_output);

1 0

avfilter/dnn: get the data type of network output from dnn execution result
by Guo, Yejun 30 Oct '19

30 Oct '19

ffmpeg | branch: master | Guo, Yejun <yejun.guo(a)intel.com> | Mon Oct 21 20:38:10 2019 +0800| [e1b45b85963b5aa9d67e23638ef9b045e7fbd875] | committer: Pedro Arthur avfilter/dnn: get the data type of network output from dnn execution result so, we can make a filter more general to accept different network models, by adding a data type convertion after getting data from network. After we add dt field into struct DNNData, it becomes the same as DNNInputData, so merge them with one struct: DNNData. Signed-off-by: Guo, Yejun <yejun.guo(a)intel.com> Signed-off-by: Pedro Arthur <bygrandao(a)gmail.com> > http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=e1b45b85963b5aa9d… --- libavfilter/dnn/dnn_backend_native.c | 3 ++- libavfilter/dnn/dnn_backend_native_layer_conv2d.c | 1 + libavfilter/dnn/dnn_backend_native_layer_depth2space.c | 1 + libavfilter/dnn/dnn_backend_native_layer_pad.c | 1 + libavfilter/dnn/dnn_backend_tf.c | 5 +++-- libavfilter/dnn_interface.h | 9 ++------- libavfilter/vf_derain.c | 4 ++-- libavfilter/vf_sr.c | 2 +- 8 files changed, 13 insertions(+), 13 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_native.c b/libavfilter/dnn/dnn_backend_native.c index ff280b5506..add1db42cf 100644 --- a/libavfilter/dnn/dnn_backend_native.c +++ b/libavfilter/dnn/dnn_backend_native.c @@ -28,7 +28,7 @@ #include "dnn_backend_native_layer_conv2d.h" #include "dnn_backend_native_layers.h" -static DNNReturnType set_input_output_native(void *model, DNNInputData *input, const char *input_name, const char **output_names, uint32_t nb_output) +static DNNReturnType set_input_output_native(void *model, DNNData *input, const char *input_name, const char **output_names, uint32_t nb_output) { ConvolutionalNetwork *network = (ConvolutionalNetwork *)model; DnnOperand *oprd = NULL; @@ -263,6 +263,7 @@ DNNReturnType ff_dnn_execute_model_native(const DNNModel *model, DNNData *output outputs[i].height = oprd->dims[1]; outputs[i].width = oprd->dims[2]; outputs[i].channels = oprd->dims[3]; + outputs[i].dt = oprd->data_type; } return DNN_SUCCESS; diff --git a/libavfilter/dnn/dnn_backend_native_layer_conv2d.c b/libavfilter/dnn/dnn_backend_native_layer_conv2d.c index 6ec0fa7a99..7b296979a9 100644 --- a/libavfilter/dnn/dnn_backend_native_layer_conv2d.c +++ b/libavfilter/dnn/dnn_backend_native_layer_conv2d.c @@ -106,6 +106,7 @@ int dnn_execute_layer_conv2d(DnnOperand *operands, const int32_t *input_operand_ output_operand->dims[1] = height - pad_size * 2; output_operand->dims[2] = width - pad_size * 2; output_operand->dims[3] = conv_params->output_num; + output_operand->data_type = operands[input_operand_index].data_type; output_operand->length = calculate_operand_data_length(output_operand); output_operand->data = av_realloc(output_operand->data, output_operand->length); if (!output_operand->data) diff --git a/libavfilter/dnn/dnn_backend_native_layer_depth2space.c b/libavfilter/dnn/dnn_backend_native_layer_depth2space.c index 174676e14a..7dab19d40f 100644 --- a/libavfilter/dnn/dnn_backend_native_layer_depth2space.c +++ b/libavfilter/dnn/dnn_backend_native_layer_depth2space.c @@ -69,6 +69,7 @@ int dnn_execute_layer_depth2space(DnnOperand *operands, const int32_t *input_ope output_operand->dims[1] = height * block_size; output_operand->dims[2] = width * block_size; output_operand->dims[3] = new_channels; + output_operand->data_type = operands[input_operand_index].data_type; output_operand->length = calculate_operand_data_length(output_operand); output_operand->data = av_realloc(output_operand->data, output_operand->length); if (!output_operand->data) diff --git a/libavfilter/dnn/dnn_backend_native_layer_pad.c b/libavfilter/dnn/dnn_backend_native_layer_pad.c index 8fa35de196..8e5959bdd1 100644 --- a/libavfilter/dnn/dnn_backend_native_layer_pad.c +++ b/libavfilter/dnn/dnn_backend_native_layer_pad.c @@ -105,6 +105,7 @@ int dnn_execute_layer_pad(DnnOperand *operands, const int32_t *input_operand_ind output_operand->dims[1] = new_height; output_operand->dims[2] = new_width; output_operand->dims[3] = new_channel; + output_operand->data_type = operands[input_operand_index].data_type; output_operand->length = calculate_operand_data_length(output_operand); output_operand->data = av_realloc(output_operand->data, output_operand->length); if (!output_operand->data) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index c8dff51744..ed91d0500d 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -83,7 +83,7 @@ static TF_Buffer *read_graph(const char *model_filename) return graph_buf; } -static TF_Tensor *allocate_input_tensor(const DNNInputData *input) +static TF_Tensor *allocate_input_tensor(const DNNData *input) { TF_DataType dt; size_t size; @@ -105,7 +105,7 @@ static TF_Tensor *allocate_input_tensor(const DNNInputData *input) input_dims[1] * input_dims[2] * input_dims[3] * size); } -static DNNReturnType set_input_output_tf(void *model, DNNInputData *input, const char *input_name, const char **output_names, uint32_t nb_output) +static DNNReturnType set_input_output_tf(void *model, DNNData *input, const char *input_name, const char **output_names, uint32_t nb_output) { TFModel *tf_model = (TFModel *)model; TF_SessionOptions *sess_opts; @@ -603,6 +603,7 @@ DNNReturnType ff_dnn_execute_model_tf(const DNNModel *model, DNNData *outputs, u outputs[i].width = TF_Dim(tf_model->output_tensors[i], 2); outputs[i].channels = TF_Dim(tf_model->output_tensors[i], 3); outputs[i].data = TF_TensorData(tf_model->output_tensors[i]); + outputs[i].dt = TF_TensorType(tf_model->output_tensors[i]); } return DNN_SUCCESS; diff --git a/libavfilter/dnn_interface.h b/libavfilter/dnn_interface.h index 057005f47e..fdefcb708b 100644 --- a/libavfilter/dnn_interface.h +++ b/libavfilter/dnn_interface.h @@ -34,15 +34,10 @@ typedef enum {DNN_NATIVE, DNN_TF} DNNBackendType; typedef enum {DNN_FLOAT = 1, DNN_UINT8 = 4} DNNDataType; -typedef struct DNNInputData{ +typedef struct DNNData{ void *data; DNNDataType dt; int width, height, channels; -} DNNInputData; - -typedef struct DNNData{ - float *data; - int width, height, channels; } DNNData; typedef struct DNNModel{ @@ -50,7 +45,7 @@ typedef struct DNNModel{ void *model; // Sets model input and output. // Should be called at least once before model execution. - DNNReturnType (*set_input_output)(void *model, DNNInputData *input, const char *input_name, const char **output_names, uint32_t nb_output); + DNNReturnType (*set_input_output)(void *model, DNNData *input, const char *input_name, const char **output_names, uint32_t nb_output); } DNNModel; // Stores pointers to functions for loading, executing, freeing DNN models for one of the backends. diff --git a/libavfilter/vf_derain.c b/libavfilter/vf_derain.c index b7bba09464..89f9d5a2ed 100644 --- a/libavfilter/vf_derain.c +++ b/libavfilter/vf_derain.c @@ -39,7 +39,7 @@ typedef struct DRContext { DNNBackendType backend_type; DNNModule *dnn_module; DNNModel *model; - DNNInputData input; + DNNData input; DNNData output; } DRContext; @@ -137,7 +137,7 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *in) int t = i * out->width * 3 + j; int t_in = (i + pad_size) * in->width * 3 + j + pad_size * 3; - out->data[0][k] = CLIP((int)((((float *)dr_context->input.data)[t_in] - dr_context->output.data[t]) * 255), 0, 255); + out->data[0][k] = CLIP((int)((((float *)dr_context->input.data)[t_in] - ((float *)dr_context->output.data)[t]) * 255), 0, 255); } } diff --git a/libavfilter/vf_sr.c b/libavfilter/vf_sr.c index 0433246e26..fff19ea693 100644 --- a/libavfilter/vf_sr.c +++ b/libavfilter/vf_sr.c @@ -41,7 +41,7 @@ typedef struct SRContext { DNNBackendType backend_type; DNNModule *dnn_module; DNNModel *model; - DNNInputData input; + DNNData input; DNNData output; int scale_factor; struct SwsContext *sws_contexts[3];

1 0

dnn: add tf.nn.conv2d support for native model
by Guo, Yejun 30 Oct '19

30 Oct '19

ffmpeg | branch: master | Guo, Yejun <yejun.guo(a)intel.com> | Mon Oct 21 20:38:03 2019 +0800| [dff39ea9f0154ec52b7548b122a4a5332df3c2c6] | committer: Pedro Arthur dnn: add tf.nn.conv2d support for native model Unlike other tf.*.conv2d layers, tf.nn.conv2d does not create many nodes (within a scope) in the graph, it just acts like other layers. tf.nn.conv2d only creates one node in the graph, and no internal nodes such as 'kernel' are created. The format of native model file is also changed, a flag named has_bias is added, so change the version number. Signed-off-by: Guo, Yejun <yejun.guo(a)intel.com> Signed-off-by: Pedro Arthur <bygrandao(a)gmail.com> > http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=dff39ea9f0154ec52… --- libavfilter/dnn/dnn_backend_native.c | 2 +- libavfilter/dnn/dnn_backend_native_layer_conv2d.c | 37 +++++++++++----- libavfilter/dnn/dnn_backend_native_layer_conv2d.h | 1 + tests/dnn/dnn-layer-conv2d-test.c | 2 + tools/python/convert_from_tensorflow.py | 54 ++++++++++++++++++++--- tools/python/convert_header.py | 4 +- 6 files changed, 82 insertions(+), 18 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_native.c b/libavfilter/dnn/dnn_backend_native.c index 06b010d90e..ff280b5506 100644 --- a/libavfilter/dnn/dnn_backend_native.c +++ b/libavfilter/dnn/dnn_backend_native.c @@ -98,7 +98,7 @@ DNNModel *ff_dnn_load_model_native(const char *model_filename) char header_expected[] = "FFMPEGDNNNATIVE"; char *buf; size_t size; - int version, header_size, major_version_expected = 0; + int version, header_size, major_version_expected = 1; ConvolutionalNetwork *network = NULL; AVIOContext *model_file_context; int file_size, dnn_size, parsed_size; diff --git a/libavfilter/dnn/dnn_backend_native_layer_conv2d.c b/libavfilter/dnn/dnn_backend_native_layer_conv2d.c index 0de890217d..6ec0fa7a99 100644 --- a/libavfilter/dnn/dnn_backend_native_layer_conv2d.c +++ b/libavfilter/dnn/dnn_backend_native_layer_conv2d.c @@ -38,27 +38,41 @@ int dnn_load_layer_conv2d(Layer *layer, AVIOContext *model_file_context, int fil conv_params->input_num = (int32_t)avio_rl32(model_file_context); conv_params->output_num = (int32_t)avio_rl32(model_file_context); conv_params->kernel_size = (int32_t)avio_rl32(model_file_context); + conv_params->has_bias = (int32_t)avio_rl32(model_file_context); + dnn_size += 28; + kernel_size = conv_params->input_num * conv_params->output_num * - conv_params->kernel_size * conv_params->kernel_size; - dnn_size += 24 + (kernel_size + conv_params->output_num << 2); + conv_params->kernel_size * conv_params->kernel_size; + dnn_size += kernel_size * 4; + if (conv_params->has_bias) + dnn_size += conv_params->output_num * 4; + if (dnn_size > file_size || conv_params->input_num <= 0 || conv_params->output_num <= 0 || conv_params->kernel_size <= 0){ av_freep(&conv_params); return 0; } + conv_params->kernel = av_malloc(kernel_size * sizeof(float)); - conv_params->biases = av_malloc(conv_params->output_num * sizeof(float)); - if (!conv_params->kernel || !conv_params->biases){ - av_freep(&conv_params->kernel); - av_freep(&conv_params->biases); + if (!conv_params->kernel) { av_freep(&conv_params); return 0; } - for (int i = 0; i < kernel_size; ++i){ + for (int i = 0; i < kernel_size; ++i) { conv_params->kernel[i] = av_int2float(avio_rl32(model_file_context)); } - for (int i = 0; i < conv_params->output_num; ++i){ - conv_params->biases[i] = av_int2float(avio_rl32(model_file_context)); + + conv_params->biases = NULL; + if (conv_params->has_bias) { + conv_params->biases = av_malloc(conv_params->output_num * sizeof(float)); + if (!conv_params->biases){ + av_freep(&conv_params->kernel); + av_freep(&conv_params); + return 0; + } + for (int i = 0; i < conv_params->output_num; ++i){ + conv_params->biases[i] = av_int2float(avio_rl32(model_file_context)); + } } layer->params = conv_params; @@ -103,7 +117,10 @@ int dnn_execute_layer_conv2d(DnnOperand *operands, const int32_t *input_operand_ for (int y = pad_size; y < height - pad_size; ++y) { for (int x = pad_size; x < width - pad_size; ++x) { for (int n_filter = 0; n_filter < conv_params->output_num; ++n_filter) { - output[n_filter] = conv_params->biases[n_filter]; + if (conv_params->has_bias) + output[n_filter] = conv_params->biases[n_filter]; + else + output[n_filter] = 0.f; for (int ch = 0; ch < conv_params->input_num; ++ch) { for (int kernel_y = 0; kernel_y < conv_params->kernel_size; ++kernel_y) { diff --git a/libavfilter/dnn/dnn_backend_native_layer_conv2d.h b/libavfilter/dnn/dnn_backend_native_layer_conv2d.h index db90b2b6f6..bf872642dd 100644 --- a/libavfilter/dnn/dnn_backend_native_layer_conv2d.h +++ b/libavfilter/dnn/dnn_backend_native_layer_conv2d.h @@ -31,6 +31,7 @@ typedef struct ConvolutionalParams{ DNNActivationFunc activation; DNNConvPaddingParam padding_method; int32_t dilation; + int32_t has_bias; float *kernel; float *biases; } ConvolutionalParams; diff --git a/tests/dnn/dnn-layer-conv2d-test.c b/tests/dnn/dnn-layer-conv2d-test.c index 9d13da37c8..2da01e5372 100644 --- a/tests/dnn/dnn-layer-conv2d-test.c +++ b/tests/dnn/dnn-layer-conv2d-test.c @@ -97,6 +97,7 @@ static int test_with_same_dilate(void) float bias[2] = { -1.6574852, -0.72915393 }; params.activation = TANH; + params.has_bias = 1; params.biases = bias; params.dilation = 2; params.input_num = 3; @@ -196,6 +197,7 @@ static int test_with_valid(void) float bias[2] = { -0.4773722, -0.19620377 }; params.activation = TANH; + params.has_bias = 1; params.biases = bias; params.dilation = 1; params.input_num = 3; diff --git a/tools/python/convert_from_tensorflow.py b/tools/python/convert_from_tensorflow.py index a663b34004..605158a32e 100644 --- a/tools/python/convert_from_tensorflow.py +++ b/tools/python/convert_from_tensorflow.py @@ -118,7 +118,7 @@ class TFConverter: return knode, bnode, dnode, anode - def dump_conv2d_to_file(self, node, f): + def dump_complex_conv2d_to_file(self, node, f): assert(node.op == 'Conv2D') self.layer_number = self.layer_number + 1 self.converted_nodes.add(node.name) @@ -153,7 +153,8 @@ class TFConverter: kernel = kernel.reshape(filter_height, filter_width, in_channels, out_channels) kernel = np.transpose(kernel, [3, 0, 1, 2]) - np.array([self.op2code[node.op], dilation, padding, self.conv_activations[activation], in_channels, out_channels, filter_height], dtype=np.uint32).tofile(f) + has_bias = 1 + np.array([self.op2code[node.op], dilation, padding, self.conv_activations[activation], in_channels, out_channels, filter_height, has_bias], dtype=np.uint32).tofile(f) kernel.tofile(f) btensor = bnode.attr['value'].tensor @@ -173,6 +174,41 @@ class TFConverter: np.array([input_operand_index, output_operand_index], dtype=np.uint32).tofile(f) + def dump_simple_conv2d_to_file(self, node, f): + assert(node.op == 'Conv2D') + self.layer_number = self.layer_number + 1 + self.converted_nodes.add(node.name) + + node0 = self.name_node_dict[node.input[0]] + node1 = self.name_node_dict[node.input[1]] + if node0.op == 'Const': + knode = node0 + input_name = node.input[1] + else: + knode = node1 + input_name = node.input[0] + + ktensor = knode.attr['value'].tensor + filter_height = ktensor.tensor_shape.dim[0].size + filter_width = ktensor.tensor_shape.dim[1].size + in_channels = ktensor.tensor_shape.dim[2].size + out_channels = ktensor.tensor_shape.dim[3].size + kernel = np.frombuffer(ktensor.tensor_content, dtype=np.float32) + kernel = kernel.reshape(filter_height, filter_width, in_channels, out_channels) + kernel = np.transpose(kernel, [3, 0, 1, 2]) + + has_bias = 0 + dilation = 1 + padding = node.attr['padding'].s.decode("utf-8") + np.array([self.op2code[node.op], dilation, self.conv_paddings[padding], self.conv_activations['None'], + in_channels, out_channels, filter_height, has_bias], dtype=np.uint32).tofile(f) + kernel.tofile(f) + + input_operand_index = self.add_operand(input_name, Operand.IOTYPE_INPUT) + output_operand_index = self.add_operand(node.name, Operand.IOTYPE_OUTPUT) + np.array([input_operand_index, output_operand_index], dtype=np.uint32).tofile(f) + + def dump_depth2space_to_file(self, node, f): assert(node.op == 'DepthToSpace') self.layer_number = self.layer_number + 1 @@ -222,10 +258,12 @@ class TFConverter: scope_name = TFConverter.get_scope_name(node.name) if scope_name in self.conv2d_scope_names: if node.op == 'Conv2D': - self.dump_conv2d_to_file(node, f) + self.dump_complex_conv2d_to_file(node, f) continue - if node.op == 'DepthToSpace': + if node.op == 'Conv2D': + self.dump_simple_conv2d_to_file(node, f) + elif node.op == 'DepthToSpace': self.dump_depth2space_to_file(node, f) elif node.op == 'MirrorPad': self.dump_mirrorpad_to_file(node, f) @@ -312,10 +350,16 @@ class TFConverter: def generate_conv2d_scope_info(self): - # conv2d is a sub block in graph, get the scope name + # mostly, conv2d is a sub block in graph, get the scope name for node in self.nodes: if node.op == 'Conv2D': scope = TFConverter.get_scope_name(node.name) + # for the case tf.nn.conv2d is called directly + if scope == '': + continue + # for the case tf.nn.conv2d is called within a scope + if scope + '/kernel' not in self.name_node_dict: + continue self.conv2d_scope_names.add(scope) # get the input name to the conv2d sub block diff --git a/tools/python/convert_header.py b/tools/python/convert_header.py index 3c2acd5b15..67672b2785 100644 --- a/tools/python/convert_header.py +++ b/tools/python/convert_header.py @@ -20,7 +20,7 @@ str = 'FFMPEGDNNNATIVE' # increase major and reset minor when we have to re-convert the model file -major = 0 +major = 1 # increase minor when we don't have to re-convert the model file -minor = 2 +minor = 0

1 0

doc: fix typo in muxers documentation
by Alfred E. Heggestad 30 Oct '19

30 Oct '19

ffmpeg | branch: master | Alfred E. Heggestad <alfred.heggestad(a)gmail.com> | Wed Oct 30 10:48:44 2019 +0100| [a269fa044b1364af1654456c33b7d45407822876] | committer: Paul B Mahol doc: fix typo in muxers documentation > http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=a269fa044b1364af1… --- doc/muxers.texi | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/muxers.texi b/doc/muxers.texi index 4c88b5daec..09ded7d48a 100644 --- a/doc/muxers.texi +++ b/doc/muxers.texi @@ -2084,7 +2084,7 @@ but supports several other algorithms. The output of the muxer consists of one line per stream of the form: @var{streamindex},@var{streamtype},@var{algo}=@var{hash}, where @var{streamindex} is the index of the mapped stream, @var{streamtype} is a -single characer indicating the type of stream, @var{algo} is a short string +single character indicating the type of stream, @var{algo} is a short string representing the hash function used, and @var{hash} is a hexadecimal number representing the computed hash.

1 0

avformat/hls: Set AVFMT_TS_DISCONT flag on HLS input format
by Philip Langdale 30 Oct '19

30 Oct '19

ffmpeg | branch: master | Philip Langdale <philipl(a)overt.org> | Sun Oct 27 11:39:08 2019 -0700| [d6ac6650b911f0957e69545d7fc25be6b7728705] | committer: Steven Liu avformat/hls: Set AVFMT_TS_DISCONT flag on HLS input format There have been many reports over the years about problems when taking an HLS stream as input to `ffmpeg` where there are timestamp discontinuities present. This is explicitly supported in the HLS spec (EXT-X-DISCONTINUITY) and often used for ad injection. Various fixes and work-arounds have been proposed over the years, but one step that seems obvious, even if it's not a complete fix, is to mark the HLS input format as supporting discontinuities. This will prevent timestamp fixup logic in ffmpeg.c kicking in that ends up mangling the timestamps unnecessarily. I've tested this out with an example provided by Joe Koberg early last year, and it is sufficient to allow `ffmpeg` to download and mux the stream correctly. Joe had briefly suggested that other situations can still be handled incorrectly, but this seems like a strict improvement. Joe's example: https://s3.amazonaws.com/playon-test-videos/discont_test_new/discont_test.m… Reviewed-by: Steven Liu <lq(a)onvideo.cn> Reviewed-by: Dennis Mungai <dmngaie(a)gmail.com> > http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=d6ac6650b911f0957… --- libavformat/hls.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libavformat/hls.c b/libavformat/hls.c index d7f4d5b442..ac151d5ca4 100644 --- a/libavformat/hls.c +++ b/libavformat/hls.c @@ -2326,7 +2326,7 @@ AVInputFormat ff_hls_demuxer = { .long_name = NULL_IF_CONFIG_SMALL("Apple HTTP Live Streaming"), .priv_class = &hls_class, .priv_data_size = sizeof(HLSContext), - .flags = AVFMT_NOGENSEARCH, + .flags = AVFMT_NOGENSEARCH | AVFMT_TS_DISCONT, .read_probe = hls_probe, .read_header = hls_read_header, .read_packet = hls_read_packet,

1 0

vp4: prevent unaligned memory access in loop filter
by Peter Ross 30 Oct '19

30 Oct '19

ffmpeg | branch: master | Peter Ross <pross(a)xvid.org> | Fri Jun 21 07:52:31 2019 +1000| [fd172185580c1ccdcfb90bbfdb59fa806fad3117] | committer: Michael Niedermayer vp4: prevent unaligned memory access in loop filter VP4 applies a loop filter during motion compensation, causing the block offset will often by unaligned. This produces a bus error on some platforms, namely ARMv7 NEON. This patch adds a unaligned version of the loop filter function pointer to VP3DSPContext. Reported-by: Mike Melanson <mike(a)multimedia.cx> Signed-off-by: Michael Niedermayer <michael(a)niedermayer.cc> > http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=fd172185580c1ccdc… --- libavcodec/vp3.c | 10 ++++++++-- libavcodec/vp3dsp.c | 4 ++-- libavcodec/vp3dsp.h | 2 ++ libavcodec/x86/vp3dsp_init.c | 4 ++-- 4 files changed, 14 insertions(+), 6 deletions(-) diff --git a/libavcodec/vp3.c b/libavcodec/vp3.c index a2bd2ef07d..9a3821a8b9 100644 --- a/libavcodec/vp3.c +++ b/libavcodec/vp3.c @@ -2031,11 +2031,17 @@ static int vp4_mc_loop_filter(Vp3DecodeContext *s, int plane, int motion_x, int plane_width, plane_height); +#define safe_loop_filter(name, ptr, stride, bounding_values) \ + if ((uintptr_t)(ptr) & 7) \ + s->vp3dsp.name##_unaligned(ptr, stride, bounding_values); \ + else \ + s->vp3dsp.name(ptr, stride, bounding_values); + if (x_offset) - s->vp3dsp.h_loop_filter(loop + loop_stride + x_offset + 1, loop_stride, bounding_values); + safe_loop_filter(h_loop_filter, loop + loop_stride + x_offset + 1, loop_stride, bounding_values); if (y_offset) - s->vp3dsp.v_loop_filter(loop + (y_offset + 1)*loop_stride + 1, loop_stride, bounding_values); + safe_loop_filter(v_loop_filter, loop + (y_offset + 1)*loop_stride + 1, loop_stride, bounding_values); } for (i = 0; i < 9; i++) diff --git a/libavcodec/vp3dsp.c b/libavcodec/vp3dsp.c index ac4c57441c..f485fba1f6 100644 --- a/libavcodec/vp3dsp.c +++ b/libavcodec/vp3dsp.c @@ -449,8 +449,8 @@ av_cold void ff_vp3dsp_init(VP3DSPContext *c, int flags) c->idct_put = vp3_idct_put_c; c->idct_add = vp3_idct_add_c; c->idct_dc_add = vp3_idct_dc_add_c; - c->v_loop_filter = vp3_v_loop_filter_8_c; - c->h_loop_filter = vp3_h_loop_filter_8_c; + c->v_loop_filter = c->v_loop_filter_unaligned = vp3_v_loop_filter_8_c; + c->h_loop_filter = c->h_loop_filter_unaligned = vp3_h_loop_filter_8_c; if (ARCH_ARM) ff_vp3dsp_init_arm(c, flags); diff --git a/libavcodec/vp3dsp.h b/libavcodec/vp3dsp.h index 32b2cad0ef..3b849ec05d 100644 --- a/libavcodec/vp3dsp.h +++ b/libavcodec/vp3dsp.h @@ -43,6 +43,8 @@ typedef struct VP3DSPContext { void (*idct_dc_add)(uint8_t *dest, ptrdiff_t stride, int16_t *block); void (*v_loop_filter)(uint8_t *src, ptrdiff_t stride, int *bounding_values); void (*h_loop_filter)(uint8_t *src, ptrdiff_t stride, int *bounding_values); + void (*v_loop_filter_unaligned)(uint8_t *src, ptrdiff_t stride, int *bounding_values); + void (*h_loop_filter_unaligned)(uint8_t *src, ptrdiff_t stride, int *bounding_values); } VP3DSPContext; void ff_vp3dsp_v_loop_filter_12(uint8_t *first_pixel, ptrdiff_t stride, int *bounding_values); diff --git a/libavcodec/x86/vp3dsp_init.c b/libavcodec/x86/vp3dsp_init.c index 1ba9576431..ba47e1c6cd 100644 --- a/libavcodec/x86/vp3dsp_init.c +++ b/libavcodec/x86/vp3dsp_init.c @@ -59,8 +59,8 @@ av_cold void ff_vp3dsp_init_x86(VP3DSPContext *c, int flags) c->idct_dc_add = ff_vp3_idct_dc_add_mmxext; if (!(flags & AV_CODEC_FLAG_BITEXACT)) { - c->v_loop_filter = ff_vp3_v_loop_filter_mmxext; - c->h_loop_filter = ff_vp3_h_loop_filter_mmxext; + c->v_loop_filter = c->v_loop_filter_unaligned = ff_vp3_v_loop_filter_mmxext; + c->h_loop_filter = c->v_loop_filter_unaligned = ff_vp3_h_loop_filter_mmxext; } }

1 0

avcodec/wmalosslessdec: Fix some integer anomalies
by Michael Niedermayer 29 Oct '19

29 Oct '19

ffmpeg | branch: master | Michael Niedermayer <michael(a)niedermayer.cc> | Mon Oct 28 00:12:59 2019 +0100| [d3dee676b8a8ab6752c599e25c9b5461f06a3959] | committer: Michael Niedermayer avcodec/wmalosslessdec: Fix some integer anomalies Fixes: left shift of negative value -341180 Fixes: 18401/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_WMALOSSLESS_fuzzer-5686380134400000 Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg Reviewed-by: Paul B Mahol <onemda(a)gmail.com> Signed-off-by: Michael Niedermayer <michael(a)niedermayer.cc> > http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=d3dee676b8a8ab675… --- libavcodec/wmalosslessdec.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/libavcodec/wmalosslessdec.c b/libavcodec/wmalosslessdec.c index 19bac949d4..d4f18b9841 100644 --- a/libavcodec/wmalosslessdec.c +++ b/libavcodec/wmalosslessdec.c @@ -766,7 +766,7 @@ static void revert_cdlms ## bits (WmallDecodeCtx *s, int ch, \ for (ilms = num_lms - 1; ilms >= 0; ilms--) { \ for (icoef = coef_begin; icoef < coef_end; icoef++) { \ int##bits##_t *prevvalues = (int##bits##_t *)s->cdlms[ch][ilms].lms_prevvalues; \ - pred = 1 << (s->cdlms[ch][ilms].scaling - 1); \ + pred = (1 << s->cdlms[ch][ilms].scaling) >> 1; \ residue = s->channel_residues[ch][icoef]; \ pred += s->dsp.scalarproduct_and_madd_int## bits (s->cdlms[ch][ilms].coefs, \ prevvalues + s->cdlms[ch][ilms].recent, \ @@ -987,9 +987,9 @@ static int decode_subframe(WmallDecodeCtx *s) for (j = 0; j < subframe_len; j++) { if (s->bits_per_sample == 16) { - *s->samples_16[c]++ = (int16_t) s->channel_residues[c][j] << padding_zeroes; + *s->samples_16[c]++ = (int16_t) s->channel_residues[c][j] * (1 << padding_zeroes); } else { - *s->samples_32[c]++ = s->channel_residues[c][j] << (padding_zeroes + 8); + *s->samples_32[c]++ = s->channel_residues[c][j] * (256 << padding_zeroes); } } }

1 0

avcodec/adpcm: Fix invalid shifts in ADPCM DTK
by Michael Niedermayer 29 Oct '19

29 Oct '19

ffmpeg | branch: master | Michael Niedermayer <michael(a)niedermayer.cc> | Sun Oct 27 23:38:47 2019 +0100| [34e701ff93b664703e1bc1b1a6073fa058b02f34] | committer: Michael Niedermayer avcodec/adpcm: Fix invalid shifts in ADPCM DTK Fixes: left shift of negative value -1 Fixes: 18397/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_ADPCM_DTK_fuzzer-5675653487132672 Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg Reviewed-by: Paul B Mahol <onemda(a)gmail.com> Signed-off-by: Michael Niedermayer <michael(a)niedermayer.cc> > http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=34e701ff93b664703… --- libavcodec/adpcm.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libavcodec/adpcm.c b/libavcodec/adpcm.c index a8e74522b3..53d8d0be6a 100644 --- a/libavcodec/adpcm.c +++ b/libavcodec/adpcm.c @@ -1723,7 +1723,7 @@ static int adpcm_decode_frame(AVCodecContext *avctx, void *data, else sampledat = sign_extend(byte >> 4, 4); - sampledat = (((sampledat << 12) >> (header & 0xf)) << 6) + prev; + sampledat = ((sampledat * (1 << 12)) >> (header & 0xf)) * (1 << 6) + prev; *samples++ = av_clip_int16(sampledat >> 6); c->status[channel].sample2 = c->status[channel].sample1; c->status[channel].sample1 = sampledat;

1 0