[FFmpeg-trac] #10119(avcodec:new): h264_amf and hevc_amf is extremely slow when use D3D11 hwaccel as input

[FFmpeg]

#10119: h264_amf and hevc_amf is extremely slow when use D3D11 hwaccel as input
-------------------------------------+-------------------------------------
             Reporter:  Reito        |                    Owner:  (none)
                 Type:  defect       |                   Status:  new
             Priority:  critical     |                Component:  avcodec
              Version:  git-master   |               Resolution:
             Keywords:  AMF amf      |               Blocked By:
  h264_amf                           |
             Blocking:               |  Reproduced by developer:  0
Analyzed by developer:  0            |
-------------------------------------+-------------------------------------
Description changed by Reito:

Old description:

> Summary of the bug:
> How to reproduce:
>
> Initialize an encoding codec using h264_amf or hevc_amf, create an D3D11
> device and initialize the hw_device and hw_frame, feed D3D11Texture2D to
> AVFrame and then avcodec_send_frame, It takes about 12ms for each frame.
>
> After doing a dig into the amfenc.c source, I find out these code cost
> about 10ms every frame and literally did nothing useful:
>
> Line 673
>
> {{{
>         if (hw_surface) {
>             AMFBuffer *frame_ref_storage_buffer;
>
>             // input HW surfaces can be vertically aligned by 16; tell
> AMF the real size
>             surface->pVtbl->SetCrop(surface, 0, 0, frame->width,
> frame->height);
>
>             frame_ref_storage_buffer =
> amf_create_buffer_with_frame_ref(frame, ctx->context);
>             AMF_RETURN_IF_FALSE(ctx, frame_ref_storage_buffer != NULL,
> AVERROR(ENOMEM), "create_buffer_with_frame_ref() returned NULL\n");
>
>             res = amf_set_property_buffer(surface, L"av_frame_ref",
> frame_ref_storage_buffer);
>             AMF_RETURN_IF_FALSE(ctx, res == AMF_OK, AVERROR_UNKNOWN,
> "SetProperty failed for \"av_frame_ref\" with error %d\n", res);
>             ctx->hwsurfaces_in_queue++;
> frame_ref_storage_buffer->pVtbl->Release(frame_ref_storage_buffer);
>         }
> }}}
>
> Line 733
>
> {{{
>             if (data->pVtbl->HasProperty(data, L"av_frame_ref")) {
>                 AMFBuffer *frame_ref_storage_buffer;
>                 res = amf_get_property_buffer(data, L"av_frame_ref",
> &frame_ref_storage_buffer);
>                 AMF_RETURN_IF_FALSE(ctx, res == AMF_OK, AVERROR_UNKNOWN,
> "GetProperty failed for \"av_frame_ref\" with error %d\n", res);
> amf_release_buffer_with_frame_ref(frame_ref_storage_buffer);
>                 ctx->hwsurfaces_in_queue--;
>             }
> }}}
>
> After remove these code the encode speed is normal (about 0.5ms per
> frame).
> In my understanding, these code is requesting an buffer from amf, clone
> an AVFrame, copy AVFrame to it, pass it to amf. After frame is submitted
> and encoded, it retrives the buffer, copy AVFrame out and release
> everything. In summary, neither amf nor ffmpeg use anything from this
> procedure, it is totally useless code.
>
> ffmpeg version
> built on master, 5.1.2, 4.4.3

New description:

 Summary of the bug:
 How to reproduce:

 Initialize an encoding codec using h264_amf or hevc_amf, create an D3D11
 device and initialize the hw_device and hw_frame, feed D3D11Texture2D to
 AVFrame and then avcodec_send_frame, It takes about 12ms for each frame.


 {{{
         AVHWFramesContext* frame =
 reinterpret_cast<AVHWFramesContext*>(hwf->data);
         frame->format = AV_PIX_FMT_D3D11;
         frame->sw_format = AV_PIX_FMT_NV12;
         frame->width = ctx->width;
         frame->height = ctx->height;
         frame->initial_pool_size = 2; // 2 is max
 }}}


 After doing a dig into the amfenc.c source, I find out these code cost
 about 10ms every frame and literally did nothing useful:

 Line 673

 {{{
         if (hw_surface) {
             AMFBuffer *frame_ref_storage_buffer;

             // input HW surfaces can be vertically aligned by 16; tell AMF
 the real size
             surface->pVtbl->SetCrop(surface, 0, 0, frame->width,
 frame->height);

             frame_ref_storage_buffer =
 amf_create_buffer_with_frame_ref(frame, ctx->context);
             AMF_RETURN_IF_FALSE(ctx, frame_ref_storage_buffer != NULL,
 AVERROR(ENOMEM), "create_buffer_with_frame_ref() returned NULL\n");

             res = amf_set_property_buffer(surface, L"av_frame_ref",
 frame_ref_storage_buffer);
             AMF_RETURN_IF_FALSE(ctx, res == AMF_OK, AVERROR_UNKNOWN,
 "SetProperty failed for \"av_frame_ref\" with error %d\n", res);
             ctx->hwsurfaces_in_queue++;
 frame_ref_storage_buffer->pVtbl->Release(frame_ref_storage_buffer);
         }
 }}}

 Line 733

 {{{
             if (data->pVtbl->HasProperty(data, L"av_frame_ref")) {
                 AMFBuffer *frame_ref_storage_buffer;
                 res = amf_get_property_buffer(data, L"av_frame_ref",
 &frame_ref_storage_buffer);
                 AMF_RETURN_IF_FALSE(ctx, res == AMF_OK, AVERROR_UNKNOWN,
 "GetProperty failed for \"av_frame_ref\" with error %d\n", res);
 amf_release_buffer_with_frame_ref(frame_ref_storage_buffer);
                 ctx->hwsurfaces_in_queue--;
             }
 }}}

 After remove these code the encode speed is normal (about 0.5ms per
 frame).

 In my understanding, these code is requesting an buffer from amf, clone an
 AVFrame, copy AVFrame to it, pass it to amf. After frame is submitted and
 encoded, it retrives the buffer, copy AVFrame out and release everything.
 In summary, neither amf nor ffmpeg use anything from this procedure, it is
 totally useless code.

 More interestingly, if i change frame->initial_pool_size = 1, it will also
 encode every frame for 12ms, so it might related to
 ctx->hwsurfaces_in_queue

 ffmpeg version
 built on master, 5.1.2, 4.4.3

--
-- 
Ticket URL: <https://trac.ffmpeg.org/ticket/10119#comment:1>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker