[FFmpeg-devel] [PATCH 2/2] lavc/videotoolboxenc: enable low-latency mode for HEVC

[mypopy at gmail.com]

On Fri, Sep 1, 2023 at 11:50 AM "zhilizhao(赵志立)" <quinkblack at foxmail.com> wrote:
>
>
>
> > On Sep 1, 2023, at 09:13, mypopy at gmail.com wrote:
> >
> > On Fri, Sep 1, 2023 at 2:13 AM Zhao Zhili <quinkblack at foxmail.com> wrote:
> >>
> >>
> >>> From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of Jun Zhao
> >>> Sent: 2023年8月31日 22:50
> >>> To: ffmpeg-devel at ffmpeg.org
> >>> Cc: Jun Zhao <barryjzhao at tencent.com>; Jun Zhao <mypopydev at gmail.com>
> >>> Subject: [FFmpeg-devel] [PATCH 2/2] lavc/videotoolboxenc: enable low-latency mode for HEVC
> >>>
> >>> Enable the HEVC encoder that supports low-latency mode.
> >>>
> >>> Signed-off-by: Jun Zhao <barryjzhao at tencent.com>
> >>> ---
> >>> libavcodec/videotoolboxenc.c | 3 ++-
> >>> 1 file changed, 2 insertions(+), 1 deletion(-)
> >>>
> >>> diff --git a/libavcodec/videotoolboxenc.c b/libavcodec/videotoolboxenc.c
> >>> index 8e493c4f7a..bfc03787a0 100644
> >>> --- a/libavcodec/videotoolboxenc.c
> >>> +++ b/libavcodec/videotoolboxenc.c
> >>> @@ -1568,7 +1568,8 @@ static int vtenc_configure_encoder(AVCodecContext *avctx)
> >>> #endif
> >>>
> >>>     // low-latency mode: eliminate frame reordering, follow a one-in-one-out encoding mode
> >>> -    if ((avctx->flags & AV_CODEC_FLAG_LOW_DELAY) && avctx->codec_id == AV_CODEC_ID_H264) {
> >>> +    if ((avctx->flags & AV_CODEC_FLAG_LOW_DELAY) &&
> >>> +        (avctx->codec_id == AV_CODEC_ID_H264 || avctx->codec_id == AV_CODEC_ID_HEVC)) {
> >>
> >> How about just remove the codec_id check?
> >> Is there any unwelcome side effects for encoders other than H264/HEVC?
> >
> > 1. One reason is that there are now other encoders available, such as
> > prores_videotoolbox.
> >
> > 2. It may be more appropriate to use a strictly qualified check
> > instead of an open condition.
>
> Both HEVC and ProRes failed with low_latency on my machine (with or without -allow_sw 1)
>
> [hevc_videotoolbox @ 0x61900002a380] Error: cannot create compression session: -12902
> [prores_videotoolbox @ 0x61900002a380] Error: cannot create compression session: -12902
>
>         kVTParameterErr                                                 = -12902,
>
I think you missed the other options, low delay in VT encoder need to
co-work with some other options

> 1. FFmpeg users should have an idea whether or not to set AV_CODEC_FLAG_LOW_DELAY.
> AV_CODEC_FLAG_LOW_DELAY doesn’t make sense for ProRes, but if user wants to try, let he/she
> do.
>
> 2. I can’t find API to query the capability. Only VideoToolBox framework itself know how
> to handle the argument. There isn’t much protect to provide for users.
>
Yes, now FFmpeg VT encoder didn't used the capability query, I think
need to used the some API VTCopySupportedPropertyDictionaryForEncoder
for capability query, this my next step
> Even with H.264, the results is unexpected: the input framerate is 25 but the output
> framerate is 5, it dropped 80 percent frames silently.
>