[FFmpeg-devel] [PATCH] fate: add VVC decoder tests

[Kieran Kunhya]

On Sat, 6 Jan 2024 at 05:10, Martin Storsjö <martin at martin.st> wrote:

> On Sat, 6 Jan 2024, Kieran Kunhya wrote:
>
> > On Sat, 6 Jan 2024 at 02:35, Nuo Mi <nuomi2021 at gmail.com> wrote:
> >
> >> On Sat, Jan 6, 2024 at 9:13 AM James Almer <jamrial at gmail.com> wrote:
> >>
> >> > On 1/5/2024 10:09 PM, Nuo Mi wrote:
> >> > > On Sat, Jan 6, 2024 at 5:09 AM James Almer <jamrial at gmail.com>
> wrote:
> >> > >
> >> > > Here are the clips and their sources:
> >> > > https://github.com/ffvvc/tests/tree/main
> >> > > passed v1 is about 194M , and each clip may have different features.
> >> It's
> >> > > better to add them all to vvc-conformance, without name change
> >> > > passed v2 is about 871M , we can pick some smaller clips that cover
> all
> >> > bit
> >> > > depths and color formats.
> >> >
> >> > Yeah, it's not so much about the size of the samples than it is about
> >> > runtime. That many tests would make a fate run take way too much, even
> >> > after we add assembly.
> >> > We need to choose a subset that cover as many decoding cases as
> possible.
> >> >
> >> Good point.
> >> If I disable the asm code on my machine, "make fate-hevc" takes about 1
> >> minute and 26 seconds. We can use this as the baseline.
> >> If we remove the largest 5 files in v1, we can decode it in 5 minutes
> and
> >> 13 seconds. We can remove more if we find it's still slow.
> >> From a feature perspective, TREE_A_HHI_3.bit may cover major features in
> >> TREE_B_HHI_3.bit, TREE_C_HHI_3.bit. It's reasonable to remove them
> >>
> >> Largest 5 files:
> >> TRANS_B_Chipsnmedia_2.bit
> >> TRANS_A_Chipsnmedia_2.bit
> >> TREE_B_HHI_3.bit
> >> LFNST_D_HHI_3.bit
> >> TREE_C_HHI_3.bit
> >>
> >
> > Could we have an option in the FATE config file to do a full run and
> allow
> > admins to turn it on at will?
> > I appreciate there are a lot of embedded devices where running all the
> VVC
> > decodes will take forever. But at the same time there are powerful
> devices
> > like M1, NUC etc where this isn't a big deal and I'd like to see VVC
> tested
> > in full.
>
> We already have an option that is somewhat like this;
> --disable-large-tests, which is documented as "disable tests that use a
> large amount of memory", primarily intended for small SBCs and similar -
> where running fate takes a long time in any case, but one doesn't want it
> to fail due to some few tests doing 8k resolutions and such.
>
> Not sure if that's the right match here, or if we should add another
> option, like --enable-extra-tests, which would be opt-in?
>
> // Martin
>

No opinion either way.

Kieran