[FFmpeg-devel] [PATCH] fate: Add a target for listing failed tests

[Martin Storsjö]

On Sun, 10 Nov 2024, Michael Niedermayer wrote:

> On Thu, Nov 07, 2024 at 11:36:24AM +0200, Martin Storsjö wrote:
>> If running tests with "make -j<N> fate", the execution will stop
>> after the first failing test. To get an overview of the whole
>> test suite, one rather would run "make -k -j<N> fate", which then
>> again buries the results about what tests actually failed further
>> up in the console log.
>>
>> Add a target so one can run "make fate-list-failing", to see a list
>> of all tests that failed the last time they were executed.
>>
>> Also add a companion target "fate-clear-results" which removes all
>> the old test results. (When executing a subset of tests, the result
>> files of all tests that aren't executed stay untouched. This also
>> allows getting rid of results for tests that no longer are present
>> in the testsuite.)
>> ---
>> So far, I've always just manually run
>> "cat tests/data/fate/*.rep | cut -f 1-2 -d : | grep -v :0", but
>> perhaps we should at least wrap it up in something more convenient
>> for the other developers.
>> ---
>>  doc/build_system.txt | 6 ++++++
>>  doc/fate.texi        | 7 +++++++
>>  tests/Makefile       | 6 ++++++
>>  3 files changed, 19 insertions(+)
>>
>
> should
> make -j32 fate-list-failing
> run the tests ? (if they did never run)
>
> It seems it does not

No, I don't see that as necessary. I would only be intended to answer the 
question of which tests failed last time they were executed. You may only 
have run e.g. fate-hevc, and then only failures within that would be 
listed. If no tests have been executed, we know of no failures either, and 
shouldn't print anything.

// Martin