[FFmpeg-devel] [PATCH] fate.sh: Allow overriding what targets to make for running the tests

[Rémi Denis-Courmont]

Le 27 novembre 2023 23:55:18 GMT+02:00, "Martin Storsjö" <martin at martin.st> a écrit :
>On Mon, 27 Nov 2023, Rémi Denis-Courmont wrote:
>
>> Le maanantaina 27. marraskuuta 2023, 14.31.18 EET Martin Storsjö a écrit :
>>> This can be useful if doing testing of uncommon CPU extensions by
>>> running tests with QEMU (by configuring with e.g.
>>> "target_exec=qemu-aarch64"), by only running the checkasm tests,
>>> to get a reasonable test coverage without excessive test runtime.
>> 
>> For the purpose of testing future or bleeding-edge CPU extensions on emulator, you would normally want to be able to actually filter those in. That is more of a matter of patching checkasm than FATE.
>
>Sorry, can you elaborate on what you mean with "filter those in" here?

You're running all checkasm tests, not just those that require the emulator.

But what's potentially much worse is that you're triggering a whole build, or it's not entirely clear from the description how you'd reuse an existing build.

For Armv8, that's just bad. For RV, that's terrible, as we need to run the same checkasm with different emulator configuration (different $QEMU_CPU in the case of QEMU): one per vector length. Armv9 will potentially have the same problem if FFmpeg grows SVE(2) support.

>
>> Considering the poor coverage of checkasm, I fear that this just gives the wrong impression, not to say a false sense of security. It feels misleading to encourage or support that paradigm into FATE, in light of that poor coverage. Afterall, if it's just about running checkasm, anybody can just run `make tests/checkasm/checkasm && tests/checkasm/checkasm`.
>
>Yes, anybody can run that - but having those results posted continuously somewhere where other can see them can be valuable as well.
>
>Anyway, the concrete case I'm considering, is that we've got AArch64 code merged, that uses the I8MM extensions. We don't have any FATE configuration that continuously test that. Whenever there are patches, I do spin up a cloud instance that supports this extension and test the patches there, but inbetween that we're pretty much blind.
>
>While checkasm's coverage isn't fantastic, for this particular case I'm not merging any AArch64 code for new extensions unless that code is covered by checkasm.
>
>The other AArch64 feature that we do have code for, which also is untested, is the assembly support for branch protection and pointer authentication. Also this is testable pretty easily with QEMU. It's of course more interesting to run the full fate suite, but if we're not looking for bugs in the compiler but only for bugs in our assembly, then checkasm should cover most of it.
>
>Yes there's potential for QEMU bugs hiding real issues, but I'd rather have a regular run of QEMU+checkasm than not have it tested at all. And I'm volunteering HW+time for testing these cases with QEMU for whatever checkasm covers, but I'm not volunteering it for full fate runs with QEMU.
>
>And sure, I can just run such configs privately, as I already do run a bunch of various regular builds for projects I care about - but as we do have FATE with the public status page, hooking it up to be reported there would feel like added value for everybody.
>
>// Martin
>_______________________________________________
>ffmpeg-devel mailing list
>ffmpeg-devel at ffmpeg.org
>https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
>To unsubscribe, visit link above, or email
>ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".