[FFmpeg-devel] [PATCH] fate: Allow running multiple rounds of tests with differing settings

[Martin Storsjö]

On Wed, 13 Dec 2023, Martin Storsjö wrote:

> On Wed, 13 Dec 2023, Rémi Denis-Courmont wrote:
>
>> Le tiistaina 12. joulukuuta 2023, 0.14.06 EET Martin Storsjö a écrit :
>>> This can be used to run tests multple times, with e.g. differing
>>> QEMU settings, by adding something like this to the FATE configuration
>>> file:
>>>
>>>     target_exec="qemu-aarch64-static"
>>>     fate_targets="fate-checkasm fate-cpu"
>>>
>>>     fate_environments="sve128 sve256 sve512"
>>>     sve128_env="QEMU_CPU=max,sve128=on"
>>>     sve256_env="QEMU_CPU=max,sve256=on"
>>>     sve512_env="QEMU_CPU=max,sve512=on"
>> 
>> I'm fine with that, but for the sake of generality, shouldn't rather the 
>> entire target_exec prefix be indirected? Some runners may want to use 
>> command line flags rather than environment variables.
>
> Yes - that's also doable. One can e.g. do this:
>
> ---
> target_exec="qemu-aarch64-static -cpu \$(MY_CPU)"
> fate_targets="fate-checkasm fate-cpu"
>
> fate_environments="sve128 sve256 sve512"
> sve128_env="MY_CPU=max,sve128=on"
> sve256_env="MY_CPU=max,sve256=on"
> sve512_env="MY_CPU=max,sve512=on"
> ---
>
> That way, one can also make the whole target_exec be e.g. \$(EXEC_CMD) and 
> set the full command via the individual envs. It's not quite as comfortable, 
> but should be doable and allow fully generic setups.

If this sounds reasonable enough, I would go ahead and land this, together 
with 
https://patchwork.ffmpeg.org/project/ffmpeg/patch/20231127123118.3622784-1-martin@martin.st/.

I guess this feature should have some documentation as well - I can draft 
up a v2 with docs before going ahead with it, if the feature overall looks 
acceptable.

// Martin