[FFmpeg-devel] [PATCH 3/3] tests/fate-run: make FATE non-interactive

[Ganesh Ajjanagadde]

On Tue, Aug 11, 2015 at 3:51 PM, Michael Niedermayer
<michael at niedermayer.cc> wrote:
> On Tue, Aug 11, 2015 at 02:53:37PM -0400, Ganesh Ajjanagadde wrote:
>> On Tue, Aug 11, 2015 at 11:35 AM, Michael Niedermayer
>> <michael at niedermayer.cc> wrote:
>> > On Fri, Jul 31, 2015 at 08:39:00AM -0400, Ganesh Ajjanagadde wrote:
>> >> FATE is non-interactive; it should not listen to user commands
>> >>
>> >> Signed-off-by: Ganesh Ajjanagadde <gajjanagadde at gmail.com>
>> >> ---
>> >>  tests/fate-run.sh | 2 +-
>> >>  1 file changed, 1 insertion(+), 1 deletion(-)
>> >>
>> >> diff --git a/tests/fate-run.sh b/tests/fate-run.sh
>> >> index b88730a..7aa5df2 100755
>> >> --- a/tests/fate-run.sh
>> >> +++ b/tests/fate-run.sh
>> >> @@ -93,7 +93,7 @@ probeframes(){
>> >>
>> >>  ffmpeg(){
>> >>      dec_opts="-hwaccel $hwaccel -threads $threads -thread_type $thread_type"
>> >> -    ffmpeg_args="-nostats -cpuflags $cpuflags"
>> >> +    ffmpeg_args="-nostdin -nostats -cpuflags $cpuflags"
>> >
>> > This is not neccessary and its not how FFmpeg is used by the majority
>> > of users scripts AFAIK.
>>
>> >
>> > I dont mind if fate tests "-nostdin" but it also should test the case
>> > without "-nostdin" as thats likely how its used most of the time
>>
>> What does testing without -nostdin add?
>> Can you give a concrete example of where by applying this patch the
>> quality/utility
>> of ffmpeg's FATE environment is being reduced?
>>
>> I am not so sure that the majority of users' scripts use -nostats either.
>
>> Applying the same logic, FATE should test without the -nostats flag as well.
>
> fate should probably run (seperate) tests with alot more flag
> combinations

I agree with this as a general philosophy.
However, there are practical issues:
1. This must be balanced with time taken to run fate.
Flags which exercise more important options (e.g cpuflags) should be prioritized
over flags which don't yield much useful info to developers (e.g
nostdin, nostats).
We want quick feedback for developers.
2. My own limited view of FATE is that it is primarily very useful for testing
the "real work" of ffmpeg, e.g encoders, decoders, muxers, etc.
Testing options beyond a point is not practically possible due to
combinatorial explosion
in possible flag combinations.
These can only be detected/observed from obscure use cases,
and we do get bug reports for these.

For a longer term solution,
I guess we should have a "normal fate" and a "long fate".

At the moment,
we should work with what we have.
The only example you give is the one where if terminal logic is broken.
Same argument could be applied to "nostats"
or any flag for that matter.

>
> [...]
> --
> Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB
>
> Let us carefully observe those good qualities wherein our enemies excel us
> and endeavor to excel them, by avoiding what is faulty, and imitating what
> is excellent in them. -- Plutarch
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>