[FFmpeg-devel] Benchmarking

[Michael Niedermayer]

On Tue, Dec 16, 2008 at 09:10:20AM +0100, Diego Biurrun wrote:
> On Tue, Dec 16, 2008 at 12:24:02AM +0100, Michael Niedermayer wrote:
> > 
> > Ive run some tests to determine how reliable benchmarks are and which
> > method is most reliable. The results are not really a surprise as they
> > 
> > 3 samples:
> > len:    1280000 correct: avg:  78 min:  61 without outliers:  78 without max:  69 without 2max:  61 
> > 
> > 5 samples:
> > len:    1280000 correct: avg:  90 min:  61 without outliers:  91 without max:  78 without 2max:  62 
> > 
> > 15 samples:
> > len:    1280000 correct: avg:  87 min:  93 min2:  80 without outliers:  98 without max:  98 without 2max:  97 
> 
> Do you have an explanation for the anomaly at this value of len?

I have a hypothesis, no proof though that it is correct

if one assumes that the 2 test "programs" each need exactly 
T1 or T2 time when undisturbed.
And then assumes that they are disturbed by some other activity that
slows them down, let us further assume this disturbance
occurs with a probability of P.

now if either all samples are disturbed almost always
P^num_samples > 0.99 and this disturbance is reasonably constant
or
or at least one sample is undisturbed almost always
P^num_samples < 0.01 (theres no need for the disturbance to be constant)
then the minimum will give the correct value (assuming our assumtations
are reasonably close to correct)

thus there are some values of P where the minimum performs poorly, the
chance to hit such a value becomes smaller (both in my tests as well as
theoretically) as more samples are used.


[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

The educated differ from the uneducated as much as the living from the
dead. -- Aristotle 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20081216/b136ce88/attachment.pgp>