[Python-checkins] r46505 -python/trunk/Tools/pybench/systimes.py

Fredrik Lundh fredrik at pythonware.com
Tue Jun 6 19:25:23 CEST 2006


M.-A. Lemburg wrote:

>>> Using these APIs, we can get an accuracy of +/- 15.6ms on each
>>> measurement, e.g. if a test tun takes 1 second, we have an accuracy
>>> of +/- 1.56%.
 >>
>> here we go again.
>>
>> instead of posting more unqualified guesses, can you *PLEASE* read the 
>> posts in the "python benchmarks" thread.
> 
> I just did, but couldn't find anything that would allow you to
> call my posting unqualified:

you stated that using rusage gives you a measurement accuracy of +/- 
1.56%.  assuming that we agree that "accuracy" means "difference between 
*actual* value (the number of cycles required to run the benchmark) and 
measured value", that's not necessarily true.  and as long as you see 
varying totals, it's definitely not true.  read on.

> If you are running a benchmark on a quiet system, the benchmark
> process *is* the process running most of the time, so your
> comments about the jiffies possibly not getting accounted to
> the benchmark process are not really all that realistic.

nope.  consider a second task running with a priority equal to or higher 
than your benchmark task.  that task can "steal" nearly a full jiffy 
every time it is run, but you'll still get a process time that's exactly 
the same as the wall time (because your process will be assigned all 
jiffies, and will thus appear to run at full speed throughout the entire 
benchmark).

however, both the resulting process time and the resulting wall time can 
be off by 2x, in a (highly hypothetical) worst case.

> In fact, if you run time.time() vs. resource.getrusage() on
> a Linux box, you'll find that both more or less show the same
> flux in timings - with an error interval of around 15ms.

which is easily explained by "cycle stealers", and is destroying the 
benchmark's precision.  and as usual, if you don't have precision, you 
don't really have accuracy (unless you have a good statistical model, 
and enough data to use it; see Andrew's posts for more on that).

</F>



More information about the Python-checkins mailing list