[Python-Dev] Python Benchmarks

M.-A. Lemburg mal at egenix.com
Fri Jun 2 12:26:02 CEST 2006


Fredrik Lundh wrote:
> M.-A. Lemburg wrote:
> 
>> Of course, but then changes to try-except logic can interfere
>> with the performance of setting up method calls. This is what
>> pybench then uncovers.
> 
> I think the only thing PyBench has uncovered is that you're convinced that it's
> always right, and everybody else is always wrong, including people who've
> spent decades measuring performance, and the hardware in your own computer.

Oh, come on. You know that's not true and I'm trying to
understand what is causing your findings, but this is
difficult, since you're not providing enough details.
E.g. the output of pybench showing the timing results
would help a lot.

I would also like to reproduce your findings. Do you have
two revision numbers in svn which I could use for this ?

>> See above (or the code in pybench.py). t1-t0 is usually
>> around 20-50 seconds:
> 
> what machines are you using?  using the default parameters, the entire run takes
> about 50 seconds on the slowest machine I could find...

If the whole suite runs in 50 seconds, the per-test
run-times are far too small to be accurate. I usually
adjust the warp factor so that each *round* takes
50 seconds.

Looks like I have to revisit the default parameters and
update the doc-strings. I'll do that when I add the new
timers.

Could you check whether you still see the same results with
running with "pybench.py -w 1" ?

>>> that's not a very good idea, given how get_process_time tends to be
>>> implemented on current-era systems (google for "jiffies")...  but it
>>> definitely explains the bogus subtest results I'm seeing, and the "magic
>>> hardware" behaviour you're seeing.
>> That's exactly the reason why tests run for a relatively long
>> time - to minimize these effects. Of course, using wall time
>> make this approach vulnerable to other effects such as current
>> load of the system, other processes having a higher priority
>> interfering with the timed process, etc.
> 
> since process time is *sampled*, not measured, process time isn't exactly in-
> vulnerable either.  it's not hard to imagine scenarios where you end up being
> assigned only a small part of the process time you're actually using, or cases
> where you're assigned more time than you've had a chance to use.
> 
> afaik, if you want true performance counters on Linux, you need to patch the
> operating system (unless something's changed in very recent versions).
> 
> I don't think that sampling errors can explain all the anomalies we've been seeing,
> but I'd wouldn't be surprised if a high-resolution wall time clock on a lightly loaded
> multiprocess system was, in practice, *more* reliable than sampled process time
> on an equally loaded system.

That's why the timers being used by pybench will become a
parameter that you can then select to adapt pybench it to
the OS your running pybench on.

Note that time.clock, the current default timer in pybench,
is a high accuracy wall-clock timer on Windows, so it should
demonstrate similar behavior to timeit.py, even more so,
since your using warp 20 and thus a similar timing strategy
as that of timeit.py.

I suspect that the calibration step is causing problems.

Steve added a parameter to change the number of calibration
runs done per test: -C n. The default is 20.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jun 02 2006)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2006-07-03: EuroPython 2006, CERN, Switzerland              30 days left

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::


More information about the Python-Dev mailing list