[Python-checkins] r46146 - sandbox/trunk/rjsh-pybench/pybench.py

M.-A. Lemburg mal at egenix.com
Wed May 24 14:16:28 CEST 2006


Steve Holden wrote:
> M.-A. Lemburg wrote:
>> Steve Holden wrote:
> [...]
>>> The sandbox copy already uses the same code timeit.py uses to
>>> determine the clock to use. 
>>
>>
>> Sorry, but that approach is just plain wrong:
>>
>> if sys.platform == "win32":
>>     # On Windows, the best timer is time.clock()
>>     default_timer = time.clock
>> else:
>>     # On most other platforms the best timer is time.time()
>>     default_timer = time.time
>>
> It's a good job you pointed that out, since that would appear to mean
> that timeit.py is also wrong, since it uses exactly the same code.
>
>> You never want to use a wall clock timer to measure
>> the speed of code execution. There are simply too many
>> other things going on in a multi-process system to make
>> this a reliable approach.
>>
> It's not a matter of what I *want* to use, it's a matter of what I *can*
> use. Easily, portably, and with reasonable repeatibility.

I understand, I would just like to have a good and
reliable timer used in pybench.

On Unix (which I use most the time), time.clock() gives
reasonable and repeatable results, so it's the
obvious choice.

On Windows, it seems that we should try to come up with
a better alternative to time.clock(). I'll give that a shot
later today.

>> On Unix time.clock() will give you process time which
>> is far more reliable as it provides feedback on the
>> time the process actually received from the kernel
>> for execution.
>>
>> Try it:
>>
>> print time.clock(); time.sleep(10); print time.clock()
>>
>> On Unix, you'll get the same results for both prints.
>> On Windows, you get wall clock results, ie. a 10 second
>> difference.
>>
> So timeit.py isn't just wrong for one platform, it's doing exactly the
> reverse of what it should, and it's just my bad luck that you didn't
> spot *that* checkin?

I'm not sure where timeit.py originated, but yes, it's
certainly doing the wrong thing on Unix.

>>> Uncle Timmy points out that the GetProcessTimes() isn't ubiquitous,
>>> so I don't think there's much point in trying to make use of it.
>>
>>
>> According to MSDN it's only available on the WinNT branch
>> of the Windows kernel. Since we're trying to optimize for
>> a current system, it should be available on all systems
>> where you'd normally be running pybench to check whether
>> an optimization makes sense or not.
>>
> So I can also ignore the comments about retaining 1.5.2 compatibility?

No, actually I think pybench should report which timer it
uses for timing things and then use the best alternative
available on the system.

Comparisons should then just work in case the same timer
was used in both runs.

>>> The one change I still do want to make is to allow the use of some
>>> bogomips-like feature to provide scaled testing to ensure that the
>>> individual test times can more easily be made large enough.
>>
>>
>> Isn't that what the warp factor already implements ?
>
> Yes, but not automatically.

You lost me there :-)

Do want pybench to automatically scale the warp factor so
that each run takes e.g. 60 seconds ?

>>> At that point I'll probably be looking to check it back into the
>>> trunk as version 1.4 - unless you have any further suggestions?
>>> Indications are that the timings do seem to be more reliable.
>>
>>
>> I disagree on the clock approach you implemented and the
>> change to use a 0 default for calibration runs, so those
>> changes should not go into the trunk.
>
> I wasn't proposing to check it back in with 0 calibration.

Ok.

> There's obviously plenty of mileage in the discussion about how to do
> the timings. I'm just not sure we will come to any reliable conclusion.

We don't have to decide things today - let's leave it cooking
for a while until we find suitable solutions for the platforms
we care about.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 24 2006)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::


More information about the Python-checkins mailing list