[Python-Dev] PyDict_SetItem hook

M.-A. Lemburg mal at egenix.com
Fri Apr 3 19:04:36 CEST 2009

On 2009-04-03 18:06, Thomas Wouters wrote:
> On Fri, Apr 3, 2009 at 11:27, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> Thomas Wouters <thomas <at> python.org> writes:
>>> Pystone is pretty much a useless benchmark. If it measures anything, it's
>> the
>> speed of the bytecode dispatcher (and it doesn't measure it particularly
>> well.)
>> PyBench isn't any better, in my experience.
>> I don't think pybench is useless. It gives a lot of performance data about
>> crucial internal operations of the interpreter. It is of course very little
>> real-world, but conversely makes you know immediately where a performance
>> regression has happened. (by contrast, if you witness a regression in a
>> high-level benchmark, you still have a lot of investigation to do to find
>> out
>> where exactly something bad happened)
> Really? Have you tried it? I get at least 5% noise between runs without any
> changes. I have gotten results that include *negative* run times. 

On which platform ? pybench 2.0 works reasonably well on Linux and
Windows, but of course can't do better than the timers available for
those platforms. If you have e.g. NTP running and it uses wall clock
timers, it is possible that you get negative round times. If you don't
and still get negative round times, you have to change the test
parameters (see below).

> And yes, I
> tried all the different settings for calibration runs and timing mechanisms.
> The tests in PyBench are not micro-benchmarks (they do way too much for
> that), they don't try to minimize overhead or noise,

That is not true. They were written as micro-benchmarks and adjusted
to have a high signal-noise ratio. For some operations this isn't easy
to do, but I certainly tried hard to get the overhead low (note that the
overhead is listed in the output).

That said, please keep in mind that the settings in pybench were last
adjusted some years ago to have the tests all run in more or less the
same wall clock time. CPUs have evolved a lot since then and this shows.

> but they are also not
> representative of real-world code.

True and they never were meant for that, since I was frustrated by
other benchmarks at the time and the whole approach in general.

Each of the tests checks one specific aspect of Python. If your
application happens to use a lot of dictionary operations, you'll
be mostly interested in those. If you do a lot of simple arithmetic,
there's another test for that.

On top of that the application is written to be easily extensible,
so it's easy to add new tests specific to whatever application space
you're after.

> That doesn't just mean "you can't infer
> the affected operation from the test name", but "you can't infer anything."
> You can just be looking at differently borrowed runtime. I have in the past
> written patches to Python that improved *every* micro-benchmark and *every*
> real-world measurement I made, except PyBench. Trying to pinpoint the
> slowdown invariably lead to tests that did too much in the measurement loop,
> introduced too much noise in the "calibration" run or just spent their time
> *in the measurement loop* on doing setup and teardown of the test. 

pybench calibrates itself to remove that kind of noise from the output.
Each test has a .calibrate() method which does all the setup and
tear down minus the actual benchmark operations.

If you get wrong numbers, try adjusting the parameters and add more
"packets" of operations. Don't forget to adjust the version number to
not compare apples and orange, though.

Perhaps it's time to readjust the pybench parameters to todays

Marc-Andre Lemburg

Professional Python Services directly from the Source  (#1, Apr 03 2009)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
2009-03-19: Released mxODBC.Connect 1.0.1      http://python.egenix.com/

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611

More information about the Python-Dev mailing list