[Python-Dev] PyDict_SetItem hook

Jeffrey Yasskin jyasskin at gmail.com
Fri Apr 3 22:17:37 CEST 2009

On Fri, Apr 3, 2009 at 12:28 PM, Michael Foord
<fuzzyman at voidspace.org.uk> wrote:
> Collin Winter wrote:
>> On Fri, Apr 3, 2009 at 2:27 AM, Antoine Pitrou <solipsis at pitrou.net>
>> wrote:
>>> Thomas Wouters <thomas <at> python.org> writes:
>>>> Pystone is pretty much a useless benchmark. If it measures anything,
>>>> it's the
>>> speed of the bytecode dispatcher (and it doesn't measure it particularly
>>> well.)
>>> PyBench isn't any better, in my experience.
>>> I don't think pybench is useless. It gives a lot of performance data
>>> about
>>> crucial internal operations of the interpreter. It is of course very
>>> little
>>> real-world, but conversely makes you know immediately where a performance
>>> regression has happened. (by contrast, if you witness a regression in a
>>> high-level benchmark, you still have a lot of investigation to do to find
>>> out
>>> where exactly something bad happened)
>>> Perhaps someone should start maintaining a suite of benchmarks,
>>> high-level and
>>> low-level; we currently have them all scattered around (pybench, pystone,
>>> stringbench, richard, iobench, and the various Unladen Swallow
>>> benchmarks; not
>>> to mention other third-party stuff that can be found in e.g. the Computer
>>> Language Shootout).
>> Already in the works :)
>> As part of the common standard library and test suite that we agreed
>> on at the PyCon language summit last week, we're going to include a
>> common benchmark suite that all Python implementations can share. This
>> is still some months off, though, so there'll be plenty of time to
>> bikeshed^Wrationally discuss which benchmarks should go in there.
> Where is the right place for us to discuss this common benchmark and test
> suite?

Dunno. Here, by default, but I'd subscribe to a tests-sig or
commonlibrary-sig or benchmark-sig if one were created.

> As the benchmark is developed I would like to ensure it can run on
> IronPython.

We want to ensure the same thing for the current unladen swallow
suite. If you find ways it currently doesn't, send us patches (until
we get it moved to the common library repository at which point you'll
be able to submit changes yourself). You should be able to check out
http://unladen-swallow.googlecode.com/svn/tests independently of the
rest of the repository. Follow the instructions at
http://code.google.com/p/unladen-swallow/wiki/Benchmarks to run
benchmarks though perf.py. You'll probably want to select benchmarks
individually rather than accepting the default of "all" because it's
currently not very resilient to tests that don't run on one of the
comparison pythons.

Personally, I'd be quite happy moving our performance tests into the
main python repository before the big library+tests move, but I don't
know what directory to put it in, and I don't know what Collin+Thomas
think of that.

> The test suite changes will need some discussion as well - Jython and
> IronPython (and probably PyPy) have almost identical changes to tests that
> currently rely on deterministic finalisation (reference counting) so it
> makes sense to test changes on both platforms and commit a single solution.

IMHO, any place in the test suite that relies on deterministic
finalization but isn't explicitly testing that CPython-specific
feature is a bug and should be fixed, even before we export it to the
new repository.


More information about the Python-Dev mailing list