[Python-Dev] [GSoC] Developing a benchmark suite (for Python 3.x)

[Anthony Scopatz]

On Thu, Apr 7, 2011 at 8:29 PM, Jesse Noller <jnoller at gmail.com> wrote:

> On Thu, Apr 7, 2011 at 7:52 PM, Michael Foord <fuzzyman at voidspace.org.uk>
> wrote:
> > On 08/04/2011 00:36, Anthony Scopatz wrote:
> >
> > On Thu, Apr 7, 2011 at 6:11 PM, Michael Foord <fuzzyman at voidspace.org.uk
> >
> > wrote:
> >>
> >> On 07/04/2011 22:41, Antoine Pitrou wrote:
> >>>
> >>> On Thu, 07 Apr 2011 17:32:24 -0400
> >>> Tres Seaver<tseaver at palladion.com>  wrote:
> >>>>>
> >>>>> Right now, we are talking about building "speed.python.org" to test
> >>>>> the speed of python interpreters, over time, and alongside one
> another
> >>>>> - cython *is not* an interpreter.
> >>>>>
> >>>>> Cython is out of scope for this.
> >>>>
> >>>> Why is it out of scope to use the benchmarks and test harness to
> answer
> >>>> questions like "can we use Cython to provide optional optimizations
> for
> >>>> the stdlib"?  I can certainly see value in havng an objective way to
> >>>> compare the macro benchmark performance of a Cython-optimized CPython
> >>>> vs. a vanilla CPython, as well as vs. PyPY, Jython, or IronPython.
> >>>
> >>> Agreed. Assuming someone wants to take care of the Cython side of
> >>> things, I don't think there's any reason to exclude it under the
> >>> dubious reason that it's "not an interpreter".
> >>> (would you exclude Psyco, if it was still alive?)
> >>>
> >>
> >> Well, sure - but within the scope of a GSOC project limiting it to "core
> >> python" seems like a more realistic goal.
> >>
> >> Adding cython later shouldn't be an issue if someone is willing to do
> the
> >> work.
> >
> > Jesse, I understand that we are talking about the benchmarks on
> > speed.pypy.org.  The current suite, and correct me if I
> > am wrong, is completely written in pure python so that any of the
> > 'interpreters' may run them.
> > My point, which I stand by, was that during the initial phase (where
> > benchmarks are defined) that the Cython crowd
> > should have a voice.  This should have an enriching effect on the whole
> > benchmarking task since they have
> > thought about this issue in a way that is largely orthogonal to the
> methods
> > PyPy developed.  I think it
> > would be a mistake to leave Cython out of the scoping study.
> >
> > Personally I think the Gsoc project should just take the pypy suite and
> run
> > with that - bikeshedding about what benchmarks to include is going to
> make
> > it hard to make progress. We can have fun with that discussion once we
> have
> > the infrastructure and *some* good benchmarks in place (and the pypy ones
> > are good ones).
> >
> > So I'm still with Jesse on this one. If there is any "discussion phase"
> as
> > part of the Gsoc project it should be very strictly bounded by time.
> >
>
> What michael said: My goal is is to get speed.pypy.org ported to be
> able to be used by $N interpreters, for $Y sets of performance
> numbers. I'm trying to constrain the problem, and the initial
> deployment so we don't spend the next year meandering about. It should
> be sufficient to port the benchmarks from speed.pypy.org, and any
> deltas from http://hg.python.org/benchmarks/ to Python 3 and the
> framework that runs the tests to start.
>
> I don't care if we eventually run cython, psyco, parrot, etc. But the
> focus at the language summit, and the continued focus of me getting
> the hardware via the PSF to host this on performance/speed.python.org
> is tightly focused on the pypy, ironpython, jython and cpython
> interpreters.
>
>

> Let's just get our basics done first before we go all crazy with adding
> stuff :)
>
>
Ahh gotcha, I think I misunderstood the scope in the short term ;).

Be Well
Anthony



>  jesse
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20110407/89a82e18/attachment.html>