Mailman 3 September 2012 - Speed

Re: [Speed] standalone PyPy benchmarks ported
by Maciej Fijalkowski 15 Sep '12

15 Sep '12

On Fri, Sep 14, 2012 at 10:19 PM, Brett Cannon <brett(a)python.org> wrote: > So I managed to get the following benchmarks moved into the unladen repo > (not pushed yet until I figure out some reasonable scaling values as some > finish probably too fast and others go for a while): > > chaos > fannkuch > meteor-contest (renamed meteor_contest) > spectral-norm (renamed spectral_norm) > telco > bm_mako (renamed bm_mako_v2; also pulled in mako 0.9.7 for this benchmark) > go > hexiom2 > json_bench (renamed json_dump_v2) > raytrace_simple (renamed raytrace) > > Most of the porting was range/xrange related. After that is was str/unicode. > I also stopped having the benchmarks write out files as it was always to > verify results and not a core part of the benchmark. > > That leaves us with the benchmarks that rely on third-party projects. The > chameleon benchmark can probably be ported as chameleon has a version > released running on Python 3. But django and html5lib have only > in-development versions that support Python 3. If we want to pull in the tip > of their repos then those benchmarks can also be ported now rather than > later. People have opinions on in-dev code vs. released for benchmarking? > > There is also the sphinx benchmark, but that requires getting CPython's docs > building under Python 3 (see http://bugs.python.org/issue10224). > > _______________________________________________ > Speed mailing list > Speed(a)python.org > http://mail.python.org/mailman/listinfo/speed > great job!

1 0

standalone PyPy benchmarks ported
by Brett Cannon 15 Sep '12

15 Sep '12

So I managed to get the following benchmarks moved into the unladen repo (not pushed yet until I figure out some reasonable scaling values as some finish probably too fast and others go for a while): chaos fannkuch meteor-contest (renamed meteor_contest) spectral-norm (renamed spectral_norm) telco bm_mako (renamed bm_mako_v2; also pulled in mako 0.9.7 for this benchmark) go hexiom2 json_bench (renamed json_dump_v2) raytrace_simple (renamed raytrace) Most of the porting was range/xrange related. After that is was str/unicode. I also stopped having the benchmarks write out files as it was always to verify results and not a core part of the benchmark. That leaves us with the benchmarks that rely on third-party projects. The chameleon benchmark can probably be ported as chameleon has a version released running on Python 3. But django and html5lib have only in-development versions that support Python 3. If we want to pull in the tip of their repos then those benchmarks can also be ported now rather than later. People have opinions on in-dev code vs. released for benchmarking? There is also the sphinx benchmark, but that requires getting CPython's docs building under Python 3 (see http://bugs.python.org/issue10224).

1 0

Re: [Speed] What PyPy benchmarks are (un)important?
by Miquel Torres 13 Sep '12

13 Sep '12

We could easily add the newer benchmarks to the front page, easy to do for the first plot. But the historical plot depends on there being data across all versions, so *that* geometric average would need to be done on the common set of 20 benchmarks for all PyPy versions, which means it will be different from the first average. Alternatively, we can drop the older PyPy versions and start with the oldest one that has data for the full set of benchmarks. Any other ideas? Miquel 2012/9/13 Maciej Fijalkowski <fijall(a)gmail.com>: > On Thu, Sep 13, 2012 at 12:35 AM, Brett Cannon <brett(a)python.org> wrote: >> I went through the list of benchmarks that PyPy has to see which ones could >> be ported to Python 3 now (others can be in the future but they depend on a >> project who has not released an official version with python 3 support): >> >> ai >> chaos >> fannkuch >> float >> meteor-contest >> nbody_modified >> richards >> spectral-norm >> telco >> >> bm_chameleon* >> bm_mako >> go >> hexiom2 >> json_bench >> pidigits >> pyflate-fast >> raytrace-simple >> sphinx* >> >> The first grouping is the 20 shown on the speed.pypy.org homepage, the rest >> are in the complete list. Anything with an asterisk has an external >> dependency that is not already in the unladen benchmarks. >> >> Are the twenty shown on the homepage of speed.pypy.org in some way special, >> or were they the first benchmarks that you were good/bad at, or what? Are >> there any benchmarks here that are particularly good or bad? I'm trying to >> prioritize what benchmarks I port so that if I hit a time crunch I got the >> critical ones moved first. > > The 20 shown on the front page are the ones that we have full > historical data, so we can compare. Others are simply newer. > > I don't think there is any priority associated, we should probably put > the others on the first page despite not having full data. > _______________________________________________ > Speed mailing list > Speed(a)python.org > http://mail.python.org/mailman/listinfo/speed

1 0

Re: [Speed] What PyPy benchmarks are (un)important?
by Maciej Fijalkowski 13 Sep '12

13 Sep '12

On Thu, Sep 13, 2012 at 12:35 AM, Brett Cannon <brett(a)python.org> wrote: > I went through the list of benchmarks that PyPy has to see which ones could > be ported to Python 3 now (others can be in the future but they depend on a > project who has not released an official version with python 3 support): > > ai > chaos > fannkuch > float > meteor-contest > nbody_modified > richards > spectral-norm > telco > > bm_chameleon* > bm_mako > go > hexiom2 > json_bench > pidigits > pyflate-fast > raytrace-simple > sphinx* > > The first grouping is the 20 shown on the speed.pypy.org homepage, the rest > are in the complete list. Anything with an asterisk has an external > dependency that is not already in the unladen benchmarks. > > Are the twenty shown on the homepage of speed.pypy.org in some way special, > or were they the first benchmarks that you were good/bad at, or what? Are > there any benchmarks here that are particularly good or bad? I'm trying to > prioritize what benchmarks I port so that if I hit a time crunch I got the > critical ones moved first. The 20 shown on the front page are the ones that we have full historical data, so we can compare. Others are simply newer. I don't think there is any priority associated, we should probably put the others on the first page despite not having full data.

1 0

What PyPy benchmarks are (un)important?
by Brett Cannon 13 Sep '12

13 Sep '12

I went through the list of benchmarks that PyPy has to see which ones could be ported to Python 3 now (others can be in the future but they depend on a project who has not released an official version with python 3 support): ai chaos fannkuch float meteor-contest nbody_modified richards spectral-norm telco bm_chameleon* bm_mako go hexiom2 json_bench pidigits pyflate-fast raytrace-simple sphinx* The first grouping is the 20 shown on the speed.pypy.org homepage, the rest are in the complete list. Anything with an asterisk has an external dependency that is not already in the unladen benchmarks. Are the twenty shown on the homepage of speed.pypy.org in some way special, or were they the first benchmarks that you were good/bad at, or what? Are there any benchmarks here that are particularly good or bad? I'm trying to prioritize what benchmarks I port so that if I hit a time crunch I got the critical ones moved first.

1 0

Re: [Speed] Are benchmarks and libraries mutable?
by Maciej Fijalkowski 02 Sep '12

02 Sep '12

On Sun, Sep 2, 2012 at 12:10 AM, Brett Cannon <brett(a)python.org> wrote: > > > On Sat, Sep 1, 2012 at 2:57 PM, Antoine Pitrou <solipsis(a)pitrou.net> wrote: >> >> On Sat, 1 Sep 2012 13:21:36 -0400 >> Brett Cannon <brett(a)python.org> wrote: >> > >> > One is moving benchmarks from PyPy over to the unladen repo on >> > hg.python.org/benchmarks. But I wanted to first make sure people don't >> > view >> > the benchmarks as immutable (e.g. as Octane does: >> > https://developers.google.com/octane/faq). Since the benchmarks are >> > always >> > relative between two interpreters their immutability isn't critical >> > compared to if we were to report some overall score. But it also means >> > that >> > any changes made would throw off historical comparisons. For instance, >> > if I >> > take PyPy's Mako benchmark (which does a lot more work), should it be >> > named >> > mako_v2, or should we just replace mako wholesale? >> >> mako_v2 sounds fine to me. Mutating benchmarks makes things confusing: >> one person may report that interpreter A is faster than interpreter B >> on a given benchmark, and another person retort that no, interpreter B >> is faster than interpreter A. >> >> Besides, if you want to have useful timelines on speed.p.o, you >> definitely need stable benchmarks. >> >> > And the second is the same question for libraries. For instance, the >> > unladen benchmarks have Django 1.1a0 as the version which is rather >> > ancient. And with 1.5 coming out with provisional Python 3 support I >> > obviously would like to update it. But the same questions as with >> > benchmarks crops up in reference to immutability. >> >> django_v2 sounds fine too :) > > > True, but having to carry around multiple copies of libraries just becomes a > pain. You just kill django when you introduce django v2 (alternatively you remove the history and keep the name django). Historical outdated benchmarks are not as interesting. > >> >> >> > (e.g. I will have to probably update the 2.7 code to use >> > io.BytesIO instead of StringIO.StringIO to be on more equal footing). >> >> I disagree. If io.BytesIO is faster than StringIO.StringIO then it's >> normal for the benchmark results to reflect that (ditto if it's slower). >> >> > If we can't find a reasonable way to handle all of this then what I will >> > do >> > is branch the unladen benchmarks for 2.x/3.x benchmarking, and then >> > create >> > another branch of the benchmark suite to just be for Python 3.x so that >> > we >> > can start fresh with a new set of benchmarks that will never change >> > themselves for benchmarking Python 3 itself. >> >> Why not simply add Python 3-specific benchmarks to the mix? >> You can then create a "py3" benchmark suite in perf.py (and perhaps >> also a "py2" one). > > > To avoid historical baggage and to start from a clean slate. I don't > necessarily want to carry around Python 2 benchmarks forever. It's not a > massive concern, just a nicety. If you guys want to have any cooperation with us, you have to carry Python 2 benchmarks for indefinite amount of time. Cheers, fijal

1 0

Re: [Speed] Are benchmarks and libraries mutable?
by Antoine Pitrou 02 Sep '12

02 Sep '12

On Sat, 1 Sep 2012 18:10:34 -0400 Brett Cannon <brett(a)python.org> wrote: > True, but having to carry around multiple copies of libraries just becomes > a pain. Apart from the initial pain of adding code to deal with multiple copies, I don't see how painful "carrying" copies can be. It's not like you have to physically carry the files :) > > > If we can't find a reasonable way to handle all of this then what I will > > do > > > is branch the unladen benchmarks for 2.x/3.x benchmarking, and then > > create > > > another branch of the benchmark suite to just be for Python 3.x so that > > we > > > can start fresh with a new set of benchmarks that will never change > > > themselves for benchmarking Python 3 itself. > > > > Why not simply add Python 3-specific benchmarks to the mix? > > You can then create a "py3" benchmark suite in perf.py (and perhaps > > also a "py2" one). > > > > To avoid historical baggage and to start from a clean slate. I don't > necessarily want to carry around Python 2 benchmarks forever. It's not a > massive concern, just a nicety. We can decide to remove some benchmarks when they become too old. It's no reason to fork a separate development branch, though. Most of the current benchmarks already run under 3.x so you would just duplicate maintenance work. Regards Antoine. -- Software development and contracting: http://pro.pitrou.net

1 0

Re: [Speed] Are benchmarks and libraries mutable?
by Antoine Pitrou 02 Sep '12

02 Sep '12

On Sat, 1 Sep 2012 13:21:36 -0400 Brett Cannon <brett(a)python.org> wrote: > > One is moving benchmarks from PyPy over to the unladen repo on > hg.python.org/benchmarks. But I wanted to first make sure people don't view > the benchmarks as immutable (e.g. as Octane does: > https://developers.google.com/octane/faq). Since the benchmarks are always > relative between two interpreters their immutability isn't critical > compared to if we were to report some overall score. But it also means that > any changes made would throw off historical comparisons. For instance, if I > take PyPy's Mako benchmark (which does a lot more work), should it be named > mako_v2, or should we just replace mako wholesale? mako_v2 sounds fine to me. Mutating benchmarks makes things confusing: one person may report that interpreter A is faster than interpreter B on a given benchmark, and another person retort that no, interpreter B is faster than interpreter A. Besides, if you want to have useful timelines on speed.p.o, you definitely need stable benchmarks. > And the second is the same question for libraries. For instance, the > unladen benchmarks have Django 1.1a0 as the version which is rather > ancient. And with 1.5 coming out with provisional Python 3 support I > obviously would like to update it. But the same questions as with > benchmarks crops up in reference to immutability. django_v2 sounds fine too :) > (e.g. I will have to probably update the 2.7 code to use > io.BytesIO instead of StringIO.StringIO to be on more equal footing). I disagree. If io.BytesIO is faster than StringIO.StringIO then it's normal for the benchmark results to reflect that (ditto if it's slower). > If we can't find a reasonable way to handle all of this then what I will do > is branch the unladen benchmarks for 2.x/3.x benchmarking, and then create > another branch of the benchmark suite to just be for Python 3.x so that we > can start fresh with a new set of benchmarks that will never change > themselves for benchmarking Python 3 itself. Why not simply add Python 3-specific benchmarks to the mix? You can then create a "py3" benchmark suite in perf.py (and perhaps also a "py2" one). Regards Antoine. -- Software development and contracting: http://pro.pitrou.net

2 1

Re: [Speed] Are benchmarks and libraries mutable?
by Maciej Fijalkowski 02 Sep '12

02 Sep '12

On Sat, Sep 1, 2012 at 9:45 PM, Gregory P. Smith <greg(a)krypto.org> wrote: > > > On Sat, Sep 1, 2012 at 10:21 AM, Brett Cannon <brett(a)python.org> wrote: >> >> Now that I can run benchmarks against Python 2.7 and 3.3 simultaneously, >> I'm ready to start updating the benchmarks. This involves two parts. >> >> One is moving benchmarks from PyPy over to the unladen repo on >> hg.python.org/benchmarks. But I wanted to first make sure people don't view >> the benchmarks as immutable (e.g. as Octane does: >> https://developers.google.com/octane/faq). Since the benchmarks are always >> relative between two interpreters their immutability isn't critical compared >> to if we were to report some overall score. But it also means that any >> changes made would throw off historical comparisons. For instance, if I take >> PyPy's Mako benchmark (which does a lot more work), should it be named >> mako_v2, or should we just replace mako wholesale? > > > I dislike benchmark immutability. The rest of the world including your > local computing environment where benchmarks run continues to change around > benchmarks which really makes using historical benchmark data from a run on > an old version for comparison to a recent modern run pointless. So far we (pypy) managed to maintain enough of the environment under control to have meaningful historical data. We have the same machine and we monitor whether changes introduce something new or not. Of course ideally, it's impossible, but for real world what we're doing is good enough. Cheers, fijal

1 0

Re: [Speed] Are benchmarks and libraries mutable?
by Gregory P. Smith 02 Sep '12

02 Sep '12

On Sat, Sep 1, 2012 at 10:21 AM, Brett Cannon <brett(a)python.org> wrote: > Now that I can run benchmarks against Python 2.7 and 3.3 simultaneously, > I'm ready to start updating the benchmarks. This involves two parts. > > One is moving benchmarks from PyPy over to the unladen repo on > hg.python.org/benchmarks. But I wanted to first make sure people don't > view the benchmarks as immutable (e.g. as Octane does: > https://developers.google.com/octane/faq). Since the benchmarks are > always relative between two interpreters their immutability isn't critical > compared to if we were to report some overall score. But it also means that > any changes made would throw off historical comparisons. For instance, if I > take PyPy's Mako benchmark (which does a lot more work), should it be named > mako_v2, or should we just replace mako wholesale? > I dislike benchmark immutability. The rest of the world including your local computing environment where benchmarks run continues to change around benchmarks which really makes using historical benchmark data from a run on an old version for comparison to a recent modern run pointless. What is needed more is benchmark *rerunability* and *repeatability*. So that an old version of a Python implementation can be built and run the current benchmark suite today within the exact same environment as a current version of a python implementation. They key is that they ran the same thing on the same hardware in the same configuration at around the same time. Nothing else is a valid comparison as too many untracked unquantified variables have changed in the interim. Where the above clearly fails: creating historical trend graphs. If you want a setup that runs the benchmarks after every commit, or at least runs them as continuously as possible _that_ benchmark suite needs to be as immutable as possible. The machine on which they are run also needs to be locked down to have no updates applied and nothing else running on it *ever*. Whenever either the bechmark suite or the historical trend benchmark running os, distro or hardware is mutated it needs to be clearly noted so deltas at that time in the results can be flagged to mark a discontinuity in the trend data as being due to the external changes. ONE way to do this is always version benchmark names. Any time one is updated, give it a new versioned name so it can't be compared with past results. Otherwise for historical data, periodically rerunning the benchmark suite on older versions (releases and betas) for use in modern comparisons is ideal. -gps > And the second is the same question for libraries. For instance, the > unladen benchmarks have Django 1.1a0 as the version which is rather > ancient. And with 1.5 coming out with provisional Python 3 support I > obviously would like to update it. But the same questions as with > benchmarks crops up in reference to immutability. Another thing is that > 2to3 can't actually be ported using 2to3 ( > http://bugs.python.org/issue15834) and so that itself will require two > versions -- a 2.x version (probably from Python 2.7's stdlib) and a 3.x > version (from the 3.2 stdlib) -- which already starts to add interesting > issues for me in terms of comparing performance (e.g. I will have to > probably update the 2.7 code to use io.BytesIO instead of StringIO.StringIO > to be on more equal footing). Similar thing goes for html5lib which has > developed its Python 3 support separately from its Python 2 code. > > If we can't find a reasonable way to handle all of this then what I will > do is branch the unladen benchmarks for 2.x/3.x benchmarking, and then > create another branch of the benchmark suite to just be for Python 3.x so > that we can start fresh with a new set of benchmarks that will never change > themselves for benchmarking Python 3 itself. That would also mean we could > start of with whatever is needed from PyPy and unladen to have the optimal > benchmark runner for speed.python.org. > > _______________________________________________ > Speed mailing list > Speed(a)python.org > http://mail.python.org/mailman/listinfo/speed >

1 0