Mailman 3 September 2012 - Speed

slow benchmark
by Antoine Pitrou Sept. 30, 2012

Sept. 30, 2012

Hello, The hexiom benchmark is very slow. Is there a reason it's included there? Regards Antoine. -- Software development and contracting: http://pro.pitrou.net

2 1

Speed and disk space
by Maciej Fijalkowski Sept. 22, 2012

Sept. 22, 2012

Hello everyone. I would like to complain that speed has still no disk space, despite having a huge empty disk space on no partition chilling there. Last time I complained was July 13th and it's a little over 2 months by now. Cheers, fijal

1 0

Re: [Speed] What PyPy benchmarks are (un)important?
by Miquel Torres Sept. 18, 2012

Sept. 18, 2012

We could easily add the newer benchmarks to the front page, easy to do for the first plot. But the historical plot depends on there being data across all versions, so *that* geometric average would need to be done on the common set of 20 benchmarks for all PyPy versions, which means it will be different from the first average. Alternatively, we can drop the older PyPy versions and start with the oldest one that has data for the full set of benchmarks. Any other ideas? Miquel 2012/9/13 Maciej Fijalkowski <fijall(a)gmail.com>: > On Thu, Sep 13, 2012 at 12:35 AM, Brett Cannon <brett(a)python.org> wrote: >> I went through the list of benchmarks that PyPy has to see which ones could >> be ported to Python 3 now (others can be in the future but they depend on a >> project who has not released an official version with python 3 support): >> >> ai >> chaos >> fannkuch >> float >> meteor-contest >> nbody_modified >> richards >> spectral-norm >> telco >> >> bm_chameleon* >> bm_mako >> go >> hexiom2 >> json_bench >> pidigits >> pyflate-fast >> raytrace-simple >> sphinx* >> >> The first grouping is the 20 shown on the speed.pypy.org homepage, the rest >> are in the complete list. Anything with an asterisk has an external >> dependency that is not already in the unladen benchmarks. >> >> Are the twenty shown on the homepage of speed.pypy.org in some way special, >> or were they the first benchmarks that you were good/bad at, or what? Are >> there any benchmarks here that are particularly good or bad? I'm trying to >> prioritize what benchmarks I port so that if I hit a time crunch I got the >> critical ones moved first. > > The 20 shown on the front page are the ones that we have full > historical data, so we can compare. Others are simply newer. > > I don't think there is any priority associated, we should probably put > the others on the first page despite not having full data. > _______________________________________________ > Speed mailing list > Speed(a)python.org > http://mail.python.org/mailman/listinfo/speed

1 0

Re: [Speed] standalone PyPy benchmarks ported
by Maciej Fijalkowski Sept. 17, 2012

Sept. 17, 2012

On Mon, Sep 17, 2012 at 8:17 PM, Brett Cannon <brett(a)python.org> wrote: > > > On Mon, Sep 17, 2012 at 11:36 AM, Maciej Fijalkowski <fijall(a)gmail.com> > wrote: >> >> On Mon, Sep 17, 2012 at 5:00 PM, Brett Cannon <brett(a)python.org> wrote: >> > >> > >> > On Sun, Sep 16, 2012 at 10:54 AM, Maciej Fijalkowski <fijall(a)gmail.com> >> > wrote: >> >> >> >> On Sun, Sep 16, 2012 at 4:43 PM, Brett Cannon <brett(a)python.org> wrote: >> >> > Quick question about the hexiom2 benchmark: what does it measure? It >> >> > is >> >> > by >> >> > far the slowest benchmark I ported, and considering it isn't a >> >> > real-world >> >> > app benchmark I want to make sure the slowness of it is worth it. >> >> > Otherwise >> >> > I would rather drop it since having something run 1/25 as many >> >> > iterations >> >> > compared to the other simple benchmarks seems to water down its >> >> > robustness. >> >> >> >> It's a puzzle solver. It got included because PyPy 1.9 got slower than >> >> 1.8 on this particular benchmark that people were actually running >> >> somewhere, so it has *some* value. >> > >> > >> > Fair enough. Just wanted to make sure that it was worth having a slow >> > execution over. >> > >> >> >> >> I wonder, does adding a fixed >> >> random number seed help the distribution? >> > >> > >> > Fix how? hexiom2 doesn't use a random value for anything. >> >> Ok, then please explain why having 1/25th of iterations kill robustness? > > > Less iterations to help smooth out any bumps in the measurements. E.g 4 > iterations compared to 100 doesn't lead to as even of a measurement. I mean > you would hope because the benchmark goes for so long that it would just > level out within a single run instead of needing multiple runs to get the > same evening out. Yes precisely :) I think the term "iterations" is a bit overloaded. The "stability" is more important. > > -Brett > >> >> >> > >> > -Brett >> > >> >> >> >> >> >> > >> >> > >> >> > On Fri, Sep 14, 2012 at 5:44 PM, Maciej Fijalkowski >> >> > <fijall(a)gmail.com> >> >> > wrote: >> >> >> >> >> >> On Fri, Sep 14, 2012 at 10:19 PM, Brett Cannon <brett(a)python.org> >> >> >> wrote: >> >> >> > So I managed to get the following benchmarks moved into the >> >> >> > unladen >> >> >> > repo >> >> >> > (not pushed yet until I figure out some reasonable scaling values >> >> >> > as >> >> >> > some >> >> >> > finish probably too fast and others go for a while): >> >> >> > >> >> >> > chaos >> >> >> > fannkuch >> >> >> > meteor-contest (renamed meteor_contest) >> >> >> > spectral-norm (renamed spectral_norm) >> >> >> > telco >> >> >> > bm_mako (renamed bm_mako_v2; also pulled in mako 0.9.7 for this >> >> >> > benchmark) >> >> >> > go >> >> >> > hexiom2 >> >> >> > json_bench (renamed json_dump_v2) >> >> >> > raytrace_simple (renamed raytrace) >> >> >> > >> >> >> > Most of the porting was range/xrange related. After that is was >> >> >> > str/unicode. >> >> >> > I also stopped having the benchmarks write out files as it was >> >> >> > always >> >> >> > to >> >> >> > verify results and not a core part of the benchmark. >> >> >> > >> >> >> > That leaves us with the benchmarks that rely on third-party >> >> >> > projects. >> >> >> > The >> >> >> > chameleon benchmark can probably be ported as chameleon has a >> >> >> > version >> >> >> > released running on Python 3. But django and html5lib have only >> >> >> > in-development versions that support Python 3. If we want to pull >> >> >> > in >> >> >> > the >> >> >> > tip >> >> >> > of their repos then those benchmarks can also be ported now rather >> >> >> > than >> >> >> > later. People have opinions on in-dev code vs. released for >> >> >> > benchmarking? >> >> >> > >> >> >> > There is also the sphinx benchmark, but that requires getting >> >> >> > CPython's >> >> >> > docs >> >> >> > building under Python 3 (see http://bugs.python.org/issue10224). >> >> >> > >> >> >> > _______________________________________________ >> >> >> > Speed mailing list >> >> >> > Speed(a)python.org >> >> >> > http://mail.python.org/mailman/listinfo/speed >> >> >> > >> >> >> >> >> >> great job! >> >> > >> >> > >> > >> > > >

1 0

Re: [Speed] standalone PyPy benchmarks ported
by Brett Cannon Sept. 17, 2012

Sept. 17, 2012

On Mon, Sep 17, 2012 at 11:36 AM, Maciej Fijalkowski <fijall(a)gmail.com>wrote: > On Mon, Sep 17, 2012 at 5:00 PM, Brett Cannon <brett(a)python.org> wrote: > > > > > > On Sun, Sep 16, 2012 at 10:54 AM, Maciej Fijalkowski <fijall(a)gmail.com> > > wrote: > >> > >> On Sun, Sep 16, 2012 at 4:43 PM, Brett Cannon <brett(a)python.org> wrote: > >> > Quick question about the hexiom2 benchmark: what does it measure? It > is > >> > by > >> > far the slowest benchmark I ported, and considering it isn't a > >> > real-world > >> > app benchmark I want to make sure the slowness of it is worth it. > >> > Otherwise > >> > I would rather drop it since having something run 1/25 as many > >> > iterations > >> > compared to the other simple benchmarks seems to water down its > >> > robustness. > >> > >> It's a puzzle solver. It got included because PyPy 1.9 got slower than > >> 1.8 on this particular benchmark that people were actually running > >> somewhere, so it has *some* value. > > > > > > Fair enough. Just wanted to make sure that it was worth having a slow > > execution over. > > > >> > >> I wonder, does adding a fixed > >> random number seed help the distribution? > > > > > > Fix how? hexiom2 doesn't use a random value for anything. > > Ok, then please explain why having 1/25th of iterations kill robustness? > Less iterations to help smooth out any bumps in the measurements. E.g 4 iterations compared to 100 doesn't lead to as even of a measurement. I mean you would hope because the benchmark goes for so long that it would just level out within a single run instead of needing multiple runs to get the same evening out. -Brett > > > > > -Brett > > > >> > >> > >> > > >> > > >> > On Fri, Sep 14, 2012 at 5:44 PM, Maciej Fijalkowski <fijall(a)gmail.com > > > >> > wrote: > >> >> > >> >> On Fri, Sep 14, 2012 at 10:19 PM, Brett Cannon <brett(a)python.org> > >> >> wrote: > >> >> > So I managed to get the following benchmarks moved into the unladen > >> >> > repo > >> >> > (not pushed yet until I figure out some reasonable scaling values > as > >> >> > some > >> >> > finish probably too fast and others go for a while): > >> >> > > >> >> > chaos > >> >> > fannkuch > >> >> > meteor-contest (renamed meteor_contest) > >> >> > spectral-norm (renamed spectral_norm) > >> >> > telco > >> >> > bm_mako (renamed bm_mako_v2; also pulled in mako 0.9.7 for this > >> >> > benchmark) > >> >> > go > >> >> > hexiom2 > >> >> > json_bench (renamed json_dump_v2) > >> >> > raytrace_simple (renamed raytrace) > >> >> > > >> >> > Most of the porting was range/xrange related. After that is was > >> >> > str/unicode. > >> >> > I also stopped having the benchmarks write out files as it was > always > >> >> > to > >> >> > verify results and not a core part of the benchmark. > >> >> > > >> >> > That leaves us with the benchmarks that rely on third-party > projects. > >> >> > The > >> >> > chameleon benchmark can probably be ported as chameleon has a > version > >> >> > released running on Python 3. But django and html5lib have only > >> >> > in-development versions that support Python 3. If we want to pull > in > >> >> > the > >> >> > tip > >> >> > of their repos then those benchmarks can also be ported now rather > >> >> > than > >> >> > later. People have opinions on in-dev code vs. released for > >> >> > benchmarking? > >> >> > > >> >> > There is also the sphinx benchmark, but that requires getting > >> >> > CPython's > >> >> > docs > >> >> > building under Python 3 (see http://bugs.python.org/issue10224). > >> >> > > >> >> > _______________________________________________ > >> >> > Speed mailing list > >> >> > Speed(a)python.org > >> >> > http://mail.python.org/mailman/listinfo/speed > >> >> > > >> >> > >> >> great job! > >> > > >> > > > > > >

1 0

Re: [Speed] standalone PyPy benchmarks ported
by Maciej Fijalkowski Sept. 17, 2012

Sept. 17, 2012

On Mon, Sep 17, 2012 at 5:00 PM, Brett Cannon <brett(a)python.org> wrote: > > > On Sun, Sep 16, 2012 at 10:54 AM, Maciej Fijalkowski <fijall(a)gmail.com> > wrote: >> >> On Sun, Sep 16, 2012 at 4:43 PM, Brett Cannon <brett(a)python.org> wrote: >> > Quick question about the hexiom2 benchmark: what does it measure? It is >> > by >> > far the slowest benchmark I ported, and considering it isn't a >> > real-world >> > app benchmark I want to make sure the slowness of it is worth it. >> > Otherwise >> > I would rather drop it since having something run 1/25 as many >> > iterations >> > compared to the other simple benchmarks seems to water down its >> > robustness. >> >> It's a puzzle solver. It got included because PyPy 1.9 got slower than >> 1.8 on this particular benchmark that people were actually running >> somewhere, so it has *some* value. > > > Fair enough. Just wanted to make sure that it was worth having a slow > execution over. > >> >> I wonder, does adding a fixed >> random number seed help the distribution? > > > Fix how? hexiom2 doesn't use a random value for anything. Ok, then please explain why having 1/25th of iterations kill robustness? > > -Brett > >> >> >> > >> > >> > On Fri, Sep 14, 2012 at 5:44 PM, Maciej Fijalkowski <fijall(a)gmail.com> >> > wrote: >> >> >> >> On Fri, Sep 14, 2012 at 10:19 PM, Brett Cannon <brett(a)python.org> >> >> wrote: >> >> > So I managed to get the following benchmarks moved into the unladen >> >> > repo >> >> > (not pushed yet until I figure out some reasonable scaling values as >> >> > some >> >> > finish probably too fast and others go for a while): >> >> > >> >> > chaos >> >> > fannkuch >> >> > meteor-contest (renamed meteor_contest) >> >> > spectral-norm (renamed spectral_norm) >> >> > telco >> >> > bm_mako (renamed bm_mako_v2; also pulled in mako 0.9.7 for this >> >> > benchmark) >> >> > go >> >> > hexiom2 >> >> > json_bench (renamed json_dump_v2) >> >> > raytrace_simple (renamed raytrace) >> >> > >> >> > Most of the porting was range/xrange related. After that is was >> >> > str/unicode. >> >> > I also stopped having the benchmarks write out files as it was always >> >> > to >> >> > verify results and not a core part of the benchmark. >> >> > >> >> > That leaves us with the benchmarks that rely on third-party projects. >> >> > The >> >> > chameleon benchmark can probably be ported as chameleon has a version >> >> > released running on Python 3. But django and html5lib have only >> >> > in-development versions that support Python 3. If we want to pull in >> >> > the >> >> > tip >> >> > of their repos then those benchmarks can also be ported now rather >> >> > than >> >> > later. People have opinions on in-dev code vs. released for >> >> > benchmarking? >> >> > >> >> > There is also the sphinx benchmark, but that requires getting >> >> > CPython's >> >> > docs >> >> > building under Python 3 (see http://bugs.python.org/issue10224). >> >> > >> >> > _______________________________________________ >> >> > Speed mailing list >> >> > Speed(a)python.org >> >> > http://mail.python.org/mailman/listinfo/speed >> >> > >> >> >> >> great job! >> > >> > > >

1 0

Re: [Speed] standalone PyPy benchmarks ported
by Brett Cannon Sept. 17, 2012

Sept. 17, 2012

On Sun, Sep 16, 2012 at 10:54 AM, Maciej Fijalkowski <fijall(a)gmail.com>wrote: > On Sun, Sep 16, 2012 at 4:43 PM, Brett Cannon <brett(a)python.org> wrote: > > Quick question about the hexiom2 benchmark: what does it measure? It is > by > > far the slowest benchmark I ported, and considering it isn't a real-world > > app benchmark I want to make sure the slowness of it is worth it. > Otherwise > > I would rather drop it since having something run 1/25 as many iterations > > compared to the other simple benchmarks seems to water down its > robustness. > > It's a puzzle solver. It got included because PyPy 1.9 got slower than > 1.8 on this particular benchmark that people were actually running > somewhere, so it has *some* value. Fair enough. Just wanted to make sure that it was worth having a slow execution over. > I wonder, does adding a fixed > random number seed help the distribution? > Fix how? hexiom2 doesn't use a random value for anything. -Brett > > > > > > > On Fri, Sep 14, 2012 at 5:44 PM, Maciej Fijalkowski <fijall(a)gmail.com> > > wrote: > >> > >> On Fri, Sep 14, 2012 at 10:19 PM, Brett Cannon <brett(a)python.org> > wrote: > >> > So I managed to get the following benchmarks moved into the unladen > repo > >> > (not pushed yet until I figure out some reasonable scaling values as > >> > some > >> > finish probably too fast and others go for a while): > >> > > >> > chaos > >> > fannkuch > >> > meteor-contest (renamed meteor_contest) > >> > spectral-norm (renamed spectral_norm) > >> > telco > >> > bm_mako (renamed bm_mako_v2; also pulled in mako 0.9.7 for this > >> > benchmark) > >> > go > >> > hexiom2 > >> > json_bench (renamed json_dump_v2) > >> > raytrace_simple (renamed raytrace) > >> > > >> > Most of the porting was range/xrange related. After that is was > >> > str/unicode. > >> > I also stopped having the benchmarks write out files as it was always > to > >> > verify results and not a core part of the benchmark. > >> > > >> > That leaves us with the benchmarks that rely on third-party projects. > >> > The > >> > chameleon benchmark can probably be ported as chameleon has a version > >> > released running on Python 3. But django and html5lib have only > >> > in-development versions that support Python 3. If we want to pull in > the > >> > tip > >> > of their repos then those benchmarks can also be ported now rather > than > >> > later. People have opinions on in-dev code vs. released for > >> > benchmarking? > >> > > >> > There is also the sphinx benchmark, but that requires getting > CPython's > >> > docs > >> > building under Python 3 (see http://bugs.python.org/issue10224). > >> > > >> > _______________________________________________ > >> > Speed mailing list > >> > Speed(a)python.org > >> > http://mail.python.org/mailman/listinfo/speed > >> > > >> > >> great job! > > > > >

1 0

Re: [Speed] standalone PyPy benchmarks ported
by Maciej Fijalkowski Sept. 16, 2012

Sept. 16, 2012

On Sun, Sep 16, 2012 at 4:43 PM, Brett Cannon <brett(a)python.org> wrote: > Quick question about the hexiom2 benchmark: what does it measure? It is by > far the slowest benchmark I ported, and considering it isn't a real-world > app benchmark I want to make sure the slowness of it is worth it. Otherwise > I would rather drop it since having something run 1/25 as many iterations > compared to the other simple benchmarks seems to water down its robustness. It's a puzzle solver. It got included because PyPy 1.9 got slower than 1.8 on this particular benchmark that people were actually running somewhere, so it has *some* value. I wonder, does adding a fixed random number seed help the distribution? > > > On Fri, Sep 14, 2012 at 5:44 PM, Maciej Fijalkowski <fijall(a)gmail.com> > wrote: >> >> On Fri, Sep 14, 2012 at 10:19 PM, Brett Cannon <brett(a)python.org> wrote: >> > So I managed to get the following benchmarks moved into the unladen repo >> > (not pushed yet until I figure out some reasonable scaling values as >> > some >> > finish probably too fast and others go for a while): >> > >> > chaos >> > fannkuch >> > meteor-contest (renamed meteor_contest) >> > spectral-norm (renamed spectral_norm) >> > telco >> > bm_mako (renamed bm_mako_v2; also pulled in mako 0.9.7 for this >> > benchmark) >> > go >> > hexiom2 >> > json_bench (renamed json_dump_v2) >> > raytrace_simple (renamed raytrace) >> > >> > Most of the porting was range/xrange related. After that is was >> > str/unicode. >> > I also stopped having the benchmarks write out files as it was always to >> > verify results and not a core part of the benchmark. >> > >> > That leaves us with the benchmarks that rely on third-party projects. >> > The >> > chameleon benchmark can probably be ported as chameleon has a version >> > released running on Python 3. But django and html5lib have only >> > in-development versions that support Python 3. If we want to pull in the >> > tip >> > of their repos then those benchmarks can also be ported now rather than >> > later. People have opinions on in-dev code vs. released for >> > benchmarking? >> > >> > There is also the sphinx benchmark, but that requires getting CPython's >> > docs >> > building under Python 3 (see http://bugs.python.org/issue10224). >> > >> > _______________________________________________ >> > Speed mailing list >> > Speed(a)python.org >> > http://mail.python.org/mailman/listinfo/speed >> > >> >> great job! > >

1 0

Re: [Speed] standalone PyPy benchmarks ported
by Brett Cannon Sept. 16, 2012

Sept. 16, 2012

Quick question about the hexiom2 benchmark: what does it measure? It is by far the slowest benchmark I ported, and considering it isn't a real-world app benchmark I want to make sure the slowness of it is worth it. Otherwise I would rather drop it since having something run 1/25 as many iterations compared to the other simple benchmarks seems to water down its robustness. On Fri, Sep 14, 2012 at 5:44 PM, Maciej Fijalkowski <fijall(a)gmail.com>wrote: > On Fri, Sep 14, 2012 at 10:19 PM, Brett Cannon <brett(a)python.org> wrote: > > So I managed to get the following benchmarks moved into the unladen repo > > (not pushed yet until I figure out some reasonable scaling values as some > > finish probably too fast and others go for a while): > > > > chaos > > fannkuch > > meteor-contest (renamed meteor_contest) > > spectral-norm (renamed spectral_norm) > > telco > > bm_mako (renamed bm_mako_v2; also pulled in mako 0.9.7 for this > benchmark) > > go > > hexiom2 > > json_bench (renamed json_dump_v2) > > raytrace_simple (renamed raytrace) > > > > Most of the porting was range/xrange related. After that is was > str/unicode. > > I also stopped having the benchmarks write out files as it was always to > > verify results and not a core part of the benchmark. > > > > That leaves us with the benchmarks that rely on third-party projects. The > > chameleon benchmark can probably be ported as chameleon has a version > > released running on Python 3. But django and html5lib have only > > in-development versions that support Python 3. If we want to pull in the > tip > > of their repos then those benchmarks can also be ported now rather than > > later. People have opinions on in-dev code vs. released for benchmarking? > > > > There is also the sphinx benchmark, but that requires getting CPython's > docs > > building under Python 3 (see http://bugs.python.org/issue10224). > > > > _______________________________________________ > > Speed mailing list > > Speed(a)python.org > > http://mail.python.org/mailman/listinfo/speed > > > > great job! >

1 0

Re: [Speed] standalone PyPy benchmarks ported
by Brett Cannon Sept. 14, 2012

Sept. 14, 2012

I just pushed the changes. On Fri, Sep 14, 2012 at 4:19 PM, Brett Cannon <brett(a)python.org> wrote: > So I managed to get the following benchmarks moved into the unladen repo > (not pushed yet until I figure out some reasonable scaling values as some > finish probably too fast and others go for a while): > > chaos > fannkuch > meteor-contest (renamed meteor_contest) > spectral-norm (renamed spectral_norm) > telco > bm_mako (renamed bm_mako_v2; also pulled in mako 0.9.7 for this benchmark) > go > hexiom2 > json_bench (renamed json_dump_v2) > raytrace_simple (renamed raytrace) > > Most of the porting was range/xrange related. After that is was > str/unicode. I also stopped having the benchmarks write out files as it was > always to verify results and not a core part of the benchmark. > > That leaves us with the benchmarks that rely on third-party projects. The > chameleon benchmark can probably be ported as chameleon has a version > released running on Python 3. But django and html5lib have only > in-development versions that support Python 3. If we want to pull in the > tip of their repos then those benchmarks can also be ported now rather than > later. People have opinions on in-dev code vs. released for benchmarking? > > There is also the sphinx benchmark, but that requires getting CPython's > docs building under Python 3 (see http://bugs.python.org/issue10224). >

1 0