Hi I'm a bit worried with our current benchmarks state. We have around 4 benchmarks that had reasonable slowdowns recently and we keep putting new features that speed up other things. How can we even say we have actually fixed the original issue? Can we have a policy of not merging new performance features before having a story why benchmarks got slower? Current list: http://speed.pypy.org/timeline/?exe=1&base=none&ben=spectral-norm&env=tannit&revs=50 http://speed.pypy.org/timeline/?exe=1&base=none&ben=spitfire&env=tannit&revs=50 This is a good example why we should not work the way we work now: http://speed.pypy.org/timeline/?exe=1&base=none&ben=slowspitfire&env=tannit&revs=200 There was an issue, then the issue was fixed, but apparently not quite (7th of June is quite a bit slower than 25th of May) and then recently we introduced something that make it faster alltogether. Can we even fish the original issue? http://speed.pypy.org/timeline/?exe=1&base=none&ben=bm_mako&env=tannit&revs=200 http://speed.pypy.org/timeline/?exe=1&base=none&ben=nbody_modified&env=tannit&revs=50 (is it relevant or just noise?) http://speed.pypy.org/timeline/?exe=1&base=none&ben=telco&env=tannit&revs=50 Cheers, fijal
On 12/07/11 01:20, Maciej Fijalkowski wrote:
Hi
I'm a bit worried with our current benchmarks state. We have around 4 benchmarks that had reasonable slowdowns recently and we keep putting new features that speed up other things. How can we even say we have actually fixed the original issue? Can we have a policy of not merging new performance features before having a story why benchmarks got slower?
I think we really need to have support for branches on codespeed. Then, we can have a policy that a branch can be merged only if none of the benchmarks slows down. I heard that someone is working on it, but nothing concrete AFAIK, so I'm considering to do the work by myself (although I would prefer to work on something more in the core :-/) ciao, Anto
On Mon, Jul 11, 2011 at 4:29 PM, Antonio Cuni <anto.cuni@gmail.com> wrote:
On 12/07/11 01:20, Maciej Fijalkowski wrote:
Hi
I'm a bit worried with our current benchmarks state. We have around 4 benchmarks that had reasonable slowdowns recently and we keep putting new features that speed up other things. How can we even say we have actually fixed the original issue? Can we have a policy of not merging new performance features before having a story why benchmarks got slower?
I think we really need to have support for branches on codespeed. Then, we can have a policy that a branch can be merged only if none of the benchmarks slows down.
I heard that someone is working on it, but nothing concrete AFAIK, so I'm considering to do the work by myself (although I would prefer to work on something more in the core :-/)
ciao, Anto _______________________________________________ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Isn't there a GSOC on that? Anyway +1 from me, if there's a regression it needs to be fixed, or reverted. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero
+1 as well, but we need to make sure we can actually identify performance regressions. I don't know if currently we can draw any conclusions from significant drops in benchmark performance. +100 for benchmarking branches. I think there's a tangentially related GSoC (moving toward speed.python.org and being multi-interpreter), but I don't remember seeing a blog post announcing the accepted projects. Cheers, Dan On Jul 11, 2011 4:32 PM, "Alex Gaynor" <alex.gaynor@gmail.com> wrote:
On Mon, Jul 11, 2011 at 4:29 PM, Antonio Cuni <anto.cuni@gmail.com> wrote:
On 12/07/11 01:20, Maciej Fijalkowski wrote:
Hi
I'm a bit worried with our current benchmarks state. We have around 4 benchmarks that had reasonable slowdowns recently and we keep putting new features that speed up other things. How can we even say we have actually fixed the original issue? Can we have a policy of not merging new performance features before having a story why benchmarks got slower?
I think we really need to have support for branches on codespeed. Then, we can have a policy that a branch can be merged only if none of the benchmarks slows down.
I heard that someone is working on it, but nothing concrete AFAIK, so I'm considering to do the work by myself (although I would prefer to work on something more in the core :-/)
ciao, Anto _______________________________________________ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Isn't there a GSOC on that?
Anyway +1 from me, if there's a regression it needs to be fixed, or reverted.
Alex
-- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero
On Tue, Jul 12, 2011 at 1:29 AM, Antonio Cuni <anto.cuni@gmail.com> wrote:
On 12/07/11 01:20, Maciej Fijalkowski wrote:
Hi
I'm a bit worried with our current benchmarks state. We have around 4 benchmarks that had reasonable slowdowns recently and we keep putting new features that speed up other things. How can we even say we have actually fixed the original issue? Can we have a policy of not merging new performance features before having a story why benchmarks got slower?
I think we really need to have support for branches on codespeed. Then, we can have a policy that a branch can be merged only if none of the benchmarks slows down.
I'll follow up on the branches, but the issue is a bit irrelevant - we still have performance regressions by trunk checkins as well.
I heard that someone is working on it, but nothing concrete AFAIK, so I'm considering to do the work by myself (although I would prefer to work on something more in the core :-/)
ciao, Anto
On 12/07/11 09:01, Maciej Fijalkowski wrote:
I'll follow up on the branches, but the issue is a bit irrelevant - we still have performance regressions by trunk checkins as well.
it's not irrelevant. I won't solve the current issues, but it will help having more in the future. ciao, Anto
Branches are already implemented. speed.pypy.org just needs to be upgraded to Codespeed 0.8.x and its data migrated. That has not yet been done because we wanted to move to ep.io, which has not yet happend, and we are working on speed.python.org and somehow things have stalled. Migrating current speed.pypy.org (I mean keeping it on the current server but upgrading) would not take me very long. But it is on the way out... 2011/7/12 Antonio Cuni <anto.cuni@gmail.com>:
On 12/07/11 09:01, Maciej Fijalkowski wrote:
I'll follow up on the branches, but the issue is a bit irrelevant - we still have performance regressions by trunk checkins as well.
it's not irrelevant. I won't solve the current issues, but it will help having more in the future.
ciao, Anto _______________________________________________ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
On Tue, Jul 12, 2011 at 9:16 AM, Miquel Torres <tobami@googlemail.com> wrote:
Branches are already implemented. speed.pypy.org just needs to be upgraded to Codespeed 0.8.x and its data migrated.
I can help you with that, but that means having local changes from speed.pypy.org commited somewhere else.
That has not yet been done because we wanted to move to ep.io, which has not yet happend, and we are working on speed.python.org and somehow things have stalled.
speed.python.org is completely irrelevant here.
Migrating current speed.pypy.org (I mean keeping it on the current server but upgrading) would not take me very long. But it is on the way out...
We either should do the ep.io move (data has been imported up until very recently) or upgrade now.
2011/7/12 Antonio Cuni <anto.cuni@gmail.com>:
On 12/07/11 09:01, Maciej Fijalkowski wrote:
I'll follow up on the branches, but the issue is a bit irrelevant - we still have performance regressions by trunk checkins as well.
it's not irrelevant. I won't solve the current issues, but it will help having more in the future.
ciao, Anto _______________________________________________ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
On 12/07/11 09:19, Maciej Fijalkowski wrote:
On Tue, Jul 12, 2011 at 9:16 AM, Miquel Torres <tobami@googlemail.com> wrote:
Branches are already implemented. speed.pypy.org just needs to be upgraded to Codespeed 0.8.x and its data migrated.
oh, this is very cool, thank you :-)
I can help you with that, but that means having local changes from speed.pypy.org commited somewhere else.
what are the local changes about?
We either should do the ep.io move (data has been imported up until very recently) or upgrade now.
+1 for doing something, whatever the something is. I am willing to help, although I'm not sure what is needed. ciao, Anto
On Tue, Jul 12, 2011 at 09:19 +0200, Maciej Fijalkowski wrote:
On Tue, Jul 12, 2011 at 9:16 AM, Miquel Torres <tobami@googlemail.com> wrote:
Branches are already implemented. speed.pypy.org just needs to be upgraded to Codespeed 0.8.x and its data migrated.
I can help you with that, but that means having local changes from speed.pypy.org commited somewhere else.
That has not yet been done because we wanted to move to ep.io, which has not yet happend, and we are working on speed.python.org and somehow things have stalled.
speed.python.org is completely irrelevant here.
Probably i am missing something but couldn't speed.pypy.org and speed.python.org mostly merge? (maybe except for the front page and default comparisons/settings but those could be easily hosted wherever) best, holger
Migrating current speed.pypy.org (I mean keeping it on the current server but upgrading) would not take me very long. But it is on the way out...
We either should do the ep.io move (data has been imported up until very recently) or upgrade now.
2011/7/12 Antonio Cuni <anto.cuni@gmail.com>:
On 12/07/11 09:01, Maciej Fijalkowski wrote:
I'll follow up on the branches, but the issue is a bit irrelevant - we still have performance regressions by trunk checkins as well.
it's not irrelevant. I won't solve the current issues, but it will help having more in the future.
ciao, Anto _______________________________________________ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
_______________________________________________ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
They *are* being merged. The question here is to have branches *now*, as we can't know how long it will take for speed.python.org to be online. All things considered, I think the way to go would be to keep current speed.pypy.org hosting if at all possible until PyPy moves to speed.python.org. If that can be done, we can upgrade the current setup in no time. 2011/7/12 holger krekel <holger@merlinux.eu>:
On Tue, Jul 12, 2011 at 09:19 +0200, Maciej Fijalkowski wrote:
On Tue, Jul 12, 2011 at 9:16 AM, Miquel Torres <tobami@googlemail.com> wrote:
Branches are already implemented. speed.pypy.org just needs to be upgraded to Codespeed 0.8.x and its data migrated.
I can help you with that, but that means having local changes from speed.pypy.org commited somewhere else.
That has not yet been done because we wanted to move to ep.io, which has not yet happend, and we are working on speed.python.org and somehow things have stalled.
speed.python.org is completely irrelevant here.
Probably i am missing something but couldn't speed.pypy.org and speed.python.org mostly merge? (maybe except for the front page and default comparisons/settings but those could be easily hosted wherever)
best, holger
Migrating current speed.pypy.org (I mean keeping it on the current server but upgrading) would not take me very long. But it is on the way out...
We either should do the ep.io move (data has been imported up until very recently) or upgrade now.
2011/7/12 Antonio Cuni <anto.cuni@gmail.com>:
On 12/07/11 09:01, Maciej Fijalkowski wrote:
I'll follow up on the branches, but the issue is a bit irrelevant - we still have performance regressions by trunk checkins as well.
it's not irrelevant. I won't solve the current issues, but it will help having more in the future.
ciao, Anto _______________________________________________ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
_______________________________________________ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
On Tue, Jul 12, 2011 at 10:53 +0200, Miquel Torres wrote:
They *are* being merged. The question here is to have branches *now*, as we can't know how long it will take for speed.python.org to be online.
yes, makes sense. Better invest your time in getting branch benchmarking to work right now, it's probably the best support for pypy development ATM.
All things considered, I think the way to go would be to keep current speed.pypy.org hosting if at all possible until PyPy moves to speed.python.org. If that can be done, we can upgrade the current setup in no time.
This is fine by me - i am still paying for the underlying machine and so it's not fine indefinitely. OTOH people could just go to our pypy.org donation page and i could ask for some hosting costs at some point (*). best, holger (*) then again, who knows what the current global currencies will be worth in a few years or months, anyway. Better put money to good use ;)
2011/7/12 holger krekel <holger@merlinux.eu>:
On Tue, Jul 12, 2011 at 09:19 +0200, Maciej Fijalkowski wrote:
On Tue, Jul 12, 2011 at 9:16 AM, Miquel Torres <tobami@googlemail.com> wrote:
Branches are already implemented. speed.pypy.org just needs to be upgraded to Codespeed 0.8.x and its data migrated.
I can help you with that, but that means having local changes from speed.pypy.org commited somewhere else.
That has not yet been done because we wanted to move to ep.io, which has not yet happend, and we are working on speed.python.org and somehow things have stalled.
speed.python.org is completely irrelevant here.
Probably i am missing something but couldn't speed.pypy.org and speed.python.org mostly merge? (maybe except for the front page and default comparisons/settings but those could be easily hosted wherever)
best, holger
Migrating current speed.pypy.org (I mean keeping it on the current server but upgrading) would not take me very long. But it is on the way out...
We either should do the ep.io move (data has been imported up until very recently) or upgrade now.
2011/7/12 Antonio Cuni <anto.cuni@gmail.com>:
On 12/07/11 09:01, Maciej Fijalkowski wrote:
I'll follow up on the branches, but the issue is a bit irrelevant - we still have performance regressions by trunk checkins as well.
it's not irrelevant. I won't solve the current issues, but it will help having more in the future.
ciao, Anto _______________________________________________ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
_______________________________________________ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Hi, On Tue, Jul 12, 2011 at 1:20 AM, Maciej Fijalkowski <fijall@gmail.com> wrote:
http://speed.pypy.org/timeline/?exe=1&base=none&ben=spitfire&env=tannit&revs=50
This was introduced by the changes we did to the GC to better support resizing big lists: 1bb155fd266f and 324a8265e420. I will look. A bientôt, Armin.
I just investigated the spitfire regression, it appears to have been caused by 27df060341f0 (merg non-null-app-dict branch) Alex On Tue, Jul 12, 2011 at 11:31 AM, Armin Rigo <arigo@tunes.org> wrote:
Hi,
On Tue, Jul 12, 2011 at 1:20 AM, Maciej Fijalkowski <fijall@gmail.com> wrote:
http://speed.pypy.org/timeline/?exe=1&base=none&ben=spitfire&env=tannit&revs=50
This was introduced by the changes we did to the GC to better support resizing big lists: 1bb155fd266f and 324a8265e420. I will look.
A bientôt,
Armin. _______________________________________________ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
-- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero
Hi Alex, On Tue, Jul 12, 2011 at 8:36 PM, Alex Gaynor <alex.gaynor@gmail.com> wrote:
I just investigated the spitfire regression, it appears to have been caused by 27df060341f0 (merg non-null-app-dict branch)
That's not the conclusion I arrived at. The speed.pypy.org website says that revision 27df060341f0 was still fast, and I have on tannit64 a pypy-c-jit from revision 76b06820d08b which is slow. The interval contains the two changes I described in the previous e-mail, plus a few other details that are really unlikely to be the cause, but not the merge of any branch. Note that by "interval" I mean "all changes in one revision and not in the other". <ad>I'm using arigo/hack/hg/hglog here, which builds a command line like "hg log -r 'reverse(ancestors(%s) and not ancestors(%s))'".</ad> A bientôt, Armin.
speed.pypy.org shows 27df (45168) as being fast, but 058e (45254) as being slow, I narrowed it down to 45168:45205, ad there's only one reasonable commit in that range. http://paste.pocoo.org/show/437119/ are my benchmark runs (64-bit, local laptop), and the hg log which seems to correspond: http://paste.pocoo.org/show/437121/ Alex On Tue, Jul 12, 2011 at 11:45 AM, Armin Rigo <arigo@tunes.org> wrote:
Hi Alex,
On Tue, Jul 12, 2011 at 8:36 PM, Alex Gaynor <alex.gaynor@gmail.com> wrote:
I just investigated the spitfire regression, it appears to have been caused by 27df060341f0 (merg non-null-app-dict branch)
That's not the conclusion I arrived at. The speed.pypy.org website says that revision 27df060341f0 was still fast, and I have on tannit64 a pypy-c-jit from revision 76b06820d08b which is slow. The interval contains the two changes I described in the previous e-mail, plus a few other details that are really unlikely to be the cause, but not the merge of any branch.
Note that by "interval" I mean "all changes in one revision and not in the other". <ad>I'm using arigo/hack/hg/hglog here, which builds a command line like "hg log -r 'reverse(ancestors(%s) and not ancestors(%s))'".</ad>
A bientôt,
Armin.
-- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero
Hi Alex, On Tue, Jul 12, 2011 at 8:50 PM, Alex Gaynor <alex.gaynor@gmail.com> wrote:
speed.pypy.org shows 27df (45168) as being fast, but 058e (45254) as being slow, I narrowed it down to 45168:45205, ad there's only one reasonable commit in that range.
The trick is that the two revisions I identify as culprit are (on tannit) revision numbers r45155 and r45156, i.e. sequentially before r45168 (which has on my repo on tannit the number r45170). We have this structure, all on the "default" branch: r45176 | r45174 / \ r45156 \ | r45170 r45155 / \ / r45154 This means that you'll miss the two revisions r45155 and r45156 if you do 45168:45205. In other words it's always subtly wrong to work with plain intervals of revision numbers in hg... The speed.pypy.org measured 45170 to be still fast, but I measured all of 45176, 45156 and 45155 to be slow. This is why I claim that the culprit is r45155. A bientôt, Armin.
On Tue, Jul 12, 2011 at 12:06 PM, Armin Rigo <arigo@tunes.org> wrote:
Hi Alex,
On Tue, Jul 12, 2011 at 8:50 PM, Alex Gaynor <alex.gaynor@gmail.com> wrote:
speed.pypy.org shows 27df (45168) as being fast, but 058e (45254) as being slow, I narrowed it down to 45168:45205, ad there's only one reasonable commit in that range.
The trick is that the two revisions I identify as culprit are (on tannit) revision numbers r45155 and r45156, i.e. sequentially before r45168 (which has on my repo on tannit the number r45170). We have this structure, all on the "default" branch:
r45176 | r45174 / \ r45156 \ | r45170 r45155 / \ / r45154
This means that you'll miss the two revisions r45155 and r45156 if you do 45168:45205. In other words it's always subtly wrong to work with plain intervals of revision numbers in hg... The speed.pypy.org measured 45170 to be still fast, but I measured all of 45176, 45156 and 45155 to be slow. This is why I claim that the culprit is r45155.
A bientôt,
Armin.
Now I'm really confused, what sha does that correspond to? https://bitbucket.org/pypy/pypy/changeset/45155 seems to indicate that it is a rather boring commit on a branch? Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero
Ahhhh, I found why i was getting nonsense information, `hg log -r<rev1> -r<rev2>` does NOT do what I wanted, you need to do `hg log -r<rev1>..<rev2>`! Alex On Tue, Jul 12, 2011 at 12:10 PM, Alex Gaynor <alex.gaynor@gmail.com> wrote:
On Tue, Jul 12, 2011 at 12:06 PM, Armin Rigo <arigo@tunes.org> wrote:
Hi Alex,
On Tue, Jul 12, 2011 at 8:50 PM, Alex Gaynor <alex.gaynor@gmail.com> wrote:
speed.pypy.org shows 27df (45168) as being fast, but 058e (45254) as being slow, I narrowed it down to 45168:45205, ad there's only one reasonable commit in that range.
The trick is that the two revisions I identify as culprit are (on tannit) revision numbers r45155 and r45156, i.e. sequentially before r45168 (which has on my repo on tannit the number r45170). We have this structure, all on the "default" branch:
r45176 | r45174 / \ r45156 \ | r45170 r45155 / \ / r45154
This means that you'll miss the two revisions r45155 and r45156 if you do 45168:45205. In other words it's always subtly wrong to work with plain intervals of revision numbers in hg... The speed.pypy.org measured 45170 to be still fast, but I measured all of 45176, 45156 and 45155 to be slow. This is why I claim that the culprit is r45155.
A bientôt,
Armin.
Now I'm really confused, what sha does that correspond to? https://bitbucket.org/pypy/pypy/changeset/45155 seems to indicate that it is a rather boring commit on a branch?
Alex
-- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero
-- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero
I *think* they are the same, but I really have no idea. Alex On Tue, Jul 12, 2011 at 3:03 PM, Antonio Cuni <anto.cuni@gmail.com> wrote:
On 12/07/11 21:28, Alex Gaynor wrote:
Ahhhh, I found why i was getting nonsense information, `hg log -r<rev1> -r<rev2>` does NOT do what I wanted, you need to do `hg log -r<rev1>..<rev2>`!
uhm, I usually do "hg log -rA:B". Is it the same as -rA..B or is again subtly different?
-- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero
On Jul 12, 2011, at 3:03 PM, Antonio Cuni wrote:
On 12/07/11 21:28, Alex Gaynor wrote:
Ahhhh, I found why i was getting nonsense information, `hg log -r<rev1> -r<rev2>` does NOT do what I wanted, you need to do `hg log -r<rev1>..<rev2>`!
uhm, I usually do "hg log -rA:B". Is it the same as -rA..B or is again subtly different?
Subtly different, hg help revset -- Philip Jenvey
I don't know why I'm surprised. Alex On Tue, Jul 12, 2011 at 3:43 PM, Philip Jenvey <pjenvey@underboss.org>wrote:
On Jul 12, 2011, at 3:03 PM, Antonio Cuni wrote:
On 12/07/11 21:28, Alex Gaynor wrote:
Ahhhh, I found why i was getting nonsense information, `hg log -r<rev1> -r<rev2>` does NOT do what I wanted, you need to do `hg log -r<rev1>..<rev2>`!
uhm, I usually do "hg log -rA:B". Is it the same as -rA..B or is again subtly different?
Subtly different, hg help revset
-- Philip Jenvey
-- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero
Hi, On Wed, Jul 13, 2011 at 12:03 AM, Antonio Cuni <anto.cuni@gmail.com> wrote:
uhm, I usually do "hg log -rA:B". Is it the same as -rA..B or is again subtly different?
...and both are different from what I find to be the *really* useful information, which is: "which checkins are contained in B but not in A"? For that you have to use -r 'ancestors(B) and not ancestors(A)', or -r 'reverse(same as before)' if you prefer the most-recent-first order. A bientôt, Armin.
On Wed, Jul 13, 2011 at 15:42, Armin Rigo <arigo@tunes.org> wrote:
...and both are different from what I find to be the *really* useful information, which is: "which checkins are contained in B but not in A"? For that you have to use -r 'ancestors(B) and not ancestors(A)',
Not sure to understand, with Git you know easily new commits since last fetch : git log origin/master..master Same command as : git log master ^origin/master (a and not b). I think it's the same thing with hg (minus inclusion/exclusion limits). I'm wrong? -- Sebastien Douche <sdouche@gmail.com> Twitter : @sdouche
On 12/07/11 01:20, Maciej Fijalkowski wrote:
Hi
I'm a bit worried with our current benchmarks state. We have around 4 benchmarks that had reasonable slowdowns recently [cut]
Ok, let's try to make a summary of what we discovered about benchmark regressions.
Current list:
http://speed.pypy.org/timeline/?exe=1&base=none&ben=spectral-norm&env=tannit&revs=50
this is weird. Maybe I did something wrong (like picking the wrong revisions to test), but I cannot reproduce the slowdown. This is on my machine, 32-bit chroot: $ ./pypy-c-jit-45360-867e8ffff7a8-linux/bin/pypy ~/pypy/benchmarks/own/spectral-norm.py --take_geo_mean 0.0256132620823 $ ./pypy-c-jit-45373-582113929b62-linux/bin/pypy ~/pypy/benchmarks/own/spectral-norm.py --take_geo_mean 0.0256120378632 $ ./pypy-c-jit-45412-3476b9be3cec-linux/bin/pypy ~/pypy/benchmarks/own/spectral-norm.py --take_geo_mean 0.025725552797 on tannit, I get similar results (a bit faster than on my machine, but all the same).
http://speed.pypy.org/timeline/?exe=1&base=none&ben=spitfire&env=tannit&revs=50
I think that armin investigated this, and the outcome was that it's because of the changes we did in the GC during the sprint. Armin, do you confirm? Do we have a solution?
This is a good example why we should not work the way we work now:
http://speed.pypy.org/timeline/?exe=1&base=none&ben=slowspitfire&env=tannit&revs=200
There was an issue, then the issue was fixed, but apparently not quite (7th of June is quite a bit slower than 25th of May) and then recently we introduced something that make it faster alltogether. Can we even fish the original issue?
did anybody look at this?
http://speed.pypy.org/timeline/?exe=1&base=none&ben=bm_mako&env=tannit&revs=200
I investigated this. The culprit is 5b62f71347c8, in particular the change to policy.py which enables tracing inside lltypesystem.rbuilder. With this patch, mako is fast again: http://paste.pocoo.org/show/439257/ The weird thing is that in jit-summary there is no obvious difference between the runs, apart the total time: http://paste.pocoo.org/show/439263/ Alex, do you feel like investigating more?
http://speed.pypy.org/timeline/?exe=1&base=none&ben=nbody_modified&env=tannit&revs=50 (is it relevant or just noise?)
http://speed.pypy.org/timeline/?exe=1&base=none&ben=telco&env=tannit&revs=50
I did not look into these two. Anybody? ciao, Anto
Hi, On Fri, Jul 15, 2011 at 10:45 AM, Antonio Cuni <anto.cuni@gmail.com> wrote:
I think that armin investigated this, and the outcome was that it's because of the changes we did in the GC during the sprint. Armin, do you confirm? Do we have a solution?
I confirm, and still have no solution. I tried to devise a solution (see PYPY_GC_LOSTCARD), which actually helps a bit on this and at least one other benchmark, but this particular benchmark is still much slower than two weeks ago. A bientôt, Armin.
On Fri, Jul 15, 2011 at 4:44 AM, Armin Rigo <arigo@tunes.org> wrote:
Hi,
On Fri, Jul 15, 2011 at 10:45 AM, Antonio Cuni <anto.cuni@gmail.com> wrote:
I think that armin investigated this, and the outcome was that it's because of the changes we did in the GC during the sprint. Armin, do you confirm? Do we have a solution?
I confirm, and still have no solution. I tried to devise a solution (see PYPY_GC_LOSTCARD), which actually helps a bit on this and at least one other benchmark, but this particular benchmark is still much slower than two weeks ago.
A bientôt,
Armin. _______________________________________________ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
I can own the mako one, ATM I have a bit no clue though. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero
Hi all, On Fri, Jul 15, 2011 at 1:44 PM, Armin Rigo <arigo@tunes.org> wrote:
On Fri, Jul 15, 2011 at 10:45 AM, Antonio Cuni <anto.cuni@gmail.com> wrote:
I think that armin investigated this, and the outcome was that it's because of the changes we did in the GC during the sprint. Armin, do you confirm? Do we have a solution?
I confirm, and still have no solution. I tried to devise a solution (see PYPY_GC_LOSTCARD), which actually helps a bit on this and at least one other benchmark, but this particular benchmark is still much slower than two weeks ago.
I found and fixed the bug in 82e5051d55c3 (and, to a less major extend, in 6b4eb34c6091). I also did 4c0d2555caa8: generate from the jit backend assembler code to set the GC card bit inline, using just 4-5 instructions, instead of in the write barrier call. Finally I reverted PYPY_GC_LOSTCARD because it sounds like a hack and isn't really compatible with 4c0d2555caa8. All in all it gives good results. I can also confirm that both chaos in 3f7e and go in e911 are randomly slowed down by jit tracer improvements, and would become fast again with a smaller --jit trace_limit... A bientôt, Armin.
On Sun, Jul 24, 2011 at 10:12 AM, Armin Rigo <arigo@tunes.org> wrote:
Hi all,
On Fri, Jul 15, 2011 at 1:44 PM, Armin Rigo <arigo@tunes.org> wrote:
On Fri, Jul 15, 2011 at 10:45 AM, Antonio Cuni <anto.cuni@gmail.com> wrote:
I think that armin investigated this, and the outcome was that it's because of the changes we did in the GC during the sprint. Armin, do you confirm? Do we have a solution?
I confirm, and still have no solution. I tried to devise a solution (see PYPY_GC_LOSTCARD), which actually helps a bit on this and at least one other benchmark, but this particular benchmark is still much slower than two weeks ago.
I found and fixed the bug in 82e5051d55c3 (and, to a less major extend, in 6b4eb34c6091). I also did 4c0d2555caa8: generate from the jit backend assembler code to set the GC card bit inline, using just 4-5 instructions, instead of in the write barrier call. Finally I reverted PYPY_GC_LOSTCARD because it sounds like a hack and isn't really compatible with 4c0d2555caa8. All in all it gives good results.
I can also confirm that both chaos in 3f7e and go in e911 are randomly slowed down by jit tracer improvements, and would become fast again with a smaller --jit trace_limit...
That probably should be addressed (maybe?) so can we decide we're done with regressions? Cheers, fijal
On Tue, Jul 12, 2011 at 1:20 AM, Maciej Fijalkowski <fijall@gmail.com> wrote:
Hi
I'm a bit worried with our current benchmarks state. We have around 4 benchmarks that had reasonable slowdowns recently and we keep putting new features that speed up other things. How can we even say we have actually fixed the original issue? Can we have a policy of not merging new performance features before having a story why benchmarks got slower?
Current list:
http://speed.pypy.org/timeline/?exe=1&base=none&ben=spectral-norm&env=tannit&revs=50
this fixed itself, recent runs are fast again (and anto could not reproduce at all)
http://speed.pypy.org/timeline/?exe=1&base=none&ben=spitfire&env=tannit&revs=50
armin will have a look one day
This is a good example why we should not work the way we work now:
http://speed.pypy.org/timeline/?exe=1&base=none&ben=slowspitfire&env=tannit&revs=200
do we say "meh" and go on?
There was an issue, then the issue was fixed, but apparently not quite (7th of June is quite a bit slower than 25th of May) and then recently we introduced something that make it faster alltogether. Can we even fish the original issue?
http://speed.pypy.org/timeline/?exe=1&base=none&ben=bm_mako&env=tannit&revs=200
no clue so far?
http://speed.pypy.org/timeline/?exe=1&base=none&ben=nbody_modified&env=tannit&revs=50 (is it relevant or just noise?)
just noise maybe?
http://speed.pypy.org/timeline/?exe=1&base=none&ben=telco&env=tannit&revs=50
nobody looked
Cheers, fijal
On Sat, Jul 16, 2011 at 1:44 PM, Maciej Fijalkowski <fijall@gmail.com>wrote:
On Tue, Jul 12, 2011 at 1:20 AM, Maciej Fijalkowski <fijall@gmail.com> wrote:
Hi
I'm a bit worried with our current benchmarks state. We have around 4 benchmarks that had reasonable slowdowns recently and we keep putting new features that speed up other things. How can we even say we have actually fixed the original issue? Can we have a policy of not merging new performance features before having a story why benchmarks got slower?
Current list:
http://speed.pypy.org/timeline/?exe=1&base=none&ben=spectral-norm&env=tannit&revs=50
this fixed itself, recent runs are fast again (and anto could not reproduce at all)
http://speed.pypy.org/timeline/?exe=1&base=none&ben=spitfire&env=tannit&revs=50
armin will have a look one day
This is a good example why we should not work the way we work now:
http://speed.pypy.org/timeline/?exe=1&base=none&ben=slowspitfire&env=tannit&revs=200
do we say "meh" and go on?
There was an issue, then the issue was fixed, but apparently not quite (7th of June is quite a bit slower than 25th of May) and then recently we introduced something that make it faster alltogether. Can we even fish the original issue?
http://speed.pypy.org/timeline/?exe=1&base=none&ben=bm_mako&env=tannit&revs=200
no clue so far?
No clue indeed, traces don't change at all, the only change is inlining something that is never called in that code. If that causes a slowdown I claim we are broken elsewhere, and this is a symptom, not a new problem.
http://speed.pypy.org/timeline/?exe=1&base=none&ben=nbody_modified&env=tannit&revs=50
(is it relevant or just noise?)
just noise maybe?
http://speed.pypy.org/timeline/?exe=1&base=none&ben=telco&env=tannit&revs=50
nobody looked
Cheers, fijal
_______________________________________________ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero
On 07/16/2011 10:44 PM Maciej Fijalkowski wrote:
On Tue, Jul 12, 2011 at 1:20 AM, Maciej Fijalkowski<fijall@gmail.com> wrote:
Hi
I'm a bit worried with our current benchmarks state. We have around 4 benchmarks that had reasonable slowdowns recently and we keep putting new features that speed up other things. How can we even say we have actually fixed the original issue? Can we have a policy of not merging new performance features before having a story why benchmarks got slower?
Current list:
http://speed.pypy.org/timeline/?exe=1&base=none&ben=spectral-norm&env=tannit&revs=50
this fixed itself, recent runs are fast again (and anto could not reproduce at all)
I am wondering what was sharing the tannit xeon with the the actual code being benchmarked. Are timing values stored via stdout to file(s)? How are they buffered? Could the process of allocating file storage space on a disk have reached a nonlinear overhead hump that was passed, and somehow impacted the benchmark temporarily, so it would seem to have "fixed itself?" I am thinking file system work possibly e.g. clobbering warm caches in some temporarily systematic way due to some convoying[1] between benchmark i/o and background file system stuff. Even though user time is not system time, some CPU and cache resources must be shared. Wonder what a dedicated SSD disk with large pre-allocated open files could do to normalize effects, if the disk were reformatted anew for each run? [1] http://en.wikipedia.org/wiki/Lock_convoy Regards, Bengt Richter
I think to summarize we're good now, except spitfire which is to be investigated by armin. Then new thing about go is a bit "we touched the world". Because the unoptimized traces are now shorter, less gets aborted, less gets run based on functions and it's slower. saying trace_limit=6000 makes it fast again. I guess "too bad" is the answer.
On 17/07/11 22:15, Maciej Fijalkowski wrote:
I think to summarize we're good now, except spitfire which is to be investigated by armin.
Then new thing about go is a bit "we touched the world". Because the unoptimized traces are now shorter, less gets aborted, less gets run based on functions and it's slower. saying trace_limit=6000 makes it fast again. I guess "too bad" is the answer.
This made me wonder how effective is our "compile the loops" idea in practice, so I benchmarked translate.py first with the default settings, and then with just the function jitting: pypy ./translate.py -Ojit [Timer] Timings: [Timer] annotate --- 453.0 s [Timer] rtype_lltype --- 310.7 s [Timer] pyjitpl_lltype --- 392.6 s [Timer] backendopt_lltype --- 132.8 s [Timer] stackcheckinsertion_lltype --- 32.1 s [Timer] database_c --- 197.1 s [Timer] source_c --- 252.0 s [Timer] compile_c --- 707.6 s [Timer] =========================================== [Timer] Total: --- 2478.0 s pypy --jit threshold=-1 ./translate -Ojit [Timer] Timings: [Timer] annotate --- 486.2 s [Timer] rtype_lltype --- 297.8 s [Timer] pyjitpl_lltype --- 396.6 s [Timer] backendopt_lltype --- 128.7 s [Timer] stackcheckinsertion_lltype --- 32.6 s [Timer] database_c --- 190.0 s [Timer] source_c --- 240.0 s [Timer] compile_c --- 594.6 s [Timer] =========================================== [Timer] Total: --- 2366.5 s As you can see, if we ignore the time spent in compile_c, the total time is almost the same. What can we conclude? That "compiling the loops" is uneffective and we only care about compiling single functions? :-( ciao, Anto
Hi Anto, On Mon, Jul 18, 2011 at 12:28 PM, Antonio Cuni <anto.cuni@gmail.com> wrote:
What can we conclude? That "compiling the loops" is uneffective and we only care about compiling single functions? :-(
Or, conversely, that compiling single functions is ineffective and we only care about compiling the loops? No. I expect that on a large and messy program like translate.py, after a while, either approach should be fine. Still, there are cases where one or the other approach is better. If you want an obvious example where compiling loops is better, write a function that runs a loop a large number of times, but is itself called only a few times. A bientôt, Armin.
On 07/18/2011 01:58 PM, Armin Rigo wrote:
Hi Anto,
On Mon, Jul 18, 2011 at 12:28 PM, Antonio Cuni<anto.cuni@gmail.com> wrote:
What can we conclude? That "compiling the loops" is uneffective and we only care about compiling single functions? :-(
Or, conversely, that compiling single functions is ineffective and we only care about compiling the loops? No.
I expect that on a large and messy program like translate.py,
offtopic, but I still want to point out that translate is indeed terribly messy. I've been reading traces some more, and it's quite scary. E.g. we have lltype._struct.__init__ has a CALL_FUNCTION bytecode that needs more than 800 traces operations, even after optimization :-). Carl Friedrich
On 18/07/11 14:10, Carl Friedrich Bolz wrote:
offtopic, but I still want to point out that translate is indeed terribly messy. I've been reading traces some more, and it's quite scary. E.g. we have lltype._struct.__init__ has a CALL_FUNCTION bytecode that needs more than 800 traces operations, even after optimization :-).
yes, I agree that translate.py is messy :-). Also, we have a speedup of ~2-2.5x which is more or less what you would expect by "just" removing the interpretation overhead. It probably indicates that we have laaaarge room for improvements, but I suppose that we already knew that :-)
Hi Anto, On Mon, Jul 18, 2011 at 2:34 PM, Antonio Cuni <anto.cuni@gmail.com> wrote:
Also, we have a speedup of ~2-2.5x which is more or less what you would expect by "just" removing the interpretation overhead. It probably indicates that we have laaaarge room for improvements, but I suppose that we already knew that :-)
No, I would argue that it's already 2x better than "just" removing the interpretation overhead, because PyPy-no-jit is up to 2x slower than CPython to start with :-) Armin
On Mon, Jul 18, 2011 at 2:34 PM, Antonio Cuni <anto.cuni@gmail.com> wrote:
On 18/07/11 14:10, Carl Friedrich Bolz wrote:
offtopic, but I still want to point out that translate is indeed terribly messy. I've been reading traces some more, and it's quite scary. E.g. we have lltype._struct.__init__ has a CALL_FUNCTION bytecode that needs more than 800 traces operations, even after optimization :-).
yes, I agree that translate.py is messy :-).
Also, we have a speedup of ~2-2.5x which is more or less what you would expect by "just" removing the interpretation overhead. It probably indicates that we have laaaarge room for improvements, but I suppose that we already knew that :-)
[citation needed] I think 2x comes from 1999 or so. It must be benchmarked again to prove that. I think there are various clues that this number has changed (even maybe dropped)
On 18/07/11 13:58, Armin Rigo wrote:
Or, conversely, that compiling single functions is ineffective and we only care about compiling the loops? No.
I expect that on a large and messy program like translate.py, after a while, either approach should be fine. Still, there are cases where one or the other approach is better. If you want an obvious example where compiling loops is better, write a function that runs a loop a large number of times, but is itself called only a few times.
yes, but this was an answer to fijal's comment that "go" is now slower because the trace_limit is too high and so we compile more loops and less functions than before. I know that you can easily write examples in which one approach is much better than the other, but it's also true that usually in complex programs we are less effective than in small ones. For example, translate.py does not use much recursion, and I would expect "loop-based" compilation to be more effective in such a program. But maybe my expectation is naive, I don't know :-). ciao, Anto
On Mon, Jul 18, 2011 at 1:58 PM, Armin Rigo <arigo@tunes.org> wrote:
Hi Anto,
On Mon, Jul 18, 2011 at 12:28 PM, Antonio Cuni <anto.cuni@gmail.com> wrote:
What can we conclude? That "compiling the loops" is uneffective and we only care about compiling single functions? :-(
Or, conversely, that compiling single functions is ineffective and we only care about compiling the loops? No.
I expect that on a large and messy program like translate.py, after a while, either approach should be fine. Still, there are cases where one or the other approach is better. If you want an obvious example where compiling loops is better, write a function that runs a loop a large number of times, but is itself called only a few times.
A bientôt,
Armin.
Or hakan's video editing code ;-) What's although worth considering is how to get stuff optimized even if we don't have loops (but I guess carl has already started)
Hi Maciek, On Mon, Jul 18, 2011 at 9:27 PM, Maciej Fijalkowski <fijall@gmail.com> wrote:
What's although worth considering is how to get stuff optimized even if we don't have loops (but I guess carl has already started)
I'm unsure what you mean here. The function_threshold stuff you did is exactly that, no? A bientôt, Armin.
On Sat, Jul 23, 2011 at 11:05 AM, Armin Rigo <arigo@tunes.org> wrote:
Hi Maciek,
On Mon, Jul 18, 2011 at 9:27 PM, Maciej Fijalkowski <fijall@gmail.com> wrote:
What's although worth considering is how to get stuff optimized even if we don't have loops (but I guess carl has already started)
I'm unsure what you mean here. The function_threshold stuff you did is exactly that, no?
Hey. No, I meant something else. Can you show up on IRC to discuss? :)
A bientôt,
Armin.
On Sat, Jul 23, 2011 at 11:05 AM, Armin Rigo <arigo@tunes.org> wrote:
Hi Maciek,
On Mon, Jul 18, 2011 at 9:27 PM, Maciej Fijalkowski <fijall@gmail.com> wrote:
What's although worth considering is how to get stuff optimized even if we don't have loops (but I guess carl has already started)
I'm unsure what you mean here. The function_threshold stuff you did is exactly that, no?
A verbose answer - the function_threshold is about it, but also the optimization level is much lower if we can't do loop invariant code motion. One example is global lookups, where carl (or his student) is working on eliminating guards even if we don't do LICM. This is what I meant by "optimizing" a no-loop scenario.
A bientôt,
Armin.
Hi, On Sat, Jul 23, 2011 at 12:39 PM, Maciej Fijalkowski <fijall@gmail.com> wrote:
A verbose answer - the function_threshold is about it, but also the optimization level is much lower if we can't do loop invariant code motion. One example is global lookups, where carl (or his student) is working on eliminating guards even if we don't do LICM. This is what I meant by "optimizing" a no-loop scenario.
Advanced. :-) Armin
participants (11)
-
Alex Gaynor -
Antonio Cuni -
Armin Rigo -
Bengt Richter -
Carl Friedrich Bolz -
Dan Roberts -
holger krekel -
Maciej Fijalkowski -
Miquel Torres -
Philip Jenvey -
Sebastien Douche