Re: [pypy-dev] Change to the frontpage of speed.pypy.org

In a message of Tue, 08 Mar 2011 20:20:06 +0100, Miquel Torres writes:
Ok, I just committed the changes.
They address two general cases: - You want to know how fast PyPy is *now* compared to CPython in different benchmark scenarios, or tasks. - You want to know how PyPy has been *improving* overall over the last re le= ases
That is now answered on the front page, and the reports are now much less prominent (I didn't change the logic because it is something I want to do properly, not just as a hack for speed.pypy). - I have not yet addressed the "smaller is better" point.
I am aware that the wording of the "faster on average" needs to be improved (I am discussing it with Holger even now ;). Please chime in so that we can have a good paragraph that is informative and short enough while at the same time not being misleading.
Miquel
The graphic is lovely. you have a sÃpelling error s/taks/task/. Many of us are at PyCon now, so working on wording may not be something we have time for now. I am not sure that the geometric mean of all benchmarks give you anything meaningful, so I would have avoided saying anything like that. More specifically, I think that there is a division between some highly mathematical programs, where you might get a speedup of 20 to 30 times CPython, and the benchmarks whcih I find much more meaningful, those that represent actualy python programs -- where I think we are typically only between 2 and 3 times faster. The only reason to have some of the benchmarks is because they are well known. So people expect them. But doing very well on them is not actually all that significant -- it would be easy to write something that is great and running these contrived, synthetic benchmarks, but really lousy at running real python code. And I don't think that you can use the geometric mean to prove a thing with this. So I think talking about it makes us look bad -- we are making claims based on either bad science, or pseudo-science. And I want the 'lestest results' stuff gone from the front page. It's as misleading as ever. And I have done a poll of developers. it's not just me. Nobody finds it valuable. Because things stay red forever, we all ignore it all the time and go directly to the raw results of runs, which is what we are interested in. This also tells us of improvements, which we are also interested in, because unexpected improvements often mean something is very wrong. The whole thing has the same problem as those popup windows 'do you really want to delete that file? confirm y/n'. You get used to typing y. Then you do it when you meant not to save the file. The red pages get ignored for precisely the same reason. We're all used to all the red, which generally refer to problems which were already fixed, and thus are things that we _should_ ignore. So we end up ignoring it all, all of the time. So it therefore doesn't serve its purpose of reminding developers of anything. The general public, however, reads the thing and thinks -- latest results, pypy has just gone to hell, look at all those slowdowns. Explaining that they aren't current, or latest results, but instead 'sometime in the past when we were bad once' is getting irritating. Can you please move it someplace else so I don't have to have this conversation with pycon attendees any more? Later, when pycon is over, we can discuss and work out a better design for informing developers that a given build may have broken things. This way is not working. Laura

And I want the 'lestest results' stuff gone from the front page. It's as misleading as ever. And I have done a poll of developers. it's not just me. Nobody finds it valuable. Because things stay red forever, we all ignore it all the time and go directly to the raw results of runs, which is what we are interested in. This also tells us of improvements, which we are also interested in, because unexpected improvements often mean something is very wrong. Ok, that is pretty clear.
Explaining that they aren't current, or latest results, but instead 'sometime in the past when we were bad once' is getting irritating. Can you please move it someplace else so I don't have to have this conversation with pycon attendees any more? Sorry about that. I have removed it from the site.
Later, when pycon is over, we can discuss and work out a better design for informing developers that a given build may have broken things. This way is not working.
Yes, we can figure out a better approach after PyCon.
And I don't think that you can use the geometric mean to prove a thing with this. So I think talking about it makes us look bad -- we are making claims based on either bad science, or pseudo-science. I agree that it wasn't the best explanation. I have removed most text, so that it doesn't explicitly say that it means PyPy *is* X times faster. It now just states that the geometric mean is X times faster. If it is still too much, we can remove all text. But then some people won't understand the numbers.
We can also remove the second plot. The mean of a set of benchmarks that does not represent a balanced real-world task mix is of course not very good. But then again the world is complicated. And, as a normal Python developer, I find the second plot extremely interesting, because it gives me a ballpark idea of where the PyPy project is heading to. Before I decide to use PyPy in production instead of CPython, I will do my own testing for *my* application, but I assure you that not having a ballpark average won't be a plus in considering to invest the time to test and adopt PyPy. But that is my opinion of course ;-) Have fun at PyCon! Miquel 2011/3/9 Laura Creighton <lac@openend.se>:
In a message of Tue, 08 Mar 2011 20:20:06 +0100, Miquel Torres writes:
Ok, I just committed the changes.
They address two general cases: - You want to know how fast PyPy is *now* compared to CPython in different benchmark scenarios, or tasks. - You want to know how PyPy has been *improving* overall over the last re le= ases
That is now answered on the front page, and the reports are now much less prominent (I didn't change the logic because it is something I want to do properly, not just as a hack for speed.pypy). - I have not yet addressed the "smaller is better" point.
I am aware that the wording of the "faster on average" needs to be improved (I am discussing it with Holger even now ;). Please chime in so that we can have a good paragraph that is informative and short enough while at the same time not being misleading.
Miquel
The graphic is lovely.
you have a sÃpelling error s/taks/task/.
Many of us are at PyCon now, so working on wording may not be something we have time for now. I am not sure that the geometric mean of all benchmarks give you anything meaningful, so I would have avoided saying anything like that. More specifically, I think that there is a division between some highly mathematical programs, where you might get a speedup of 20 to 30 times CPython, and the benchmarks whcih I find much more meaningful, those that represent actualy python programs -- where I think we are typically only between 2 and 3 times faster.
The only reason to have some of the benchmarks is because they are well known. So people expect them. But doing very well on them is not actually all that significant -- it would be easy to write something that is great and running these contrived, synthetic benchmarks, but really lousy at running real python code.
And I don't think that you can use the geometric mean to prove a thing with this. So I think talking about it makes us look bad -- we are making claims based on either bad science, or pseudo-science.
And I want the 'lestest results' stuff gone from the front page. It's as misleading as ever. And I have done a poll of developers. it's not just me. Nobody finds it valuable. Because things stay red forever, we all ignore it all the time and go directly to the raw results of runs, which is what we are interested in. This also tells us of improvements, which we are also interested in, because unexpected improvements often mean something is very wrong.
The whole thing has the same problem as those popup windows 'do you really want to delete that file? confirm y/n'. You get used to typing y. Then you do it when you meant not to save the file. The red pages get ignored for precisely the same reason. We're all used to all the red, which generally refer to problems which were already fixed, and thus are things that we _should_ ignore. So we end up ignoring it all, all of the time. So it therefore doesn't serve its purpose of reminding developers of anything.
The general public, however, reads the thing and thinks -- latest results, pypy has just gone to hell, look at all those slowdowns. Explaining that they aren't current, or latest results, but instead 'sometime in the past when we were bad once' is getting irritating. Can you please move it someplace else so I don't have to have this conversation with pycon attendees any more?
Later, when pycon is over, we can discuss and work out a better design for informing developers that a given build may have broken things. This way is not working.
Laura

Hey. Awesome job Miquel, I love it. This is exactly what we wanted

Cool! ;-) PS: Currently, only 20 benchmarks are shown (and averaged!) because older pypy revisions (mostly PyPy 1.3) don't have numbers for the newest benchmarks. If Maciej reruns those revisions the rest will be displayed ;-) 2011/3/9 Maciej Fijalkowski <fijall@gmail.com>:
Hey.
Awesome job Miquel, I love it. This is exactly what we wanted

I really, really like the new display! And it motivated me to dig into the data ... which is a great result on its own. The first question for myself was "hey, why is it slow on slowspitfire, and, btw, what is slowspitfire? Could that be something that my application does, too?" But I was unable to find out what slowspitfire is doing ... I found spitfire, which does some HTML templating stuff, and deducted, that slowspitfire will do some slow HTML templating stuff. Where did I click wrong? Is there a path down to the slowspitfire.py file or an explanation what slowspitfire is doing? Harald -- GHUM GmbH Harald Armin Massa Spielberger Straße 49 70435 Stuttgart 0173/9409607 Amtsgericht Stuttgart, HRB 734971 - persuadere. et programmare

On Wed, Mar 9, 2011 at 8:19 AM, Massa, Harald Armin <chef@ghum.de> wrote:
I really, really like the new display!
And it motivated me to dig into the data ... which is a great result on its own.
The first question for myself was "hey, why is it slow on slowspitfire, and, btw, what is slowspitfire? Could that be something that my application does, too?"
But I was unable to find out what slowspitfire is doing ... I found spitfire, which does some HTML templating stuff, and deducted, that slowspitfire will do some slow HTML templating stuff. Where did I click wrong? Is there a path down to the slowspitfire.py file or an explanation what slowspitfire is doing?
Harald
https://bitbucket.org/pypy/benchmarks/src/b93caae762a0/unladen_swallow/perfo... It's creating a very large template table (1000x1000 elements I think) The explanation "why it's slow" is a bit longish. It's a combination of factors, including very long lists with GC objects in it, using ''.join(list) instead of cStringIO (the latter is faster and yes, it is a bug) and a bit of other factors.
-- GHUM GmbH Harald Armin Massa Spielberger Straße 49 70435 Stuttgart 0173/9409607
Amtsgericht Stuttgart, HRB 734971 - persuadere. et programmare

Hey Miquel. A small feature request ;-) Can we get favicon? Cheers, fijal

Hi all, On Wed, Mar 9, 2011 at 14:21, Maciej Fijalkowski <fijall@gmail.com> wrote:
On Wed, Mar 9, 2011 at 8:19 AM, Massa, Harald Armin <chef@ghum.de> wrote:
I really, really like the new display! Congratulations to Miquel for the great work!
A minor comment about the homepage: the answer to "How fast is PyPy?" would better stay close to the question, i.e. above Plot 1 (at least with the current wording).
But I was unable to find out what slowspitfire is doing ... I found spitfire, which does some HTML templating stuff, and deducted, that slowspitfire will do some slow HTML templating stuff. Where did I click wrong?
Is there a path down to the slowspitfire.py file or an explanation what slowspitfire is doing?
Is there a place the answer this information to the website? I propose a link to the source in each benchmark page. Additionally, on the frontpage the individual benchmark names could be links to the benchmark page, like in the grid view. The specific benchmark has no description: http://speed.pypy.org/timeline/?exe=1%2C3&base=2%2B35&ben=slowspitfire&env=tannit&revs=200 Following the spitfire_cstringio template, I propose the following rewording of the below answer (I'm not entirely happy but I guess Maciej can fix it easily if it's too bad): slowspitfire: Uses the Spitfire template system to build a 1000x1000-cell HTML table; it differs from spitfire which is slower on PyPy: it uses .join(list) instead of cStringIO module, has very long lists with GC objects in it, and some other smaller problems.
https://bitbucket.org/pypy/benchmarks/src/b93caae762a0/unladen_swallow/perfo...
It's creating a very large template table (1000x1000 elements I think)
The explanation "why it's slow" is a bit longish. It's a combination of factors, including very long lists with GC objects in it, using ''.join(list) instead of cStringIO (the latter is faster and yes, it is a bug) and a bit of other factors.
Another small problem I had with zooming (which is really cool, BTW):
There is an easy solution for that, at least for the moment: enabling zooming. I just did that, and you can now use zooming in a timeline plot to select a narrower yaxis range or just view a particular area in detail. A single click resets the zoom level.
While trying this I clicked on a revision; I immediately clicked on "back", but I was brought too much backwards, to the grid of all benchmarks, which loads slow enough for one to notice. If you instead click "Back" from a specific benchmark page, you are brought back to the home. Fixing this without loading a separate page for each plot seems hard; however, it seems that e.g. Facebook handles this by modifying the URL part after #, so that the page is not reloaded from scratch, but I'm no web developer, so you probably know better than me. Cheers, -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/

The specific benchmark has no description: http://speed.pypy.org/timeline/?exe=1%2C3&base=2%2B35&ben=slowspitfire&env=tannit&revs=200
Thanks, I have added your description.
Is there a place the answer this information to the website? I propose a link to the source in each benchmark page. Yes, I will try to find a way to put links to the code in each benchmark page.
Additionally, on the frontpage the individual benchmark names could be links to the benchmark page, like in the grid view. I first thought that it would be difficult because the benchmark names themselves are tick labels for the axes, but I think I can actually make the bars clickable. I will see.
While trying this I clicked on a revision; I immediately clicked on "back", but I was brought too much backwards, to the grid of all benchmarks, which loads slow enough for one to notice. If you instead click "Back" from a specific benchmark page, you are brought back to the home. Fixing this without loading a separate page for each plot seems hard; however, it seems that e.g. Facebook handles this by modifying the URL part after #, so that the page is not reloaded from scratch, but I'm no web developer, so you probably know better than me.
Yes, we are looking at a solution there. See issue #17 : https://github.com/tobami/codespeed/issues#issue/17 Stefan Marr has already implemented it. There were some issues at first, and he uses a development un-minified version of a plugin, but it works. So we should be able to integrate the feature soon. Miquel 2011/3/9 Paolo Giarrusso <pgiarrusso@mathematik.uni-marburg.de>:
Hi all,
On Wed, Mar 9, 2011 at 14:21, Maciej Fijalkowski <fijall@gmail.com> wrote:
On Wed, Mar 9, 2011 at 8:19 AM, Massa, Harald Armin <chef@ghum.de> wrote:
I really, really like the new display! Congratulations to Miquel for the great work!
A minor comment about the homepage: the answer to "How fast is PyPy?" would better stay close to the question, i.e. above Plot 1 (at least with the current wording).
But I was unable to find out what slowspitfire is doing ... I found spitfire, which does some HTML templating stuff, and deducted, that slowspitfire will do some slow HTML templating stuff. Where did I click wrong?
Is there a path down to the slowspitfire.py file or an explanation what slowspitfire is doing?
Is there a place the answer this information to the website? I propose a link to the source in each benchmark page. Additionally, on the frontpage the individual benchmark names could be links to the benchmark page, like in the grid view.
The specific benchmark has no description: http://speed.pypy.org/timeline/?exe=1%2C3&base=2%2B35&ben=slowspitfire&env=tannit&revs=200
Following the spitfire_cstringio template, I propose the following rewording of the below answer (I'm not entirely happy but I guess Maciej can fix it easily if it's too bad): slowspitfire: Uses the Spitfire template system to build a 1000x1000-cell HTML table; it differs from spitfire which is slower on PyPy: it uses .join(list) instead of cStringIO module, has very long lists with GC objects in it, and some other smaller problems.
https://bitbucket.org/pypy/benchmarks/src/b93caae762a0/unladen_swallow/perfo...
It's creating a very large template table (1000x1000 elements I think)
The explanation "why it's slow" is a bit longish. It's a combination of factors, including very long lists with GC objects in it, using ''.join(list) instead of cStringIO (the latter is faster and yes, it is a bug) and a bit of other factors.
Another small problem I had with zooming (which is really cool, BTW):
There is an easy solution for that, at least for the moment: enabling zooming. I just did that, and you can now use zooming in a timeline plot to select a narrower yaxis range or just view a particular area in detail. A single click resets the zoom level.
While trying this I clicked on a revision; I immediately clicked on "back", but I was brought too much backwards, to the grid of all benchmarks, which loads slow enough for one to notice. If you instead click "Back" from a specific benchmark page, you are brought back to the home. Fixing this without loading a separate page for each plot seems hard; however, it seems that e.g. Facebook handles this by modifying the URL part after #, so that the page is not reloaded from scratch, but I'm no web developer, so you probably know better than me.
Cheers, -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ _______________________________________________ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev

But I was unable to find out what slowspitfire is doing ... I found spitfire, which does some HTML templating stuff, and deducted, that slowspitfire will do some slow HTML templating stuff. Where did I click wrong? Is there a path down to the slowspitfire.py file or an explanation what slowspitfire is doing?
You definitely have a point, and there are features planned that should make benchmark-discovery more user-friendly. 2011/3/9 Massa, Harald Armin <chef@ghum.de>:
I really, really like the new display!
And it motivated me to dig into the data ... which is a great result on its own.
The first question for myself was "hey, why is it slow on slowspitfire, and, btw, what is slowspitfire? Could that be something that my application does, too?"
But I was unable to find out what slowspitfire is doing ... I found spitfire, which does some HTML templating stuff, and deducted, that slowspitfire will do some slow HTML templating stuff. Where did I click wrong? Is there a path down to the slowspitfire.py file or an explanation what slowspitfire is doing?
Harald
-- GHUM GmbH Harald Armin Massa Spielberger Straße 49 70435 Stuttgart 0173/9409607
Amtsgericht Stuttgart, HRB 734971 - persuadere. et programmare
participants (5)
-
Laura Creighton
-
Maciej Fijalkowski
-
Massa, Harald Armin
-
Miquel Torres
-
Paolo Giarrusso