[pypy-dev] Change to the frontpage of speed.pypy.org

[Miquel Torres]

> And I want the 'lestest results' stuff gone from the front page.
> It's as misleading as ever.  And I have done a poll of developers.  it's
> not just me.  Nobody finds it valuable.  Because things stay red forever,
> we all ignore it all the time and go directly to the raw results of
> runs, which is what we are interested in.  This also tells us of
> improvements, which we are also interested in, because unexpected
> improvements often mean something is very wrong.
Ok, that is pretty clear.

> Explaining that they aren't current, or latest results, but instead
> 'sometime in the past when we were bad once' is getting irritating.
> Can you please move it someplace else so I don't have to have this
> conversation with pycon attendees any more?
Sorry about that.
I have removed it from the site.

> Later, when pycon is over, we can discuss and work out a better design
> for informing developers that a given build may have broken things.  This
> way is not working.

Yes, we can figure out a better approach after PyCon.

> And I don't think that you can use the geometric mean to prove a thing
> with this.  So I think talking about it makes us look bad -- we are
> making claims based on either bad science, or pseudo-science.
I agree that it wasn't the best explanation. I have removed most text,
so that it doesn't explicitly say that it means PyPy *is* X times
faster. It now just states that the geometric mean is X times faster.
If it is still too much, we can remove all text. But then some people
won't understand the numbers.

We can also remove the second plot.

The mean of a set of benchmarks that does not represent a balanced
real-world task mix is of course not very good. But then again the
world is complicated. And, as a normal Python developer, I find the
second plot extremely interesting, because it gives me a ballpark idea
of where the PyPy project is heading to. Before I decide to use PyPy
in production instead of CPython, I will do my own testing for *my*
application, but I assure you that not having a ballpark average won't
be a plus in considering to invest the time to test and adopt PyPy.
But that is my opinion of course ;-)

Have fun at PyCon!
Miquel


2011/3/9 Laura Creighton <lac at openend.se>:
> In a message of Tue, 08 Mar 2011 20:20:06 +0100, Miquel Torres writes:
>>Ok, I just committed the changes.
>>
>>They address two general cases:
>>- You want to know how fast PyPy is *now* compared to CPython in
>>different benchmark scenarios, or tasks.
>>- You want to know how PyPy has been *improving* overall over the last re
>>le=
>>ases
>>
>>That is now answered on the front page, and the reports are now much
>>less prominent (I didn't change the logic because it is something I
>>want to do properly, not just as a hack for speed.pypy).
>>- I have not yet addressed the "smaller is better" point.
>>
>>I am aware that the wording of the "faster on average" needs to be
>>improved (I am discussing it with Holger even now ;). Please chime in
>>so that we can have a good paragraph that is informative and short
>>enough while at the same time not being misleading.
>>
>>Miquel
>
> The graphic is lovely.
>
> you have a sÃpelling error s/taks/task/.
>
> Many of us are at PyCon now, so working on wording may not be
> something we have time for now.  I am not sure that the geometric
> mean of all benchmarks give you anything meaningful, so I would
> have avoided saying anything like that.  More specifically, I think
> that there is a division between some highly mathematical programs,
> where you might get a speedup of 20 to 30 times CPython, and the
> benchmarks whcih I find much more meaningful, those that represent
> actualy python programs -- where I think we are typically only between
> 2 and 3 times faster.
>
> The only reason to have some of the benchmarks is because they are
> well known.  So people expect them.   But doing very well on them is
> not actually all that significant -- it would be easy to write something
> that is great and running these contrived, synthetic benchmarks, but
> really lousy at running real python code.
>
> And I don't think that you can use the geometric mean to prove a thing
> with this.  So I think talking about it makes us look bad -- we are
> making claims based on either bad science, or pseudo-science.
>
> And I want the 'lestest results' stuff gone from the front page.
> It's as misleading as ever.  And I have done a poll of developers.  it's
> not just me.  Nobody finds it valuable.  Because things stay red forever,
> we all ignore it all the time and go directly to the raw results of
> runs, which is what we are interested in.  This also tells us of
> improvements, which we are also interested in, because unexpected
> improvements often mean something is very wrong.
>
> The whole thing has the same problem as those popup windows 'do you
> really want to delete that file? confirm y/n'.  You get used to typing
> y.  Then you do it when you meant not to save the file.  The red pages
> get ignored for precisely the same reason.  We're all used to all the
> red, which generally refer to problems which were already fixed, and
> thus are things that we _should_ ignore.  So we end up ignoring it all,
> all of the time.  So it therefore doesn't serve its purpose of reminding
> developers of anything.
>
> The general public, however, reads the thing and thinks -- latest
> results, pypy has just gone to hell, look at all those slowdowns.
> Explaining that they aren't current, or latest results, but instead
> 'sometime in the past when we were bad once' is getting irritating.
> Can you please move it someplace else so I don't have to have this
> conversation with pycon attendees any more?
>
> Later, when pycon is over, we can discuss and work out a better design
> for informing developers that a given build may have broken things.  This
> way is not working.
>
> Laura
>