[pypy-dev] Change to the frontpage of speed.pypy.org

Laura Creighton lac at openend.se
Wed Mar 9 06:17:41 CET 2011

In a message of Tue, 08 Mar 2011 20:20:06 +0100, Miquel Torres writes:
>Ok, I just committed the changes.
>They address two general cases:
>- You want to know how fast PyPy is *now* compared to CPython in
>different benchmark scenarios, or tasks.
>- You want to know how PyPy has been *improving* overall over the last re
>That is now answered on the front page, and the reports are now much
>less prominent (I didn't change the logic because it is something I
>want to do properly, not just as a hack for speed.pypy).
>- I have not yet addressed the "smaller is better" point.
>I am aware that the wording of the "faster on average" needs to be
>improved (I am discussing it with Holger even now ;). Please chime in
>so that we can have a good paragraph that is informative and short
>enough while at the same time not being misleading.

The graphic is lovely.

you have a sÃpelling error s/taks/task/.

Many of us are at PyCon now, so working on wording may not be
something we have time for now.  I am not sure that the geometric
mean of all benchmarks give you anything meaningful, so I would
have avoided saying anything like that.  More specifically, I think
that there is a division between some highly mathematical programs,
where you might get a speedup of 20 to 30 times CPython, and the
benchmarks whcih I find much more meaningful, those that represent
actualy python programs -- where I think we are typically only between 
2 and 3 times faster.

The only reason to have some of the benchmarks is because they are
well known.  So people expect them.   But doing very well on them is
not actually all that significant -- it would be easy to write something
that is great and running these contrived, synthetic benchmarks, but
really lousy at running real python code.

And I don't think that you can use the geometric mean to prove a thing
with this.  So I think talking about it makes us look bad -- we are
making claims based on either bad science, or pseudo-science.

And I want the 'lestest results' stuff gone from the front page.
It's as misleading as ever.  And I have done a poll of developers.  it's
not just me.  Nobody finds it valuable.  Because things stay red forever,
we all ignore it all the time and go directly to the raw results of
runs, which is what we are interested in.  This also tells us of
improvements, which we are also interested in, because unexpected
improvements often mean something is very wrong.  

The whole thing has the same problem as those popup windows 'do you
really want to delete that file? confirm y/n'.  You get used to typing
y.  Then you do it when you meant not to save the file.  The red pages
get ignored for precisely the same reason.  We're all used to all the
red, which generally refer to problems which were already fixed, and
thus are things that we _should_ ignore.  So we end up ignoring it all,
all of the time.  So it therefore doesn't serve its purpose of reminding
developers of anything.  

The general public, however, reads the thing and thinks -- latest
results, pypy has just gone to hell, look at all those slowdowns.
Explaining that they aren't current, or latest results, but instead
'sometime in the past when we were bad once' is getting irritating.
Can you please move it someplace else so I don't have to have this
conversation with pycon attendees any more?

Later, when pycon is over, we can discuss and work out a better design
for informing developers that a given build may have broken things.  This
way is not working.


More information about the Pypy-dev mailing list