Hi Miquel, all, regarding the new speed.pypy.org host I realized that the current setting is not suited for setting up another VM. Too few resources and old-ish software. I plan to order a new host, setup speed.pypy.org there (and migrate existing VMs later). Takes a week or two i guess. Hope this doesn't block you. Could maybe someone here on the list (temporarily) offer a suitable VM to run Django Apps/DBs where Miquel can get an account? i'd then point the "speed.pypy.org" DNS entry to it. The earlier the site gets running the earlier we can all follow speed developments with the PyPy JIT :) best, holger -- Metaprogramming, Python, Testing: http://tetamap.wordpress.com Python, PyPy, pytest contracting: http://merlinux.eu
Hi Holger, for the time being it is not a problem. I can begin the implementation locally and later continue on the server VM. One thing I would need though: a speed.pypy wiki page. I would like that you (pypy core developers) write down the most important use cases for speed.pypy.org For example: - See right away the last x trunk revisions. A line plot (and table?) will be shown for each individual benchmark in the benchmark suite. - Allow to select multiple, arbitrary trunk revisions for comparing - Show different data series in the same graph for different compile options (like tuatara's benchmarks??) ... I have my own ideas, but you will be the main users, so you need to tell me what you need implemented first. Another wiki page could be about what benchmarks could be useful for the suite. It would be great if all benchmarks results were points, or all seconds (less is better), (or x times faster/slower than cpython) because that way results can be shown in a much more compact form. For example stacked: http://www.codinghorror.com/blog/images/browser-javascript-performance-graph... * * or combined in a single chart: http://www.codinghorror.com/blog/images/browser-javascript-performance-graph... Anyway the lack of a VM is not a problem for me right now. Cheers, Miquel <http://www.codinghorror.com/blog/images/browser-javascript-performance-graph...> 2009/10/2 holger krekel <holger@merlinux.eu>
Hi Miquel, May I point out a couple of comments about http://speed.pypy.org/ ? The first is that it looks great, indeed; thank you very much for doing such a site! :-) The comments are mostly about the graphics in /timeline: * they should also display, or allow to display, the speed of CPython for comparison; * they should have a longer maximum history than 100, to see more clearly the long-term evolution. All in all it looks great! A bientot, Armin.
I also have a bunch of comments, all are my opinion and should not be taken as demands (or even as a good review). - When you first visit the site I think it would be better to be on the timeline like on http://buildbot.pypy.org/plotsummary.html that shows all benchmarks on the same page, but just the last 50 revisions or it becomes too hard to see the recent improvements. (being able to show more history is important too, maybe there is a way to show the trend and the last x revisions clearly). - Overview window comments: - rename the column "result" to "time" (or timing?); - create a new column called "previous time" with the last time of the previous revision. - rename "current change" to "improvement" and invert the values (instead of a green -2% you end up with a green 2%); - with the last two ones it will make understanding the improvement and trends a lot easier; - trend should have the number of revisions on the label (and the values reversed like the "improvement" column); - rename the column "times cpython" to something else (eg. "compared to cpython") and maybe have values like "2x slow down" and "2.5x speedup"; - the graph should say what it is graphing; - maybe invert the order of results. - Timeline window comments: - showing all benchmarks in the same page would be good - Pypy with jit should come first in the list of interpreters, and while you only test with hybrid gc there is no need to state that explicitly; - see all benchmarks together, so you don't have to hunt around (maybe have a detailed view); - describe all axis; - as armin said have a selected by default cpython comparison so people can have a baseline. I think this speed center could be used for the common mercurial benchmark repository that is being setup, that would be very very cool. On Feb 25, 2010, at 6:18 AM, Armin Rigo wrote:
-- Leonardo Santagada santagada at gmail.com
Thanks for the nice comments. I was waiting for the test hook to be implemented before emailing pypy-dev about it, but seeing as It already got out I may do it now as well ;-) @Armin
@Leonardo phew! those were a lot of suggestions, thanks. I will start implementing some of them, though I would like the changes to be discussed so that they get done right and all agree on what's best (in the cases where it can't be an option). I already implemented a couple of changes suggested by fijal: logarithmic scale for the overview bars and variable width so it fits on small screens, down to 980px width. I hope he is happy now :-) I'll start a new thread for the explanation and plans. Cheers! 2010/2/25 Leonardo Santagada <santagada@gmail.com>:
I have commited a bunch of changes: - Added cpython baseline to timeline - Changed timeline defaults to both interpreters selected (plus baseline) - Performance improvements in Timeline's ajax: due to above changes, I worried that for a lot of interpreters selected the queries could be too slow. Even though it is known that Premature optimization is the root of all evil (tm), I had a look and optimized django database calls a little bit (or more precisely made my implementation suck less). So instead of nearly 200 ms per interpreter, it now takes less than 100ms (measured locally, add >100ms for network latency). Plot rerendering should feel faster now. I find it interesting how quick django is. Sometimes you may even forget that the data is being pulled from the server each time a different plot is selected. - Timeline: Added labels for the axes - pypy-c-jit appears now listed first - Added tooltips for Host info and for some benchmarks. Copied the descriptions for the benchmarks listed at the unladen site. Cheers, Miquel 2010/2/25 Leonardo Santagada <santagada@gmail.com>:
Hi there Armin Rigo wrote:
Yes - it looks both nice and very useful indeed! Do we have a link from PyPys "main page" http://codespeak.net/pypy/dist/pypy/doc/ to this page so people can find it more easily? Or did I miss that? Cheers Bea
On Fri, Feb 26, 2010 at 5:44 AM, Bea During <bea@changemaker.nu> wrote:> Hi there
Our main website is a mess right now. I would like to have it redesigned, so we have actually interesting links from there, and obviously the one to graphs (we have two of those now) would be great. Note that for now speed.pypy.org is not automatically updated - it requires adjustments to buildbot run. Cheers, fijal
Hi Holger, for the time being it is not a problem. I can begin the implementation locally and later continue on the server VM. One thing I would need though: a speed.pypy wiki page. I would like that you (pypy core developers) write down the most important use cases for speed.pypy.org For example: - See right away the last x trunk revisions. A line plot (and table?) will be shown for each individual benchmark in the benchmark suite. - Allow to select multiple, arbitrary trunk revisions for comparing - Show different data series in the same graph for different compile options (like tuatara's benchmarks??) ... I have my own ideas, but you will be the main users, so you need to tell me what you need implemented first. Another wiki page could be about what benchmarks could be useful for the suite. It would be great if all benchmarks results were points, or all seconds (less is better), (or x times faster/slower than cpython) because that way results can be shown in a much more compact form. For example stacked: http://www.codinghorror.com/blog/images/browser-javascript-performance-graph... * * or combined in a single chart: http://www.codinghorror.com/blog/images/browser-javascript-performance-graph... Anyway the lack of a VM is not a problem for me right now. Cheers, Miquel <http://www.codinghorror.com/blog/images/browser-javascript-performance-graph...> 2009/10/2 holger krekel <holger@merlinux.eu>
Hi Miquel, May I point out a couple of comments about http://speed.pypy.org/ ? The first is that it looks great, indeed; thank you very much for doing such a site! :-) The comments are mostly about the graphics in /timeline: * they should also display, or allow to display, the speed of CPython for comparison; * they should have a longer maximum history than 100, to see more clearly the long-term evolution. All in all it looks great! A bientot, Armin.
I also have a bunch of comments, all are my opinion and should not be taken as demands (or even as a good review). - When you first visit the site I think it would be better to be on the timeline like on http://buildbot.pypy.org/plotsummary.html that shows all benchmarks on the same page, but just the last 50 revisions or it becomes too hard to see the recent improvements. (being able to show more history is important too, maybe there is a way to show the trend and the last x revisions clearly). - Overview window comments: - rename the column "result" to "time" (or timing?); - create a new column called "previous time" with the last time of the previous revision. - rename "current change" to "improvement" and invert the values (instead of a green -2% you end up with a green 2%); - with the last two ones it will make understanding the improvement and trends a lot easier; - trend should have the number of revisions on the label (and the values reversed like the "improvement" column); - rename the column "times cpython" to something else (eg. "compared to cpython") and maybe have values like "2x slow down" and "2.5x speedup"; - the graph should say what it is graphing; - maybe invert the order of results. - Timeline window comments: - showing all benchmarks in the same page would be good - Pypy with jit should come first in the list of interpreters, and while you only test with hybrid gc there is no need to state that explicitly; - see all benchmarks together, so you don't have to hunt around (maybe have a detailed view); - describe all axis; - as armin said have a selected by default cpython comparison so people can have a baseline. I think this speed center could be used for the common mercurial benchmark repository that is being setup, that would be very very cool. On Feb 25, 2010, at 6:18 AM, Armin Rigo wrote:
-- Leonardo Santagada santagada at gmail.com
Thanks for the nice comments. I was waiting for the test hook to be implemented before emailing pypy-dev about it, but seeing as It already got out I may do it now as well ;-) @Armin
@Leonardo phew! those were a lot of suggestions, thanks. I will start implementing some of them, though I would like the changes to be discussed so that they get done right and all agree on what's best (in the cases where it can't be an option). I already implemented a couple of changes suggested by fijal: logarithmic scale for the overview bars and variable width so it fits on small screens, down to 980px width. I hope he is happy now :-) I'll start a new thread for the explanation and plans. Cheers! 2010/2/25 Leonardo Santagada <santagada@gmail.com>:
I have commited a bunch of changes: - Added cpython baseline to timeline - Changed timeline defaults to both interpreters selected (plus baseline) - Performance improvements in Timeline's ajax: due to above changes, I worried that for a lot of interpreters selected the queries could be too slow. Even though it is known that Premature optimization is the root of all evil (tm), I had a look and optimized django database calls a little bit (or more precisely made my implementation suck less). So instead of nearly 200 ms per interpreter, it now takes less than 100ms (measured locally, add >100ms for network latency). Plot rerendering should feel faster now. I find it interesting how quick django is. Sometimes you may even forget that the data is being pulled from the server each time a different plot is selected. - Timeline: Added labels for the axes - pypy-c-jit appears now listed first - Added tooltips for Host info and for some benchmarks. Copied the descriptions for the benchmarks listed at the unladen site. Cheers, Miquel 2010/2/25 Leonardo Santagada <santagada@gmail.com>:
Hi there Armin Rigo wrote:
Yes - it looks both nice and very useful indeed! Do we have a link from PyPys "main page" http://codespeak.net/pypy/dist/pypy/doc/ to this page so people can find it more easily? Or did I miss that? Cheers Bea
On Fri, Feb 26, 2010 at 5:44 AM, Bea During <bea@changemaker.nu> wrote:> Hi there
Our main website is a mess right now. I would like to have it redesigned, so we have actually interesting links from there, and obviously the one to graphs (we have two of those now) would be great. Note that for now speed.pypy.org is not automatically updated - it requires adjustments to buildbot run. Cheers, fijal
participants (7)
-
Armin Rigo -
Bea During -
holger krekel -
Leonardo Santagada -
Maciej Fijalkowski -
Miquel Torres -
Samuel Reis