New version of Codespeed (0.7) for speed.pypy.org
Hi all, I want to announce the release of Codespeed 0.7, the version which is now powering speed.pypy.org The announcement has been made on http://groups.google.com/group/codespeed/browse_thread/thread/3dd2d9491b26d0..., but I will paste it here: -------------------------------------------------- I will briefly describe the most significant improvements. Result Reports -------------- While the previous Codespeed version allowed a project to analyse and visualize performance changes and compare different versions or even "competitors", it didn't address well a use-case: a core developer that is interested in the daily performance variations. You had to go to the Changes view, click on the desired revision, maybe another executable, and again for every revision you wanted to check. That use case is now addressed by Reports. They are a summary of the most significant changes on every run. It's a ultra-condensed Changes table. It is shown on the main page, and you can even subscribe to a RSS feed. That way you only visit the page when you are interested in some particular revision or change. After a few weeks of use, you may find out that a different "algorithm" is better (for example dropping trends or trends triggering the red state...), or that the thresholds need to be tweaked. And there are surely other interesting run statistics that could be gathered and shown. I eagerly await the feedback and ensuing discussion ;-) Support for small screen sizes ------------------------------ Using CSS media queries, the site layout adapts to smaller screens. It is optimized for 3 sizes: - Desktop: normal layout for big screens - Netbook, tablets (~1000px): Similar layout but smaller margins, smaller plots. Coincidentally, it is about the same width of Maciej's laptop screen ;-) - Smartphones: iPhones and Androids in landscape mode should be now able to visit the site without problems. Though it can be tested just by resizing the browser window, it would be great to get some feedback of anyone actually using the above mentioned devices! Performance improvements ------------------------ Exarkun made the timeline view faster by pre-fetching all related data. The timeline grid took 1.46 seconds to load, and a single plot 0.45 seconds. They now need 1.26 and 0.26 seconds respectively. A big improvement! There is yet another improvement waiting in the wings, but we still need to figure out some regression it introduces. The Changes view is also much faster (from 480ms to 210ms), because it now uses the cached table stored in the corresponding report (only when viewing with the default trend=10 ). A last detail --------------- Another change you may notice is that the default checked executables in the Comparison view are now configurable (I didn't forget Maciej! :). So they are not all selected per default any more, so the plot shouldn't look any longer like the Flying Spaghetti Monster ;-) I'll leave it there. I hope you enjoy the changes! Miquel
Great! On Thu, Jan 20, 2011 at 11:00 PM, Miquel Torres <tobami@googlemail.com> wrote:
Hi all,
I want to announce the release of Codespeed 0.7, the version which is now powering speed.pypy.org
The announcement has been made on http://groups.google.com/group/codespeed/browse_thread/thread/3dd2d9491b26d0..., but I will paste it here:
-------------------------------------------------- I will briefly describe the most significant improvements.
Result Reports -------------- While the previous Codespeed version allowed a project to analyse and visualize performance changes and compare different versions or even "competitors", it didn't address well a use-case: a core developer that is interested in the daily performance variations.
You had to go to the Changes view, click on the desired revision, maybe another executable, and again for every revision you wanted to check.
That use case is now addressed by Reports. They are a summary of the most significant changes on every run. It's a ultra-condensed Changes table. It is shown on the main page, and you can even subscribe to a RSS feed. That way you only visit the page when you are interested in some particular revision or change.
After a few weeks of use, you may find out that a different "algorithm" is better (for example dropping trends or trends triggering the red state...), or that the thresholds need to be tweaked. And there are surely other interesting run statistics that could be gathered and shown. I eagerly await the feedback and ensuing discussion ;-)
Support for small screen sizes ------------------------------
Using CSS media queries, the site layout adapts to smaller screens. It is optimized for 3 sizes: - Desktop: normal layout for big screens - Netbook, tablets (~1000px): Similar layout but smaller margins, smaller plots. Coincidentally, it is about the same width of Maciej's laptop screen ;-) - Smartphones: iPhones and Androids in landscape mode should be now able to visit the site without problems.
Though it can be tested just by resizing the browser window, it would be great to get some feedback of anyone actually using the above mentioned devices!
Performance improvements ------------------------
Exarkun made the timeline view faster by pre-fetching all related data. The timeline grid took 1.46 seconds to load, and a single plot 0.45 seconds. They now need 1.26 and 0.26 seconds respectively. A big improvement!
There is yet another improvement waiting in the wings, but we still need to figure out some regression it introduces.
The Changes view is also much faster (from 480ms to 210ms), because it now uses the cached table stored in the corresponding report (only when viewing with the default trend=10 ).
A last detail ---------------
Another change you may notice is that the default checked executables in the Comparison view are now configurable (I didn't forget Maciej! :). So they are not all selected per default any more, so the plot shouldn't look any longer like the Flying Spaghetti Monster ;-)
I'll leave it there. I hope you enjoy the changes!
Miquel _______________________________________________ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Hi Miquel On 20/01/11 22:00, Miquel Torres wrote:
Hi all,
I want to announce the release of Codespeed 0.7, the version which is now powering speed.pypy.org
wow, that's very nice, thank you once more :-). From my side, there is only one feature that I miss a lot, which is the possibility to benchmark a branch. IIRC, at the moment if you just run benchmarks on a branch, they are appended to the list of results, which can be a bit confusing especially if you look at them weeks later. I think that maybe it's possible to do it by (ab)using an additional environment, but probably a more builtin solution is better. Do you have any idea/opinion/suggestion on this? ciao, Anto
@Anto Yes, branches are a pending item that has been requested a couple of times now. The current solution is actually not to abuse an environment like you say, but to create a new project for a branch. That way it gets cleanly separated, and in a way branches are like different projects. But it is of course not optimal. Technically it is very easy to come up with several solutions (add a branch dimension, for example), but interface-wise it is not easy to find something that doesn't clutter things. So one idea would be: - Every revision has a branch associated to it (a problem though is that subversion has the same revision number for all branches, while mercurial and git not) - Branch would be a selectable parameter on the left menu box, just like executables or environments - You would be able to compare a branch using the comparison view, but another possibility is that the timeline view doesn't have branch as a radio button, but as a checkbox. That means that a plot is associated only to an executable and environment, and that you can choose to display either all branches in the same plot (trunk or master + any branch) or only the ones you select. - There are problems with this approach. In the Changes view, when you select a revision that doesn't have results for the currently selected branch, nothing would be displayed. You would need to blindly search for your results. An option is to always display all branches for a revision, but sometimes a revision will contain results for a branch, sometimes for others. Another idea would be to associate executables to a branch. This avoids the problem in the Changes view. But it is not really much different from the current solution of creating a different project for each branch. I would gladly hear other ideas. Cheers, Miquel 2011/1/20 Antonio Cuni <anto.cuni@gmail.com>:
Hi Miquel
On 20/01/11 22:00, Miquel Torres wrote:
Hi all,
I want to announce the release of Codespeed 0.7, the version which is now powering speed.pypy.org
wow, that's very nice, thank you once more :-).
From my side, there is only one feature that I miss a lot, which is the possibility to benchmark a branch.
IIRC, at the moment if you just run benchmarks on a branch, they are appended to the list of results, which can be a bit confusing especially if you look at them weeks later.
I think that maybe it's possible to do it by (ab)using an additional environment, but probably a more builtin solution is better. Do you have any idea/opinion/suggestion on this?
ciao, Anto
Hi Miquel!
I would gladly hear other ideas. Here are my two cents!
On Fri, Jan 21, 2011 at 08:49, Miquel Torres <tobami@googlemail.com> wrote:
So one idea would be: - Every revision has a branch associated to it (a problem though is that subversion has the same revision number for all branches, while mercurial and git not) This suggests (to me) that a revision would be associated to a specific branch, but then I don't get expressions like "all the branches for a revision". Maybe you mean that you would replace revisions with pairs (Revision, Branch)? [...] - There are problems with this approach. In the Changes view, when you select a revision that doesn't have results for the currently selected branch, nothing would be displayed.
You're referring to an SVN context, right? Otherwise I can't make sense of it. I guess you can use, on other branches, the "closest" revision available, after defining a linear order on revisions across branches. You have that order in SVN; since you don't have that in Git, why don't you just use the _date_ of a revision as the ordering key? [For the data structures to implement this, I've seen even seen interfaces to tree-based dictionaries allowing navigating to the key-predecessor and key-successor, which might be especially handy here (such as NavigableMap in Java, dunno about Python).] Of course, that's suitable just as default, so you need to be able to navigate between revisions independently for each branch. Please note that with complex merge histories, using the date does not necessarily make sense - so that at least Git tools provide many linearization algorithms. The problem is that a commit might be made on a certain date in some repository, but be merged on mainline much later, especially when many developers are not committers themselves; one solution is to consider only revisions which were committed directly on the branch or merged shortly after. I also would believe this use case does not appear for PyPy. See help about git log --topo-order and --date-order (such docs used to be more complete, though).
You would need to blindly search for your results. An option is to always display all branches for a revision, but sometimes a revision will contain results for a branch, sometimes for others. Outside of the SVN context, I'm not sure I get what you mean here, because revisions in different branches are not necessarily comparable. -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/
Hi Paolo,
- There are problems with this approach. In the Changes view, when you select a revision that doesn't have results for the currently selected branch, nothing would be displayed.
You're referring to an SVN context, right? Otherwise I can't make sense of it.
The opposite. For subversion, revisions mark the whole repository, so a revision could have results for both trunk and a branch. That is not the case in mercurial. Regarding my explanation, I understand it is a bit difficult to visualize this UI issues without images.
You would need to blindly search for your results. An option is to always display all branches for a revision, but sometimes a revision will contain results for a branch, sometimes for others. Outside of the SVN context, I'm not sure I get what you mean here, because revisions in different branches are not necessarily comparable.
Same as above. So the differences between version control systems really suggest that you cannot add a branch dimension to a revision. More like a result is not only associated to a project, revision, and executable, but also to a branch... 2011/1/21 Paolo Giarrusso <p.giarrusso@gmail.com>:
Hi Miquel!
I would gladly hear other ideas. Here are my two cents!
On Fri, Jan 21, 2011 at 08:49, Miquel Torres <tobami@googlemail.com> wrote:
So one idea would be: - Every revision has a branch associated to it (a problem though is that subversion has the same revision number for all branches, while mercurial and git not) This suggests (to me) that a revision would be associated to a specific branch, but then I don't get expressions like "all the branches for a revision". Maybe you mean that you would replace revisions with pairs (Revision, Branch)? [...] - There are problems with this approach. In the Changes view, when you select a revision that doesn't have results for the currently selected branch, nothing would be displayed.
You're referring to an SVN context, right? Otherwise I can't make sense of it.
I guess you can use, on other branches, the "closest" revision available, after defining a linear order on revisions across branches. You have that order in SVN; since you don't have that in Git, why don't you just use the _date_ of a revision as the ordering key? [For the data structures to implement this, I've seen even seen interfaces to tree-based dictionaries allowing navigating to the key-predecessor and key-successor, which might be especially handy here (such as NavigableMap in Java, dunno about Python).] Of course, that's suitable just as default, so you need to be able to navigate between revisions independently for each branch.
Please note that with complex merge histories, using the date does not necessarily make sense - so that at least Git tools provide many linearization algorithms. The problem is that a commit might be made on a certain date in some repository, but be merged on mainline much later, especially when many developers are not committers themselves; one solution is to consider only revisions which were committed directly on the branch or merged shortly after. I also would believe this use case does not appear for PyPy. See help about git log --topo-order and --date-order (such docs used to be more complete, though).
You would need to blindly search for your results. An option is to always display all branches for a revision, but sometimes a revision will contain results for a branch, sometimes for others. Outside of the SVN context, I'm not sure I get what you mean here, because revisions in different branches are not necessarily comparable. -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/
On 21/01/11 08:49, Miquel Torres wrote:
@Anto
Yes, branches are a pending item that has been requested a couple of times now.
yes, I think most of the requests has been by me :)
The current solution is actually not to abuse an environment like you say, but to create a new project for a branch. That way it gets cleanly separated, and in a way branches are like different projects. But it is of course not optimal. Technically it is very easy to come up with several solutions (add a branch dimension, for example), but interface-wise it is not easy to find something that doesn't clutter things.
Uhm, I don't think that using a different project is a good idea. For branches, we are usually not much interested in the branch history, but in the comparison with trunk (ideally, with trunk at the point we created the branch, or at the point of the last merge from trunk). As for visualize changes, I think that we don't need anything fancy, for example it would be already immensely useful to have the possibility of displaying the Changes page in a way that it compares the performances of the branch against trunk. ciao, Anto
On 06:14 pm, anto.cuni@gmail.com wrote:
On 21/01/11 08:49, Miquel Torres wrote:
@Anto
Yes, branches are a pending item that has been requested a couple of times now.
yes, I think most of the requests has been by me :)
The current solution is actually not to abuse an environment like you say, but to create a new project for a branch. That way it gets cleanly separated, and in a way branches are like different projects. But it is of course not optimal. Technically it is very easy to come up with several solutions (add a branch dimension, for example), but interface-wise it is not easy to find something that doesn't clutter things.
Uhm, I don't think that using a different project is a good idea. For branches, we are usually not much interested in the branch history, but in the comparison with trunk (ideally, with trunk at the point we created the branch, or at the point of the last merge from trunk).
As for visualize changes, I think that we don't need anything fancy, for example it would be already immensely useful to have the possibility of displaying the Changes page in a way that it compares the performances of the branch against trunk.
How about a "Branches" checkbox (per project? per executable? per graph? One of those maybe). When it's checked, branch results within the revision horizon (last 10, 50, 200, etc) get plotted on the same graph as the trunk data is plotted. Each branch could be a different color, perhaps (but would at least have hover info telling you what it is). This implies adding a branch column to the results table (or is it the revisions table?). Maybe that's just the obvious way to do it and everyone else already thought of and discarded it already, though. Actually, in general I'd like a way to plot more things on one graph. So maybe this is just a special case of that. Jean-Paul
It doesn't sound like a bad idea. But how would you save the branch data? 2011/1/21 <exarkun@twistedmatrix.com>:
On 06:14 pm, anto.cuni@gmail.com wrote:
On 21/01/11 08:49, Miquel Torres wrote:
@Anto
Yes, branches are a pending item that has been requested a couple of times now.
yes, I think most of the requests has been by me :)
The current solution is actually not to abuse an environment like you say, but to create a new project for a branch. That way it gets cleanly separated, and in a way branches are like different projects. But it is of course not optimal. Technically it is very easy to come up with several solutions (add a branch dimension, for example), but interface-wise it is not easy to find something that doesn't clutter things.
Uhm, I don't think that using a different project is a good idea. For branches, we are usually not much interested in the branch history, but in the comparison with trunk (ideally, with trunk at the point we created the branch, or at the point of the last merge from trunk).
As for visualize changes, I think that we don't need anything fancy, for example it would be already immensely useful to have the possibility of displaying the Changes page in a way that it compares the performances of the branch against trunk.
How about a "Branches" checkbox (per project? per executable? per graph? One of those maybe). When it's checked, branch results within the revision horizon (last 10, 50, 200, etc) get plotted on the same graph as the trunk data is plotted. Each branch could be a different color, perhaps (but would at least have hover info telling you what it is).
This implies adding a branch column to the results table (or is it the revisions table?).
Maybe that's just the obvious way to do it and everyone else already thought of and discarded it already, though.
Actually, in general I'd like a way to plot more things on one graph. So maybe this is just a special case of that.
Jean-Paul _______________________________________________ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
On 07:37 pm, tobami@googlemail.com wrote:
It doesn't sound like a bad idea. But how would you save the branch data?
What do you need beyond an extra column in the revisions table? Everything else would be the same, except when you upload data, include the branch it is for, and when you query for data, limit yourself to trunk/default/whatever unless you know you want more. Jean-Paul
2011/1/21 <exarkun@twistedmatrix.com>:
On 06:14 pm, anto.cuni@gmail.com wrote:
On 21/01/11 08:49, Miquel Torres wrote:
@Anto
Yes, branches are a pending item that has been requested a couple of times now.
yes, I think most of the requests has been by me :)
The current solution is actually not to abuse an environment like you say, but to create a new project for a branch. That way it gets cleanly separated, and in a way branches are like different projects. But it is of course not optimal. Technically it is very easy to come up with several solutions (add a branch dimension, for example), but interface-wise it is not easy to find something that doesn't clutter things.
Uhm, I don't think that using a different project is a good idea. For branches, we are usually not much interested in the branch history, but in the comparison with trunk (ideally, with trunk at the point we created the branch, or at the point of the last merge from trunk).
As for visualize changes, I think that we don't need anything fancy, for example it would be already immensely useful to have the possibility of displaying the Changes page in a way that it compares the performances of the branch against trunk.
How about a "Branches" checkbox (per project? �per executable? �per graph? �One of those maybe). �When it's checked, branch results within the revision horizon (last 10, 50, 200, etc) get plotted on the same graph as the trunk data is plotted. �Each branch could be a different color, perhaps (but would at least have hover info telling you what it is).
This implies adding a branch column to the results table (or is it the revisions table?).
Maybe that's just the obvious way to do it and everyone else already thought of and discarded it already, though.
Actually, in general I'd like a way to plot more things on one graph. So maybe this is just a special case of that.
Jean-Paul _______________________________________________ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
participants (5)
-
Antonio Cuni -
exarkun＠twistedmatrix.com -
Maciej Fijalkowski -
Miquel Torres -
Paolo Giarrusso