Hi there,
The twistedmatrix deployment is here BTW:
http://speed.twistedmatrix.com/ This is an interesting idea (and Codespeed is ace!). I have a few things that Codespeed could help with, although I'm, not sure I could justify paying for them, but certainly this is something that would interest me. One of my undergrads (Sam Giles) used it in his final year thesis and it worked very well for him.
A few random thoughts for you:
- To be more useful for academics Codespeed would need to generate more statistical information and much better quality charts. Statistical tests on improvements between versions, confidence intervals, etc. would be handy. This piece of work has some very nice ideas:
https://github.com/jamesbornholt/plotty This would move Codespeed from being able to provide a basic "smoke test" for performance improvement, to something much more rigorous. I guess not everyone would necessarily need that, but I think it would not do anyone harm to have it.
- Table output in LaTeX would also be useful for academic types, as would some way to make the charts more "configurable", or maybe just being able to dump the Python code (+ data) that creates the charts would do, then people could mess around offline. Another option would be to use
https://plot.ly/ which gives you the code to generate each chart you create on its API (e.g. click on the "code" tab on the left of this:
https://plot.ly/~snim2/43/ ). You could go a step further and have something like an "open my results in LaTeX / Overleaf [
https://www.overleaf.com/devs ] or ShareLaTeX. The big thing here is workflow - how does the user go from pushing a changeset in their code to generating high quality analysis and output for a blog or paper publication or a business pitch? There are several ways to reduce friction there, you can probably think of more ideas than I have.
- The style files could do with an update, which probably doesn't matter much to people on this list, but may matter more if you want to commercialise
- For any user ease of use is really the key thing, but also flexibility. It is important to be able to get at the data in different formats, but people will probably find uses for Codespeed that you hadn't intended (and I guess not all future users would necessarily be full time software developers), and there are cases where the PyPy way of doing things isn't quite what you want. Here's a simple example use case: maybe instead of tracking the progress of some software project, I want to compare benchmarks for different pieces of hardware. When I push changes to the benchmark code and re-run the analysis I would not necessarily expect the results to change over time (although they might with updated firmware / SDKs, etc) but I might want to add more hardware platforms to the data and more benchmarks to the code. Codespeed doesn't quite match that use case at the moment, but I don't see why it shouldn't be possible to reconfigure it to be useful in that situation.
Cheers,
Sarah