On 20 December 2010 19:21, William ML Leslie <william.leslie.ttg@gmail.com> wrote:
for example http://shootout.alioth.debian.org/ doesn't say which pypy version is used, what options, doesn't have performance figures for multithreaded/multicore also benchmarks are kinda small, most of them are not docuemented, and I couldn't find any info if the same python code was used for cpython and pypy (both shootout and speed pypy) or interpreter-specific verions were used, that is each tweaked for best performance given the known tradeoffs for each implementation.further the most standard benchmarks, pystone, etc. completely ignore the fact that real world programs are large and only a few small paths are execured often. another temp problem with speed pypy is that it's terrubly slow in ff beta. it also occasionally grinds in stable ff and opera, but I guess this can be forgiven for the sake of simplicity / developer effort. if you google for 'python performance' you don't get a single link to pypy on the first page, as a matter of fact, codespeak is poorly indexed, it took me quite some time to get some of my questions answered with a search. also if you look up 'pypy gc' you get a page on codespeak, but to interpret what the data actually means is so far beyond me. a good overview is found in the mainling list http://codespeak.net/pipermail/pypy-dev/2010q1/005757.html then again slowspitfire and spambayes bits are outdated by now. the definitive good thing about pypy is that it's much easier to find out about its inner workings than that of cpython! hopefully a bit more of end-user stuff get known. let's call it pypy public outreach (feature request)
Oh I didn't know there were so many, and I mistakenly though that js was a target, not implmented langauge. In any case I read somewhere that js support was retired...
Makes perfect sense, after all any given other language hardly has the same data model as python.
Hi Dima,
Well, speed.pypy is actually fast in all modern browsers. The problem you are referring to is probably caused by a bug in the javascript plotting library (jqPplot) that is triggered in the comparison view when there are some results with 0 values. It only appears for some plot types, but it is very annoying because it grinds the browser to a halt like you say. Is that what you meant? We are looking into it, and will fix that library if necessary. Cheers, Miquel 2010/12/21 Dima Tisnek <dimaqq@gmail.com>:
Yes it does grind ff 4b7 to an almost halt, little cpu usage, reasonable memory size and constant disk activity for several minutes, very weird... So far, the difference between browsers is so large that I doubt it's dependent on data. Also it seems to tirgger as I remove columns progressively, thus new zero values should not appear, right? I'll invesitage some more... On 21 December 2010 01:06, Miquel Torres <tobami@googlemail.com> wrote:
I sort of figured it out, although I don't have a ready solution. This affects opera 11, stable ff, ff 4.0b7; amazingly it does not affect ie8. Basically memory consumption of the plot appears to be proportional to plot area. Normal plot that you get by default at http://speed.pypy.org/comparison/ costs about 100M of browser memory consumption: opera 130M, stable ff 70M, beta ff 90M at window size 1680x1050; opera 80M, stable ff 55M, beta ff 70M at window size 1024x600; Switching to "horizontal" produces a tall plot of same width and costs about 300~700M of browser memory: opera 720M, stable ff 370M, beta ff 370M at window wize 1680x1050; opera 350M, stable ff 370M, beta ff 370M at window size 1024x600; Suprisingly window size only matters while javascript produces the plot, and not when window is resized, even though plot it resized with the window correctly. This alone is pretty heavy, but doesn't grind the browser. What really grinds is that every time you change a tickbox on the left, a plot is redrawn and another 200M of browser memory is wasted. This is not double buffering, as next change adds yet another 200M or so and so on, it appears that either js doesn't free something, or browser caches or saves the previous page state. As memory consumption grows, at some point browser hits the wall, causes heavy swapping for some time, and I think garbage collection, because practical (but not virtual) memory usage first drops to something like 20~50M and then returns to "normal" 300M. opera ~30 seconds, stable ff about a minute, beta ff several minutes (total system mem 1G, cpu Atom @1.6GHz) Perhaps OS also plays a role in the grind, as it is clearly swapping and swaps out too much? or triggers gc too late and gc has to pull the pages back from disk to perform collection? ie8 doesn't use that much memory, as a matter of fact memory consumption starts little (40M) and changes very little (only +10M) if you go to horizonatal view; the price is very slow rendering, more than 10 seconds per column change. I'll post this on firefox bugzilla too, let's see if someone has a solution. Meanwhile perhaps pypy speed center could start with a smaller plot area (or fewer columns as that makes horizontal plot smaller) to accomodate varying hardware and system mem usage that users might have? The simplest would be a warning next to "horizontal" checkbox. On 21 December 2010 01:06, Miquel Torres <tobami@googlemail.com> wrote:
Mmm, yes, it appears that the memory is not properly freed. I can try experimenting by destroying the dom element and recreating it each time that the plot changes... Btw., instead of continuing here "polluting" the pypy-dev mailing list, we can move to http://groups.google.com/group/codespeed if needed. Thanks for reporting this. Cheers, Miquel 2010/12/22 Dima Tisnek <dimaqq@gmail.com>:
Firefox people point the blame squarely at jqplot: --- Comment #4 from Boris Zbarsky (:bz) <bzbarsky@mit.edu> 2010-12-22 13:43:23 PST --- So what I see this page do, in horizontal mode, is create 17 canvases each of which is width="816" height="3587". That means that each of them has a backing store of 816*3587*4 = 11,707,968 bytes. So that's about 200MB of memory usage right there. I have no idea why they feel a need for 17 huge canvases, but if they want them, that's how much memory they'll take... ( https://bugzilla.mozilla.org/show_bug.cgi?id=620883 ) On 22 December 2010 01:24, Miquel Torres <tobami@googlemail.com> wrote:
Ok, so it seems from reading comments further down that bug report that FF's garbage collection does not immediately free memory for old jqplot canvases. As I said, as soon as I find the time I will experiment with more aggressive DOM element destruction/creation and see if it improves how quickly memory is freed. I do think it will at least partly circumvent jqPlot's bug. 2010/12/22 Dima Tisnek <dimaqq@gmail.com>:
Hi Dima, On Wed, Dec 22, 2010 at 11:21 PM, Dima Tisnek <dimaqq@gmail.com> wrote:
That looks very similar to an issue with PyPy's own GC, in which ctypes.create_string_buffer() returns objects which tend to be GC'ed late. That's because the string buffer object in ctypes appears (to PyPy's GC) to be just a small object, even though it actually references a potentially large piece of raw memory. Similarly, my vague guess about the above is that the 17*11MB of memory are hold by 17 small objects which firefox's GC think don't need to be collected particularly aggressively. A bientôt, Armin.
On Thu, Dec 23, 2010 at 12:14, Armin Rigo <arigo@tunes.org> wrote:
Bigger objects are not collected sooner by typical GC algorithms* - they might just fill the heap more quickly and trigger earlier a GC invocation. the raw-memory allocator to create and update such stats; then the heap-overflow check could consider them to decide whether to trigger GC. The details of the modified heap-overflow check would probably not be entirely trivial, but still doable. * "Typical" GC algorithms here includes copying, mark-sweep, and mark-compact collectors, including generational variants; even collectors having a large object space (typically reclaimed through mark-sweep) should still typically collect small objects often (by using the heuristic described above to trigger GC). Best regards
-- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/
Hi all, I had a look at the issue and could finally pin it (more or less). Codespeed does destroy and recreate the whole element containing the canvas, so it is NOT a but in jqPlot (nor in Codespeed). It is a tricky GC issue where memory taken by big canvases is not properly freed. It probably affects several bowsers with modern JS JITs. Actually, in Firefox 3.6 it is so bad that not even closing the tab frees up the memory for a while! Chrome exhibits similar behaviour, but it does free old canvases after 20-40s. I am sure you will be interested in the fact that Konqueror 4.4 (with KHTML and kjs), which probably doesn't have such an aggressive JIT (or no JIT at all?), does NOT suffer this problem. Every refresh properly frees up the memory of the old canvas. For anyone further interested, Dmtrii did an excellent job filling a new bug for it with simple example code that triggers the problem (I added my observations there): https://bugzilla.mozilla.org/show_bug.cgi?id=621416 So what about speed.pypy.org? It being a multi-browser GC bug, I did the only thing I could do: limit the size of the sometimes ridiculously big horizontal plots ;-) It now has a 2000px limit. It does not completely fix the issue, but it is much more bearable. In the first 2 refreshes (selecting or -deselecting benchmarks), FF even manages to free up the memory. After that it keeps on eating more, but its much better. Thanks Dmitrii for caring! :-D (Please tell me whether it got better for you.) Cheers, Miquel 2010/12/23 Armin Rigo <arigo@tunes.org>:
Hi Dima,
Well, speed.pypy is actually fast in all modern browsers. The problem you are referring to is probably caused by a bug in the javascript plotting library (jqPplot) that is triggered in the comparison view when there are some results with 0 values. It only appears for some plot types, but it is very annoying because it grinds the browser to a halt like you say. Is that what you meant? We are looking into it, and will fix that library if necessary. Cheers, Miquel 2010/12/21 Dima Tisnek <dimaqq@gmail.com>:
Yes it does grind ff 4b7 to an almost halt, little cpu usage, reasonable memory size and constant disk activity for several minutes, very weird... So far, the difference between browsers is so large that I doubt it's dependent on data. Also it seems to tirgger as I remove columns progressively, thus new zero values should not appear, right? I'll invesitage some more... On 21 December 2010 01:06, Miquel Torres <tobami@googlemail.com> wrote:
I sort of figured it out, although I don't have a ready solution. This affects opera 11, stable ff, ff 4.0b7; amazingly it does not affect ie8. Basically memory consumption of the plot appears to be proportional to plot area. Normal plot that you get by default at http://speed.pypy.org/comparison/ costs about 100M of browser memory consumption: opera 130M, stable ff 70M, beta ff 90M at window size 1680x1050; opera 80M, stable ff 55M, beta ff 70M at window size 1024x600; Switching to "horizontal" produces a tall plot of same width and costs about 300~700M of browser memory: opera 720M, stable ff 370M, beta ff 370M at window wize 1680x1050; opera 350M, stable ff 370M, beta ff 370M at window size 1024x600; Suprisingly window size only matters while javascript produces the plot, and not when window is resized, even though plot it resized with the window correctly. This alone is pretty heavy, but doesn't grind the browser. What really grinds is that every time you change a tickbox on the left, a plot is redrawn and another 200M of browser memory is wasted. This is not double buffering, as next change adds yet another 200M or so and so on, it appears that either js doesn't free something, or browser caches or saves the previous page state. As memory consumption grows, at some point browser hits the wall, causes heavy swapping for some time, and I think garbage collection, because practical (but not virtual) memory usage first drops to something like 20~50M and then returns to "normal" 300M. opera ~30 seconds, stable ff about a minute, beta ff several minutes (total system mem 1G, cpu Atom @1.6GHz) Perhaps OS also plays a role in the grind, as it is clearly swapping and swaps out too much? or triggers gc too late and gc has to pull the pages back from disk to perform collection? ie8 doesn't use that much memory, as a matter of fact memory consumption starts little (40M) and changes very little (only +10M) if you go to horizonatal view; the price is very slow rendering, more than 10 seconds per column change. I'll post this on firefox bugzilla too, let's see if someone has a solution. Meanwhile perhaps pypy speed center could start with a smaller plot area (or fewer columns as that makes horizontal plot smaller) to accomodate varying hardware and system mem usage that users might have? The simplest would be a warning next to "horizontal" checkbox. On 21 December 2010 01:06, Miquel Torres <tobami@googlemail.com> wrote:
Mmm, yes, it appears that the memory is not properly freed. I can try experimenting by destroying the dom element and recreating it each time that the plot changes... Btw., instead of continuing here "polluting" the pypy-dev mailing list, we can move to http://groups.google.com/group/codespeed if needed. Thanks for reporting this. Cheers, Miquel 2010/12/22 Dima Tisnek <dimaqq@gmail.com>:
Firefox people point the blame squarely at jqplot: --- Comment #4 from Boris Zbarsky (:bz) <bzbarsky@mit.edu> 2010-12-22 13:43:23 PST --- So what I see this page do, in horizontal mode, is create 17 canvases each of which is width="816" height="3587". That means that each of them has a backing store of 816*3587*4 = 11,707,968 bytes. So that's about 200MB of memory usage right there. I have no idea why they feel a need for 17 huge canvases, but if they want them, that's how much memory they'll take... ( https://bugzilla.mozilla.org/show_bug.cgi?id=620883 ) On 22 December 2010 01:24, Miquel Torres <tobami@googlemail.com> wrote:
Ok, so it seems from reading comments further down that bug report that FF's garbage collection does not immediately free memory for old jqplot canvases. As I said, as soon as I find the time I will experiment with more aggressive DOM element destruction/creation and see if it improves how quickly memory is freed. I do think it will at least partly circumvent jqPlot's bug. 2010/12/22 Dima Tisnek <dimaqq@gmail.com>:
Hi Dima, On Wed, Dec 22, 2010 at 11:21 PM, Dima Tisnek <dimaqq@gmail.com> wrote:
That looks very similar to an issue with PyPy's own GC, in which ctypes.create_string_buffer() returns objects which tend to be GC'ed late. That's because the string buffer object in ctypes appears (to PyPy's GC) to be just a small object, even though it actually references a potentially large piece of raw memory. Similarly, my vague guess about the above is that the 17*11MB of memory are hold by 17 small objects which firefox's GC think don't need to be collected particularly aggressively. A bientôt, Armin.
On Thu, Dec 23, 2010 at 12:14, Armin Rigo <arigo@tunes.org> wrote:
Bigger objects are not collected sooner by typical GC algorithms* - they might just fill the heap more quickly and trigger earlier a GC invocation. the raw-memory allocator to create and update such stats; then the heap-overflow check could consider them to decide whether to trigger GC. The details of the modified heap-overflow check would probably not be entirely trivial, but still doable. * "Typical" GC algorithms here includes copying, mark-sweep, and mark-compact collectors, including generational variants; even collectors having a large object space (typically reclaimed through mark-sweep) should still typically collect small objects often (by using the heuristic described above to trigger GC). Best regards
-- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/
Hi all, I had a look at the issue and could finally pin it (more or less). Codespeed does destroy and recreate the whole element containing the canvas, so it is NOT a but in jqPlot (nor in Codespeed). It is a tricky GC issue where memory taken by big canvases is not properly freed. It probably affects several bowsers with modern JS JITs. Actually, in Firefox 3.6 it is so bad that not even closing the tab frees up the memory for a while! Chrome exhibits similar behaviour, but it does free old canvases after 20-40s. I am sure you will be interested in the fact that Konqueror 4.4 (with KHTML and kjs), which probably doesn't have such an aggressive JIT (or no JIT at all?), does NOT suffer this problem. Every refresh properly frees up the memory of the old canvas. For anyone further interested, Dmtrii did an excellent job filling a new bug for it with simple example code that triggers the problem (I added my observations there): https://bugzilla.mozilla.org/show_bug.cgi?id=621416 So what about speed.pypy.org? It being a multi-browser GC bug, I did the only thing I could do: limit the size of the sometimes ridiculously big horizontal plots ;-) It now has a 2000px limit. It does not completely fix the issue, but it is much more bearable. In the first 2 refreshes (selecting or -deselecting benchmarks), FF even manages to free up the memory. After that it keeps on eating more, but its much better. Thanks Dmitrii for caring! :-D (Please tell me whether it got better for you.) Cheers, Miquel 2010/12/23 Armin Rigo <arigo@tunes.org>:
participants (5)
-
Armin Rigo -
Dima Tisnek -
Massa, Harald Armin -
Miquel Torres -
Paolo Giarrusso