[pypy-dev] funding/popularity?

Miquel Torres tobami at googlemail.com
Thu Dec 23 08:29:40 CET 2010


Ok, so it seems from reading comments further down that bug report
that FF's garbage collection does not immediately free memory for old
jqplot canvases.

As I said, as soon as I find the time I will experiment with more
aggressive DOM element destruction/creation and see if it improves how
quickly memory is freed. I do think it will at least partly circumvent
jqPlot's bug.


2010/12/22 Dima Tisnek <dimaqq at gmail.com>:
> Firefox people point the blame squarely at jqplot:
>
> --- Comment #4 from Boris Zbarsky (:bz) <bzbarsky at mit.edu> 2010-12-22
> 13:43:23 PST ---
> So what I see this page do, in horizontal mode, is create 17 canvases each of
> which is width="816" height="3587".  That means that each of them has a backing
> store of 816*3587*4 = 11,707,968 bytes.  So that's about 200MB of memory usage
> right there.
>
> I have no idea why they feel a need for 17 huge canvases, but if they want
> them, that's how much memory they'll take...
>
> ( https://bugzilla.mozilla.org/show_bug.cgi?id=620883 )
>
> On 22 December 2010 01:24, Miquel Torres <tobami at googlemail.com> wrote:
>> Mmm, yes, it appears that the memory is not properly freed. I can try
>> experimenting by destroying the dom element and recreating it each
>> time that the plot changes...
>>
>> Btw., instead of continuing here "polluting" the pypy-dev mailing
>> list, we can move to http://groups.google.com/group/codespeed if
>> needed.
>>
>> Thanks for reporting this.
>>
>> Cheers,
>> Miquel
>>
>>
>> 2010/12/22 Dima Tisnek <dimaqq at gmail.com>:
>>> I sort of figured it out, although I don't have a ready solution.
>>>
>>> This affects opera 11, stable ff, ff 4.0b7; amazingly it does not affect ie8.
>>> Basically memory consumption of the plot appears to be proportional to
>>> plot area.
>>> Normal plot that you get by default at
>>> http://speed.pypy.org/comparison/ costs about 100M of browser memory
>>> consumption:
>>> opera 130M, stable ff 70M, beta ff 90M at window size 1680x1050;
>>> opera  80M, stable ff 55M, beta ff 70M at window size 1024x600;
>>> Switching to "horizontal" produces a tall plot of same width and costs
>>> about 300~700M of browser memory:
>>> opera 720M, stable ff 370M, beta ff 370M at window wize 1680x1050;
>>> opera 350M, stable ff 370M, beta ff 370M at window size 1024x600;
>>>
>>> Suprisingly window size only matters while javascript produces the
>>> plot, and not when window is resized, even though plot it resized with
>>> the window correctly.
>>>
>>> This alone is pretty heavy, but doesn't grind the browser.
>>> What really grinds is that every time you change a tickbox on the
>>> left, a plot is redrawn and another 200M of browser memory is wasted.
>>> This is not double buffering, as next change adds yet another 200M or
>>> so and so on, it appears that either js doesn't free something, or
>>> browser caches or saves the previous page state.
>>>
>>> As memory consumption grows, at some point browser hits the wall,
>>> causes heavy swapping for some time, and I think garbage collection,
>>> because practical (but not virtual) memory usage first drops to
>>> something like 20~50M and then returns to "normal" 300M.
>>> opera ~30 seconds, stable ff about a minute, beta ff several minutes
>>> (total system mem 1G, cpu Atom @1.6GHz)
>>>
>>> Perhaps OS also plays a role in the grind, as it is clearly swapping
>>> and swaps out too much? or triggers gc too late and gc has to pull the
>>> pages back from disk to perform collection?
>>>
>>> ie8 doesn't use that much memory, as a matter of fact memory
>>> consumption starts little (40M) and changes very little (only +10M) if
>>> you go to horizonatal view; the price is very slow rendering, more
>>> than 10 seconds per column change.
>>>
>>> I'll post this on firefox bugzilla too, let's see if someone has a solution.
>>>
>>> Meanwhile perhaps pypy speed center could start with a smaller plot
>>> area (or fewer columns as that makes horizontal plot smaller) to
>>> accomodate varying hardware and system mem usage that users might
>>> have?
>>> The simplest would be a warning next to "horizontal" checkbox.
>>>
>>> On 21 December 2010 01:06, Miquel Torres <tobami at googlemail.com> wrote:
>>>> Hi Dima,
>>>>
>>>>> another temp problem with speed pypy is that it's terrubly slow in ff
>>>>> beta. it also occasionally grinds in stable ff and opera, but I guess
>>>>> this can be forgiven for the sake of simplicity / developer effort.
>>>>
>>>> Well, speed.pypy is actually fast in all modern browsers. The problem
>>>> you are referring to is probably caused by a bug in the javascript
>>>> plotting library (jqPplot) that is triggered in the comparison view
>>>> when there are some results with 0 values. It only appears for some
>>>> plot types, but it is very annoying because it grinds the browser to a
>>>> halt like you say. Is that what you meant?
>>>>
>>>> We are looking into it, and will fix that library if necessary.
>>>>
>>>> Cheers,
>>>> Miquel
>>>>
>>>>
>>>> 2010/12/21 Dima Tisnek <dimaqq at gmail.com>:
>>>>> On 20 December 2010 19:21, William ML Leslie
>>>>> <william.leslie.ttg at gmail.com> wrote:
>>>>>> On 21 December 2010 11:59, Dima Tisnek <dimaqq at gmail.com> wrote:
>>>>>>> More visibility for performance achievements would do
>>>>>>> good too.
>>>>>>
>>>>>> Where are pypy's performance achievements *not* visible, but should be?
>>>>>
>>>>> for example http://shootout.alioth.debian.org/
>>>>> doesn't say which pypy version is used, what options, doesn't have
>>>>> performance figures for multithreaded/multicore
>>>>>
>>>>> also benchmarks are kinda small, most of them are not docuemented, and
>>>>> I couldn't find any info if the same python code was used for cpython
>>>>> and pypy (both shootout and speed pypy) or interpreter-specific
>>>>> verions were used, that is each tweaked for best performance given the
>>>>> known tradeoffs for each implementation.further the most standard
>>>>> benchmarks, pystone, etc. completely ignore the fact that real world
>>>>> programs are large and only a few small paths are execured often.
>>>>>
>>>>> another temp problem with speed pypy is that it's terrubly slow in ff
>>>>> beta. it also occasionally grinds in stable ff and opera, but I guess
>>>>> this can be forgiven for the sake of simplicity / developer effort.
>>>>>
>>>>> if you google for 'python performance' you don't get a single link to
>>>>> pypy on the first page, as a matter of fact, codespeak is poorly
>>>>> indexed, it took me quite some time to get some of my questions
>>>>> answered with a search. also if you look up 'pypy gc' you get a page
>>>>> on codespeak, but to interpret what the data actually means is so far
>>>>> beyond me.
>>>>>
>>>>> a good overview is found in the mainling list
>>>>> http://codespeak.net/pipermail/pypy-dev/2010q1/005757.html then again
>>>>> slowspitfire and spambayes bits are outdated by now.
>>>>>
>>>>> the definitive good thing about pypy is that it's much easier to find
>>>>> out about its inner workings than that of cpython!
>>>>>
>>>>> hopefully a bit more of end-user stuff get known.
>>>>> let's call it pypy public outreach (feature request)
>>>>>
>>>>>>
>>>>>>> Sidetracking... one day when pypy jit/gc/etc are all worked out, how
>>>>>>> hard would it be to use same techniques and most of backends for some
>>>>>>> unrelated language that doesn't have jit yet, e.g. php?
>>>>>>
>>>>>> You know that pypy already has implementations of other languages,
>>>>>> right - Scheme, Javascript, Prolog, IO and smallTalk? They aren't
>>>>>> integrated with the translated pypy-c, but they show that it is not
>>>>>> too difficult to write a runtime for any dynamic language you choose.
>>>>>
>>>>> Oh I didn't know there were so many, and I mistakenly though that js
>>>>> was a target, not implmented langauge. In any case I read somewhere
>>>>> that js support was retired...
>>>>>
>>>>>>
>>>>>>> And how hard
>>>>>>> would it be to marry two dynamic languages, so that modules from one
>>>>>>> could be used in the other? Or that modules written in rpython could
>>>>>>> be used in several langs?
>>>>>>
>>>>>> It's in the "interesting problems" bucket, and the effort required
>>>>>> depends on the level of integration between languages you want.  There
>>>>>> are several projects already attempting to do decent integration
>>>>>> between several languages, besides the approach used on the JVM, there
>>>>>> are also GNU Guile, Racket, and Parrot, among others.  It might be
>>>>>> worth waiting to see how these different projects pan out, before
>>>>>> spending a bunch of effort just to be an also-ran in the
>>>>>> multi-language runtime market.
>>>>>>
>>>>>> However, implementing more languages in rpython has the side-effect of
>>>>>> propagating the l * o * p problem: it introduces more and more
>>>>>> implementations that then have to be maintained, so good
>>>>>> cross-language integration probably belongs /outside/ pypy itself, so
>>>>>> existing runtimes can hook into it.
>>>>>
>>>>> Makes perfect sense, after all any given other language hardly has the
>>>>> same data model as python.
>>>>>
>>>>>>
>>>>>> But it would be an interesting experiment (to marry the various
>>>>>> interpreters pypy ships with), if you wanted to try it.
>>>>>>
>>>>>> My two cents.
>>>>>>
>>>>>> --
>>>>>> William Leslie
>>>>>>
>>>>> _______________________________________________
>>>>> pypy-dev at codespeak.net
>>>>> http://codespeak.net/mailman/listinfo/pypy-dev
>>>>
>>>
>>
>



More information about the Pypy-dev mailing list