[Python-ideas] cross platform memory usage high water mark for improved gc?

[Aaron Watters]

>
> You're concerned that a new feature may increase how high of a
> threshold you need, yet it could also exceed the "maximum" of your
> adaptive scheme.
>
> I'm not convinced you need that high of a threshold anyway.  I'd like
> to see a benchmark showing how your app performs at different levels.


You are absolutely right that I can set a threshold high enough.
With the default values Python is extremely slow for certain cases.
I'm arguing it should automatically detect when it is being stupid
and attempt to fix it.  In particular I would set the maximum very
high and start at the minimum, which might be near the current
defaults.

For example I get the following in a simple test (python2.6):

> python gctest.py
gc not disabled
elapsed 19.2473409176
> python gctest.py disable
gc disabled
elapsed 4.88715791702

In this case the interpreter is spending 80% of its time trying
to collect non-existent garbage.  Now a newbie who
didn't know to go fiddling with the garbage collector
might just conclude "python is ssslllooowwww" and go
back to using Perl or Ruby or whatever in a case like this.
Maybe the powers that be couldn't care less about it, I don't
know.  (I know newbies can be irritating).

The problem is quadratic also: if I double the limit the
penalty goes up by a factor of 4.

Here is the source:

def test(disable=False, limit=1000000):
    from time import time
    import gc
    if disable:
        gc.disable()
        print "gc disabled"
    else:
        print "gc not disabled"
    now = time()
    D = {}
    for i in range(limit):
        D[ (hex(i), oct(i)) ] = str(i)+repr(i)
    L = [ (y,x) for (x,y) in D.iteritems() ]
    elapsed = time()-now
    print "elapsed", elapsed

if __name__=="__main__":
    import sys
    disable = False
    if "disable" in sys.argv:
        disable = True
    test(disable)


-- Aaron Watters

===
http://www.xfeedme.com/nucular/pydistro.py/go?FREETEXT=being+anal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20080311/27c6ed0b/attachment.html>