On Mon, Mar 24, 2003 at 07:51:00AM -0600, Jeff Epler wrote:

On Mon, Mar 24, 2003 at 08:43:16AM -0500, Neal Norwitz wrote:

...

One ref may be leaked the first time gc.collect() is called with garbage (23150 -> 23151). But after that, no more refs are leaked (ref count stays at 23151).

If that's true, then running the 'def' block repeatedly will leak references, right? I think from Christian's original message this is what he meant, but I'm not sure.

I misread the original message. Running the 'def' block does indeed leak a reference and collect() has no effect. Similarly: [23154 refs] >>> def conjoin(gs): ... def gen(): ... gs # unbreakable cycle ... gen # unless one is commented out ... [23194 refs] >>> del conjoin [23155 refs] Neal

Neal Norwitz wrote:

On Mon, Mar 24, 2003 at 01:40:44PM +0100, Christian Tismer wrote:

...

I just tested generators and found a memory leak. (Has nothing to do with generators). The following code adds one to the overall refcount and gc cannot reclaim it.

def conjoin(gs): def gen(): gs # unbreakable cycle gen # unless one is commented out

With current CVS: ...

One ref may be leaked the first time gc.collect() is called with garbage (23150 -> 23151). But after that, no more refs are leaked (ref count stays at 23151).

No, this is not the point. Don't call the function at all, just execute the above code and call gc.collect(). You will see one reference eaten every time you repeat this. ciao - chris -- Christian Tismer :^) mailto:tismer@tismer.com Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

Tim Peters wrote:

[Christian Tismer]

...

No, this is not the point. Don't call the function at all, just execute the above code and call gc.collect(). You will see one reference eaten every time you repeat this.

Can you show explicit evidence instead of trying to describe it? Here's what I tried:

Sorry, I had to re-read your message several times until I understood where I wasn't clear: By "execute" I meant exec() this piece of Python code. I actually pasted it in, watching the refcount grow. cheers - chris -- Christian Tismer :^) mailto:tismer@tismer.com Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

OK, there's no leaking memory here, but there is a leaking refcount: the refcount on the int 0 keeps going up. The compiler has leaked references to little integers before, but offhand I don't recall the details. ----- old stuff ----- OK, *this* program leaks a reference each time around; probably a missing decref in the compiler: source = """\ def conjoin(gs): def gen(): gs # unbreakable cycle gen # unless one is commented out """ def one(): exec source in {} import sys, gc lastrc = 0 while 1: one() gc.collect() thisrc = sys.gettotalrefcount() print thisrc - lastrc, lastrc = thisrc

On Mon, 24 Mar 2003, Michael Hudson wrote:

Tim Peters writes:

...

OK, there's no leaking memory here, but there is a leaking refcount: the refcount on the int 0 keeps going up. The compiler has leaked references to little integers before, but offhand I don't recall the details.

This seems to be all it takes:

Your patch isn't a 100% fix, since a reference can still be leaked if the PyDict_SetItem fails. If nobody beats me to it, I can do a validation pass through compile.c and see how many I can squash. -Kevin

Index: compile.c =================================================================== RCS file: /cvsroot/python/python/dist/src/Python/compile.c,v retrieving revision 2.275 diff -c -C7 -r2.275 compile.c *** compile.c 12 Feb 2003 16:56:51 -0000 2.275 --- compile.c 24 Mar 2003 16:43:28 -0000 *************** *** 4524,4537 **** --- 4564,4578 ---- d = PyDict_New(); for (i = PyList_GET_SIZE(list); --i >= 0; ) { v = PyInt_FromLong(i); if (v == NULL) goto fail; if (PyDict_SetItem(d, PyList_GET_ITEM(list, i), v) < 0) goto fail; + Py_DECREF(v); if (PyDict_DelItem(*cellvars, PyList_GET_ITEM(list, i)) < 0) goto fail; } pos = 0; i = PyList_GET_SIZE(list); Py_DECREF(list); while (PyDict_Next(*cellvars, &pos, &v, &w)) {

... found by the obscure strategy of searching for "PyInt_FromLong" in Python/compile.c ...

A quick eyeballing suggests there are a bunch more of these, but only on error returns.

Cheers, M.

-- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com

Tim Peters writes: [me]

...

A quick eyeballing suggests there are a bunch more of these, but only on error returns.

Possibly. If a dict setitem call fails, it's almost certainly because we're out of memory, and the program is going to die soon regardless. How much pain it's worth to die with a refcount that's not one too large is open to debate <wink>.

This occurred to me too. I don't think I care enough to do anything about it today. Cheers, M. -- In case you're not a computer person, I should probably point out that "Real Soon Now" is a technical term meaning "sometime before the heat-death of the universe, maybe". -- Scott Fahlman

Tim> Heh. Here at the PyCon sprint ... So how's it going? S