Hi Tim et al, I just tested generators and found a memory leak. (Has nothing to do with generators). The following code adds one to the overall refcount and gc cannot reclaim it. def conjoin(gs): def gen(): gs # unbreakable cycle gen # unless one is commented out Should I send a bug report, or is this known? The above holds for Python 2.2.2 upto the current CVS version. ciao - chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
On Mon, Mar 24, 2003 at 01:40:44PM +0100, Christian Tismer wrote:
I just tested generators and found a memory leak. (Has nothing to do with generators). The following code adds one to the overall refcount and gc cannot reclaim it.
def conjoin(gs): def gen(): gs # unbreakable cycle gen # unless one is commented out
With current CVS: >>> gc.collect() 0 [23150 refs] >>> conjoin(1) [23160 refs] >>> conjoin(1) [23170 refs] >>> gc.collect() 8 [23151 refs] >>> conjoin(1) [23161 refs] >>> conjoin(1) [23171 refs] >>> gc.collect() 8 [23151 refs] >>> conjoin(1) [23161 refs] >>> conjoin(1) [23171 refs] >>> gc.collect() 8 [23151 refs] One ref may be leaked the first time gc.collect() is called with garbage (23150 -> 23151). But after that, no more refs are leaked (ref count stays at 23151). Neal
On Mon, Mar 24, 2003 at 08:43:16AM -0500, Neal Norwitz wrote:
One ref may be leaked the first time gc.collect() is called with garbage (23150 -> 23151). But after that, no more refs are leaked (ref count stays at 23151).
If that's true, then running the 'def' block repeatedly will leak references, right? I think from Christian's original message this is what he meant, but I'm not sure. Jeff
On Mon, Mar 24, 2003 at 07:51:00AM -0600, Jeff Epler wrote:
On Mon, Mar 24, 2003 at 08:43:16AM -0500, Neal Norwitz wrote:
One ref may be leaked the first time gc.collect() is called with garbage (23150 -> 23151). But after that, no more refs are leaked (ref count stays at 23151).
If that's true, then running the 'def' block repeatedly will leak references, right? I think from Christian's original message this is what he meant, but I'm not sure.
I misread the original message. Running the 'def' block does indeed leak a reference and collect() has no effect. Similarly: [23154 refs] >>> def conjoin(gs): ... def gen(): ... gs # unbreakable cycle ... gen # unless one is commented out ... [23194 refs] >>> del conjoin [23155 refs] Neal
Jeff Epler wrote:
On Mon, Mar 24, 2003 at 08:43:16AM -0500, Neal Norwitz wrote:
One ref may be leaked the first time gc.collect() is called with garbage (23150 -> 23151). But after that, no more refs are leaked (ref count stays at 23151).
If that's true, then running the 'def' block repeatedly will leak references, right? I think from Christian's original message this is what he meant, but I'm not sure.
Exactly. -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
Neal Norwitz wrote:
On Mon, Mar 24, 2003 at 01:40:44PM +0100, Christian Tismer wrote:
I just tested generators and found a memory leak. (Has nothing to do with generators). The following code adds one to the overall refcount and gc cannot reclaim it.
def conjoin(gs): def gen(): gs # unbreakable cycle gen # unless one is commented out
With current CVS: ...
One ref may be leaked the first time gc.collect() is called with garbage (23150 -> 23151). But after that, no more refs are leaked (ref count stays at 23151).
No, this is not the point. Don't call the function at all, just execute the above code and call gc.collect(). You will see one reference eaten every time you repeat this. ciao - chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
[Christian Tismer]
No, this is not the point. Don't call the function at all, just execute the above code and call gc.collect(). You will see one reference eaten every time you repeat this.
Can you show explicit evidence instead of trying to describe it? Here's what I tried: def one(): def conjoin(gs): def gen(): gs # unbreakable cycle gen # unless one is commented out import sys, gc lastrc = 0 while 1: one() gc.collect() thisrc = sys.gettotalrefcount() print thisrc - lastrc, lastrc = thisrc Running that program under a debug-build CVS Python shows no growth in sys.gettotalrefcount() after the first two iterations. It also displays no process-size growth. IOW, I see no evidence of any flavor of leak. I don't claim that you don't, but I don't know what "just execute the above code ... one reference eaten every time" *means*. It can't mean executing the specific program I pasted in above, because that simply doesn't eat a reference each time.
Tim Peters wrote:
[Christian Tismer]
No, this is not the point. Don't call the function at all, just execute the above code and call gc.collect(). You will see one reference eaten every time you repeat this.
Can you show explicit evidence instead of trying to describe it? Here's what I tried:
Sorry, I had to re-read your message several times until I understood where I wasn't clear: By "execute" I meant exec() this piece of Python code. I actually pasted it in, watching the refcount grow. cheers - chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
OK, *this* program leaks a reference each time around; probably a missing decref in the compiler: source = """\ def conjoin(gs): def gen(): gs # unbreakable cycle gen # unless one is commented out """ def one(): exec source in {} import sys, gc lastrc = 0 while 1: one() gc.collect() thisrc = sys.gettotalrefcount() print thisrc - lastrc, lastrc = thisrc
OK, there's no leaking memory here, but there is a leaking refcount: the refcount on the int 0 keeps going up. The compiler has leaked references to little integers before, but offhand I don't recall the details. ----- old stuff ----- OK, *this* program leaks a reference each time around; probably a missing decref in the compiler: source = """\ def conjoin(gs): def gen(): gs # unbreakable cycle gen # unless one is commented out """ def one(): exec source in {} import sys, gc lastrc = 0 while 1: one() gc.collect() thisrc = sys.gettotalrefcount() print thisrc - lastrc, lastrc = thisrc
Tim Peters <tim.one@comcast.net> writes:
OK, there's no leaking memory here, but there is a leaking refcount: the refcount on the int 0 keeps going up. The compiler has leaked references to little integers before, but offhand I don't recall the details.
This seems to be all it takes: Index: compile.c =================================================================== RCS file: /cvsroot/python/python/dist/src/Python/compile.c,v retrieving revision 2.275 diff -c -C7 -r2.275 compile.c *** compile.c 12 Feb 2003 16:56:51 -0000 2.275 --- compile.c 24 Mar 2003 16:43:28 -0000 *************** *** 4524,4537 **** --- 4564,4578 ---- d = PyDict_New(); for (i = PyList_GET_SIZE(list); --i >= 0; ) { v = PyInt_FromLong(i); if (v == NULL) goto fail; if (PyDict_SetItem(d, PyList_GET_ITEM(list, i), v) < 0) goto fail; + Py_DECREF(v); if (PyDict_DelItem(*cellvars, PyList_GET_ITEM(list, i)) < 0) goto fail; } pos = 0; i = PyList_GET_SIZE(list); Py_DECREF(list); while (PyDict_Next(*cellvars, &pos, &v, &w)) { ... found by the obscure strategy of searching for "PyInt_FromLong" in Python/compile.c ... A quick eyeballing suggests there are a bunch more of these, but only on error returns. Cheers, M. -- ... when all the programmes on all the channels actually were made by actors with cleft pallettes speaking lines by dyslexic writers filmed by blind cameramen instead of merely seeming like that, it somehow made the whole thing more worthwhile. -- HHGTG, Episode 11
On Mon, 24 Mar 2003, Michael Hudson wrote:
Tim Peters <tim.one@comcast.net> writes:
OK, there's no leaking memory here, but there is a leaking refcount: the refcount on the int 0 keeps going up. The compiler has leaked references to little integers before, but offhand I don't recall the details.
This seems to be all it takes:
Your patch isn't a 100% fix, since a reference can still be leaked if the PyDict_SetItem fails. If nobody beats me to it, I can do a validation pass through compile.c and see how many I can squash. -Kevin
Index: compile.c =================================================================== RCS file: /cvsroot/python/python/dist/src/Python/compile.c,v retrieving revision 2.275 diff -c -C7 -r2.275 compile.c *** compile.c 12 Feb 2003 16:56:51 -0000 2.275 --- compile.c 24 Mar 2003 16:43:28 -0000 *************** *** 4524,4537 **** --- 4564,4578 ---- d = PyDict_New(); for (i = PyList_GET_SIZE(list); --i >= 0; ) { v = PyInt_FromLong(i); if (v == NULL) goto fail; if (PyDict_SetItem(d, PyList_GET_ITEM(list, i), v) < 0) goto fail; + Py_DECREF(v); if (PyDict_DelItem(*cellvars, PyList_GET_ITEM(list, i)) < 0) goto fail; } pos = 0; i = PyList_GET_SIZE(list); Py_DECREF(list); while (PyDict_Next(*cellvars, &pos, &v, &w)) {
... found by the obscure strategy of searching for "PyInt_FromLong" in Python/compile.c ...
A quick eyeballing suggests there are a bunch more of these, but only on error returns.
Cheers, M.
-- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com
[Michael Hudson]
This seems to be all it takes:
[Kevin Jacobs]
Your patch isn't a 100% fix, since a reference can still be leaked if the PyDict_SetItem fails.
The patch I checked in paid attention to that.
If nobody beats me to it, I can do a validation pass through compile.c and see how many I can squash. ... A quick eyeballing suggests there are a bunch more of these, but only on error returns.
Possibly. If a dict setitem call fails, it's almost certainly because we're out of memory, and the program is going to die soon regardless. How much pain it's worth to die with a refcount that's not one too large is open to debate <wink>.
Tim Peters <tim.one@comcast.net> writes: [me]
A quick eyeballing suggests there are a bunch more of these, but only on error returns.
Possibly. If a dict setitem call fails, it's almost certainly because we're out of memory, and the program is going to die soon regardless. How much pain it's worth to die with a refcount that's not one too large is open to debate <wink>.
This occurred to me too. I don't think I care enough to do anything about it today. Cheers, M. -- In case you're not a computer person, I should probably point out that "Real Soon Now" is a technical term meaning "sometime before the heat-death of the universe, maybe". -- Scott Fahlman <sef@cs.cmu.edu>
[Tim]
Heh. Here at the PyCon sprint ...
[Skip Montanaro]
So how's it going?
I wouldn't have guessed it, but legions of Indonesian houseboys giving sprinters foot massages really does increase productivity! I'm not so sure about the ubiquitous champagne fountains, though. roughing-it-ly y'rs - tim
participants (7)
-
Christian Tismer -
Jeff Epler -
Kevin Jacobs -
Michael Hudson -
Neal Norwitz -
Skip Montanaro -
Tim Peters