[Twisted-Python] help with refcounts and memleaks
Hello,
I was just shoked today when I noticed this:
------------------- import sys
class A(object): y = None def x(self): pass def __del__(self): print 'deleted'
a = A() print sys.getrefcount(a) if 1: a.y = a.x print sys.getrefcount(a) del a -------------------
I understood the cross references memleaks well, like "x.y = y; y.x= x; del x,y", but I didn't imagine that "a.y = a.x" would be enough to generate a memleak. "a.y = a.x" isn't referencing another structure, it's referencing itself only. Infact if I do this the memleak goes away!!
------------------- import sys
class A(object): def x(self): pass y = x def __del__(self): print 'deleted'
a = A() print sys.getrefcount(a) a.x() a.y() print a.x, a.y del a -------------------
Now the fact a static field doesn't generate a reference but a dynamic one does is quite confusing to me and it also opened a can of worms in my code. I can handle that now that I know about it, but I wonder what people recommends to solve memleaks of this kind.
I'd also like to know how other languages like ruby and java behave in terms of self-references of objects. Can't the language understand it's a self reference, and in turn it's the same as an integer or a string, like it already does when the member is initialized statically?
Infact can't the language be smart enough to even understand when two cross referenced objects lost visibility from all points of view, and drop both objects even if they hold a reference on each other? I understand this is a lot more complicated but wouldn't it be possible in theory? What does the garbage collection of other languages like ruby and java, the same as python or more advanced?
So far my python programs never really cared to released memory (so my not full understanding of python refcounts wasn't a problem), but now since I'm dealing with a server I must make sure that the "proto" is released after a loseConnection invocation. So I must cleanup all cross and self! references in loseConnection and use weakrefs where needed.
Now those structures that I'm leaking (like the protocol object) are so tiny that there's no chance that I could ever notice the memleak in real life, so I had to add debugging code to trap memleaks. You can imagine my server code like this:
class cpushare_protocol(Int32StringReceiver): def connectionMade(self): [..] self.hard_handlers = { PROTO_SECCOMP : self.seccomp_handler, PROTO_LOG : self.log_handler, } [..] def log_handler(self, string): [..] def seccomp_handler(self, string): [..] def __del__(self): print 'protocol deleted' def connectionLost(self, reason): [..] # memleaks del self.hard_handlers print 'protocol refcount:', sys.getrefcount(self) #assert sys.getrefcount(self) == 4
For things like hard_handlers (that are self-referencing callbacks) I can't even use the weakref.WeakValueDictionary, because it wouldn't hold itself, the object gets released immediately. So the only chance I have to release the memory of the protocol object when the connection is dropped, is to do an explicit del self.hard_handlers in loseConnection.
I wonder what other twisted developers do to avoid those troubles. Perhaps I shouldn't use self referencing callbacks to hold the state machine, and do like the smpt protocol that does this:
def lookupMethod(self, command): return getattr(self, 'do_' + command.upper(), None)
basically working with strings instead of pointers. Or I can simply make sure to cleanup all structures when I stop using them (like with the del self.hard_handlers above), but then I'll lose part of the automatic garbage collection features of python. I really want garbage collection or I could have written this in C++ if I'm forced to cleanup by hand.
Help is appreciated, thanks!
PS. Merry Christmas and Happy New 2006!
On Mon, 26 Dec 2005 17:07:35 +0100, Andrea Arcangeli andrea@cpushare.com wrote:
Hello,
Hey,
This is really a question for a Python list. However, I've attached some comments below.
I was just shoked today when I noticed this:
import sys
class A(object): y = None def x(self): pass def __del__(self): print 'deleted'
a = A() print sys.getrefcount(a) if 1: a.y = a.x print sys.getrefcount(a) del a
I understood the cross references memleaks well, like "x.y = y; y.x= x; del x,y", but I didn't imagine that "a.y = a.x" would be enough to generate a memleak. "a.y = a.x" isn't referencing another structure, it's referencing itself only. Infact if I do this the memleak goes away!!
I'm not sure how far you've gotten into this, but here's the basic explanation: "a.x" gives you a "bound method instance"; since you might do anything at all with the object it evaluates to, it wraps up a reference to the object "a" references, so it knows what object to use as "self"; this has the effect of increasing the reference count of "a", but it doesn't actually leak any memory.
Of course, in creating a cycle which contains an object with an implementation of __del__, you have created a leak, since Python's GC cannot collect that kind of graph.
Hopefully the __del__ implementation is only included as an aid to understanding what is going on, and you don't actually need it in any of your actual applications. Once removed, the cycle will be collectable by Python.
Another strategy is to periodically examine gc.garbage and manually break cycles. This way, if you do have any __del__ implementations, they will no longer be part of a cycle, and Python will again be able to collect these objects.
import sys
class A(object): def x(self): pass y = x def __del__(self): print 'deleted'
a = A() print sys.getrefcount(a) a.x() a.y() print a.x, a.y del a
Now the fact a static field doesn't generate a reference but a dynamic one does is quite confusing to me and it also opened a can of worms in my code. I can handle that now that I know about it, but I wonder what people recommends to solve memleaks of this kind.
This is an interesting case. Python does not do what you probably expect here. When you define a class with methods, Python does not actually create any method objects! It is the actual attribute lookup on an instance which creates the method object. You can see this in the following example:
>>> class X: ... def y(self): pass ... >>> a = X() >>> a.y is a.y False >>> a.y is X.__dict__['y'] False >>> X.__dict__['y'] is X.__dict__['y'] True >>>
So when you added "y" to your class "A", Python didn't care, because there aren't even any method objects until you access an attribute which is bound to a function. Continuing the above example:
>>> sys.getrefcount(a) 2 >>> L = [a.y, a.y, a.y, a.y] >>> sys.getrefcount(a) 6 >>>
I'd also like to know how other languages like ruby and java behave in terms of self-references of objects. Can't the language understand it's a self reference, and in turn it's the same as an integer or a string, like it already does when the member is initialized statically?
I don't know Ruby well enough to comment directly, but I believe Ruby's GC is much simpler (and less capable) than Python's. Java doesn't have bound methods (or unbound methods, or heck, functions): the obvious way in which you would construct them on top of the primitives the language does offer seems to me as though it would introduce the same "problem" you are seeing in Python, but that may just be due to the influence Python has had on my thinking.
Infact can't the language be smart enough to even understand when two cross referenced objects lost visibility from all points of view, and drop both objects even if they hold a reference on each other? I understand this is a lot more complicated but wouldn't it be possible in theory? What does the garbage collection of other languages like ruby and java, the same as python or more advanced?
When you have "two cross referenced objects", that's a cycle, and Python will indeed clean it up. The only exception is if there is a __del__ implementation, as I mentioned above. This is a general problem with garbage collection. If you have two objects which refer to each other and which each wish to perform some finalization, which finalizer do you call first?
So far my python programs never really cared to released memory (so my not full understanding of python refcounts wasn't a problem), but now since I'm dealing with a server I must make sure that the "proto" is released after a loseConnection invocation. So I must cleanup all cross and self! references in loseConnection and use weakrefs where needed.
Now those structures that I'm leaking (like the protocol object) are so tiny that there's no chance that I could ever notice the memleak in real life, so I had to add debugging code to trap memleaks. You can imagine my server code like this:
You might be surprised :) These things tend to build up, if your process is long-running.
class cpushare_protocol(Int32StringReceiver): def connectionMade(self): [..] self.hard_handlers = { PROTO_SECCOMP : self.seccomp_handler, PROTO_LOG : self.log_handler, } [..] def log_handler(self, string): [..] def seccomp_handler(self, string): [..] def __del__(self): print 'protocol deleted' def connectionLost(self, reason): [..] # memleaks del self.hard_handlers print 'protocol refcount:', sys.getrefcount(self) #assert sys.getrefcount(self) == 4
For things like hard_handlers (that are self-referencing callbacks) I can't even use the weakref.WeakValueDictionary, because it wouldn't hold itself, the object gets released immediately. So the only chance I have to release the memory of the protocol object when the connection is dropped, is to do an explicit del self.hard_handlers in loseConnection.
I wonder what other twisted developers do to avoid those troubles. Perhaps I shouldn't use self referencing callbacks to hold the state machine, and do like the smpt protocol that does this:
def lookupMethod(self, command): return getattr(self, 'do_' + command.upper(), None)
basically working with strings instead of pointers. Or I can simply make sure to cleanup all structures when I stop using them (like with the del self.hard_handlers above), but then I'll lose part of the automatic garbage collection features of python. I really want garbage collection or I could have written this in C++ if I'm forced to cleanup by hand.
(You can probably guess what I'm going to say here. ;) In general, I avoid implementing __del__. My programs may end up with cycles, but as long as I don't have __del__, Python can figure out how to free the objects. Note that it does sometimes take it a while (and this has implications for peak memory usage which may be important to you), but if you find a case that it doesn't handle, then you've probably found a bug in the GC that python-dev will fix.
Hope this helps, and happy holidays,
Jean-Paul
On Mon, Dec 26, 2005 at 12:21:29PM -0500, Jean-Paul Calderone wrote:
Of course, in creating a cycle which contains an object with an implementation of __del__, you have created a leak, since Python's GC cannot collect that kind of graph.
Ah this explains many things. I didn't realize that having a __del__ callback made any difference from a garbage collection point of view, so while trying to fix memleaks I probably added them ;).
Sorry for posting it here and not a python list, but my basic problem is to make sure the "protocol" object is being collected away, and the protocol object is a very twisted thing, so I thought it would be at on topic here since everyone of us needs the protocol object garbage collected properly. Now it turned out more a language thing than I thought originally...
Ok, going back to how this thing started. I happened to allocate 50M of ram somehow attached to a protocol object, and then I noticed that the reconnectingclientfactory was leaking memory after a disconnect/reconnect event. Every time I restarted the server, 50M were added to the RSS of the task. That was definitely a memleak, and I never had a __del__ method. Then I started adding debugging aid to figure out what was going wrong. By removing the self and cross references the memleak was fixed in the client. So then I figured out the same self-references were in the server as well, and I added more debugging in the server as well. That lead me in the current situation. So something was definitely going wrong w.r.t. memleaks even before I started messing with the __del__ methods.
But I'm very relieived to know that python gets it right if __del__ isn't implemented.
Hopefully the __del__ implementation is only included as an aid to understanding what is going on, and you don't actually need it in any of your actual applications. Once removed, the cycle will be collectable by Python.
Correct, it was only an aid, it didn't exist until today.
When you have "two cross referenced objects", that's a cycle, and Python will indeed clean it up. The only exception is if there is a
Well, I never cared about cyclic references until today, because I thought python would understand it automatically like I think it's possible infact.
But then while trying to debug the 50M leak in the client at every server restart (so very visible), I quickly into this:
http://www.nightmare.com/medusa/memory-leaks.html
class thing: pass
a = thing() b = thing() a.other = b b.other = a
del a del b
Code like above is very common in my twisted based server. Note that there's no __del__ method in the class "thing". So what you say seems in disagreement with the above url. Perhaps I got bitten by the common mistake "I found it on the internet so it must be true"... I really hope you're the one being right, my code was all written with your ideas in mind but that seems to collide strong with the above url. I guess I should have checked the date, it's from 99, perhaps it has been true a long time ago?
__del__ implementation, as I mentioned above. This is a general problem with garbage collection. If you have two objects which refer to each other and which each wish to perform some finalization, which finalizer do you call first?
Why would it matter which one you call first? Random no? Better to call it random than to leak memory, no? At least python should spawn a gigantic warning that there's a cross reference leaking, instead of silenty not calling __del__.
You might be surprised :) These things tend to build up, if your process is long-running.
I think you're right there was no memleak generated by self/cross cyclic references, but then the load is pretty low at the moment so I could have overlooked it. I periodically monitor the rss of all tasks. I never had problems before noticing the reconnectingclientfactory memleak (which btw I can't reproduce anymore after removing the cross references).
(You can probably guess what I'm going to say here. ;) In general, I avoid implementing __del__. My programs may end up with cycles, but as long as I don't have __del__, Python can figure out how to free the objects. Note that it does sometimes take it a while (and this has implications for peak memory usage which may be important to you), but if you find a case that it doesn't handle, then you've probably found a bug in the GC that python-dev will fix.
Hope this helps, and happy holidays,
Thanks a lot, things looks much better now, I'm relieved that python can figure out how to free objects, I always thought it was able to do so infact ;). Happy holidays to you too.
So, I'll backout all my latest changes, and I'll try to find the real cause of the reconnectingclientfactory memleak which definitely happened even though there was no __del__ method implemented.
On Tue, 27 Dec 2005 01:18:33 +0100, Andrea Arcangeli andrea@cpushare.com wrote:
But then while trying to debug the 50M leak in the client at every server restart (so very visible), I quickly into this:
http://www.nightmare.com/medusa/memory-leaks.html
class thing: pass
a = thing() b = thing() a.other = b b.other = a
del a del b
Code like above is very common in my twisted based server. Note that there's no __del__ method in the class "thing". So what you say seems in disagreement with the above url. Perhaps I got bitten by the common mistake "I found it on the internet so it must be true"...
the document at that URL is six years old, and so dates from a time when python's garbage collector couldn't properly deal with plain cyclical references, AFAICT.
-- Moe Aboulkheir
On Tue, Dec 27, 2005 at 02:39:30AM +0200, Moe Aboulkheir wrote:
the document at that URL is six years old, and so dates from a time when python's garbage collector couldn't properly deal with plain cyclical references, AFAICT.
cool, thanks for confirming this. I admit I didn't pay attention to the date before reading your replies here... also because I could reproduce the problem here due the __del__ heisenbug in my testcases.
I'm just sending an email to the owner of the obsolete info, that page scored at the top of my google search for python memleaks and it created me lots of unnecessary confusion ;). But after all it's good because now I learnt about the __del__ effect on the gc. Sorry for the noise.
The only remaining thing to understand is why by default the collect() method is never invoked and I've to invoke it explicitly to avoid a gigantic leak (see my previous email with the fix for my reconnectingclientfactory pratical memleak that made me look into this).
On Tue, Dec 27, 2005 at 01:18:33AM +0100, Andrea Arcangeli wrote:
So, I'll backout all my latest changes, and I'll try to find the real
Now after understanding the __del__ heisenbug in my testcases, and a 1000 lines backout (dropping all weakrefs and hacks I did to try to remove those pratical memleaks) here my findings on the real life reconnectingclientfactory memleak that made me look into this in the first place: It wasn't a memleak in theory, it was only in practice... gc.collect() seems not invoked frequently enough.
Here it is a ps v of my reconnectingclientfactory with an artificial disconnect from the server at every second (as said before about 50M were attached to every new protocol instance).
while :; do ps v |grep [c]pushare-0; sleep 1; done 20113 pts/6 SNl+ 0:00 0 2 94453 51492 3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 RNl+ 0:00 0 2 115017 70120 4.5 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:00 0 2 135513 92528 6.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:00 0 2 135513 92528 6.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:00 0 2 135513 92528 6.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:01 0 2 176497 133512 8.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:01 0 2 176497 133512 8.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:01 0 2 176497 133512 8.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:01 0 2 217481 174496 11.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:01 0 2 217481 174496 11.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:01 0 2 217481 174496 11.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 RNl+ 0:01 0 2 278957 216252 14.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:02 0 2 94529 51568 3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:02 0 2 94529 51568 3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 RNl+ 0:02 0 2 125269 82252 5.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:02 0 2 135513 92584 6.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:02 0 2 135513 92584 6.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:02 0 2 135513 92584 6.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:02 0 2 176497 133580 8.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:02 0 2 176497 133580 8.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:02 0 2 176497 133596 8.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:02 0 2 135513 92584 6.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:02 0 2 135513 92584 6.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:02 0 2 135513 92584 6.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:02 0 2 176497 133580 8.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:02 0 2 176497 133580 8.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:02 0 2 176497 133596 8.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:03 0 2 217481 174588 11.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:03 0 2 217481 174588 11.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:03 0 2 217481 174596 11.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:03 0 2 258465 215580 14.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:03 0 2 258465 215580 14.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:03 0 2 258465 215580 14.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:03 0 2 299449 256584 16.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:03 0 2 299449 256584 16.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:03 0 2 299449 256584 16.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 RNl+ 0:04 0 2 319937 275096 17.8 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:04 0 2 340433 297568 19.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:04 0 2 340433 297568 19.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:04 0 2 340433 297584 19.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:04 0 2 381417 338568 22.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:04 0 2 381417 338568 22.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:04 0 2 381417 338568 22.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 RNl+ 0:05 0 2 412221 369312 24.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:05 0 2 422469 379560 24.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:04 0 2 381417 338568 22.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 RNl+ 0:05 0 2 412221 369312 24.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:05 0 2 422469 379560 24.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:05 0 2 422469 379560 24.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:05 0 2 422537 379572 24.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:05 0 2 463457 420556 27.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:05 0 2 463457 420556 27.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:05 0 2 463457 420556 27.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 RNl+ 0:06 0 2 524933 472712 30.7 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:06 0 2 504441 461548 30.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:06 0 2 504441 461548 30.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 RNl+ 0:06 0 2 524929 482044 31.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:07 0 2 545425 502540 32.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:07 0 2 545425 502540 32.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:07 0 2 545425 502540 32.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 RNl+ 0:07 0 2 586413 534096 34.7 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:07 0 2 586409 543532 35.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:07 0 2 586409 543532 35.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:07 0 2 586497 543540 35.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:08 0 2 627417 584524 38.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:08 0 2 627417 584524 38.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:08 0 2 627417 584524 38.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 RNl+ 0:08 0 2 647905 605016 39.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:08 0 2 668401 625512 40.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:08 0 2 668401 625512 40.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:08 0 2 668401 625512 40.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 RNl+ 0:09 0 2 709389 663952 43.1 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:09 0 2 709385 666500 43.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:09 0 2 709385 666500 43.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:09 0 2 709385 666500 43.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 RNl+ 0:09 0 2 740121 697240 45.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:10 0 2 750369 707488 45.9 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:10 0 2 750369 707488 45.9 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:10 0 2 750369 707488 45.9 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 RNl+ 0:10 0 2 791357 744356 48.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:10 0 2 791353 748248 48.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:10 0 2 791353 748248 48.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 RNl+ 0:10 0 2 791353 748252 48.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 RNl+ 0:11 0 2 842581 790452 51.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:11 0 2 832337 788608 51.2 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:11 0 2 832337 788608 51.2 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:11 0 2 832337 788608 51.2 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:11 0 2 863077 819348 53.2 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:12 0 2 873321 829596 53.9 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:12 0 2 873321 829596 53.9 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:11 0 2 832337 788608 51.2 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:11 0 2 832337 788608 51.2 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:11 0 2 863077 819348 53.2 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:12 0 2 873321 829596 53.9 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:12 0 2 873321 829596 53.9 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:12 0 2 873321 829596 53.9 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:12 0 2 904061 860200 55.9 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 DNl+ 0:12 0 2 924549 874220 56.8 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 DNl+ 0:12 0 2 934797 883008 57.4 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 DNl+ 0:12 0 2 934797 890176 57.8 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:12 0 2 914305 869848 56.5 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 SNl+ 0:12 0 2 914305 869848 56.5 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 RNl+ 0:12 0 2 914305 869848 56.5 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 DNl+ 0:13 0 2 945041 900420 58.5 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 DNl+ 0:13 0 2 965533 916084 59.5 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 20113 pts/6 DNl+ 0:13 0 2 975781 925724 60.1 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
At the end the system was heavy into swap: huge memelak. But it shouldn't have happened, I know how to write code not to generate memleaks, the only thing left were cyclical references and self references, but there was no visibility at all of the objects that were supposed to be freed.
Now see the right fix that made the original gigantic memleak go away!
Index: cpushare/proto.py =================================================================== RCS file: /home/andrea/crypto/cvs/cpushare/client/cpushare/cpushare/proto.py,v retrieving revision 1.62 diff -u -p -r1.62 proto.py --- cpushare/proto.py 27 Dec 2005 00:43:34 -0000 1.62 +++ cpushare/proto.py 27 Dec 2005 00:50:13 -0000 @@ -206,6 +206,8 @@ class cpushare_factory(ReconnectingClien
def buildProtocol(self, addr): self.resetDelay() + import gc + gc.collect() return ReconnectingClientFactory.buildProtocol(self, addr)
def clientConnectionLost(self, connector, reason):
And now see the output of the same command with the same reconnectingclientfactory being restarted once every second, but with the above two liner fix applied.
21219 pts/6 RNl+ 0:00 0 2 73773 30924 2.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 21219 pts/6 SNl+ 0:00 0 2 94233 51428 3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 21219 pts/6 SNl+ 0:00 0 2 94233 51428 3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 21219 pts/6 SNl+ 0:00 0 2 94233 51428 3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 21219 pts/6 RNl+ 0:01 0 2 83985 41188 2.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 21219 pts/6 SNl+ 0:01 0 2 94233 51436 3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 21219 pts/6 SNl+ 0:01 0 2 94233 51436 3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 21219 pts/6 RNl+ 0:01 0 2 73737 30944 2.0 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 21219 pts/6 SNl+ 0:01 0 2 94233 51440 3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 21219 pts/6 SNl+ 0:01 0 2 94233 51440 3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 21219 pts/6 RNl+ 0:01 0 2 63497 17496 1.1 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 21219 pts/6 SNl+ 0:01 0 2 94233 51440 3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 21219 pts/6 SNl+ 0:01 0 2 94233 51440 3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 21219 pts/6 RNl+ 0:02 0 2 53249 10464 0.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 21219 pts/6 SNl+ 0:02 0 2 94233 51448 3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 21219 pts/6 SNl+ 0:02 0 2 94233 51448 3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 21219 pts/6 SNl+ 0:02 0 2 53249 10468 0.6 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 21219 pts/6 SNl+ 0:02 0 2 94233 51452 3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 21219 pts/6 SNl+ 0:02 0 2 94233 51452 3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 21219 pts/6 RNl+ 0:02 0 2 63497 15368 0.9 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 21219 pts/6 SNl+ 0:03 0 2 94233 51476 3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 21219 pts/6 SNl+ 0:03 0 2 94233 51476 3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 21219 pts/6 SNl+ 0:03 0 2 94233 51476 3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 21219 pts/6 SNl+ 0:03 0 2 94233 51476 3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 21219 pts/6 SNl+ 0:03 0 2 94233 51476 3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 21219 pts/6 SNl+ 0:03 0 2 94233 51476 3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 21219 pts/6 SNl+ 0:03 0 2 94233 51480 3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 21219 pts/6 SNl+ 0:03 0 2 94233 51480 3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py 21219 pts/6 SNl+ 0:03 0 2 94233 51480 3.3 python /home/andrea/bin/x86_64/python/bin/twistd --pidfile cpushare-0.pid -noy cpushare/tap.py
Now the RSS of the task never exceeds 50M, this is perfectly correct and the expected behaviour since the first place.
The memleak disappeared completely. So it wasn't a memleak, nor a bug in my code, but it was only the lack of garbage collection invocation! Of course my previous attempts of removing the cyclical references made the problem go away artificially but it made my code a total mess (similar to c++ style), because then the refcount of the protocol was forced to go down to 0 and so the memory was released synchronously.
So the question now, is how frequently is the garbage collection invoked, and why do I need to invoke it by myself to avoid a gigantic "pratical" memleaks?
Of course now I'll keep an explicit gc.collect() into the reconnecting handler. Should I add an explicit timer rearming itself and invoking the garbage collection periodically in the server too?
I'm really relieved the article dating back to 99 was totally obsolete since my code has always been written for 2005 python garbage collection ;)
Thanks a lot for the quick help!
On Dec 26, 2005, at 8:02 PM, Andrea Arcangeli wrote:
On Tue, Dec 27, 2005 at 01:18:33AM +0100, Andrea Arcangeli wrote:
So, I'll backout all my latest changes, and I'll try to find the real
Now after understanding the __del__ heisenbug in my testcases, and a 1000 lines backout (dropping all weakrefs and hacks I did to try to remove those pratical memleaks) here my findings on the real life reconnectingclientfactory memleak that made me look into this in the first place: It wasn't a memleak in theory, it was only in practice... gc.collect() seems not invoked frequently enough.
Take a look at: http://docs.python.org/lib/module-gc.html
You can adjust the threshold for each GC generation to suit your application better.
-bob
On Mon, Dec 26, 2005 at 08:22:24PM -0500, Bob Ippolito wrote:
Take a look at: http://docs.python.org/lib/module-gc.html
You can adjust the threshold for each GC generation to suit your application better.
gc.set_threshold(1, 1, 1) fixed it too, any other setting didn't (sometime it increases to 100m). If (1,1,1) is as good as it can get, I'll keep doing the gc.collect() during the factory restart since the 50m allocation only happens after the connectionMade callback and never again in the context of any given protocol. So the gc.collect() seems the optimal fix to me for now.
I wish the size of the task would be taken into account any way in the threshold tunables. I'd like to say "gc.set_mem_threshold({30*1024*1024 : (10, 10, 10), 50*1024*1024 : (1, 1, 1),})", which mean it's a dynamic threshold. It should be possible to implement this in O(1), the interpreter should easily track how much anonymous memory it has allocated with malloc at any given time. The more anonymous memory, the less generations it should wait. It could be a linear function too. However for now I'm happy with the gc.collect().
Thanks a lot for all help.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Wow, Andrea Arcangeli, of Linux Kernel fame. What an honor :).
import gcgc.collect()
Haha. I was just writing you about manual garbage collector invocations.
Manual collections can be costly, nevertheless. Since the manual collector is only needed to free cyclic references, I guess that Twisted people would rather prefer a patch to break cyclic references in Twisted code. For example, using "__del__" methods or "weakref".
You can easily see what objects and references keep the cycles alive using "gc.garbage" and "gc.get_referrers()". Usually a single "a.value=None" can do miracles :-). Weakref can be very very helpful, also.
http://docs.python.org/lib/module-gc.html http://docs.python.org/lib/module-weakref.html
Merry Christmas and happy 2006, by the way.
- -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@argo.es http://www.argo.es/~jcea/ _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ _/_/_/_/_/ PGP Key Available at KeyServ _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz
On 12/27/05, Jesus Cea jcea@argo.es wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Wow, Andrea Arcangeli, of Linux Kernel fame. What an honor :).
import gcgc.collect()Haha. I was just writing you about manual garbage collector invocations.
Manual collections can be costly, nevertheless. Since the manual collector is only needed to free cyclic references, I guess that Twisted people would rather prefer a patch to break cyclic references in Twisted code. For example, using "__del__" methods or "weakref".
You can easily see what objects and references keep the cycles alive using "gc.garbage" and "gc.get_referrers()". Usually a single "a.value=None" can do miracles :-). Weakref can be very very helpful, also.
Just some points of clarification, as I don't think Jesus has got the whole story:
- gc.collect() is the equivalent of what's run occassionally by Python. So explicitly calling it is *not* required to break all circular references. In this case, it only helped because Andrea was accumulating objects quicker than the Python-scheduled gc.collect was being run. - I really doubt we need a patch to Twisted to break any cyclic references. :) - gc.garbage contains what _won't_ be collected by a call to gc.collect(). gc.garbage contains only broken cycles (e.g., those with objects that have __del__ methods), not all cycles. - Ever since the new gc system was enabled by default in Python, it should never be necessary to manually break cycles unless you have __del__ crap involved.
-- Twisted | Christopher Armstrong: International Man of Twistery Radix | -- http://radix.twistedmatrix.com | Release Manager, Twisted Project \\V/// | -- http://twistedmatrix.com |o O| | w----v----w-+
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Thank you for ypur correction, Christopher. You are right, of course. My fault. I apologize. 04:08 in the morning in Spain :-p. Better go to bed now...
- -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@argo.es http://www.argo.es/~jcea/ _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ _/_/_/_/_/ PGP Key Available at KeyServ _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz
On Tue, Dec 27, 2005 at 01:47:42PM +1100, Christopher Armstrong wrote:
- gc.collect() is the equivalent of what's run occassionally by
IMHO the "occasionally" is the only wrong thing of the whole story. It must not be "occasionally", it must be "occasionally _or_ when the task is growing to an insanse size". The total amount of anonymous memory allocated by the interpreter must be tracked in O(1) and a collect() should have been invokved at least every time the amount of memory doubled.
The reason it took me so long before I could suspect the gc, is that coming from a vm kernel background, I couldn't even dream that after the task grown up to >1G and the system was into swap, the python gc didn't even yet try to prune all potentially freeable objects. The gc should definitely be in function of "size" too, and currently it's not. There is definitely room for improvements in the gc by adding heuristics in function of "size of anonymous memory allocated", and it doesn't seem difficult to add it, nor it should impact performance since the deep gc.collect() (the only costly thing) would very rarely be invoked more frequently.
Quoting Andrea Arcangeli andrea@cpushare.com:
On Tue, Dec 27, 2005 at 01:47:42PM +1100, Christopher Armstrong wrote:
- gc.collect() is the equivalent of what's run occassionally by
IMHO the "occasionally" is the only wrong thing of the whole story. It must not be "occasionally", it must be "occasionally _or_ when the task is growing to an insanse size".
One good way to see the gc in action is to add 'gc.set_debug(gc.DEBUG_LEAK)' at the start of your program. Most of the time I realize it's my program fault when I use this :).
But you may have to adjust the gc threshold in real life application (note that a reconnectingclientfactory with a disconnect every second is not exactly a use case).
On Tue, Dec 27, 2005 at 09:35:03AM +0100, Thomas HERVE wrote:
But you may have to adjust the gc threshold in real life application (note that a reconnectingclientfactory with a disconnect every second is not exactly a use case).
You just need to send a buffer larger than MAX_LENGTH of the Int32StringReceiver to trigger a disconnect on the other end, this is what my testcase did. Of course it's not the normal behaviour, but python shouldn't allocate >1G of ram in that case, exactly because this is not a common load, you should only have to worry about why the other end dropped the connection, and not about the memory management.
On Tue, Dec 27, 2005 at 03:28:30AM +0100, Jesus Cea wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Wow, Andrea Arcangeli, of Linux Kernel fame. What an honor :).
import gcgc.collect()Haha. I was just writing you about manual garbage collector invocations.
;) Thanks! I couldn't even think that it could have been the garbage collector because the thing was >1G large. I definitely expected any sane garbage collector to kick in at least once every time the task was doubling in size and certainly before the thing could grow to the gigabytes. Perhaps this is what should really happen. Every time the task doubles in size we could force a gc.collect(). The growth rate that should trigger a collect() could be a threshold tunable too (default None to be backwards compatible even though I think None is not a safe/sane default).
Manual collections can be costly, nevertheless. Since the manual collector is only needed to free cyclic references, I guess that Twisted people would rather prefer a patch to break cyclic references in Twisted code. For example, using "__del__" methods or "weakref".
You can easily see what objects and references keep the cycles alive using "gc.garbage" and "gc.get_referrers()". Usually a single "a.value=None" can do miracles :-). Weakref can be very very helpful, also.
a.value=None + weakrefs is what did the miracles indeed, but I did it in my code not in twisted code. twisted code has no cyclic references. All cyclic references were in my code sitting on top of twisted. It's me implementing the protocol.
The a.value=None made the code a mess, the weakref complicated things too (especially because I had to hash one of those weak values, so I had to use weakref.ref and I had to change lots of code to add "()"). So I'm soooo glad to have backed out all those changes because they were unnecessary ;). Plus a.value = None is similar to freeing memory in a normal language without garbage collection, not very different from free(a.value). So I certainly prefer to use the more advanced features of the garbage collector, even if I have to invoke it by hand... ;)
On 12/26/05, Jean-Paul Calderone exarkun@divmod.com wrote:
On Mon, 26 Dec 2005 17:07:35 +0100, Andrea Arcangeli andrea@cpushare.com wrote:
I'd also like to know how other languages like ruby and java behave in terms of self-references of objects. Can't the language understand it's a self reference, and in turn it's the same as an integer or a string, like it already does when the member is initialized statically?
I don't know Ruby well enough to comment directly, but I believe Ruby's GC is much simpler (and less capable) than Python's. Java doesn't have bound methods (or unbound methods, or heck, functions): the obvious way in which you would construct them on top of the primitives the language does offer seems to me as though it would introduce the same "problem" you are seeing in Python, but that may just be due to the influence Python has had on my thinking.
(A little late jumping into this discussion) C# (and the CLR in genral), being the better Java that it is, has dynamic functions in the form of delegates, which can be "bound methods" in the sense that Python has them. In addition, it has destructors which are very similar to python's __del__. The CLR's garbage collector will collect cycles with destructors, unlike Python. This means that destructors in C# don't always have access to all of their members (some may "mysteriously" be null, and no order is guaranteed so it will appear random) because they've already been collected. To complexify the matter even more, the CLR's garbage collector may (and in windows does) run in a seperate thread, making safe robust CLR deconstructors a fine art.
On Dec 26, 2005, at 11:07 AM, Andrea Arcangeli wrote:
I was just shoked today when I noticed this:
import sys
class A(object): y = None def x(self): pass def __del__(self): print 'deleted'
a = A() print sys.getrefcount(a) if 1: a.y = a.x print sys.getrefcount(a) del a
I understood the cross references memleaks well, like "x.y = y; y.x= x; del x,y", but I didn't imagine that "a.y = a.x" would be enough to generate a memleak. "a.y = a.x" isn't referencing another structure, it's referencing itself only. Infact if I do this the memleak goes away!!
Objects that use __del__ don't participate in cyclic garbage collection. You really want to write code that doesn't need it, or that uses weakref callbacks instead of __del__.
-bob
On Mon, Dec 26, 2005 at 12:31:32PM -0500, Bob Ippolito wrote:
Objects that use __del__ don't participate in cyclic garbage collection. You really want to write code that doesn't need it, or that uses weakref callbacks instead of __del__.
Thanks for the hint! I didn't know that __del__ made a difference to the garabage collection decisions, that combined with an obsolete webpage written in 99 was the source of quite some confusion to me.
participants (8)
-
Andrea Arcangeli -
Bob Ippolito -
Christopher Armstrong -
Jean-Paul Calderone -
Jesus Cea -
Moe Aboulkheir -
Thomas HERVE -
Timothy Fitz