[Twisted-Python] Woven application, objects not collected.
I have a serious problem with some woven applications where deferreds and their results returned from wmfactory_ methods are not garbage collected. They just stack up in gc.garbage. The python documentation states that the contents of gc.garbage are objects that are part of a cycle can't be collected because they have a __del__ method. Any help on this issue would be really appreciated.
On Tue, Nov 25, 2003 at 01:10:09PM +0100, Syver Enstad wrote:
Deferreds have a __del__ method, used to log unhandled errors, which has often proved to be a valuable debugging aid (people often forget to .addErrback(log.err)). Unfortunately, this means that Deferreds in reference cycles can't be automatically garbage collected. Can you boil this down to a minimal test case? This sounds like the probable cause of a memory leak some users have reported with Twisted Web. The trick will be to figure out what the reference cycle is, and breaking it. It may help to do: twisted.internet.defer.Deferred.debug = True and then inspect the .creator attribute of the Deferreds the end up in gc.garbage. You could also try manually poking around with gc.get_referrers on those Deferreds. If we can find a good enough test case, we might be able to write a unit test to trigger it and then check gc.garbage, so that we can include it in the test suite and stop it from happening again. I'm very keen to find out what's causing this -- let me know if there's anything I can help with. -Andrew.
On Wed, Nov 26, 2003 at 12:13:38AM +1100, Andrew Bennetts wrote:
Thinking about this further, I just wrote the following hack for trial: bash-2.05b$ cvs diff -u twisted/scripts/trial.py Index: twisted/scripts/trial.py =================================================================== RCS file: /cvs/Twisted/twisted/scripts/trial.py,v retrieving revision 1.51 diff -u -r1.51 trial.py --- twisted/scripts/trial.py 17 Oct 2003 21:48:16 -0000 1.51 +++ twisted/scripts/trial.py 25 Nov 2003 13:17:02 -0000 @@ -328,6 +328,11 @@ suite.run, reporter, config['random']) else: suite.run(reporter, config['random']) + import gc + gc.collect() + if gc.garbage: + print "GARBAGE: %d objects" % len(gc.garbage) + print "GARBAGE: " + repr(gc.garbage) sys.exit(not reporter.allPassed()) if __name__ == '__main__': I've only just started playing with this, but none of the web or woven tests show anything. test_pb seems to leak a whole bunch of stuff, though... -Andrew.
Andrew Bennetts <andrew@puzzling.org> writes:
The deferreds that are leaking are coming from pb (aReferenceable.callRemote). I think that a minimal test case to show the problem would be: (I'll test this later, I have to go in five minutes time). Pseudo code: class Page: def initialize(self, ...): self._deferred = defer.succeed([1, 2, 3]) def wmfactory_oneTwoThree(self, request): return self._deferred My guess is that self._deferred will end up in gc.garbage. I will test this later.
On Tue, Nov 25, 2003 at 02:56:26PM +0100, Syver Enstad wrote:
Presumably at some later point you will have a line like "self._deferred.callback(value)". Immediately after this (actually, before is preferable, how is left as an exercise for the reader), you should "self._deferred = None". The cycle will be broken and the garbage will be collectable. Jp
Hi there !! I'm new to twisted, I jump into cuz it looks that is gonna make my work lot lot easier, I have to write a telnet client code to connect to systems. I have been looking for telnet client samples out there but no luck so far, can anyone point me in the right direction ??? :)) Any help will be well appreciated.
Jp Calderone <exarkun@intarweb.us> writes:
Presumably at some later point you will have a line like "self._deferred.callback(value)".
No I just return the deferred to woven (with defer.succeed the callback is already called).
I think I'll have to override Page.pageRenderComplete and set it to None there. -- Syver Enstad
[list admins: I originally posted this from the wrong address. Please delete the dupe from the moderation queue.] On Wed, Nov 26, 2003 at 12:13:38AM +1100, Andrew Bennetts wrote:
Thinking about this further, I just wrote the following hack for trial: bash-2.05b$ cvs diff -u twisted/scripts/trial.py Index: twisted/scripts/trial.py =================================================================== RCS file: /cvs/Twisted/twisted/scripts/trial.py,v retrieving revision 1.51 diff -u -r1.51 trial.py --- twisted/scripts/trial.py 17 Oct 2003 21:48:16 -0000 1.51 +++ twisted/scripts/trial.py 25 Nov 2003 13:17:02 -0000 @@ -328,6 +328,11 @@ suite.run, reporter, config['random']) else: suite.run(reporter, config['random']) + import gc + gc.collect() + if gc.garbage: + print "GARBAGE: %d objects" % len(gc.garbage) + print "GARBAGE: " + repr(gc.garbage) sys.exit(not reporter.allPassed()) if __name__ == '__main__': I've only just started playing with this, but none of the web or woven tests show anything. test_pb seems to leak a whole bunch of stuff, though... -Andrew.
On Tue, Nov 25, 2003 at 01:10:09PM +0100, Syver Enstad wrote:
Deferreds have a __del__ method, used to log unhandled errors, which has often proved to be a valuable debugging aid (people often forget to .addErrback(log.err)). Unfortunately, this means that Deferreds in reference cycles can't be automatically garbage collected. Can you boil this down to a minimal test case? This sounds like the probable cause of a memory leak some users have reported with Twisted Web. The trick will be to figure out what the reference cycle is, and breaking it. It may help to do: twisted.internet.defer.Deferred.debug = True and then inspect the .creator attribute of the Deferreds the end up in gc.garbage. You could also try manually poking around with gc.get_referrers on those Deferreds. If we can find a good enough test case, we might be able to write a unit test to trigger it and then check gc.garbage, so that we can include it in the test suite and stop it from happening again. I'm very keen to find out what's causing this -- let me know if there's anything I can help with. -Andrew.
On Wed, Nov 26, 2003 at 12:13:38AM +1100, Andrew Bennetts wrote:
Thinking about this further, I just wrote the following hack for trial: bash-2.05b$ cvs diff -u twisted/scripts/trial.py Index: twisted/scripts/trial.py =================================================================== RCS file: /cvs/Twisted/twisted/scripts/trial.py,v retrieving revision 1.51 diff -u -r1.51 trial.py --- twisted/scripts/trial.py 17 Oct 2003 21:48:16 -0000 1.51 +++ twisted/scripts/trial.py 25 Nov 2003 13:17:02 -0000 @@ -328,6 +328,11 @@ suite.run, reporter, config['random']) else: suite.run(reporter, config['random']) + import gc + gc.collect() + if gc.garbage: + print "GARBAGE: %d objects" % len(gc.garbage) + print "GARBAGE: " + repr(gc.garbage) sys.exit(not reporter.allPassed()) if __name__ == '__main__': I've only just started playing with this, but none of the web or woven tests show anything. test_pb seems to leak a whole bunch of stuff, though... -Andrew.
Andrew Bennetts <andrew@puzzling.org> writes:
The deferreds that are leaking are coming from pb (aReferenceable.callRemote). I think that a minimal test case to show the problem would be: (I'll test this later, I have to go in five minutes time). Pseudo code: class Page: def initialize(self, ...): self._deferred = defer.succeed([1, 2, 3]) def wmfactory_oneTwoThree(self, request): return self._deferred My guess is that self._deferred will end up in gc.garbage. I will test this later.
On Tue, Nov 25, 2003 at 02:56:26PM +0100, Syver Enstad wrote:
Presumably at some later point you will have a line like "self._deferred.callback(value)". Immediately after this (actually, before is preferable, how is left as an exercise for the reader), you should "self._deferred = None". The cycle will be broken and the garbage will be collectable. Jp
Hi there !! I'm new to twisted, I jump into cuz it looks that is gonna make my work lot lot easier, I have to write a telnet client code to connect to systems. I have been looking for telnet client samples out there but no luck so far, can anyone point me in the right direction ??? :)) Any help will be well appreciated.
Jp Calderone <exarkun@intarweb.us> writes:
Presumably at some later point you will have a line like "self._deferred.callback(value)".
No I just return the deferred to woven (with defer.succeed the callback is already called).
I think I'll have to override Page.pageRenderComplete and set it to None there. -- Syver Enstad
[list admins: I originally posted this from the wrong address. Please delete the dupe from the moderation queue.] On Wed, Nov 26, 2003 at 12:13:38AM +1100, Andrew Bennetts wrote:
Thinking about this further, I just wrote the following hack for trial: bash-2.05b$ cvs diff -u twisted/scripts/trial.py Index: twisted/scripts/trial.py =================================================================== RCS file: /cvs/Twisted/twisted/scripts/trial.py,v retrieving revision 1.51 diff -u -r1.51 trial.py --- twisted/scripts/trial.py 17 Oct 2003 21:48:16 -0000 1.51 +++ twisted/scripts/trial.py 25 Nov 2003 13:17:02 -0000 @@ -328,6 +328,11 @@ suite.run, reporter, config['random']) else: suite.run(reporter, config['random']) + import gc + gc.collect() + if gc.garbage: + print "GARBAGE: %d objects" % len(gc.garbage) + print "GARBAGE: " + repr(gc.garbage) sys.exit(not reporter.allPassed()) if __name__ == '__main__': I've only just started playing with this, but none of the web or woven tests show anything. test_pb seems to leak a whole bunch of stuff, though... -Andrew.
participants (6)
-
Andrew Bennetts -
Andrew Bennetts -
Carlos A. Osoria -
Jp Calderone -
Syver Enstad -
Syver Enstad