Finding memory leaks in Nevow/Twisted
Hi I have managed to write my nevow/twisted site. I am using a lot of Athena elements on my site and they are all working well from a functional point of view. However, the Athena elements are eating up RAM and as I leave things running the Twisted process slowly consumes more and more RAM until the process dies. This only happens for the pages with Athena elements which suggests that something isn't been cleaned up when AJAX methods are being called. Can anyone suggest a possible cause? Or what I really need is a suggestion in how to track down the memory leak. Is there a way of doing memory profiling to find the culprit? Any advice is welcomed. AW
On Tue, 16 Dec 2008 18:07:41 +0300, Abdul-Wahid Paterson <abdulwahid@gmail.com> wrote:
Hi
I have managed to write my nevow/twisted site. I am using a lot of Athena elements on my site and they are all working well from a functional point of view. However, the Athena elements are eating up RAM and as I leave things running the Twisted process slowly consumes more and more RAM until the process dies.
This only happens for the pages with Athena elements which suggests that something isn't been cleaned up when AJAX methods are being called.
Can anyone suggest a possible cause? Or what I really need is a suggestion in how to track down the memory leak. Is there a way of doing memory profiling to find the culprit?
I've found http://guppy-pe.sourceforge.net/ to be useful for tracking down memory issues. Jean-Paul
On Dec 16, 2008, at 10:07 AM, Abdul-Wahid Paterson wrote:
I have managed to write my nevow/twisted site. I am using a lot of Athena elements on my site and they are all working well from a functional point of view. However, the Athena elements are eating up RAM and as I leave things running the Twisted process slowly consumes more and more RAM until the process dies.
The first thing would be to check that you're running at least Python 2.5. Previous versions have garbage collection problems on long- running apps like a Twisted web app. Athena-based apps should not inherently leak memory, but adding leaks of your own creation isn't too hard ;-) If you are running 2.5 or later, you should try to reduce your app to the smallest possible test version that demonstrates the leak, and someone here can probably be of more help. -phil
On Tue, 16 Dec 2008 10:23:16 -0500, Phil Christensen <phil@bubblehouse.org> wrote:
On Dec 16, 2008, at 10:07 AM, Abdul-Wahid Paterson wrote:
I have managed to write my nevow/twisted site. I am using a lot of Athena elements on my site and they are all working well from a functional point of view. However, the Athena elements are eating up RAM and as I leave things running the Twisted process slowly consumes more and more RAM until the process dies.
The first thing would be to check that you're running at least Python 2.5. Previous versions have garbage collection problems on long- running apps like a Twisted web app.
I think this is an overstatement of the memory allocation issues in Python 2.4 and older. :) Python 2.5 does a better job of backing off from peak allocation, but it's entirely possible to have a long running Python 2.4 process which doesn't grow in memory usage without bounds. I have a Python 2.4 server which often runs for months at a time and has stable memory usage - at the moment, it's been running since Nov 7 and is using 133MB of memory, a size it reached within a couple hours of starting up. Jean-Paul
Hi, I am using Python 2.5. I will try to use guppy as suggested in another post to so if it will help. Otherwise yes, I will try to reduce it to the smallest possible code to demonstrate the leak. At the moment the code is a bit complex so is hard to track down. Thanks for the help. AW On Tue, Dec 16, 2008 at 6:23 PM, Phil Christensen <phil@bubblehouse.org>wrote:
On Dec 16, 2008, at 10:07 AM, Abdul-Wahid Paterson wrote:
I have managed to write my nevow/twisted site. I am using a lot of Athena elements on my site and they are all working well from a functional point of view. However, the Athena elements are eating up RAM and as I leave things running the Twisted process slowly consumes more and more RAM until the process dies.
The first thing would be to check that you're running at least Python 2.5. Previous versions have garbage collection problems on long-running apps like a Twisted web app.
Athena-based apps should not inherently leak memory, but adding leaks of your own creation isn't too hard ;-)
If you are running 2.5 or later, you should try to reduce your app to the smallest possible test version that demonstrates the leak, and someone here can probably be of more help.
-phil
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
Hi I was bitten by the same problem and went so far to track down all the fishy cycles which easily can be created. Most of the cycles I detected were multi-object cycles which were created by me just storing references to objects to have them handy. Certain types of cycles cannot be broken by the gc, you get a runaway situation either in Python24 or Python25. My code now runs without problems for months serving more than 10k users a day providing a card gaming environment. Things to check: - the object hierarchy on the server and the client should be in sync, if not detach might not work correctly - do you use stateful LivePages? if so, check the toremember list in the request if its growing ad libidum - do you use guard/sessions? if so, be careful with the session/mind object and what you store in mind But without seeing your code it's close to impossible to give you more than hints. I include some code which I wrote to peek into my running server with a manhole connection. dumpobjects is loosely based on several Python snippets and might not win a contest in programming but it helps to track the number of objects of certain types in the running system when used on the commandline from inside your server. HTH, Werner To create a manhole in the same process you started with your .tac file use the following code: --------------------------------------------------------------- def ManholeFactory(namespace, **passwords): realm = manhole_ssh.TerminalRealm() def getManhole(_): return manhole.Manhole(namespace) realm.chainedProtocolFactory.protocolFactory = getManhole p = portal.Portal(realm) p.registerChecker(checkers.InMemoryUsernamePasswordDatabaseDontUse(**passwords)) f = manhole_ssh.ConchFactory(p) return f console = ManholeFactory(globals(), admin='admin') internet.TCPServer(2222, console, interface='localhost').setServiceParent(application) --------------------------------------------------------------- exc = [ "function", "type", "list", "dict", "tuple", "wrapper_descriptor", "module", "method_descriptor", "member_descriptor", "instancemethod", "builtin_function_or_method", "frame", "classmethod", "classmethod_descriptor", "_Environ", "MemoryError", "_Printer", "_Helper", "getset_descriptor", "weakreaf" ] inc = [ 'YourObject_One', 'YourObject_Two' ] prev = {} def dumpObjects(delta=True, limit=0, include=inc, exclude=[]): global prev if include != [] and exclude != []: print 'cannot use include and exclude at the same time' return print 'working with:' print ' delta: ', delta print ' limit: ', limit print ' include: ', include print ' exclude: ', exclude objects = {} gc.collect() oo = gc.get_objects() for o in oo: if getattr(o, "__class__", None): name = o.__class__.__name__ if ((exclude == [] and include == []) or \ (exclude != [] and name not in exclude) or \ (include != [] and name in include)): objects[name] = objects.get(name, 0) + 1 ## if more: ## print o pk = prev.keys() pk.sort() names = objects.keys() names.sort() for name in names: if limit == 0 or objects[name] > limit: if not prev.has_key(name): prev[name] = objects[name] dt = objects[name] - prev[name] if delta or dt != 0: print '%0.6d -- %0.6d -- ' % (dt, objects[name]), name prev[name] = objects[name] Abdul-Wahid Paterson wrote:
Hi,
I am using Python 2.5.
I will try to use guppy as suggested in another post to so if it will help. Otherwise yes, I will try to reduce it to the smallest possible code to demonstrate the leak. At the moment the code is a bit complex so is hard to track down.
Thanks for the help.
AW
On Tue, Dec 16, 2008 at 6:23 PM, Phil Christensen <phil@bubblehouse.org <mailto:phil@bubblehouse.org>> wrote:
On Dec 16, 2008, at 10:07 AM, Abdul-Wahid Paterson wrote:
I have managed to write my nevow/twisted site. I am using a lot of Athena elements on my site and they are all working well from a functional point of view. However, the Athena elements are eating up RAM and as I leave things running the Twisted process slowly consumes more and more RAM until the process dies.
The first thing would be to check that you're running at least Python 2.5. Previous versions have garbage collection problems on long-running apps like a Twisted web app.
Athena-based apps should not inherently leak memory, but adding leaks of your own creation isn't too hard ;-)
If you are running 2.5 or later, you should try to reduce your app to the smallest possible test version that demonstrates the leak, and someone here can probably be of more help.
-phil
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com <mailto:Twisted-web@twistedmatrix.com> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
------------------------------------------------------------------------
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
Werner, Thanks for your post. Is very useful. I am using Guard so that might be one place to start looking although I personally didn't develop that part of the code. (Which kinda make me even more suspicious). The Manhole stuff looks interesting too. I will give that a go if I can't find anything by disabling the Guard code. Thanks again. AW On Tue, Dec 16, 2008 at 10:39 PM, Werner Thie <wthie@thiengineering.ch>wrote:
Hi
I was bitten by the same problem and went so far to track down all the fishy cycles which easily can be created. Most of the cycles I detected were multi-object cycles which were created by me just storing references to objects to have them handy. Certain types of cycles cannot be broken by the gc, you get a runaway situation either in Python24 or Python25. My code now runs without problems for months serving more than 10k users a day providing a card gaming environment.
Things to check: - the object hierarchy on the server and the client should be in sync, if not detach might not work correctly
- do you use stateful LivePages? if so, check the toremember list in the request if its growing ad libidum
- do you use guard/sessions? if so, be careful with the session/mind object and what you store in mind
But without seeing your code it's close to impossible to give you more than hints.
I include some code which I wrote to peek into my running server with a manhole connection. dumpobjects is loosely based on several Python snippets and might not win a contest in programming but it helps to track the number of objects of certain types in the running system when used on the commandline from inside your server.
HTH, Werner
To create a manhole in the same process you started with your .tac file use the following code:
--------------------------------------------------------------- def ManholeFactory(namespace, **passwords): realm = manhole_ssh.TerminalRealm()
def getManhole(_): return manhole.Manhole(namespace)
realm.chainedProtocolFactory.protocolFactory = getManhole p = portal.Portal(realm)
p.registerChecker(checkers.InMemoryUsernamePasswordDatabaseDontUse(**passwords)) f = manhole_ssh.ConchFactory(p) return f
console = ManholeFactory(globals(), admin='admin') internet.TCPServer(2222, console, interface='localhost').setServiceParent(application) ---------------------------------------------------------------
exc = [ "function", "type", "list", "dict", "tuple", "wrapper_descriptor", "module", "method_descriptor", "member_descriptor", "instancemethod", "builtin_function_or_method", "frame", "classmethod", "classmethod_descriptor", "_Environ", "MemoryError", "_Printer", "_Helper", "getset_descriptor", "weakreaf" ]
inc = [ 'YourObject_One', 'YourObject_Two' ]
prev = {}
def dumpObjects(delta=True, limit=0, include=inc, exclude=[]): global prev if include != [] and exclude != []: print 'cannot use include and exclude at the same time' return print 'working with:' print ' delta: ', delta print ' limit: ', limit print ' include: ', include print ' exclude: ', exclude objects = {} gc.collect() oo = gc.get_objects() for o in oo: if getattr(o, "__class__", None): name = o.__class__.__name__ if ((exclude == [] and include == []) or \ (exclude != [] and name not in exclude) or \ (include != [] and name in include)): objects[name] = objects.get(name, 0) + 1 ## if more: ## print o pk = prev.keys() pk.sort() names = objects.keys() names.sort() for name in names: if limit == 0 or objects[name] > limit: if not prev.has_key(name): prev[name] = objects[name] dt = objects[name] - prev[name] if delta or dt != 0: print '%0.6d -- %0.6d -- ' % (dt, objects[name]), name prev[name] = objects[name]
Abdul-Wahid Paterson wrote:
Hi,
I am using Python 2.5.
I will try to use guppy as suggested in another post to so if it will help. Otherwise yes, I will try to reduce it to the smallest possible code to demonstrate the leak. At the moment the code is a bit complex so is hard to track down.
Thanks for the help.
AW
On Tue, Dec 16, 2008 at 6:23 PM, Phil Christensen <phil@bubblehouse.org<mailto: phil@bubblehouse.org>> wrote:
On Dec 16, 2008, at 10:07 AM, Abdul-Wahid Paterson wrote:
I have managed to write my nevow/twisted site. I am using a lot of Athena elements on my site and they are all working well from a functional point of view. However, the Athena elements are eating up RAM and as I leave things running the Twisted process slowly consumes more and more RAM until the process dies.
The first thing would be to check that you're running at least Python 2.5. Previous versions have garbage collection problems on long-running apps like a Twisted web app.
Athena-based apps should not inherently leak memory, but adding leaks of your own creation isn't too hard ;-)
If you are running 2.5 or later, you should try to reduce your app to the smallest possible test version that demonstrates the leak, and someone here can probably be of more help.
-phil
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com <mailto:Twisted-web@twistedmatrix.com> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
------------------------------------------------------------------------
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
participants (4)
-
Abdul-Wahid Paterson -
Jean-Paul Calderone -
Phil Christensen -
Werner Thie