[Twisted-Python] strategies for tracking down memory issues
some recent changes to a very-happy twisted daemon have resulted in a process that grows in memory until it crashes the box. boo! looking through the code and logs, i'm wondering if i''ve coded things in such a way that defferds or deferrd lists are somehow not getting cleaned up if an unhandled exception occurs. i've been looking through all my former notes and some questions on stack overflow, and I've seen a lot of info on using heapy and other tools to find issues on a function-by-function basis. i'm wondering if anyone has experience in simply monitoring the lifecycle of deferreds ?
On 1/17/14 10:43 AM, Jonathan Vanasco wrote:
some recent changes to a very-happy twisted daemon have resulted in a process that grows in memory until it crashes the box. boo!
looking through the code and logs, i'm wondering if i''ve coded things in such a way that defferds or deferrd lists are somehow not getting cleaned up if an unhandled exception occurs.
i've been looking through all my former notes and some questions on stack overflow, and I've seen a lot of info on using heapy and other tools to find issues on a function-by-function basis.
i'm wondering if anyone has experience in simply monitoring the lifecycle of deferreds ?
Hi A few years ago I ran into the problem of non collectable cycles being produced by keeping references around in objects locally. Please find attached a snippet of code I used over and over again for hunting down such cycles. It's for sure not a simple cure, but with a manhole service being run in your process of question, it gives you the telltale signs your looking for. Use it like from dumpObjects import dumpObjects dumpObjects() Various include/exclude sets should help you to narrow the search in consecutive runs. As for the lifecycle of deferreds I never ever had a problem with deferreds not being cleaned up, it has always been me who produced those non collectable cycles, usually under the false assumption, that it was safe to keep a ref handy. Werner
On Jan 17, 2014, at 12:43 PM, Jonathan Vanasco <twisted-python@2xlp.com> wrote:
some recent changes to a very-happy twisted daemon have resulted in a process that grows in memory until it crashes the box. boo!
looking through the code and logs, i'm wondering if i''ve coded things in such a way that defferds or deferrd lists are somehow not getting cleaned up if an unhandled exception occurs.
i've been looking through all my former notes and some questions on stack overflow, and I've seen a lot of info on using heapy and other tools to find issues on a function-by-function basis.
i'm wondering if anyone has experience in simply monitoring the lifecycle of deferreds ?
First off, just manhole in and inspect gc.garbage :-). Second, I've had pretty good luck with <http://www.aminus.net/wiki/Dowser>, although it would be nice if it were a little easier to set up without writing a weird little custom code hook and having to decide where to put it. However, you can sort reachable objects by count and type, so you can see which of your classes or Twisted's classes are hanging around in memory and that will usually very quickly give you an idea of where the issue might be. -glyph
On Fri, Jan 17, 2014 at 6:06 PM, Glyph Lefkowitz <glyph@twistedmatrix.com>wrote:
On Jan 17, 2014, at 12:43 PM, Jonathan Vanasco <twisted-python@2xlp.com> wrote:
some recent changes to a very-happy twisted daemon have resulted in a process that grows in memory until it crashes the box. boo!
looking through the code and logs, i'm wondering if i''ve coded things in such a way that defferds or deferrd lists are somehow not getting cleaned up if an unhandled exception occurs.
i've been looking through all my former notes and some questions on stack overflow, and I've seen a lot of info on using heapy and other tools to find issues on a function-by-function basis.
i'm wondering if anyone has experience in simply monitoring the lifecycle of deferreds ?
First off, just manhole in and inspect gc.garbage :-).
<snip>
-glyph
I had a similar situation several years ago, and messed around with heapy and some other Python memory profiling tools, but the manhole + gc.garbage was both the easiest and most effective. One other thing I did was to set up a separate Twisted Service that would run a memory profiling function periodically (I think it just looked at gc.garbage, and sorted things nicely) and log it. I used txScheduler (which I wrote) for that. In fact that's part of why I wrote it. I can't give you much more detail than that, though. It was over 5 years ago, and I don't have access to that code any more. -- Kevin Horn
On Jan 17, 2014, at 5:32 PM, Kevin Horn <kevin.horn@gmail.com> wrote:
I used txScheduler (which I wrote) for that.
Available here: <https://bitbucket.org/khorn/txscheduler>.
In fact that's part of why I wrote it.
Not to be confused with TxScheduling: <https://pypi.python.org/pypi/TxScheduling>. -glyph
participants (4)
-
Glyph Lefkowitz -
Jonathan Vanasco -
Kevin Horn -
Werner Thie