[Twisted-Python] MemoryError in twisted causes the program to exit, or not
Hi, I am trying to understand what will happen in my long-running twisted server program when the available memory is low. If I run the following program with increasing numeric seeds as arguments, 0, 1, 2, 3, 4, … , some of the time the out of memory condition will crash and exit the program right away (when the MemoryError is raised inside twisted core code), and other times it will just carry on with an UnhandledError (if the MemoryError is raised inside my hi() function below). Sample code: --- #!/usr/bin/env python import sys, resource, random, traceback from twisted.internet import reactor resource.setrlimit(resource.RLIMIT_AS, (180 * 1024 * 1024, resource.RLIM_INFINITY)) data = [] random.seed(int(sys.argv[1])) def hi(): data.append('x' * random.randint(1, 10000)) reactor.callLater(0, hi) def main(): reactor.callLater(0, hi) reactor.callWhenRunning(main) reactor.run() --- I am using python 2.7.1, twisted 11.0.0. Sample runs follow: bash$ n=0; while true; do echo "starting rep $n"; ./twisted-oom.py $n; n=$(($n+1)); echo; done starting rep 0 Traceback (most recent call last): File “./twisted-oom.py", line 21, in <module> reactor.run() File "/sw/external/twisted-py27-11.0.0/lib/python/twisted/internet/base.py", line 1162, in run self.mainLoop() File "/sw/external/twisted-py27-11.0.0/lib/python/twisted/internet/base.py", line 1177, in mainLoop log.err() File "/sw/external/twisted-py27-11.0.0/lib/python/twisted/python/log.py", line 197, in err _stuff = failure.Failure() MemoryError starting rep 1 Traceback (most recent call last): File ". /twisted-oom.py", line 21, in <module> reactor.run() File "/sw/external/twisted-py27-11.0.0/lib/python/twisted/internet/base.py", line 1162, in run self.mainLoop() File "/sw/external/twisted-py27-11.0.0/lib/python/twisted/internet/base.py", line 1177, in mainLoop log.err() MemoryError starting rep 2 Unhandled Error Traceback (most recent call last): File "./twisted-oom.py", line 21, in <module> reactor.run() File "/sw/external/twisted-py27-11.0.0/lib/python/twisted/internet/base.py", line 1162, in run self.mainLoop() File "/sw/external/twisted-py27-11.0.0/lib/python/twisted/internet/base.py", line 1171, in mainLoop self.runUntilCurrent() --- <exception caught here> --- File "/sw/external/twisted-py27-11.0.0/lib/python/twisted/internet/base.py", line 793, in runUntilCurrent call.func(*call.args, **call.kw) File "/home/ruttbe/dev/oneoff/twisted-oom.py", line 14, in hi data.append('x' * random.randint(1, 10000)) exceptions.MemoryError: On the first two runs, the program exited quickly, but on the last rep ("starting rep 2"), it just carried on running with the reactor still running. It would be nice if there was a way to make the out of memory behavior consistent, so I would know that it would either always carry on running with a MemoryError stack trace, or always crash out and end the program. Maybe runUntilCurrent should not catch MemoryError from user code and should let it propagate out? Also wondering if there are other different behaviors in other areas of the code (such as callbacks into user code such as dataReceived, outReceived, lineReceived, etc.). i.e. is the observed behavior (that MemoryError in user code is caught and MemoryError in twisted core is not) intentional, or an accident? Thanks, -- Benjamin Rutt
Le Sep 27, 2012 à 7:41 PM, Benjamin Rutt <rutt.4@osu.edu> a écrit :
Hi, I am trying to understand what will happen in my long-running twisted server program when the available memory is low.
Probably nothing good.
If I run the following program with increasing numeric seeds as arguments, 0, 1, 2, 3, 4, … , some of the time the out of memory condition will crash and exit the program right away (when the MemoryError is raised inside twisted core code), and other times it will just carry on with an UnhandledError (if the MemoryError is raised inside my hi() function below).
Yes. Resource exhaustion testing is tough, especially in Python, where literally every function call may (non-deterministically) allocate memory. MemoryError is an exception that you may see at any time, in any context, without warning. Most notably you may encounter a MemoryError when attempting to handle a MemoryError and there's not much you can do about it at that point. Ostensibly, Twisted could deal with this "better" by special-casing MemoryError in top-level error handlers and falling through to exit, rather than attempting to log; but, in cases where only a localized area is experiencing memory pressure and the exception causes cleanup to happen correctly, this would overzealously exit your service even in cases where it could recover. So: don't exhaust your available memory. It will cause a lot of unpredictable problems. If you have a better idea for how Twisted might handle this generally though, I'm happy to hear it :). -glyph
On Fri, 28 Sep 2012 00:09:23 -0700 Glyph <glyph@twistedmatrix.com> wrote:
I agree this would be unwelcome behaviour.
If you have a better idea for how Twisted might handle this generally though, I'm happy to hear it :).
Calling gc.collect() might improve things, temporarily or not, in some contexts. Perhaps enough to be able to log the error, so that the developer isn't left without a clue. (whether Python itself should try to GC-collect when MemoryError is raised is an open question) Regards Antoine. -- Software development and contracting: http://pro.pitrou.net
Le Sep 28, 2012 à 1:10 AM, Antoine Pitrou <solipsis@pitrou.net> a écrit :
(whether Python itself should try to GC-collect when MemoryError is raised is an open question)
Shouldn't it be doing a GC collect _before_ MemoryError is raised? (Isn't this rather the whole point of having a garbage collector?) -glyph
On Fri, 28 Sep 2012 11:40:54 -0700 Glyph <glyph@twistedmatrix.com> wrote:
Indeed, it also could. Regards Antoine. -- Software development and contracting: http://pro.pitrou.net
Le Sep 27, 2012 à 7:41 PM, Benjamin Rutt <rutt.4@osu.edu> a écrit :
Hi, I am trying to understand what will happen in my long-running twisted server program when the available memory is low.
Probably nothing good.
If I run the following program with increasing numeric seeds as arguments, 0, 1, 2, 3, 4, … , some of the time the out of memory condition will crash and exit the program right away (when the MemoryError is raised inside twisted core code), and other times it will just carry on with an UnhandledError (if the MemoryError is raised inside my hi() function below).
Yes. Resource exhaustion testing is tough, especially in Python, where literally every function call may (non-deterministically) allocate memory. MemoryError is an exception that you may see at any time, in any context, without warning. Most notably you may encounter a MemoryError when attempting to handle a MemoryError and there's not much you can do about it at that point. Ostensibly, Twisted could deal with this "better" by special-casing MemoryError in top-level error handlers and falling through to exit, rather than attempting to log; but, in cases where only a localized area is experiencing memory pressure and the exception causes cleanup to happen correctly, this would overzealously exit your service even in cases where it could recover. So: don't exhaust your available memory. It will cause a lot of unpredictable problems. If you have a better idea for how Twisted might handle this generally though, I'm happy to hear it :). -glyph
On Fri, 28 Sep 2012 00:09:23 -0700 Glyph <glyph@twistedmatrix.com> wrote:
I agree this would be unwelcome behaviour.
If you have a better idea for how Twisted might handle this generally though, I'm happy to hear it :).
Calling gc.collect() might improve things, temporarily or not, in some contexts. Perhaps enough to be able to log the error, so that the developer isn't left without a clue. (whether Python itself should try to GC-collect when MemoryError is raised is an open question) Regards Antoine. -- Software development and contracting: http://pro.pitrou.net
Le Sep 28, 2012 à 1:10 AM, Antoine Pitrou <solipsis@pitrou.net> a écrit :
(whether Python itself should try to GC-collect when MemoryError is raised is an open question)
Shouldn't it be doing a GC collect _before_ MemoryError is raised? (Isn't this rather the whole point of having a garbage collector?) -glyph
On Fri, 28 Sep 2012 11:40:54 -0700 Glyph <glyph@twistedmatrix.com> wrote:
Indeed, it also could. Regards Antoine. -- Software development and contracting: http://pro.pitrou.net
participants (3)
-
Antoine Pitrou -
Benjamin Rutt -
Glyph