[Twisted-Python] Another newbie question... out of file descriptors?
Can anyone explain this result to me? It looks to me as though the SelectReactor is leaking file descriptors. Granted, it's a pathological case, but it's not uncommon if you're running a bunch of tests. --rich rich@black.noir.com> cat demo.py from twisted.internet.selectreactor import SelectReactor for i in xrange(1024): SelectReactor() rich@black.noir.com> python demo.py Traceback (most recent call last): File "demo.py", line 4, in <module> SelectReactor() File "/home/rich/projects/sprained/sprained-dev/lib/python2.6/site-packages/Twisted-9.0.0_r28543-py2.6-linux-x86_64.egg/twisted/internet/selectreactor.py", line 72, in __init__ posixbase.PosixReactorBase.__init__(self) File "/home/rich/projects/sprained/sprained-dev/lib/python2.6/site-packages/Twisted-9.0.0_r28543-py2.6-linux-x86_64.egg/twisted/internet/base.py", line 471, in __init__ self.installWaker() File "/home/rich/projects/sprained/sprained-dev/lib/python2.6/site-packages/Twisted-9.0.0_r28543-py2.6-linux-x86_64.egg/twisted/internet/posixbase.py", line 266, in installWaker self.waker = _Waker(self) File "/home/rich/projects/sprained/sprained-dev/lib/python2.6/site-packages/Twisted-9.0.0_r28543-py2.6-linux-x86_64.egg/twisted/internet/posixbase.py", line 128, in __init__ self.i, self.o = os.pipe() OSError: [Errno 24] Too many open files rich@black.noir.com> lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu lucid (development branch) Release: 10.04 Codename: lucid
On 03:55 am, rich@noir.com wrote:
Can anyone explain this result to me?
It looks to me as though the SelectReactor is leaking file descriptors. Granted, it's a pathological case, but it's not uncommon if you're running a bunch of tests.
See ticket #3063. Jean-Paul
On Feb 25, 2010, at 10:00 PM, exarkun@twistedmatrix.com wrote:
On 03:55 am, rich@noir.com wrote:
Can anyone explain this result to me?
It looks to me as though the SelectReactor is leaking file descriptors. Granted, it's a pathological case, but it's not uncommon if you're running a bunch of tests.
See ticket #3063.
Jean-Paul
For your clicking (and google's indexing) enjoyment, that's <http://twistedmatrix.com/trac/ticket/3063>.
Glyph Lefkowitz wrote:
On Feb 25, 2010, at 10:00 PM, exarkun@twistedmatrix.com wrote:
On 03:55 am, rich@noir.com wrote:
Can anyone explain this result to me?
It looks to me as though the SelectReactor is leaking file descriptors. Granted, it's a pathological case, but it's not uncommon if you're running a bunch of tests.
See ticket #3063.
Jean-Paul
For your clicking (and google's indexing) enjoyment, that's <http://twistedmatrix.com/trac/ticket/3063>.
Thanks. Is it a reasonable workaround, or are there any nonobvious downsides to using reactor.waker.connectionLost(None) either... a) immediately after instantiating my reactor, (assuming I don't need a waker), or... b) immediately before or after calling reactor.stop I guess I'm asking why that hasn't been included in reactor.stop already. --rich
On Feb 26, 2010, at 12:26 AM, K. Richard Pixley wrote:
Is it a reasonable workaround, or are there any nonobvious downsides to using reactor.waker.connectionLost(None) either...
Yes. Lots. This is an _extremely_ hazardous area of Twisted to be mucking around with, as it may appear to work until it completely locks up your process.
a) immediately after instantiating my reactor, (assuming I don't need a waker), or...
You always need a waker. Twisted uses threads internally for several things: the two that come to mind immediately are name resolution and signal handling. If you remove the waker, you will get hangs and crashes.
b) immediately before or after calling reactor.stop
You need to wait for the reactor to fully shut down (i.e. 'run' exits, not 'stop' has been called) since service cleanup may be asynchronous and need to do things like resolve DNS names and handle further signals during shutdown (especially SIGCHLD, as subprocesses will typically exit during shutdown).
I guess I'm asking why that hasn't been included in reactor.stop already.
The implementation is tricky, and you have to audit all the internal state that the reactor keeps. To make matters worse, the test coverage for this area of Twisted is worse than others, because it's old code that works well enough and therefore (A) was written before Twisted had strong quality requirements such as complete test coverage, and (B) since it works well enough, it hasn't been extensively modified and therefore hasn't had to be re-examined to improve its test coverage to 100%. But please, contribute patches to fix it! It isn't *that* hard, mostly it's just that we've had lots of other stuff to do. (See the weekly emails from trac about how many open tickets there are!)
participants (3)
-
exarkun＠twistedmatrix.com -
Glyph Lefkowitz -
K. Richard Pixley