[Twisted-Python] segfaults with deferToThread

This is on Freebsd 5.3-release-p2 with python 2.4. The script below calls a blocking method using deferToThread. When I have 5 or so clients all hitting it at the same time I get a segfault or bus error within minutes. I recompiled python with --with-pydebug and set OPT to -g, but it didn't give me any more information in the backtrace which is also below. Then I ran gdb on the server process and something funny happened. I didn't get a segfault, but the server started catching exceptions and printing them to the screen. As soon as I detached gdb the exceptions stop but it dumps core a minute later (or less). The exception is pasted below also. Python was compiled with pthreads, which is the freebsd kse threads as far as I know. Anyone have an idea what is going on? Server code: ----------------------------------------------------------- from twisted.internet.protocol import Protocol, Factory from twisted.internet import defer, reactor from twisted.python import threadable from twisted.internet import threads threadable.init(1) from otransact.otdirect import process from twisted.python import log class GetData: # this calls my blocking method. THe method blocks when doing database calls with psycopg and when using urllib2 to do an https POST. Other than that everything is non blocking. psycopg is supposed to be thread safe, and I would assume urllib2 is also. def Do(self,data): response = process(data) return response ### Protocol Implementation # This is just about the simplest possible protocol class Echo(Protocol): def dataReceived(self, data): """As soon as any data is received, write it back.""" reactor.callLater(0, self.Start,data) def PrintData(self,data): self.transport.write("%s\r\n" % data) self.transport.loseConnection() #reactor.stop() def Start(self,data): c = GetData() d = threads.deferToThread(c.Do,data) d.addCallback(self.PrintData) d.addErrback(log.err) def main(): f = Factory() f.protocol = Echo reactor.listenTCP(8000, f) reactor.run() if __name__ == '__main__': main() Exception raised by server when gdb was running. ----------------------------------------------------------------------- File "otssl.py", line 42, in ? reactor.run() File "/usr/local/lib/python2.4/site-packages/twisted/internet/default.py", line 126, in run self.mainLoop() File "/usr/local/lib/python2.4/site-packages/twisted/internet/default.py", line 134, in mainLoop self.runUntilCurrent() --- <exception caught here> --- File "/usr/local/lib/python2.4/site-packages/twisted/internet/base.py", line 423, in runUntilCurrent call.func(*call.args, **call.kw) File "/usr/home/otransact/ssl/ots.py", line 72, in Start d = threads.deferToThread(c.Do,data) File "/usr/local/lib/python2.4/site-packages/twisted/internet/threads.py", line 48, in deferToThread reactor.callInThread(_putResultInDeferred, d, f, args, kwargs) File "/usr/local/lib/python2.4/site-packages/twisted/internet/base.py", line 453, in callInThread self.threadpool.callInThread(_callable, *args, **kwargs) File "/usr/local/lib/python2.4/site-packages/twisted/python/threadpool.py", line 130, in callInThread self.q.put(o) File "/usr/local/lib/python2.4/Queue.py", line 91, in put self.not_full.release() thread.error: release unlocked lock Backtrace: ------------------------------------------------ #0 0x28286dbb in pthread_testcancel () from /usr/lib/libpthread.so.1 #1 0x2827eb06 in pthread_mutexattr_init () from /usr/lib/libpthread.so.1 #2 0x00000000 in ?? ()

On Mon, Feb 07, 2005 at 07:53:46PM -0800, snacktime wrote:
This is on Freebsd 5.3-release-p2 with python 2.4.
The script below calls a blocking method using deferToThread. When I have 5 or so clients all hitting it at the same time I get a segfault or bus error within minutes.
This looks like it may be related to http://python.org/sf/1080660. I've added a comment to that bug; you might like to follow-up there yourself. The bug is almost certainly in Python itself (or a 3rd-party extension module you're using, like psycopg), and not Twisted. -Andrew.
participants (2)
-
Andrew Bennetts
-
snacktime