[Twisted-Python] intermittent traceback using adbapi with pyPgSQL

Twisted gurus, I didn't want to create a bug report yet because I'm not sure that it's a Twisted bug. I have only observed it while using pyPgSQL as the adaptor; haven't seen it yet (and have run the exact same code that produces it) with psycopg. I see this traceback intermittently occur at the end of chains of deferreds that include 2 or more adbapi calls. I can provide some example code, but I should really develop a concise example (as opposed to my app code, which is spread over 2 or 3 modules) that might be useful as the basis for a unit test in case this is a Twisted problem -- it's not clear to me where the problem is: my code, Twisted, or pyPgSQL. My natural assumption would be the former, but the intermittency and the adaptor-dependence makes me wonder. I can run the exact same code several times without incident and then suddenly get this traceback, which makes me think it might be some allocated limit being exceeded (threads in the thread pool? sorry, clueless here). Some observations: * all database operations in the deferred chain complete successfully before the traceback occurs. * it only occurs with pyPgSQL -- not with psycopg (at least I haven't yet seen it with psycopg, and believe me I've tried to make it happen! ... perhaps psycopg's thread management prevents it?) * this always occurs at the end of a "test" in which there is a succeed(result) callback that prints the result and calls reactor.stop() -- I wonder if stopping the reactor (which wouldn't normally happen in an app, of course) breaks the pipe? * I'm using - PostgreSQL 7.4.2 - pyPgSQL 2.4 - psycopg 1.1.15 - Twisted 1.3.0 - Python Python 2.3.3 (#1, Jul 22 2004, 15:58:51) [GCC 3.3.2 20031022 (Red Hat Linux 3.3.2-1)] on linux2 - Fedora Core 1, 2.4.22-1.2197.nptl Anyway, here's the traceback: --------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python2.3/threading.py", line 436, in __bootstrap self.run() File "/usr/local/lib/python2.3/threading.py", line 416, in run self.__target(*self.__args, **self.__kwargs) --- <exception caught here> --- File "/usr/local/lib/python2.3/site-packages/twisted/python/threadpool.py", line 157, in _worker context.call(ctx, function, *args, **kwargs) File "/usr/local/lib/python2.3/site-packages/twisted/python/context.py", line 64, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/local/lib/python2.3/site-packages/twisted/python/context.py", line 43, in callWithContext return func(*args,**kw) File "/usr/local/lib/python2.3/site-packages/twisted/internet/threads.py", line 42, in _putResultInDeferred reactor.callFromThread(deferred.callback, result) File "/usr/local/lib/python2.3/site-packages/twisted/internet/base.py", line 201, in callFromThread self.wakeUp() File "/usr/local/lib/python2.3/site-packages/twisted/internet/base.py", line 207, in wakeUp self.waker.wakeUp() File "/usr/local/lib/python2.3/site-packages/twisted/internet/default.py", line 399, in wakeUp self.o.flush() exceptions.IOError: [Errno 32] Broken pipe --------------------------------------------------------------------- Has anyone else observed this? Cheers, Steve

It sounds like you have a database command running in the threadpool after the reactor has already been shut down. Which suggests you just want to make sure reactor.stop() is called after you're certain there are no more database calls running (and you mustn't call reactor.stop() in a thread of course - reactor.callFromThread(reactor.stop) if you want to).
participants (2)
-
Itamar Shtull-Trauring
-
Stephen Waterbury