[Twisted-Python] Reactor callback from the wrong thread
Hi! I have an issue where the reactor calls the callback from a different thread than the one the reactor is running in. The usecase is as follows; We are running tests using PyFit (Fitnesse), and where some of the tests depends on receiving SNMP traps in the background. So at the start we start a trap deamon defined as class TrapDeamon(Thread, netsnmp.Session) We use the pynetsnmp.twistedsnmp package and has a run() method that looks like this def run(self): self.awaitTraps(self.hostPort) twistedsnmp.updateReactor() reactor.run(installSignalHandlers=0) In the callback from the reactor, we get the pdu of the trap which is then sent into a trap buffer. On the PyFit side of things, we execute some commands, and at some points we need a confirmation from a trap before we can continue. This is done by waiting on a queue (Queue.Queue). The trap buffer has a register over queues and awaited trap OID's, and when the correct one is added to the buffer from the reactor callback, it is put onto the queue, the test rejoice, and continues. However, I noticed that fairly often (say every 3rd wait), the wait timed out even though I had seen the trap on wireshark. So I improved the logging, and noticed that in the cases where the wait timed out, the reactor callback for the trap I waited for happened in the main thread (at least main thread id is reported by python logging), and thus did not appear until immediately after the wait had failed and unblocked. Most of the trap callbacks (both before and after these misses) happens in the correct (reactor) thread though. What can cause this and how may I debug it? AFAIK, PyFit does not use twisted or threading at all, and we only have two simple background threads our selves in addition to the one running the trap deamon. Best regards, Lars Ivar Igesund
On Wed, 27 May 2009 15:08:49 +0200, Lars Ivar Igesund <larsivi@gmail.com> wrote:
Hi!
I have an issue where the reactor calls the callback from a different thread than the one the reactor is running in.
Generally speaking, the only callbacks the reactor invokes are protocol methods (like dataReceived and connectionLost) and timed events (things you pass to reactor.callLater). It always calls these in the thread it is running in. What callbacks are you seeing be invoked in the "wrong" thread?
[snip]
What can cause this and how may I debug it? AFAIK, PyFit does not use twisted or threading at all, and we only have two simple background threads our selves in addition to the one running the trap deamon.
The most likely explanation is that your code (perhaps by way of PyFit, I'm not sure -- I've never used PyFit) is calling a Twisted API from a thread other than the reactor thread. So, examine all the places you call Twisted APIs (including APIs in twistedsnmp which may call Twisted APIs) and make sure they're only run in the reactor thread. Jean-Paul
Hi, thanks for the reply. On Wed, May 27, 2009 at 5:43 PM, Jean-Paul Calderone <exarkun@divmod.com> wrote:
On Wed, 27 May 2009 15:08:49 +0200, Lars Ivar Igesund <larsivi@gmail.com> wrote:
Hi!
I have an issue where the reactor calls the callback from a different thread than the one the reactor is running in.
Generally speaking, the only callbacks the reactor invokes are protocol methods (like dataReceived and connectionLost) and timed events (things you pass to reactor.callLater). It always calls these in the thread it is running in. What callbacks are you seeing be invoked in the "wrong" thread?
Sorry for being a bit unclear; these callbacks originates from the SNMP trap listen agent which at the bottom is Net-SNMP wrapped in Python and a protocol for Twisted (incorporated in pynetsnmp, package python-pynetsnmp in Ubuntu/Debian).
[snip]
What can cause this and how may I debug it? AFAIK, PyFit does not use twisted or threading at all, and we only have two simple background threads our selves in addition to the one running the trap deamon.
The most likely explanation is that your code (perhaps by way of PyFit, I'm not sure -- I've never used PyFit) is calling a Twisted API from a thread other than the reactor thread. So, examine all the places you call Twisted APIs (including APIs in twistedsnmp which may call Twisted APIs) and make sure they're only run in the reactor thread.
Googling further, it appears that NetSNMP is not considered thread safe, and this is probably the reason for my troubles (there are normal netsnmp calls going the other way from the apparently hijacked thread). PyFit did not have any twisted dependencies, so I don't consider it an option at this point. I have also tested the alternative Twisted SNMP package (based on PySNMP), but unfortunately both of those appear to mature enough for our use. At this point I'm probably left with two options; either make all of the SNMP communication go via pynetsnmp (will cause at least some rewriting), or move the trap deamon part into its own application that communicates with the test framework via a sockets. The latter would guarantee no snmp thread confusion, but may be a bit overkill. If anyone has a good advice, then I'd appreciate it. Thanks!
On Thu, 2009-05-28 at 14:04 +0200, Lars Ivar Igesund wrote:
If anyone has a good advice, then I'd appreciate it.
You can call any reactor API from another thread using reactor.callFromThread, or the more useful twisted.internet.threads.blockingCallFromThread. So just wrap all places that call into the reactor. Hopefully one day we will have feature to help debug this sort of problem: http://twistedmatrix.com/trac/ticket/792
Thanks for the tip, however - in which version was blockingCallFromThread added? It did not wish to be imported from the version I installed via Ubuntu which appears to be 2.5.0. (Bit confused about the version numbering though since Twisted appears to be at something like 8.2) Best, Lars Ivar On Thu, May 28, 2009 at 3:21 PM, Itamar Shtull-Trauring <itamar@itamarst.org> wrote:
On Thu, 2009-05-28 at 14:04 +0200, Lars Ivar Igesund wrote:
If anyone has a good advice, then I'd appreciate it.
You can call any reactor API from another thread using reactor.callFromThread, or the more useful twisted.internet.threads.blockingCallFromThread. So just wrap all places that call into the reactor.
Hopefully one day we will have feature to help debug this sort of problem: http://twistedmatrix.com/trac/ticket/792
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Fri, May 29, 2009 at 3:02 AM, Lars Ivar Igesund <larsivi@gmail.com> wrote:
Thanks for the tip,
however - in which version was blockingCallFromThread added? It did not wish to be imported from the version I installed via Ubuntu which appears to be 2.5.0. (Bit confused about the version numbering though since Twisted appears to be at something like 8.2)
Well, using Ubuntu, you should find the new versioning scheme somewhat familiar ;) http://twistedmatrix.com/trac/wiki/FrequentlyAskedQuestions#Whydidtheversion... -Drew
participants (4)
-
Drew Smathers -
Itamar Shtull-Trauring -
Jean-Paul Calderone -
Lars Ivar Igesund