[Twisted-Python] Issue with calling back deferreds from threads
Introduced in one of the recent versions of Twisted was a patch described here. I'm cc'ing itamar because he created the patch: http://twistedmatrix.com/pipermail/twisted-commits/2003-November/010059.html This makes sure you can't callback with Deferred objects. However, some of my code depends on doing just that because of the blocking DNS issue: def mymethod(*args): def go(*args): return xmlproxy.callRemote(*args) twisted.internet.threads.deferToThread(go) As you can see, I'm deferring something that should not be blocking to a thread. The purpose for doing so is that sometimes resolving the hostname can take a while for this call. (Unrelated document issue with Twisted: DNS is blocking.) So, in this case, it is desirable (although hackish) to defer the creation of a deferred object to a thread, and itamar's patch prevents this process. One option I have is to make go() blocking, but that will hold a thread open for longer than is necessary. Another option is to comment out the assertion, but that's probably not a good idea. Until the DNS blocking issue is resolved, I'm not sure how to solve this one. Suggestions? Thanks Ken
On Wed, Jul 14, 2004 at 07:26:13PM -0400, Bob Ippolito wrote:
On Jul 14, 2004, at 6:27 PM, Ken Kinder wrote:
[...]
There is, although not clearly documented anywhere I know of. It's: from twisted.names import client from twisted.internet import reactor reactor.installResolver(client.theResolver) If you need a resolver with different behavior to the default (e.g. you want to tell it to use a specific DNS server), use: resolver = client.createResolver(servers=[('127.0.0.1', 53)]) reactor.installResolver(resolver) There's also a ThreadedResolver, which just uses Python's builtin socket.gethostbyname in a thread. You could install that one with: reactor.installResolver(client.ThreadedResolver()) -Andrew.
On Wed, 2004-07-14 at 18:27, Ken Kinder wrote:
1. Twisted is NOT THREAD SAFE. Code that talks to Twisted in any way will break if run in a thread in parallel to the main reactor thread. Use reactor.callFromThread to call Twisted code in the reactor thread from other threads. 2. Twisted supports non-blocking DNS, see other posts. 3. If you *do* need to connect two Deferreds together, you might want to look at twisted.internet.defer.maybeDeferred and Deferred.chainDeferred, functions. -- Itamar Shtull-Trauring http://itamarst.org
On Wed, Jul 14, 2004 at 07:26:13PM -0400, Bob Ippolito wrote:
On Jul 14, 2004, at 6:27 PM, Ken Kinder wrote:
[...]
There is, although not clearly documented anywhere I know of. It's: from twisted.names import client from twisted.internet import reactor reactor.installResolver(client.theResolver) If you need a resolver with different behavior to the default (e.g. you want to tell it to use a specific DNS server), use: resolver = client.createResolver(servers=[('127.0.0.1', 53)]) reactor.installResolver(resolver) There's also a ThreadedResolver, which just uses Python's builtin socket.gethostbyname in a thread. You could install that one with: reactor.installResolver(client.ThreadedResolver()) -Andrew.
On Wed, 2004-07-14 at 18:27, Ken Kinder wrote:
1. Twisted is NOT THREAD SAFE. Code that talks to Twisted in any way will break if run in a thread in parallel to the main reactor thread. Use reactor.callFromThread to call Twisted code in the reactor thread from other threads. 2. Twisted supports non-blocking DNS, see other posts. 3. If you *do* need to connect two Deferreds together, you might want to look at twisted.internet.defer.maybeDeferred and Deferred.chainDeferred, functions. -- Itamar Shtull-Trauring http://itamarst.org
participants (4)
-
Andrew Bennetts -
Bob Ippolito -
Itamar Shtull-Trauring -
Ken Kinder