[Twisted-Python] Running callbacks in a non-main thread
Hey, I realize that the point of deferreds is to have process/network intensive tasks work on in their own thread and send the results to callback function in the main thread, but is there away to have a deferred process it's callback in a non-main thread? I'm sure someone will want to know why, and maybe solving my meta-problem would show me the errors in my ways in wanting to warp the purpose of the mighty deferred. Async is great, but I'm slowly replacing some code that uses xmlrpc (blocking) with prospect broker objects (returns deferred, hence non-blocking). What I'm trying to hack is a method to make a non-blocking call in the main thread blocking until the deferred returns a result, hence emulating the blocking nature of xmlrpc. This dependency is something I would like to phase out, but certain things need it, so work with me eh? My hack is to simply use a threading.Event to wait for the callback to be called in a non-waiting thread, save the value and do the let the main thread return the result. Of course this doesn't work when callbacks run in the main thread because the main thread is blocked. #Bits of hack code: class Block def waitFor(self, function, *args, **kwargs): if not hasattr(self,'async_done'): self.async_done = threading.Event() self.result = None #ceate a thread to wait for callback. threads.deferToThread(function, *args, **kwargs).addCallback(self._cbWait).addErrback(self._ebWait) #block on results of thread to be set self.async_done.wait() self.async_done.clear() return self.result #This is never called, because main thread is blocked def _cbWait(self, deferred): #expect original function to return a deferred deferred.addBoth(self._cbValue) def _ebWait(self, reason): print "ERROR in getting remote function: ",reason def _cbValue(self, v): self.result = v self.async_done.set() Any ideas O masters of the twisted? Gabe Rudy
On Feb 3, 2006, at 2:29 PM, Gabe Rudy wrote:
No, that's not the point of deferreds. A Deferred is just an object. It doesn't know anything at all about threads, and it sure as hell doesn't implement any sort of thread safety. All it really does it encapsulate a call chain. Technically a Deferred will process its callbacks in the same thread that callback or errback was invoked in. If you feel the need to do that, your design is probably fubar though. ... there is *one* function in Twisted that will invoke a callable in a thread and return a Deferred, but that's hardly a common usage. -bob
On Friday 03 February 2006 5:13 pm, Bob Ippolito wrote:
Ok, thanks for setting me strait. I see that my problem was really I was blocking the main thread, and hence the reactor from doing _anything_. I got a workable hack now that processes reactor events in a loop until the worker thread finishes, then returns from the function with the results (making the function appear synchronous to the caller). --Gabe
On Fri, 3 Feb 2006 18:06:52 -0700, Gabe Rudy <rudy@goldenhelix.com> wrote:
(making the function appear synchronous to the caller).
I'm not sure *exactly* what you mean by that, but generally trying to make functions look synchronous is a source of problems (*especially* if both threads are running the reactor. I hope that's not what you mean, is it?) http://glyf.livejournal.com/40037.html
On Fri, 3 Feb 2006 18:06:52 -0700, Gabe Rudy <rudy@goldenhelix.com> wrote:
Hack is exactly the right word to describe this. The only way to implement this is by going behind Twisted's back, using unsupported APIs, and risking having the whole thing fall apart in the next point release (that's assuming you get it to work reliably at all, which is a long shot at best). I recommend working very hard to eliminate this style of usage from your code, and not introducing any further instances of it. Jean-Paul
Gabe Rudy wrote:
No. The exact opposite in fact. The purpose of a deferred is to act as a placeholder for a result that will be available later, and to call callbacks or errbacks that receive that result, in a single threaded event-driven system.
in the main thread, but is there away to have a deferred process it's callback in a non-main thread?
If I understand you, the deferred callback will take a long time. Your code is indicative of a misunderstanding of deferreds, which you need not feel bad about - they're not an easy concept at first. Broadly, if you have code that looks like this under synchronous frameworks: def operation(): res = blockingCall() # the following function takes time res2 = doCalc(res) return res2.field ...that needs to be turned into: def asyncOperation(): deferred = asyncCall() deferred.addCallbacks(asyncOperation_2, log.err) return deferred def asyncOperation_2(value): # doCalc is still expensive deferred = deferToThread(doCalc, value) deferred.addCallbacks(asyncOperation_3, log.err) # by returning a deferred, the deferred that calls us will be # paused until this "INNER" deferred finishes, so it all "just works" return deferred def asyncOperation_3(doCalc_result): # this is a trivial reformatting operation return doCalc_result.field
On Feb 3, 2006, at 2:29 PM, Gabe Rudy wrote:
No, that's not the point of deferreds. A Deferred is just an object. It doesn't know anything at all about threads, and it sure as hell doesn't implement any sort of thread safety. All it really does it encapsulate a call chain. Technically a Deferred will process its callbacks in the same thread that callback or errback was invoked in. If you feel the need to do that, your design is probably fubar though. ... there is *one* function in Twisted that will invoke a callable in a thread and return a Deferred, but that's hardly a common usage. -bob
On Friday 03 February 2006 5:13 pm, Bob Ippolito wrote:
Ok, thanks for setting me strait. I see that my problem was really I was blocking the main thread, and hence the reactor from doing _anything_. I got a workable hack now that processes reactor events in a loop until the worker thread finishes, then returns from the function with the results (making the function appear synchronous to the caller). --Gabe
On Fri, 3 Feb 2006 18:06:52 -0700, Gabe Rudy <rudy@goldenhelix.com> wrote:
(making the function appear synchronous to the caller).
I'm not sure *exactly* what you mean by that, but generally trying to make functions look synchronous is a source of problems (*especially* if both threads are running the reactor. I hope that's not what you mean, is it?) http://glyf.livejournal.com/40037.html
On Fri, 3 Feb 2006 18:06:52 -0700, Gabe Rudy <rudy@goldenhelix.com> wrote:
Hack is exactly the right word to describe this. The only way to implement this is by going behind Twisted's back, using unsupported APIs, and risking having the whole thing fall apart in the next point release (that's assuming you get it to work reliably at all, which is a long shot at best). I recommend working very hard to eliminate this style of usage from your code, and not introducing any further instances of it. Jean-Paul
Gabe Rudy wrote:
No. The exact opposite in fact. The purpose of a deferred is to act as a placeholder for a result that will be available later, and to call callbacks or errbacks that receive that result, in a single threaded event-driven system.
in the main thread, but is there away to have a deferred process it's callback in a non-main thread?
If I understand you, the deferred callback will take a long time. Your code is indicative of a misunderstanding of deferreds, which you need not feel bad about - they're not an easy concept at first. Broadly, if you have code that looks like this under synchronous frameworks: def operation(): res = blockingCall() # the following function takes time res2 = doCalc(res) return res2.field ...that needs to be turned into: def asyncOperation(): deferred = asyncCall() deferred.addCallbacks(asyncOperation_2, log.err) return deferred def asyncOperation_2(value): # doCalc is still expensive deferred = deferToThread(doCalc, value) deferred.addCallbacks(asyncOperation_3, log.err) # by returning a deferred, the deferred that calls us will be # paused until this "INNER" deferred finishes, so it all "just works" return deferred def asyncOperation_3(doCalc_result): # this is a trivial reformatting operation return doCalc_result.field
participants (5)
-
Bob Ippolito -
Gabe Rudy -
glyph＠divmod.com -
Jean-Paul Calderone -
Phil Mayers