[Twisted-Python] wired problem with deferreds
hi im stuck with the following deferred construct - the problem is that cb2 is called with None instead of the expected argument from twisted.internet import defer cd = None def req1(): return defer.Deferred().addCallback(req2) def req2(a): print 'req2', a global cd if not cd: cd = defer.Deferred().addCallback(req3) return cd def req3(a): print 'req3', a return a def cb1(a): print 'cb1', a return a def cb2(a): print 'cb2', a return a d1 = req1().addCallback(cb1) d1.callback('X') d2 = req1().addCallback(cb2) d2.callback('X') cd.callback('S') output: req2 X req2 X req3 S cb1 S cb2 None # <- ? req1 fetches a mapping for given arguments - req2 tries to create an object + additional resquests by the result of req1 - but these objects should only be created once - so far i didn't find an acceptable workaround for that - req3 applys some modifications but the problem occurs also without the step in req3 is there something i overlooked so that cb2 isn't called with the required arg? thanks markus
On Feb 3, 2008 2:41 PM, markus espenhain <me@mocisoft.com> wrote:
is there something i overlooked so that cb2 isn't called with the required arg?
What are you trying to do? This is quite a convoluted example. Nonetheless, this does make me wonder what the heck is going on. To appease my curiosity, I added debug statements to Defer._runCallbacks: ... try: self._runningCallbacks = True try: print 'calling %r (%r)' % (callback, self) self.result = callback(self.result, *args, **kw) print 'Got result', self.result finally: self._runningCallbacks = False if isinstance(self.result, Deferred): ... And this is the output: ... calling <function cb1 at 0x82a3e9c> (<Deferred at 0xb7a34c8cL current result: 'S'>) cb1 S Got result S Got result None ... WTF? How on earth can this even happen??? -- \\\\\/\"/\\\\\\\\\\\ \\\\/ // //\/\\\\\\\ \\\/ \\// /\ \/\\\\ \\/ /\/ / /\/ /\ \\\ \/ / /\/ /\ /\\\ \\ / /\\\ /\\\ \\\\\/\ \/\\\\\/\\\\\/\\\\\\ d.p.s
Hi, it looks to me like the chain of deferred callbacks is broken by this. At least i think deferreds aren't supposed to return the result of last callbacked function in the chain. If that is true, then your code fails on this: You have d1, d2, cd. Before callbacking cd, the callback chains will look like this: xd1 = half-done d1, callback chain: [cb1] xd2 = half-done d2, callback chain: [cb2] cd, callback chain: [req3, xd1, xd2] the cd's chain processing will take req3 and it's return value, 'S' will give to the xd1 (_continue method). xd1 will unpause itself and give the 'S' to cb1. The point is here, that although cb1 returns 'S', it is only relevant to xd1 aka d1 chain processing. The d1 deferred does return None. So that goes to next in cd's chain, the xd2. xd2 will give None to cb2 and there goes your output. The deferred chaining works, because if some addCallback'ed' function returns a deferred, then the current deffered adds it's '_continue' method to that deferred's chain. That way it gets the result values and can process the chain further then. The short answer is probably: You can return a deferred(s) from addCallbackeds methods and wait for them, but should only wait in one place for one deferred. If returning the same deferred in multiple places, only the first will get the result of that deferred chain (and i thint that this is certainly not the common use case ;) Think about some tree-like-pattern, that branches only one way. Otherwise it's really twisted ;) Probably you should use an array of distinct deferreds in req3, although I'm not sure of the purpose of the code. Hope this helps. Petr markus espenhain wrote:
markus espenhain <me@mocisoft.com> writes:
I think it's because you're effectively pausing two callback chains on the same nested deferred, and it's the continuation processing of that deferred for the first callback chain that is clearing your result for the second callback chain. There are three deferred's involved above, with these initial callbacks: d1 - Callbacks req2, then cb1 d2 - Callbacks req2 then cb2 cd - Callback req3 Now in each of the d1.callback() and d2.callback() calls, the callback chain runs the first callback req2(), and then pauses because req2 returns a deferred (cb) that hasn't finished. It's important to realize that the way the deferred object "pauses" a callback is to simply insert its own continuation method as a final callback on the nested deferred. So, after those two callback() methods, the cd deferred actually has a callback chain of req3, d1._continue, and d2._continue. It's also important to note that the internal _continue method of the deferred object has no return (thus None). That's where your result is being lost along the way to cb2. So, when you finally call cd.callback('S'), I believe you actually trigger a sequence of events such as: req3('S'), returning 'S' d1._continue('S'), returning None and unpausing d1 (recursively) cb1('S') due to the next callback in d1's callback chain. d2._continue(None), returning None and unpausing d2 (recursively) cb2(None) due to the next callback in d2's callback chain. You'll find that if you temporarily add code to Deferred._continue to pass through the result it is handed that you see it show up. I don't know if sharing a nested deferred chain among more than one outer deferred chain something that can be expected to have sane behavior. Then again, I'm not sure it's really a bad thing to have the Deferred object pass through the result in addition to using it for itself (e.g., adding a "return result" to the Deferred._continue method). You might find that if you need to perform some sort of singleton action on the underlying operation, that you would be best off manually handling how that callback affects others than relying on the default Twisted behavior of nested deferreds. -- David
On Feb 3, 2008 2:41 PM, markus espenhain <me@mocisoft.com> wrote:
is there something i overlooked so that cb2 isn't called with the required arg?
What are you trying to do? This is quite a convoluted example. Nonetheless, this does make me wonder what the heck is going on. To appease my curiosity, I added debug statements to Defer._runCallbacks: ... try: self._runningCallbacks = True try: print 'calling %r (%r)' % (callback, self) self.result = callback(self.result, *args, **kw) print 'Got result', self.result finally: self._runningCallbacks = False if isinstance(self.result, Deferred): ... And this is the output: ... calling <function cb1 at 0x82a3e9c> (<Deferred at 0xb7a34c8cL current result: 'S'>) cb1 S Got result S Got result None ... WTF? How on earth can this even happen??? -- \\\\\/\"/\\\\\\\\\\\ \\\\/ // //\/\\\\\\\ \\\/ \\// /\ \/\\\\ \\/ /\/ / /\/ /\ \\\ \/ / /\/ /\ /\\\ \\ / /\\\ /\\\ \\\\\/\ \/\\\\\/\\\\\/\\\\\\ d.p.s
Hi, it looks to me like the chain of deferred callbacks is broken by this. At least i think deferreds aren't supposed to return the result of last callbacked function in the chain. If that is true, then your code fails on this: You have d1, d2, cd. Before callbacking cd, the callback chains will look like this: xd1 = half-done d1, callback chain: [cb1] xd2 = half-done d2, callback chain: [cb2] cd, callback chain: [req3, xd1, xd2] the cd's chain processing will take req3 and it's return value, 'S' will give to the xd1 (_continue method). xd1 will unpause itself and give the 'S' to cb1. The point is here, that although cb1 returns 'S', it is only relevant to xd1 aka d1 chain processing. The d1 deferred does return None. So that goes to next in cd's chain, the xd2. xd2 will give None to cb2 and there goes your output. The deferred chaining works, because if some addCallback'ed' function returns a deferred, then the current deffered adds it's '_continue' method to that deferred's chain. That way it gets the result values and can process the chain further then. The short answer is probably: You can return a deferred(s) from addCallbackeds methods and wait for them, but should only wait in one place for one deferred. If returning the same deferred in multiple places, only the first will get the result of that deferred chain (and i thint that this is certainly not the common use case ;) Think about some tree-like-pattern, that branches only one way. Otherwise it's really twisted ;) Probably you should use an array of distinct deferreds in req3, although I'm not sure of the purpose of the code. Hope this helps. Petr markus espenhain wrote:
markus espenhain <me@mocisoft.com> writes:
I think it's because you're effectively pausing two callback chains on the same nested deferred, and it's the continuation processing of that deferred for the first callback chain that is clearing your result for the second callback chain. There are three deferred's involved above, with these initial callbacks: d1 - Callbacks req2, then cb1 d2 - Callbacks req2 then cb2 cd - Callback req3 Now in each of the d1.callback() and d2.callback() calls, the callback chain runs the first callback req2(), and then pauses because req2 returns a deferred (cb) that hasn't finished. It's important to realize that the way the deferred object "pauses" a callback is to simply insert its own continuation method as a final callback on the nested deferred. So, after those two callback() methods, the cd deferred actually has a callback chain of req3, d1._continue, and d2._continue. It's also important to note that the internal _continue method of the deferred object has no return (thus None). That's where your result is being lost along the way to cb2. So, when you finally call cd.callback('S'), I believe you actually trigger a sequence of events such as: req3('S'), returning 'S' d1._continue('S'), returning None and unpausing d1 (recursively) cb1('S') due to the next callback in d1's callback chain. d2._continue(None), returning None and unpausing d2 (recursively) cb2(None) due to the next callback in d2's callback chain. You'll find that if you temporarily add code to Deferred._continue to pass through the result it is handed that you see it show up. I don't know if sharing a nested deferred chain among more than one outer deferred chain something that can be expected to have sane behavior. Then again, I'm not sure it's really a bad thing to have the Deferred object pass through the result in addition to using it for itself (e.g., adding a "return result" to the Deferred._continue method). You might find that if you need to perform some sort of singleton action on the underlying operation, that you would be best off manually handling how that callback affects others than relying on the default Twisted behavior of nested deferreds. -- David
participants (4)
-
David Bolen -
Drew Smathers -
markus espenhain -
Petr Mifek