[Twisted-Python] defer.gatherResults & Deferreds chaining
I'm trying to understand why a chain of Deferreds I'm dealing with hangs and why return values from some callbacks are not always making it to other callbacks down the chain of Deferreds. Is it possible that there could be a race condition between Deferred chaining and Deferred result gathering<http://twistedmatrix.com/documents/current/core/howto/defer.html> ? I.e., Chaining Deferreds<http://twistedmatrix.com/documents/current/core/howto/defer.html#auto13> says that "*If you need one Deferred to wait on another, all you need to do is return a Deferred from a method added to addCallbacks. Specifically, if you return Deferred B from a method added to Deferred A using A.addCallbacks, Deferred A's processing chain will stop until Deferred B's .callback() method is called; at that point, the next callback in A will be passed the result of the last callback in Deferred B's processing chain at the time*." Now at the same time, DeferredList<http://twistedmatrix.com/documents/current/core/howto/defer.html#auto8> (on top of which gatherResults is built) warns that "*If you want to apply callbacks to the individual Deferreds that go into the DeferredList, you should be careful about when those callbacks are added. The act of adding a Deferred to a DeferredList inserts a callback into that Deferred ... The important thing to remember is that it is this callback which records the value that goes into the result list handed to the DeferredList's callback.* *Therefore, if you add a callback to the Deferred after adding the Deferred to the DeferredList, the value returned by that callback will not be given to the DeferredList's callback. To avoid confusion, we recommend not adding callbacks to a Deferred once it has been used in a DeferredList*" Given these two explanations, say I have something like this: --------------------------------------------------------------------------- def slowFuncReturningDeferred_A(): ... def slowFuncReturningDeferred_B(): ... d = defer.Deferred() d.addCallback(labmda _: slowFuncReturningDeferred_A()) d.addCallback(labmda _: slowFuncReturningDeferred_B()) defer.gatherResults([d]) --------------------------------------------------------------------------- Wouldn't it be possible that defer.gatherResults inserts its callback into d before say slowFuncReturningDeferred_B gets to run and return its own Deferred (and therefore chain it to what gatherRestults is ultimately waiting for?). If so, wouldn't the results returned by slowFuncReturningDeferred_B never make it to the results gathered by defer.gatherResults? Thanks, Jorge
On Jul 29, 2013, at 1:29 PM, Jorge Gonzalez <gjorge@google.com> wrote:
Wouldn't it be possible that defer.gatherResults inserts its callback into d before say slowFuncReturningDeferred_B gets to run and return its own Deferred (and therefore chain it to what gatherRestults is ultimately waiting for?). If so, wouldn't the results returned by slowFuncReturningDeferred_B never make it to the results gathered by defer.gatherResults?
No, that's not what happens. You call addCallback(lambda _: slowFunc...A()), then you call gatherResults. That means gatherResult's callback is after slowFuncReturningA() in d's callback chain. Can you attach an actual runnable code example with some behavior that you didn't expect, so we can actually run it and explain what's happening? No need for you to spin the reactor or anything, just instantiate a Deferred and then fire it. -glyph
Thanks Glyph. I actually am not able to reduce the behavior to a simpler runnable example yet, and that's why I tried running my theory first through this list. I did try a simple example just instantiating some Deferreds, firing them and gathering their results into a DeferredList (no reactor involved) and what you're saying holds true. So I'm working on getting a simpler version of my code that exposes the behavior I'm seeing and will post it once I have it. Jorge On Mon, Jul 29, 2013 at 4:04 PM, Glyph <glyph@twistedmatrix.com> wrote:
On Jul 31, 2013, at 10:08 AM, Jorge Gonzalez <gjorge@google.com> wrote:
I found the root cause, and it has nothing to do with Deferreds nor DeferredLists. It was something else in my code unrelated to twisted.
As is often the case :). This is why we ask for complete examples. Just because you see Twisted frames in your traceback does not mean Twisted is the cause of the bug :). -glyph
On Jul 29, 2013, at 1:29 PM, Jorge Gonzalez <gjorge@google.com> wrote:
Wouldn't it be possible that defer.gatherResults inserts its callback into d before say slowFuncReturningDeferred_B gets to run and return its own Deferred (and therefore chain it to what gatherRestults is ultimately waiting for?). If so, wouldn't the results returned by slowFuncReturningDeferred_B never make it to the results gathered by defer.gatherResults?
No, that's not what happens. You call addCallback(lambda _: slowFunc...A()), then you call gatherResults. That means gatherResult's callback is after slowFuncReturningA() in d's callback chain. Can you attach an actual runnable code example with some behavior that you didn't expect, so we can actually run it and explain what's happening? No need for you to spin the reactor or anything, just instantiate a Deferred and then fire it. -glyph
Thanks Glyph. I actually am not able to reduce the behavior to a simpler runnable example yet, and that's why I tried running my theory first through this list. I did try a simple example just instantiating some Deferreds, firing them and gathering their results into a DeferredList (no reactor involved) and what you're saying holds true. So I'm working on getting a simpler version of my code that exposes the behavior I'm seeing and will post it once I have it. Jorge On Mon, Jul 29, 2013 at 4:04 PM, Glyph <glyph@twistedmatrix.com> wrote:
On Jul 31, 2013, at 10:08 AM, Jorge Gonzalez <gjorge@google.com> wrote:
I found the root cause, and it has nothing to do with Deferreds nor DeferredLists. It was something else in my code unrelated to twisted.
As is often the case :). This is why we ask for complete examples. Just because you see Twisted frames in your traceback does not mean Twisted is the cause of the bug :). -glyph
participants (2)
-
Glyph -
Jorge Gonzalez