Steve Freitas <sflist@ihonk.com> writes:
Hi all,
I've got an XML-RPC app that has to grab some data from a SQL database, then depending on the results, make some more SQL calls, make some more decisions, then finally return a result. So, I'm hip-deep in Deferreds. So, here's the idiom I've come up with, and I'm running into a limitation with it, namely that I can't seem to add callbacks to a Deferred after it has started running. Here's what I'm doing in abbreviated form, and it's an idiom I'd like to stick with, if possible:
You can certainly add a callback to a Deferred that has already fired (it will immediately run synchronously), but yes, you can't do it during the callback/errback operation (e.g., adding to a Deferred from within one of that Deferred's callbacks/errbacks as it is running).
class MyXmlRpcClass(xmlrpc.XMLRPC): def__init__(self): self.db = adbapi.ConnectionPool(blah)
def xmlrpc_foo(self): return FooClass(self.db).step1() # This returns a deferred, see below
class FooClass(General): def __init__(self, db): self.db = db
def step1(self): self.d = self.db.runQuery(blah) self.d.addCallback(self.step2) return self.d
def step2(self, query): if query == 'yadda': return 'This bit works' else: self.d.chainDeferred(self.db.runOperation(blah)) self.d.addCallback(self.step3)
def step3(self, data): return "Never gets here!"
If I get to the part where it adds the callback for step3, it ends up giving an AlreadyCalled exception in Deferred. So, I expect there's a good way to add to a running Deferred, no?
I don't think you actually want to "add to a running Deferred" here, since even if your chainDeferred call worked, it would be chaining at the _end_ of the Deferred's callback/errback chain, which might be after a whole slew of other operations that your callers (who got the deferred from step1) already added. So it wouldn't occur in the sequence you want anyway. What you really want to do (I believe) is the opposite chain operation - you want to take the Deferred from the runOperation(blah), and chain it to your current Deferred, so that your current Deferred's callback/errback chain waits on the runOperation result to continue working. The actual chaining is easy (just change your use of chainDeferred), but the hard part is making the current callback chain suspend itself until the new Deferred finishes. Luckily, Twisted already provides explicit support for this behavior (which is sort of crucial to permit deferrable operations to interact properly anyway). What I think you want (similar to what I showed in an earlier message of mine in this thread) is simply to return the new deferred from within your callback. Twisted will automatically notice that the callback itself is awaiting a response via the deferred, and it establishes the necessary linkage. Once the new internal Deferred finishes, it is its result (whether successful or a Failure object) that will gate whether the original Deferreds chain continues up callback or errback. You can peek at the _runCallbacks inside the Deferred class definition if you want to see the mechanism. Or check out the first paragraph in the "Chaining Deferreds" section of the Using Deferreds HOWTO. Interestingly enough, the HOWTO (next paragraph) makes mention of it potentially being confusing but that the reader will probably recognize the need when they run into it. Clearly that doesn't always happen :-) There's two ways, I think you can do FooClass, depending on how you want to conceptually insert step3 into the processing: The first, fairly literal adjustment of your class would be: [Case 1] class FooClass(General): def __init__(self, db): self.db = db def step1(self): return self.db.runQuery(blah).addCallback(self.step2) def step2(self, query): if query == 'yadda': return 'This bit works' else: return self.db.runOperation(blah).addCallback(self.step3) def step3(self, data): # 'data' is result of self.db.runOperation(blah) # Note that the result of this method will become the callback # result of the self.db.runOperation(blah) Deferred, and thus # in turn the next result passed up the callback chain for the # first self.db.runQuery(blah) Deferred as the answer from step2 # in the non-yadda case. return "I got here!' or: [Case 2] class FooClass(General): def __init__(self, db): self.db = db def step1(self): d = self.db.runQuery(blah) return d.addCallback(self.step2).addCallback(self.step2) def step2(self, query): if query == 'yadda': return 'This bit works' else: return self.db.runOperation(blah) def step3(self, data): # 'data' is result of step2 # Note that the result of this method will become the callback # in the main callback chain for the first self.db.runQuery(blah) # Deferred, and will thus replace that of step2 return "I got here!' (Actually, there's also a third way in which you manually pause and unpause the main Deferred callback chain, and set up a callback on the secondary Deferred chain to resume the main one, but that's precisely what Twisted does for you when you just return the underlying Deferred from your callback, so I don't see any benefit of doing it yourself). In both of these cases there are two deferred chains running (arising from the two discrete deferrable operations that themselves are creating Deferreds), but that in the even of the non-yadda path in step2, the first callback chain is suspended while awaiting completion of the second. In Case 1, the addCallback of step3 is to the secondary callback chain, so it provides additional post-processing of the runOperation result, but does not get involved in other paths/results from step2. In Case 2, step3 is added to the main callback (runQuery) chain, so it will see any results of step2 (both yadda and runOperation) and can then post-process them, becoming in all cases the callback result for the main callback chain. Note a key point here - even though step3 is next in the main callback chain after step2, in the event step2 returns the Deferred from runOperation, the main chain suspends (so step3 doesn't run immediately) until that second Deferred finishes, at which point it is the result that step3 sees. So what you end up (in the non-yadda case) is something like (in poor ASCII diagraming): Case 1: Case 2: runQuery runQuery | | step2 step2 | | | no | no +-yadda->-runOperation +-yadda->-runOperation | | | | yadda v yadda v | | | | +---<------ step3 +---<----------+ | | v step3 | v Hope this helps clarify things somewhat. -- David