[Twisted-Python] Twisted is now under the MIT license.
![](https://secure.gravatar.com/avatar/46f609c9bea026767ebae519e0656656.jpg?s=120&d=mm&r=g)
I just switched everything in Subversion from LGPL to MIT, at glyph's command. The policy that he outlined: 1) All contributions or inclusions must be MIT licensed. 1.a) If you want to include someone else's code in Twisted, it still has to be MIT licensed, but keep their own copyright notice intact. 2) Anybody who's a contributor and wants their name listed as a copyright holder can have their name added to the LICENSE file (I copied everyone from CREDITS into LICENSE initally). 3) Put this preamble at the top of files: # Copyright (c) 2001-2004 Twisted Matrix Laboratories. # See LICENSE for details. Have fun. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com
![](https://secure.gravatar.com/avatar/3bef09da3292c944649ffc28673df870.jpg?s=120&d=mm&r=g)
On Wed, Aug 25, 2004, Christopher Armstrong wrote:
Can you clarify this a bit? If individual contributions are now copyright their original author, surely 2 only applies to Twisted code as it stood at the time of your last commits. Copyright can't be blanket asserted in the LICENSE file over code that is added after that point. Or am I misunderstanding what the LICENSE file is saying? -Mary
![](https://secure.gravatar.com/avatar/3bef09da3292c944649ffc28673df870.jpg?s=120&d=mm&r=g)
On Wed, Aug 25, 2004, Mary Gardiner wrote:
OK, looking at it it's unclear whether the LICENSE file asserts that: - the people listed there jointly hold copyright over all of Twisted (which may be true now, but won't be as pieces get added[1]) or - the people listed there individually hold copyright over pieces of Twisted -Mary [1] I assume joint copyright is fine under US law in theory, it's just that individual pieces of Twisted are apparently going to retain their copyright now.
![](https://secure.gravatar.com/avatar/46f609c9bea026767ebae519e0656656.jpg?s=120&d=mm&r=g)
To Mary and Alex: Ask a lawyer (or someone else who's silly enough to answer :). This is how a lot of open source project's are run. ESR's Licensing HOWTO may or may not be accurate, but if you assume it is correct, it is enlightening. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com
![](https://secure.gravatar.com/avatar/46f609c9bea026767ebae519e0656656.jpg?s=120&d=mm&r=g)
On Wed, 25 Aug 2004 17:33:22 -0700, Andrew Bennetts <andrew-twisted@puzzling.org> wrote:
Well, I was focusing on his "what's the difference", which I can't answer. I guess I'll risk it: The rationale is that MIT (presumably) allows people to derive from the work without keeping derived works open, and that's what glyph wanted now that Twisted has matured. Also, the Twisted codebase is (presumably) not at the whim of glyph any more, since all contributors (if they want) are listed as copyright holders, not just glyph. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com
![](https://secure.gravatar.com/avatar/b3407ff6ccd34c6e7c7a9fdcfba67a45.jpg?s=120&d=mm&r=g)
On Wed, Aug 25, 2004 at 08:43:57PM -0400, Christopher Armstrong wrote:
To be clear on this: contributions are now *only* copyright their authors, (rather than the previous joint copyright-assignment to glyph)? And so when you say "copyright holders", you're referring to everyone that has copyright in any part of Twisted's source, rather than saying that everyone is now joint holders of copyright? Also, the new source file boilerplate says: # Copyright (c) 2001-2004 Twisted Matrix Laboratories. # See LICENSE for details. But as far as I know, Twisted Matrix Laboratories isn't a legally recognised organisation (in any country), and regardless it isn't mentioned in the LICENSE file, despite the boilerplate's promise that it would have details. This is probably a minor issue, but it is confusing. -Andrew.
![](https://secure.gravatar.com/avatar/46f609c9bea026767ebae519e0656656.jpg?s=120&d=mm&r=g)
On Wed, 25 Aug 2004 18:42:17 -0700, Andrew Bennetts <andrew-twisted@puzzling.org> wrote: [A bunch of horrible scary questions about copyright and license] This is getting too hairy for me. Glyph, you take it. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com
![](https://secure.gravatar.com/avatar/3c4988f83703127d279406fc6eea7079.jpg?s=120&d=mm&r=g)
Hi all, I've got an XML-RPC app that has to grab some data from a SQL database, then depending on the results, make some more SQL calls, make some more decisions, then finally return a result. So, I'm hip-deep in Deferreds. So, here's the idiom I've come up with, and I'm running into a limitation with it, namely that I can't seem to add callbacks to a Deferred after it has started running. Here's what I'm doing in abbreviated form, and it's an idiom I'd like to stick with, if possible: class MyXmlRpcClass(xmlrpc.XMLRPC): def__init__(self): self.db = adbapi.ConnectionPool(blah) def xmlrpc_foo(self): return FooClass(self.db).step1() # This returns a deferred, see below class FooClass(General): def __init__(self, db): self.db = db def step1(self): self.d = self.db.runQuery(blah) self.d.addCallback(self.step2) return self.d def step2(self, query): if query == 'yadda': return 'This bit works' else: self.d.chainDeferred(self.db.runOperation(blah)) self.d.addCallback(self.step3) def step3(self, data): return "Never gets here!" If I get to the part where it adds the callback for step3, it ends up giving an AlreadyCalled exception in Deferred. So, I expect there's a good way to add to a running Deferred, no? Thanks! Steve
![](https://secure.gravatar.com/avatar/c61adf126fc39fcc328e9fb244d925f2.jpg?s=120&d=mm&r=g)
Steve Freitas wrote:
This wasn't spectacularly obvious to me the first time either, but I think it's covered somewhere in the deferred execution howto; in your callback/errback, return a new deferred and the processing of the deferred chain will pause until that new deferred fires. Here's an example of code that does that... df = self.shouldRetrieve( urlRecord ) def ifWeShould( judgement, urlRecord ): if judgement: df = urlRecord.retrieve( self.timeout, self.userAgent ) df.addCallback( self.onSuccess, urlRecord = urlRecord ) df.addErrback( self.onFailure, urlRecord = urlRecord ) return df else: return self.onFailure( failure.Failure( RobotDenied(urlRecord)), urlRecord = urlRecord ) df.addCallback( ifWeShould, urlRecord=urlRecord ) In the one branch, we return the result of a call, in the other, we return a new Deferred. Note that the new deferred shows up between the current callback and any Deferred higher up in the chain, so the Deferred's higher up the chain will get the result of the new Deferred. If that's not desirable, you can define a handler for the new deferred that returns the original result. Hope I understood the question properly and that this helped, Mike ________________________________________________ Mike C. Fletcher Designer, VR Plumber, Coder http://www.vrplumber.com http://blog.vrplumber.com
![](https://secure.gravatar.com/avatar/3c4988f83703127d279406fc6eea7079.jpg?s=120&d=mm&r=g)
Thanks for your reply, Mike! I don't thoroughly understand your code yet, but I'm workin' on it. One of the things I don't like about the examples is that they don't demonstrate a conditional addition of a callback. One of the things I did try was returning a new Deferred from step2(), but it didn't work, and I'm not sure why. I'm checking out defer.py now to try to figure out how it works... Steve
![](https://secure.gravatar.com/avatar/3c4988f83703127d279406fc6eea7079.jpg?s=120&d=mm&r=g)
This wasn't spectacularly obvious to me the first time either, but I think it's covered somewhere in the deferred execution howto
Ah ha! I think I figured it out. If I do this, it seems to work... class XMLRPCResponseClass(): def step1(self): self.d = self.db.runQuery(blah) self.d.addCallback(self.step2) return self.d def step2(self, query): # Here's where the change is if query == 'yadda': return 'This bit works' else: self.d = self.db.runOperation(blah) # Rebinding self.d to new # Deferred() self.d.addCallback(self.step3) # add callbacks to new one return self.d # And return ref to the new one def step3(self, data): return 'Now it gets here!' The trick is that I need to add any subsequent callbacks to a new deferred which I return from the function. Strangely (to me), if I try using chainDeferred() instead of returning the new deferred from step2, it fails with AlreadyCalled, which I don't yet understand, but that's fine, 'cause this seems to work well enough. I really like this technique because I don't have to use a big state machine at the root level of the xmlrpc calls, or preassign possibly unnecessary callbacks, or toss around callables as variables until I'm thoroughly confused. Instead, it allows me to place all the (client||server) state information for a given method call inside a class instance which persists only until the xmlrpc method returns. And multiple calls to this method while it's being simultaneously Deferred() for other clients don't require any extra coding to handle, as I've got one class instance per connection. Very clean, very well separated, which is hard to figure out how to do with this wonderful framework. Alright, I'll stop ranting now. :-) I still have to give this thing a workout to make sure it does all I want it to. Thanks for the help! Steve
![](https://secure.gravatar.com/avatar/6cd37343ce1da139b1d2ff5ee717f5d5.jpg?s=120&d=mm&r=g)
Steve, I'm all too familar with the situation you describe, and I've come up with a pattern that works well for me. Here's how I would have written your example: class XMLRPCResponseClass(): def step1(self): finished = defer.Deferred() self.db.runQuery(blah).addCallback( self.step2, finished).addErrback(finished.errback) return finished def step2(self, result, finished): if result == 'yadda': finished.callback('This bit works') else: self.d.runOperation(blah).chainDeferred(finished) It takes a while to really understand how Deferreds work, but I'll try to explain this example. In step1, we create a new Deferred called 'finished'. This Deferred didn't come from a method we called; we created it from scratch. This means that it won't be triggered automatically when some event finishes. Instead, we have the freedom to call it whenever we want. We then kick off the asyncronous operation self.db.runQuery. We set up the callbacks as follows: 1. If runQuery completes successfully, call step2(), passing finished as an additional argument. 2. If runQuery fails, pass the error on to finished. Then we return finished, our Deferred. At this point finished.errback will get called if runQuery fails. But what if runQuery succeeds? That will be handled by step2. In step2, we take the result of runQuery , as well as the 'finished' Deferred we created in step1. Now we have the results of the first operation, as well as a reference to the deferred that was returned in step1. If the result of runQuery was 'yadda', we callback finished with a successful value. Otherwise, we kick of a second asyncronous operation, (self.d.runOperation) and chain the results to finished. Remember that a.chainDeferred(b) doesn't do anything special, it's just shorthand for: a.addCallback(b.callback) a.addErrback(b.errback) So the result is that finished ends up calling back or erring back with the results of self.d.runOperation, even though it wasn't bound to runOperation when it was first created. The only thing to be careful of with this technique is that you have to make sure any operation that could possibly fail sends its error to finished.errback - otherwise finished will never be finished :-) I hope that's helpful! I'm happy to answer questions if you have any. Abe p.s. - It's possible there's a better way to do this sort of thing. But I'm using the above technique extensively in Hep and Yarn without any problems. Steve Freitas wrote:
![](https://secure.gravatar.com/avatar/2c498e6b589e4a4318a8280da536fb36.jpg?s=120&d=mm&r=g)
Abe Fettig <abe@fettig.net> writes:
I'm not sure I follow the need for the extra deferred. I would think that the following would work just as well: class XMLRPCResponseClass(): def step1(self): return self.db.runQuery(blah).addCallback(self.step2) def step2(self, result): if result == 'yadda': return 'This bit works' else: return self.db.runOperation(blah) In this case, the primary deferred is the original runQuery. The first callback on that forwards you to step 2. A failure would flow up the errback chain which you don't process locally but leave to the caller to handle. In the success case, if you need to additionally process the result you just return the new value (the 'yadda' test), otherwise you spin off another deferred operation, returning that deferred from your callback. Returning a deferred within a callback is an implicit chaining operation as Twisted will wait for that new deferred to finish before taking its result and propagating it up the callback or errback chain appropriately. Remember that when you add your callbacks/errbacks you are in control of the sequence of execution as the callback/errback chain fires, and thus you'll always get control in advance of any other callback/errbacks that callers who you return the same deferred to might be attaching. That, plus the implicit chaining Twisted permits by an individual function in the callback/errback chain itself returning a deferred is pretty flexible. It's certainly been rare for me to need my own internally generated deferred unless I was the root of the deferrable operation. -- David
![](https://secure.gravatar.com/avatar/2c498e6b589e4a4318a8280da536fb36.jpg?s=120&d=mm&r=g)
Steve Freitas <sflist@ihonk.com> writes:
(should have finished reading ahead since you figured out some of my prior response :-)) The above may or may not be true though, since it depends on what the subsequent callbacks are supposed to do. For example, if step3 is processing of step2 results (anything step2 can return) then you want it added to your main callback chain for your first Deferred. If however, step3 is only supposed to post-process the results of the conditional path through step2 that invokes another deferrable routine, yes, the right place to hang it is on that Deferred and not the main Deferred. Oh, and not that I probably wouldn't suggest overwriting self.d with the intermediate deferreds if you could avoid it, if only because it's bound to be confusing down the road when maintaining the code. Since you should know everyone that is going to be involved in a callback chain at the point when a particular chain is being created (either your first one, or one that is created within a callback) you should be able to add callbacks to each chain as it is created. If they are generic processing in sequence that should work with any result from - see Case 2 in my prior response - depending on when you want the subsequent processing to run. -- David
![](https://secure.gravatar.com/avatar/2c498e6b589e4a4318a8280da536fb36.jpg?s=120&d=mm&r=g)
Steve Freitas <sflist@ihonk.com> writes:
You can certainly add a callback to a Deferred that has already fired (it will immediately run synchronously), but yes, you can't do it during the callback/errback operation (e.g., adding to a Deferred from within one of that Deferred's callbacks/errbacks as it is running).
I don't think you actually want to "add to a running Deferred" here, since even if your chainDeferred call worked, it would be chaining at the _end_ of the Deferred's callback/errback chain, which might be after a whole slew of other operations that your callers (who got the deferred from step1) already added. So it wouldn't occur in the sequence you want anyway. What you really want to do (I believe) is the opposite chain operation - you want to take the Deferred from the runOperation(blah), and chain it to your current Deferred, so that your current Deferred's callback/errback chain waits on the runOperation result to continue working. The actual chaining is easy (just change your use of chainDeferred), but the hard part is making the current callback chain suspend itself until the new Deferred finishes. Luckily, Twisted already provides explicit support for this behavior (which is sort of crucial to permit deferrable operations to interact properly anyway). What I think you want (similar to what I showed in an earlier message of mine in this thread) is simply to return the new deferred from within your callback. Twisted will automatically notice that the callback itself is awaiting a response via the deferred, and it establishes the necessary linkage. Once the new internal Deferred finishes, it is its result (whether successful or a Failure object) that will gate whether the original Deferreds chain continues up callback or errback. You can peek at the _runCallbacks inside the Deferred class definition if you want to see the mechanism. Or check out the first paragraph in the "Chaining Deferreds" section of the Using Deferreds HOWTO. Interestingly enough, the HOWTO (next paragraph) makes mention of it potentially being confusing but that the reader will probably recognize the need when they run into it. Clearly that doesn't always happen :-) There's two ways, I think you can do FooClass, depending on how you want to conceptually insert step3 into the processing: The first, fairly literal adjustment of your class would be: [Case 1] class FooClass(General): def __init__(self, db): self.db = db def step1(self): return self.db.runQuery(blah).addCallback(self.step2) def step2(self, query): if query == 'yadda': return 'This bit works' else: return self.db.runOperation(blah).addCallback(self.step3) def step3(self, data): # 'data' is result of self.db.runOperation(blah) # Note that the result of this method will become the callback # result of the self.db.runOperation(blah) Deferred, and thus # in turn the next result passed up the callback chain for the # first self.db.runQuery(blah) Deferred as the answer from step2 # in the non-yadda case. return "I got here!' or: [Case 2] class FooClass(General): def __init__(self, db): self.db = db def step1(self): d = self.db.runQuery(blah) return d.addCallback(self.step2).addCallback(self.step2) def step2(self, query): if query == 'yadda': return 'This bit works' else: return self.db.runOperation(blah) def step3(self, data): # 'data' is result of step2 # Note that the result of this method will become the callback # in the main callback chain for the first self.db.runQuery(blah) # Deferred, and will thus replace that of step2 return "I got here!' (Actually, there's also a third way in which you manually pause and unpause the main Deferred callback chain, and set up a callback on the secondary Deferred chain to resume the main one, but that's precisely what Twisted does for you when you just return the underlying Deferred from your callback, so I don't see any benefit of doing it yourself). In both of these cases there are two deferred chains running (arising from the two discrete deferrable operations that themselves are creating Deferreds), but that in the even of the non-yadda path in step2, the first callback chain is suspended while awaiting completion of the second. In Case 1, the addCallback of step3 is to the secondary callback chain, so it provides additional post-processing of the runOperation result, but does not get involved in other paths/results from step2. In Case 2, step3 is added to the main callback (runQuery) chain, so it will see any results of step2 (both yadda and runOperation) and can then post-process them, becoming in all cases the callback result for the main callback chain. Note a key point here - even though step3 is next in the main callback chain after step2, in the event step2 returns the Deferred from runOperation, the main chain suspends (so step3 doesn't run immediately) until that second Deferred finishes, at which point it is the result that step3 sees. So what you end up (in the non-yadda case) is something like (in poor ASCII diagraming): Case 1: Case 2: runQuery runQuery | | step2 step2 | | | no | no +-yadda->-runOperation +-yadda->-runOperation | | | | yadda v yadda v | | | | +---<------ step3 +---<----------+ | | v step3 | v Hope this helps clarify things somewhat. -- David
![](https://secure.gravatar.com/avatar/2c498e6b589e4a4318a8280da536fb36.jpg?s=120&d=mm&r=g)
David Bolen <db3l@fitlinxx.com> writes: (oops - minor correction to my last post)
this should be (step3 on the final callback): return d.addCallback(self.step2).addCallback(self.step3) -- David
![](https://secure.gravatar.com/avatar/3c4988f83703127d279406fc6eea7079.jpg?s=120&d=mm&r=g)
Thanks to all of you for your very informative responses, Mike, Abe and David! I've learned a lot, and gotten the information I need to continue on with my work. Everything was explained thoroughly, so I think I'm all set now. It's great to have such a helpful list to lean on. Thanks! Steve
![](https://secure.gravatar.com/avatar/3bef09da3292c944649ffc28673df870.jpg?s=120&d=mm&r=g)
On Wed, Aug 25, 2004, Christopher Armstrong wrote:
Can you clarify this a bit? If individual contributions are now copyright their original author, surely 2 only applies to Twisted code as it stood at the time of your last commits. Copyright can't be blanket asserted in the LICENSE file over code that is added after that point. Or am I misunderstanding what the LICENSE file is saying? -Mary
![](https://secure.gravatar.com/avatar/3bef09da3292c944649ffc28673df870.jpg?s=120&d=mm&r=g)
On Wed, Aug 25, 2004, Mary Gardiner wrote:
OK, looking at it it's unclear whether the LICENSE file asserts that: - the people listed there jointly hold copyright over all of Twisted (which may be true now, but won't be as pieces get added[1]) or - the people listed there individually hold copyright over pieces of Twisted -Mary [1] I assume joint copyright is fine under US law in theory, it's just that individual pieces of Twisted are apparently going to retain their copyright now.
![](https://secure.gravatar.com/avatar/46f609c9bea026767ebae519e0656656.jpg?s=120&d=mm&r=g)
To Mary and Alex: Ask a lawyer (or someone else who's silly enough to answer :). This is how a lot of open source project's are run. ESR's Licensing HOWTO may or may not be accurate, but if you assume it is correct, it is enlightening. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com
![](https://secure.gravatar.com/avatar/46f609c9bea026767ebae519e0656656.jpg?s=120&d=mm&r=g)
On Wed, 25 Aug 2004 17:33:22 -0700, Andrew Bennetts <andrew-twisted@puzzling.org> wrote:
Well, I was focusing on his "what's the difference", which I can't answer. I guess I'll risk it: The rationale is that MIT (presumably) allows people to derive from the work without keeping derived works open, and that's what glyph wanted now that Twisted has matured. Also, the Twisted codebase is (presumably) not at the whim of glyph any more, since all contributors (if they want) are listed as copyright holders, not just glyph. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com
![](https://secure.gravatar.com/avatar/b3407ff6ccd34c6e7c7a9fdcfba67a45.jpg?s=120&d=mm&r=g)
On Wed, Aug 25, 2004 at 08:43:57PM -0400, Christopher Armstrong wrote:
To be clear on this: contributions are now *only* copyright their authors, (rather than the previous joint copyright-assignment to glyph)? And so when you say "copyright holders", you're referring to everyone that has copyright in any part of Twisted's source, rather than saying that everyone is now joint holders of copyright? Also, the new source file boilerplate says: # Copyright (c) 2001-2004 Twisted Matrix Laboratories. # See LICENSE for details. But as far as I know, Twisted Matrix Laboratories isn't a legally recognised organisation (in any country), and regardless it isn't mentioned in the LICENSE file, despite the boilerplate's promise that it would have details. This is probably a minor issue, but it is confusing. -Andrew.
![](https://secure.gravatar.com/avatar/46f609c9bea026767ebae519e0656656.jpg?s=120&d=mm&r=g)
On Wed, 25 Aug 2004 18:42:17 -0700, Andrew Bennetts <andrew-twisted@puzzling.org> wrote: [A bunch of horrible scary questions about copyright and license] This is getting too hairy for me. Glyph, you take it. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com
![](https://secure.gravatar.com/avatar/3c4988f83703127d279406fc6eea7079.jpg?s=120&d=mm&r=g)
Hi all, I've got an XML-RPC app that has to grab some data from a SQL database, then depending on the results, make some more SQL calls, make some more decisions, then finally return a result. So, I'm hip-deep in Deferreds. So, here's the idiom I've come up with, and I'm running into a limitation with it, namely that I can't seem to add callbacks to a Deferred after it has started running. Here's what I'm doing in abbreviated form, and it's an idiom I'd like to stick with, if possible: class MyXmlRpcClass(xmlrpc.XMLRPC): def__init__(self): self.db = adbapi.ConnectionPool(blah) def xmlrpc_foo(self): return FooClass(self.db).step1() # This returns a deferred, see below class FooClass(General): def __init__(self, db): self.db = db def step1(self): self.d = self.db.runQuery(blah) self.d.addCallback(self.step2) return self.d def step2(self, query): if query == 'yadda': return 'This bit works' else: self.d.chainDeferred(self.db.runOperation(blah)) self.d.addCallback(self.step3) def step3(self, data): return "Never gets here!" If I get to the part where it adds the callback for step3, it ends up giving an AlreadyCalled exception in Deferred. So, I expect there's a good way to add to a running Deferred, no? Thanks! Steve
![](https://secure.gravatar.com/avatar/c61adf126fc39fcc328e9fb244d925f2.jpg?s=120&d=mm&r=g)
Steve Freitas wrote:
This wasn't spectacularly obvious to me the first time either, but I think it's covered somewhere in the deferred execution howto; in your callback/errback, return a new deferred and the processing of the deferred chain will pause until that new deferred fires. Here's an example of code that does that... df = self.shouldRetrieve( urlRecord ) def ifWeShould( judgement, urlRecord ): if judgement: df = urlRecord.retrieve( self.timeout, self.userAgent ) df.addCallback( self.onSuccess, urlRecord = urlRecord ) df.addErrback( self.onFailure, urlRecord = urlRecord ) return df else: return self.onFailure( failure.Failure( RobotDenied(urlRecord)), urlRecord = urlRecord ) df.addCallback( ifWeShould, urlRecord=urlRecord ) In the one branch, we return the result of a call, in the other, we return a new Deferred. Note that the new deferred shows up between the current callback and any Deferred higher up in the chain, so the Deferred's higher up the chain will get the result of the new Deferred. If that's not desirable, you can define a handler for the new deferred that returns the original result. Hope I understood the question properly and that this helped, Mike ________________________________________________ Mike C. Fletcher Designer, VR Plumber, Coder http://www.vrplumber.com http://blog.vrplumber.com
![](https://secure.gravatar.com/avatar/3c4988f83703127d279406fc6eea7079.jpg?s=120&d=mm&r=g)
Thanks for your reply, Mike! I don't thoroughly understand your code yet, but I'm workin' on it. One of the things I don't like about the examples is that they don't demonstrate a conditional addition of a callback. One of the things I did try was returning a new Deferred from step2(), but it didn't work, and I'm not sure why. I'm checking out defer.py now to try to figure out how it works... Steve
![](https://secure.gravatar.com/avatar/3c4988f83703127d279406fc6eea7079.jpg?s=120&d=mm&r=g)
This wasn't spectacularly obvious to me the first time either, but I think it's covered somewhere in the deferred execution howto
Ah ha! I think I figured it out. If I do this, it seems to work... class XMLRPCResponseClass(): def step1(self): self.d = self.db.runQuery(blah) self.d.addCallback(self.step2) return self.d def step2(self, query): # Here's where the change is if query == 'yadda': return 'This bit works' else: self.d = self.db.runOperation(blah) # Rebinding self.d to new # Deferred() self.d.addCallback(self.step3) # add callbacks to new one return self.d # And return ref to the new one def step3(self, data): return 'Now it gets here!' The trick is that I need to add any subsequent callbacks to a new deferred which I return from the function. Strangely (to me), if I try using chainDeferred() instead of returning the new deferred from step2, it fails with AlreadyCalled, which I don't yet understand, but that's fine, 'cause this seems to work well enough. I really like this technique because I don't have to use a big state machine at the root level of the xmlrpc calls, or preassign possibly unnecessary callbacks, or toss around callables as variables until I'm thoroughly confused. Instead, it allows me to place all the (client||server) state information for a given method call inside a class instance which persists only until the xmlrpc method returns. And multiple calls to this method while it's being simultaneously Deferred() for other clients don't require any extra coding to handle, as I've got one class instance per connection. Very clean, very well separated, which is hard to figure out how to do with this wonderful framework. Alright, I'll stop ranting now. :-) I still have to give this thing a workout to make sure it does all I want it to. Thanks for the help! Steve
![](https://secure.gravatar.com/avatar/6cd37343ce1da139b1d2ff5ee717f5d5.jpg?s=120&d=mm&r=g)
Steve, I'm all too familar with the situation you describe, and I've come up with a pattern that works well for me. Here's how I would have written your example: class XMLRPCResponseClass(): def step1(self): finished = defer.Deferred() self.db.runQuery(blah).addCallback( self.step2, finished).addErrback(finished.errback) return finished def step2(self, result, finished): if result == 'yadda': finished.callback('This bit works') else: self.d.runOperation(blah).chainDeferred(finished) It takes a while to really understand how Deferreds work, but I'll try to explain this example. In step1, we create a new Deferred called 'finished'. This Deferred didn't come from a method we called; we created it from scratch. This means that it won't be triggered automatically when some event finishes. Instead, we have the freedom to call it whenever we want. We then kick off the asyncronous operation self.db.runQuery. We set up the callbacks as follows: 1. If runQuery completes successfully, call step2(), passing finished as an additional argument. 2. If runQuery fails, pass the error on to finished. Then we return finished, our Deferred. At this point finished.errback will get called if runQuery fails. But what if runQuery succeeds? That will be handled by step2. In step2, we take the result of runQuery , as well as the 'finished' Deferred we created in step1. Now we have the results of the first operation, as well as a reference to the deferred that was returned in step1. If the result of runQuery was 'yadda', we callback finished with a successful value. Otherwise, we kick of a second asyncronous operation, (self.d.runOperation) and chain the results to finished. Remember that a.chainDeferred(b) doesn't do anything special, it's just shorthand for: a.addCallback(b.callback) a.addErrback(b.errback) So the result is that finished ends up calling back or erring back with the results of self.d.runOperation, even though it wasn't bound to runOperation when it was first created. The only thing to be careful of with this technique is that you have to make sure any operation that could possibly fail sends its error to finished.errback - otherwise finished will never be finished :-) I hope that's helpful! I'm happy to answer questions if you have any. Abe p.s. - It's possible there's a better way to do this sort of thing. But I'm using the above technique extensively in Hep and Yarn without any problems. Steve Freitas wrote:
![](https://secure.gravatar.com/avatar/2c498e6b589e4a4318a8280da536fb36.jpg?s=120&d=mm&r=g)
Abe Fettig <abe@fettig.net> writes:
I'm not sure I follow the need for the extra deferred. I would think that the following would work just as well: class XMLRPCResponseClass(): def step1(self): return self.db.runQuery(blah).addCallback(self.step2) def step2(self, result): if result == 'yadda': return 'This bit works' else: return self.db.runOperation(blah) In this case, the primary deferred is the original runQuery. The first callback on that forwards you to step 2. A failure would flow up the errback chain which you don't process locally but leave to the caller to handle. In the success case, if you need to additionally process the result you just return the new value (the 'yadda' test), otherwise you spin off another deferred operation, returning that deferred from your callback. Returning a deferred within a callback is an implicit chaining operation as Twisted will wait for that new deferred to finish before taking its result and propagating it up the callback or errback chain appropriately. Remember that when you add your callbacks/errbacks you are in control of the sequence of execution as the callback/errback chain fires, and thus you'll always get control in advance of any other callback/errbacks that callers who you return the same deferred to might be attaching. That, plus the implicit chaining Twisted permits by an individual function in the callback/errback chain itself returning a deferred is pretty flexible. It's certainly been rare for me to need my own internally generated deferred unless I was the root of the deferrable operation. -- David
![](https://secure.gravatar.com/avatar/2c498e6b589e4a4318a8280da536fb36.jpg?s=120&d=mm&r=g)
Steve Freitas <sflist@ihonk.com> writes:
(should have finished reading ahead since you figured out some of my prior response :-)) The above may or may not be true though, since it depends on what the subsequent callbacks are supposed to do. For example, if step3 is processing of step2 results (anything step2 can return) then you want it added to your main callback chain for your first Deferred. If however, step3 is only supposed to post-process the results of the conditional path through step2 that invokes another deferrable routine, yes, the right place to hang it is on that Deferred and not the main Deferred. Oh, and not that I probably wouldn't suggest overwriting self.d with the intermediate deferreds if you could avoid it, if only because it's bound to be confusing down the road when maintaining the code. Since you should know everyone that is going to be involved in a callback chain at the point when a particular chain is being created (either your first one, or one that is created within a callback) you should be able to add callbacks to each chain as it is created. If they are generic processing in sequence that should work with any result from - see Case 2 in my prior response - depending on when you want the subsequent processing to run. -- David
![](https://secure.gravatar.com/avatar/2c498e6b589e4a4318a8280da536fb36.jpg?s=120&d=mm&r=g)
Steve Freitas <sflist@ihonk.com> writes:
You can certainly add a callback to a Deferred that has already fired (it will immediately run synchronously), but yes, you can't do it during the callback/errback operation (e.g., adding to a Deferred from within one of that Deferred's callbacks/errbacks as it is running).
I don't think you actually want to "add to a running Deferred" here, since even if your chainDeferred call worked, it would be chaining at the _end_ of the Deferred's callback/errback chain, which might be after a whole slew of other operations that your callers (who got the deferred from step1) already added. So it wouldn't occur in the sequence you want anyway. What you really want to do (I believe) is the opposite chain operation - you want to take the Deferred from the runOperation(blah), and chain it to your current Deferred, so that your current Deferred's callback/errback chain waits on the runOperation result to continue working. The actual chaining is easy (just change your use of chainDeferred), but the hard part is making the current callback chain suspend itself until the new Deferred finishes. Luckily, Twisted already provides explicit support for this behavior (which is sort of crucial to permit deferrable operations to interact properly anyway). What I think you want (similar to what I showed in an earlier message of mine in this thread) is simply to return the new deferred from within your callback. Twisted will automatically notice that the callback itself is awaiting a response via the deferred, and it establishes the necessary linkage. Once the new internal Deferred finishes, it is its result (whether successful or a Failure object) that will gate whether the original Deferreds chain continues up callback or errback. You can peek at the _runCallbacks inside the Deferred class definition if you want to see the mechanism. Or check out the first paragraph in the "Chaining Deferreds" section of the Using Deferreds HOWTO. Interestingly enough, the HOWTO (next paragraph) makes mention of it potentially being confusing but that the reader will probably recognize the need when they run into it. Clearly that doesn't always happen :-) There's two ways, I think you can do FooClass, depending on how you want to conceptually insert step3 into the processing: The first, fairly literal adjustment of your class would be: [Case 1] class FooClass(General): def __init__(self, db): self.db = db def step1(self): return self.db.runQuery(blah).addCallback(self.step2) def step2(self, query): if query == 'yadda': return 'This bit works' else: return self.db.runOperation(blah).addCallback(self.step3) def step3(self, data): # 'data' is result of self.db.runOperation(blah) # Note that the result of this method will become the callback # result of the self.db.runOperation(blah) Deferred, and thus # in turn the next result passed up the callback chain for the # first self.db.runQuery(blah) Deferred as the answer from step2 # in the non-yadda case. return "I got here!' or: [Case 2] class FooClass(General): def __init__(self, db): self.db = db def step1(self): d = self.db.runQuery(blah) return d.addCallback(self.step2).addCallback(self.step2) def step2(self, query): if query == 'yadda': return 'This bit works' else: return self.db.runOperation(blah) def step3(self, data): # 'data' is result of step2 # Note that the result of this method will become the callback # in the main callback chain for the first self.db.runQuery(blah) # Deferred, and will thus replace that of step2 return "I got here!' (Actually, there's also a third way in which you manually pause and unpause the main Deferred callback chain, and set up a callback on the secondary Deferred chain to resume the main one, but that's precisely what Twisted does for you when you just return the underlying Deferred from your callback, so I don't see any benefit of doing it yourself). In both of these cases there are two deferred chains running (arising from the two discrete deferrable operations that themselves are creating Deferreds), but that in the even of the non-yadda path in step2, the first callback chain is suspended while awaiting completion of the second. In Case 1, the addCallback of step3 is to the secondary callback chain, so it provides additional post-processing of the runOperation result, but does not get involved in other paths/results from step2. In Case 2, step3 is added to the main callback (runQuery) chain, so it will see any results of step2 (both yadda and runOperation) and can then post-process them, becoming in all cases the callback result for the main callback chain. Note a key point here - even though step3 is next in the main callback chain after step2, in the event step2 returns the Deferred from runOperation, the main chain suspends (so step3 doesn't run immediately) until that second Deferred finishes, at which point it is the result that step3 sees. So what you end up (in the non-yadda case) is something like (in poor ASCII diagraming): Case 1: Case 2: runQuery runQuery | | step2 step2 | | | no | no +-yadda->-runOperation +-yadda->-runOperation | | | | yadda v yadda v | | | | +---<------ step3 +---<----------+ | | v step3 | v Hope this helps clarify things somewhat. -- David
![](https://secure.gravatar.com/avatar/2c498e6b589e4a4318a8280da536fb36.jpg?s=120&d=mm&r=g)
David Bolen <db3l@fitlinxx.com> writes: (oops - minor correction to my last post)
this should be (step3 on the final callback): return d.addCallback(self.step2).addCallback(self.step3) -- David
![](https://secure.gravatar.com/avatar/3c4988f83703127d279406fc6eea7079.jpg?s=120&d=mm&r=g)
Thanks to all of you for your very informative responses, Mike, Abe and David! I've learned a lot, and gotten the information I need to continue on with my work. Everything was explained thoroughly, so I think I'm all set now. It's great to have such a helpful list to lean on. Thanks! Steve
participants (9)
-
Abe Fettig
-
Alex Levy
-
Andrew Bennetts
-
Christopher Armstrong
-
David Bolen
-
Glyph Lefkowitz
-
Mary Gardiner
-
Mike C. Fletcher
-
Steve Freitas