[Twisted-Python] async code structure question
Since I'm new to having to structure everything for async programming, I thought I would see if my idea on how to restructure my code makes sense.
Say I have a class with several functions that are called in order. Each function relies on the results of the previous function in order to proceed. To write this asynchronously, do I chain them together so that each function is called as a deferred, with the callback pointing to the next function and so forth down the list?
Chris
On Wed, Feb 09, 2005, snacktime wrote:
Say I have a class with several functions that are called in order. Each function relies on the results of the previous function in order to proceed. To write this asynchronously, do I chain them together so that each function is called as a deferred, with the callback pointing to the next function and so forth down the list?
It depends on whether these are synchronous or asynchonrous functions.
For synchronous, let's assume you call one function that returns a Deferred, and then the rest of them manipulate the results of that function. You just add all the manipulating functions as callbacks:
from twisted.internet import defer
def getDummyDeferred(): """ Dummy method which returns a Deferred for us to play with Usually this would be some library function that does something async """ # defer.succeed returns a Deferred fired with its argument return defer.succeed(3)
def multiplyByTwo(result): return result * 2
def square(result): return result * result
def printResult(result): print "result is %d" % result
def main(): d = getDummyDeferred() d.addCallback(multiplyByTwo) d.addCallback(square) d.addCallback(printResult)
if __name__ == '__main__': main()
For asynchronous, you chain the Deferreds:
from twisted.internet import defer
def getDummyDeferred(): return defer.succeed(2)
def getDummyDeferredTwo(result): return defer.succeed(result * 3)
def printResult(result): print result
def main(): d = getDummyDeferred() d.addCallback(getDummyDeferredTwo) d.addCallback(printResult)
You could have some fun converting these all to use reactor.callLater in the getDummyDeferred and getDummyDeferredTwo methods rather than using defer.succeed to generate an already fired Deferred :) (This will mean using reactor.run() in main.)
Relevant examples are at http://twistedmatrix.com/documents/current/howto/defer#auto5
For (a tiny bit) more on chaining Deferreds together, have a look at http://twistedmatrix.com/documents/current/howto/defer#auto12. There's already a bug filed about improving this bit: http://twistedmatrix.com/bugs/issue596
Comments welcome.
-Mary
Thanks for the info Mary. Now that I've gotten a bit further I have another design issue that I really don't know how to deal with.
The below code is my main server loop. When data is received I call a non blocking method (Do()). So far so good. Inside Do() I have a number of reactor.callLater's that fire off blocking database calls inside adbapi and are chained using callbacks. Do() returns immediately. What I need is a way to write back the data to the client once all the callbacks have fired. The code below doesn't actually work because it prints a response to the client before the actual response is even processed.
I can think of a couple of possible ways to deal with it. One would if there is a way to use transport.write from within Do() (Not sure if that's possible), or maybe there is a way to call a method in the protocol handler from inside another class? Something like the latter would be preferrable because I would like to keep the server separate from the rest of the code if I can. Sometimes people will just want to import ProcessTransaction and call Do() directly instead of using the server component.
Does my problem make sense? Not sure if I am explaining it very clearly.
Chris
class OT(Protocol): def dataReceived(self, data): c = ProcessTransaction() res = c.Do(data) self.transport.write("%s\r\n" % res) self.transport.loseConnection()
application = service.Application("otransact") OTService = service.IServiceCollection(application) OTfactory = Factory() OTfactory.protocol = OT OTServer = internet.SSLServer(8000, OTfactory,ServerContextFactory()) OTServer.setServiceParent(OTService) print dir(OTServer)
On Thu, 2005-02-10 at 09:44 -0800, snacktime wrote:
Thanks for the info Mary. Now that I've gotten a bit further I have another design issue that I really don't know how to deal with.
The below code is my main server loop. When data is received I call a non blocking method (Do()). So far so good. Inside Do() I have a number of reactor.callLater's that fire off blocking database calls inside adbapi and are chained using callbacks. Do() returns immediately.
Do() should return a Deferred that has result when processing is done, and then you can Do().addCallback(lambda: self.transport.write("hello")) or something.
e.g.
def Do(): d = deferToThread(foo) d.addCallback(processResult) return d
or
def Do(): # processing in this case is waiting for one second d = Deferred() reactor.callLater(d.callback, 1) return d
On Thu, 10 Feb 2005 13:08:15 -0500, Itamar Shtull-Trauring itamar@itamarst.org wrote:
Do() should return a Deferred that has result when processing is done, and then you can Do().addCallback(lambda: self.transport.write("hello")) or something.
e.g.
def Do(): d = deferToThread(foo) d.addCallback(processResult) return d
or
def Do(): # processing in this case is waiting for one second d = Deferred() reactor.callLater(d.callback, 1)
reactor.callLater(1, d.callback, "foo")
But this is much less realistic than the first example :)
Jp
def Do(): d = deferToThread(foo) d.addCallback(processResult) return d
or
def Do(): # processing in this case is waiting for one second d = Deferred() reactor.callLater(d.callback, 1)
reactor.callLater(1, d.callback, "foo")
But this is much less realistic than the first example :)
The first example is what I was doing and it worked just fine. But now I've rewritten everything so it's non blocking and I no longer need to use deferToThread. I found out that it's a bit more complicated now though:)
On Thu, 10 Feb 2005 11:28:52 -0800, snacktime snacktime@gmail.com wrote:
def Do(): d = deferToThread(foo) d.addCallback(processResult) return d
or
def Do(): # processing in this case is waiting for one second d = Deferred() reactor.callLater(d.callback, 1)
reactor.callLater(1, d.callback, "foo")
But this is much less realistic than the first example :)
The first example is what I was doing and it worked just fine. But now I've rewritten everything so it's non blocking and I no longer need to use deferToThread. I found out that it's a bit more complicated now though:)
How? Not with callLater, I hope. callLater doesn't make anything non-blocking.
Jp
The first example is what I was doing and it worked just fine. But now I've rewritten everything so it's non blocking and I no longer need to use deferToThread. I found out that it's a bit more complicated now though:)
How? Not with callLater, I hope. callLater doesn't make anything non-blocking.
Actually it's just a lot of code restructuring now that I am starting to learn what does and doesn't work.
I appreciate everyone's help and advice, thank you.
Chris
By the way, this is all for a (partially) open source project that I will put on the web asap. It's an application that connects directly to bank networks for processing credit card/ach transactions, and eliminates the need to use online payment processors such as authorizenet (which themselves are using applications just like this one)
All of these direct processors have strict NDA's for their api's and require distribution in object code only. As soon as I can cleanly separate out the processor specific api implementations from the generic components I will be making it public. The processor specific stuff will be in shared libraries made with pyrex.
I do have permission to let other developers see the entire source as long as they sign the same NDA, but I still haven't decided the best way to go about that.
Do() should return a Deferred that has result when processing is done, and then you can Do().addCallback(lambda: self.transport.write("hello")) or something.
e.g.
def Do(): d = deferToThread(foo) d.addCallback(processResult) return d
or
def Do(): # processing in this case is waiting for one second d = Deferred() reactor.callLater(d.callback, 1) return d
Thanks Itamar. I think I understand these examples to a point, but what I don't understand is how to implement it when you are calling methods several levels deep into other classes. For example the method authorize() in class Authorization below is where the final data is that I need to print back to the client via Printdata in the protocol handler. In authorize there are several chained deferreds also, and the last one needs to somehow trigger the very first callback that was added in dataReceived().
Specifically what I am having the most difficulty with is this. In Do() is where the Deferred is created that needs to have it's callbacks fired. How can I fire off that callback from authorize() and have authorize() pass it the final data?
class OT(Protocol): def dataReceived(self, data): def Do(): d = defer.Deferred() c = ProcessTransaction() reactor.callLater(0, c.Do,data) d.addCallback(self.PrintData) d.addErrback(log.err)
def PrintData(self,data): self.transport.write("%s\r\n" % data) self.transport.loseConnection()
class ProcessTransaction:
def Do(self,data): ''' Do some sanity checks on data and see what network to process the transaction through''' if network == 'vital': d = Vital() response = d.Submit() elif... else...
class Vital: ''' Run method based on the transaction type '''
if transaction_type == 'AUTHORIZE': r = Authorization(req) response = r.Submit() elif... else...
class Authorization:
def authorize(self): ''' Here we have several chained deferreds to handle some database operations, submit the transaction via a post, and finally format a final response which needs to be printed to the client.
On Thu, Feb 10, 2005, snacktime wrote:
Specifically what I am having the most difficulty with is this. In Do() is where the Deferred is created that needs to have it's callbacks fired. How can I fire off that callback from authorize() and have authorize() pass it the final data?
Somehow authorize needs to have the deferred accessible. There's no magical Deferred way of doing this, you do it the way that you'd make any other piece of data accessible to a method: pass it to the method, or having it accessible as self.d would be the standard ways.
So that means that Do() will change slightly
def Do(): d = defer.Deferred() c = ProcessTransaction() reactor.callLater(0, c.Do,data) d.addCallback(self.PrintData) d.addErrback(log.err)
# finish here by storing d somewhere so that authorize can later # get access to it
And then authorize does something like this to fire off the callback with data:
# assume that the Deferred is accessible to authorize as self.d self.d.callback(data)
Once you use .callback, data will be passed to the first callback method (self.PrintData) and so on.
The precise method by which you pass d around to the various classes is as much a matter of individual program design as passing any other bit of data around: config variables, strings, ints, etc etc.
-Mary
Somehow authorize needs to have the deferred accessible. There's no magical Deferred way of doing this, you do it the way that you'd make any other piece of data accessible to a method: pass it to the method, or having it accessible as self.d would be the standard ways.
Ok that's what I was afraid of but I wanted to double check. I do have a global request object that I use throughout most of my classes so that won't be too much work.
Chris
Mary Gardiner mary-twisted@puzzling.org writes:
Somehow authorize needs to have the deferred accessible. There's no magical Deferred way of doing this, you do it the way that you'd make any other piece of data accessible to a method: pass it to the method, or having it accessible as self.d would be the standard ways.
Alternatively, the authorization class/code can just make its own deferred (either directly or by performing a deferred operation) and returns it to its caller. The caller of authorize then sees it as a deferable operation (returning a deferred) onto which it can hang any callbacks, which which might be as simple as chaining to the initial deferred.
If there's an intermediate layer (such as the ProcessTransaction class in the example), it can just return the authorize deferred as its own result without doing anything to it (unless it does in fact want to do some post processing to authorizes result) up the chain.
snacktime, in your sample, this could avoid your use of callLater in OT, since you could just call c.Do(data) and use the deferred that it returned to hang your self.PrintData callback on. Within your ProcessTransaction and Vital classes, your "response" object would be the deferred being passed up from the authentication layer, possibly after some callback processing (although your example code didn't show those objects manipulating response in any way).
If you think about this from the bottom up, your authentication operations become the ones which are promising an eventual answer (and on which all the rest of the processing rests) by returning a deferred, either by generating one locally or already being layered on a deferrable operation, via the network, a deferToThread blocking call, a adbapi call, etc...
In turn, the other layers that sit above that deferrable operation simply make use of that low level Deferred object on which they hang whatever processing they need.
-- David
On Thu, Feb 10, 2005, David Bolen wrote:
Alternatively, the authorization class/code can just make its own deferred (either directly or by performing a deferred operation) and returns it to its caller. The caller of authorize then sees it as a deferable operation (returning a deferred) onto which it can hang any callbacks, which which might be as simple as chaining to the initial deferred.
True. It's generally a good idea to have a distinction between the library code which returns various Deferreds, and application code which adds callbacks to them in order to feed the data around in the right order and then act depending on the result.
-Mary
On Thu, 10 Feb 2005 14:41:32 -0800 (PST), David Bolen db3l@fitlinxx.com wrote:
Mary Gardiner mary-twisted@puzzling.org writes:
Somehow authorize needs to have the deferred accessible. There's no magical Deferred way of doing this, you do it the way that you'd make any other piece of data accessible to a method: pass it to the method, or having it accessible as self.d would be the standard ways.
Alternatively, the authorization class/code can just make its own deferred (either directly or by performing a deferred operation) and returns it to its caller. The caller of authorize then sees it as a deferable operation (returning a deferred) onto which it can hang any callbacks, which which might be as simple as chaining to the initial deferred.
If there's an intermediate layer (such as the ProcessTransaction class in the example), it can just return the authorize deferred as its own result without doing anything to it (unless it does in fact want to do some post processing to authorizes result) up the chain.
snacktime, in your sample, this could avoid your use of callLater in OT, since you could just call c.Do(data) and use the deferred that it returned to hang your self.PrintData callback on. Within your ProcessTransaction and Vital classes, your "response" object would be the deferred being passed up from the authentication layer, possibly after some callback processing (although your example code didn't show those objects manipulating response in any way).
If you think about this from the bottom up, your authentication operations become the ones which are promising an eventual answer (and on which all the rest of the processing rests) by returning a deferred, either by generating one locally or already being layered on a deferrable operation, via the network, a deferToThread blocking call, a adbapi call, etc...
In turn, the other layers that sit above that deferrable operation simply make use of that low level Deferred object on which they hang whatever processing they need.
I thought about this approach, but I kind of like the idea of having the deferred that fires off PrintData at the very top and pass the deferred around in the request object for the request. That way I can trigger a message back to the client from anywhere.
Chris
On Thu, Feb 10, 2005, Mary Gardiner wrote:
You could have some fun converting these all to use reactor.callLater in the getDummyDeferred and getDummyDeferredTwo methods rather than using defer.succeed to generate an already fired Deferred :) (This will mean using reactor.run() in main.)
Re Jp's later post I will emphasise that while this is a nice exercise to test your understanding, it is, indeed, not the way to make a method that takes time suddenly non-blocking.
If anyone wants to rant at me on or off-list about blocking and non-blocking code, I will take your words and Lore-ify them: that's ahole in the docs.
-Mary
participants (5)
-
David Bolen -
Itamar Shtull-Trauring -
Jp Calderone -
Mary Gardiner -
snacktime