Thanks for the info Mary. Now that I've gotten a bit further I have another design issue that I really don't know how to deal with. The below code is my main server loop. When data is received I call a non blocking method (Do()). So far so good. Inside Do() I have a number of reactor.callLater's that fire off blocking database calls inside adbapi and are chained using callbacks. Do() returns immediately. What I need is a way to write back the data to the client once all the callbacks have fired. The code below doesn't actually work because it prints a response to the client before the actual response is even processed. I can think of a couple of possible ways to deal with it. One would if there is a way to use transport.write from within Do() (Not sure if that's possible), or maybe there is a way to call a method in the protocol handler from inside another class? Something like the latter would be preferrable because I would like to keep the server separate from the rest of the code if I can. Sometimes people will just want to import ProcessTransaction and call Do() directly instead of using the server component. Does my problem make sense? Not sure if I am explaining it very clearly. Chris class OT(Protocol): def dataReceived(self, data): c = ProcessTransaction() res = c.Do(data) self.transport.write("%s\r\n" % res) self.transport.loseConnection() application = service.Application("otransact") OTService = service.IServiceCollection(application) OTfactory = Factory() OTfactory.protocol = OT OTServer = internet.SSLServer(8000, OTfactory,ServerContextFactory()) OTServer.setServiceParent(OTService) print dir(OTServer)

On Thu, 10 Feb 2005 13:08:15 -0500, Itamar Shtull-Trauring wrote:

Do() should return a Deferred that has result when processing is done, and then you can Do().addCallback(lambda: self.transport.write("hello")) or something.

e.g.

def Do(): d = deferToThread(foo) d.addCallback(processResult) return d

or

def Do(): # processing in this case is waiting for one second d = Deferred() reactor.callLater(d.callback, 1)

reactor.callLater(1, d.callback, "foo") But this is much less realistic than the first example :) Jp

On Thu, 10 Feb 2005 11:28:52 -0800, snacktime wrote:

...

...

def Do(): d = deferToThread(foo) d.addCallback(processResult) return d

or

def Do(): # processing in this case is waiting for one second d = Deferred() reactor.callLater(d.callback, 1)

reactor.callLater(1, d.callback, "foo")

But this is much less realistic than the first example :)

The first example is what I was doing and it worked just fine. But now I've rewritten everything so it's non blocking and I no longer need to use deferToThread. I found out that it's a bit more complicated now though:)

How? Not with callLater, I hope. callLater doesn't make anything non-blocking. Jp

By the way, this is all for a (partially) open source project that I will put on the web asap. It's an application that connects directly to bank networks for processing credit card/ach transactions, and eliminates the need to use online payment processors such as authorizenet (which themselves are using applications just like this one) All of these direct processors have strict NDA's for their api's and require distribution in object code only. As soon as I can cleanly separate out the processor specific api implementations from the generic components I will be making it public. The processor specific stuff will be in shared libraries made with pyrex. I do have permission to let other developers see the entire source as long as they sign the same NDA, but I still haven't decided the best way to go about that.

On Thu, Feb 10, 2005, snacktime wrote:

Specifically what I am having the most difficulty with is this. In Do() is where the Deferred is created that needs to have it's callbacks fired. How can I fire off that callback from authorize() and have authorize() pass it the final data?

Somehow authorize needs to have the deferred accessible. There's no magical Deferred way of doing this, you do it the way that you'd make any other piece of data accessible to a method: pass it to the method, or having it accessible as self.d would be the standard ways. So that means that Do() will change slightly

def Do(): d = defer.Deferred() c = ProcessTransaction() reactor.callLater(0, c.Do,data) d.addCallback(self.PrintData) d.addErrback(log.err)

# finish here by storing d somewhere so that authorize can later # get access to it And then authorize does something like this to fire off the callback with data: # assume that the Deferred is accessible to authorize as self.d self.d.callback(data) Once you use .callback, data will be passed to the first callback method (self.PrintData) and so on. The precise method by which you pass d around to the various classes is as much a matter of individual program design as passing any other bit of data around: config variables, strings, ints, etc etc. -Mary

Mary Gardiner writes:

Somehow authorize needs to have the deferred accessible. There's no magical Deferred way of doing this, you do it the way that you'd make any other piece of data accessible to a method: pass it to the method, or having it accessible as self.d would be the standard ways.

Alternatively, the authorization class/code can just make its own deferred (either directly or by performing a deferred operation) and returns it to its caller. The caller of authorize then sees it as a deferable operation (returning a deferred) onto which it can hang any callbacks, which which might be as simple as chaining to the initial deferred. If there's an intermediate layer (such as the ProcessTransaction class in the example), it can just return the authorize deferred as its own result without doing anything to it (unless it does in fact want to do some post processing to authorizes result) up the chain. snacktime, in your sample, this could avoid your use of callLater in OT, since you could just call c.Do(data) and use the deferred that it returned to hang your self.PrintData callback on. Within your ProcessTransaction and Vital classes, your "response" object would be the deferred being passed up from the authentication layer, possibly after some callback processing (although your example code didn't show those objects manipulating response in any way). If you think about this from the bottom up, your authentication operations become the ones which are promising an eventual answer (and on which all the rest of the processing rests) by returning a deferred, either by generating one locally or already being layered on a deferrable operation, via the network, a deferToThread blocking call, a adbapi call, etc... In turn, the other layers that sit above that deferrable operation simply make use of that low level Deferred object on which they hang whatever processing they need. -- David

On Thu, 10 Feb 2005 14:41:32 -0800 (PST), David Bolen wrote:

Mary Gardiner writes:

...

Somehow authorize needs to have the deferred accessible. There's no magical Deferred way of doing this, you do it the way that you'd make any other piece of data accessible to a method: pass it to the method, or having it accessible as self.d would be the standard ways.

Alternatively, the authorization class/code can just make its own deferred (either directly or by performing a deferred operation) and returns it to its caller. The caller of authorize then sees it as a deferable operation (returning a deferred) onto which it can hang any callbacks, which which might be as simple as chaining to the initial deferred.

If there's an intermediate layer (such as the ProcessTransaction class in the example), it can just return the authorize deferred as its own result without doing anything to it (unless it does in fact want to do some post processing to authorizes result) up the chain.

snacktime, in your sample, this could avoid your use of callLater in OT, since you could just call c.Do(data) and use the deferred that it returned to hang your self.PrintData callback on. Within your ProcessTransaction and Vital classes, your "response" object would be the deferred being passed up from the authentication layer, possibly after some callback processing (although your example code didn't show those objects manipulating response in any way).

If you think about this from the bottom up, your authentication operations become the ones which are promising an eventual answer (and on which all the rest of the processing rests) by returning a deferred, either by generating one locally or already being layered on a deferrable operation, via the network, a deferToThread blocking call, a adbapi call, etc...

In turn, the other layers that sit above that deferrable operation simply make use of that low level Deferred object on which they hang whatever processing they need.

I thought about this approach, but I kind of like the idea of having the deferred that fires off PrintData at the very top and pass the deferred around in the request object for the request. That way I can trigger a message back to the client from anywhere. Chris

On Fri, 2005-02-11 at 08:44 +1100, Mary Gardiner wrote:

If anyone wants to rant at me on or off-list about blocking and non-blocking code, I will take your words and Lore-ify them: that's ahole in the docs.

"Lore and the Non-Blockers" would be a great name for a band.