[Twisted-Python] How to "monitor" parallel executions of Deferreds
Hello, I need to implement a function (say, myfunc) that fires several independent Deferreds (using deferToThread), and that returns a deferred itself. A client of this function (say, client) is expected to do something like this: def client(): def done(success) if success: print 'd1 and d2 are done' d = myfunc() d.addCallback(done) ... myfuct() above does something like this: def myfunc(): def checkFunc1(success): ... def checkFunc2(success): ... d = defer.Deferred() d1 = deferToThread(func1).addCallback(checkFunc1) d2 = deferToThread(func2).addCallback(checkFunc2) d.addCallback(????) return d Clearly d, d1, and d2 in the above are disconnected. And I don't want to serialize d1 and d2 which would be a way to connect them all. The problem is I don't know how d should be defined. The idea is that the 'done' function in the client function is only called when both threads d1 and d2 are done. Can you give me any suggestions on how to implement myfunc()? Thank you, -- Pedro
Hi, I'm trying to use twisted.news.nntp.NNTPClient. As I understand it, to connect to the server I need to call makeConnection, to which I need to pass a transport. How do I create a suitable transport? I've looked at what implements ITransport, and nothing appeals. Many thanks, Graham
On Thu, 2005-08-25 at 17:50 +0100, Graham Stratton wrote:
I'm trying to use twisted.news.nntp.NNTPClient. As I understand it, to connect to the server I need to call makeConnection,
Nope, you want to use reactor.connectTCP with a factory - see the client howto in the main Twisted documentation.
Pedro Sanchez wrote:
Hello,
I need to implement a function (say, myfunc) that fires several independent Deferreds (using deferToThread), and that returns a deferred itself. A client of this function (say, client) is expected to do something like this:
def client(): def done(success) if success: print 'd1 and d2 are done'
d = myfunc() d.addCallback(done) ...
myfuct() above does something like this:
def myfunc(): def checkFunc1(success): ...
def checkFunc2(success): ...
d = defer.Deferred() d1 = deferToThread(func1).addCallback(checkFunc1) d2 = deferToThread(func2).addCallback(checkFunc2) d.addCallback(????)
return d
Clearly d, d1, and d2 in the above are disconnected. And I don't want to serialize d1 and d2 which would be a way to connect them all.
The problem is I don't know how d should be defined. The idea is that the 'done' function in the client function is only called when both threads d1 and d2 are done.
I think what you're looking for is DeferredList. You pass it a list of deferreds you want to wait on and DeferredList will only fire its callback method once all of the other deferred are complete. Read the docs because the first callback you add to it will get a list of results from all the defereds. So for your above code, you'd want to do something like... d1 = deferToThread(...) d2 = deferToThread(...) dl = defer.DeferredList([d1, d2]) dl.addCallback(...) # If needed here. return dl Just be aware to not add any additional callbacks to d1 or d2 after you add them to the deferredlist. Hope this helps. :) -- Jason Fritcher Software Engineer Core Infrastructure Services & Strategy Earthlink, Inc fritcher@corp.earthlink.net (404) 748-7262, x22262
I already tested DeferredList and it does just what I was looking for. There was another suggestion to use defer.gatherResults() for which I didn't find any documentation. I have yet to browse the code to see what it does. Thanks for the reply. -- Pedro On Thu, 2005-25-08 at 18:00 -0400, Jason Fritcher wrote:
Pedro Sanchez wrote:
Hello,
I need to implement a function (say, myfunc) that fires several independent Deferreds (using deferToThread), and that returns a deferred itself. A client of this function (say, client) is expected to do something like this:
def client(): def done(success) if success: print 'd1 and d2 are done'
d = myfunc() d.addCallback(done) ...
myfuct() above does something like this:
def myfunc(): def checkFunc1(success): ...
def checkFunc2(success): ...
d = defer.Deferred() d1 = deferToThread(func1).addCallback(checkFunc1) d2 = deferToThread(func2).addCallback(checkFunc2) d.addCallback(????)
return d
Clearly d, d1, and d2 in the above are disconnected. And I don't want to serialize d1 and d2 which would be a way to connect them all.
The problem is I don't know how d should be defined. The idea is that the 'done' function in the client function is only called when both threads d1 and d2 are done.
I think what you're looking for is DeferredList. You pass it a list of deferreds you want to wait on and DeferredList will only fire its callback method once all of the other deferred are complete. Read the docs because the first callback you add to it will get a list of results from all the defereds.
So for your above code, you'd want to do something like...
d1 = deferToThread(...) d2 = deferToThread(...) dl = defer.DeferredList([d1, d2]) dl.addCallback(...) # If needed here. return dl
Just be aware to not add any additional callbacks to d1 or d2 after you add them to the deferredlist.
Hope this helps. :)
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
participants (5)
-
Graham Stratton -
Itamar Shtull-Trauring -
Jason Fritcher -
Pedro Sanchez -
Tommi Virtanen