Re: [Twisted-Python] blocking and threads
Thanks, how embarassing... I suggest adding it to the "using threads" howto (I'd do it myself if I knew how). Another idea: make an asynchronous file-handling module that wraps Python's file IO with something that has a deferred-oriented interface. Is there already something like that too? Joe --- Jp Calderone <exarkun@divmod.com> wrote:
Hi all,
I am just wondering: from the Twisted how-tos, it appears that doing anything that could possibly take arbitrarily long to execute should not be done in
reactor's main thread; i.e. it should be done using an asynchronous library (such as Twisted's network communication facilities), or in a different
On Wed, 27 Apr 2005 05:49:01 -0700 (PDT), Joachim Boomberschloss <boomberschloss@yahoo.com> wrote: the thread,
using the reactor's thread pool.
So it would seem that many things that may be considered "primitive" in Python, such as file IO, require some kind of patching if they are to be immediately usable by a Twisted application. I came up with the following solution, which enables calling any function in a different thread with a deferred interface; I just wanted to make sure that I'm not completely missing some point:
def deferToThread(func, *args, **kargs): """executes the given function in a thread, and passes the return value to the deferred we return""" d = defer.Deferred() reactor.callInThread(_calledInThread, d, func, *args, **kargs) return d
def _calledInThread(d, func, *args, **kargs): try: retval = func(*args, **kargs) except Exception, x: reactor.callFromThread(d.errback, x) else: reactor.callFromThread(d.callback, retval)
Looks like you've got a pretty firm understanding. The one thing you did miss is the twisted.internet.threads module, which provides the deferToThread function.
Jp
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
__________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
On Wed, 27 Apr 2005 07:07:15 -0700 (PDT), Joachim Boomberschloss <boomberschloss@yahoo.com> wrote:
Thanks, how embarassing... I suggest adding it to the "using threads" howto (I'd do it myself if I knew how). Another idea: make an asynchronous file-handling module that wraps Python's file IO with something that has a deferred-oriented interface. Is there already something like that too?
There's not, yet (although likely the VFS project Chris Armstrong, Andy Gayton, Tim Stebbing, and Andrew Bennetts recently started on will probably encompass this at some point). For many projects, asynchronous disk file i/o isn't actually that important. They either only do a little bit of it, or they can fragment it over multiple reactor iterations manually without too much effort. This is probably the main reason there isn't something like adbapi for files in Twisted already. For projects that do a lot of disk IO (like an ftp server, say :), or have latency requirements more stringent than average, an asynchronous file API would definitely be useful. Jp
On 4/27/05, Jp Calderone <exarkun@divmod.com> wrote:
For many projects, asynchronous disk file i/o isn't actually that important. They either only do a little bit of it, or they can fragment it over multiple reactor iterations manually without too much effort.
can you explain the "fragement it over multiple reactor itertations" please? I actually have a project that it is very import and we are trying to figure out the best way to do this type of thing
jarrod roberson wrote:
On 4/27/05, *Jp Calderone* <exarkun@divmod.com <mailto:exarkun@divmod.com>> wrote:
For many projects, asynchronous disk file i/o isn't actually that important. They either only do a little bit of it, or they can fragment it over multiple reactor iterations manually without too much effort.
can you explain the "fragement it over multiple reactor itertations" please?
see if this thread is helpful: http://twistedmatrix.com/pipermail/twisted-python/2005-March/009950.html I'm using this technique which boils down to reactor.callLater(0, recursive_function, args) I also tried reactor.iterate() (http://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.I... ) - although this prevented blocking for some reason it had some undesirable side effects with my app. regards, Eugene Coetzee -- -- =============================================== Reedflute Software Solutions Telephone -> +27 18 293 3236 General information -> info@reedflute.com Project information -> projects@reedflute.com Web -> www.reedflute.com ===============================================
On Tue, 2005-05-03 at 18:07 +0200, Eugene Coetzee wrote:
I also tried reactor.iterate()
(http://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.I... )
Don't do that, reactor.iterate() is not something Twisted applications should use.
Itamar Shtull-Trauring wrote:
On Tue, 2005-05-03 at 18:07 +0200, Eugene Coetzee wrote:
I also tried reactor.iterate()
(http://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.I... )
Don't do that, reactor.iterate() is not something Twisted applications should use.
Ok, perhaps that was a bit of wishful thinking :-) It would have been kind of neat if one could call something like reactor.yield(thisfunc,delay) from within a time consuming loop instead of the recursive workaround or resorting to a multithreaded approach.
participants (5)
-
Eugene Coetzee -
Itamar Shtull-Trauring -
jarrod roberson -
Joachim Boomberschloss -
Jp Calderone