Itamar Shtull-Trauring ?????:

On Wed, 17 Sep 2003 12:58:29 +0400 Egor Cheshkov <egor@iplus.ru> wrote:

...

Hello! Suppose I run task in worker thread (e.g. database transaction)

and I need to do some IO (PB call) and get the result from it during transaction. Since I run not in reactor's thread it's ok to block and wait for the result. I know that there is callFromThread but it never return any result. Is it possible to implement something like callFromThread that can block, wait for result and return it?

As documented in the threading howto, you can use twisted.internet.threads.deferToThread.

If I understand right, I should use deferToThread to shedule long running task from main thread (that running event loop), i need different thing. I need to shedule a method to be run in the main event loop from another thread. It is the thing that callFromThread does, but callFromThread does not return anything. I need to get result from such call. Is it possible? Egor.

On Thu, Sep 18, 2003 at 11:46:03AM +0200, Syver Enstad wrote:

Egor Cheshkov <egor@iplus.ru> writes:

...

If I understand right, I should use deferToThread to shedule long running task from main thread (that running event loop), i need different thing. I need to shedule a method to be run in the main event loop from another thread. It is the thing that callFromThread does, but callFromThread does not return anything. I need to get result from such call. Is it possible?

Can't you pass yourself and have the method that you pass to callFromThread callback on a method on yourself with the results?

But that won't run the callback in the worker thread. Here's the situation in dodgy ascii art: Main Thread (Twisted reactor, etc) | v | \ |\ | ->----------------------callInThread starts v Worker Thread running | worker_function() | | | v | | | / | /. callFromThread------------------------<- . causes other_func() to . run in Main Thread . | . v . | Worker Thread waits | for other_func result | . | . But how does the . Main Thread pass a result . to Worker Thread and . resume its execution? ? There needs to be a wrapper function around callFromThread that blocks the worker thread on a threading.Event or similar, and then causes the Event to be fired when the function finishes, and returns the result. Or, in untested code: ----- import threading, sys from twisted.internet import reactor def blockingCallFromThread(func): e = threading.Event() l = [] def wrapped_func(): try: l.append(func) except: l.append(sys.exc_info()) l.append(0) else: l.append(1) e.set() reactor.callFromThread(wrapped_func) e.wait() result, ok = l if ok: return result else: # Whee! Cross-thread exceptions! raise result[0], result[1], result[2] ---- This is utterly untested. It probably should be tested and added to twisted.internet.util or t.i.threads, though. -Andrew.

On Mon, Oct 06, 2003 at 10:11:42PM +0200, Matthias Urlichs wrote:

Hi, Andrew Bennetts wrote:

...

But how does the . Main Thread pass a result . to Worker Thread and . resume its execution? ?

Just put it in a Queue and wait on it in the thread. Standard Python.

It seems cleaner to me for a worker thread to say: result = blockingCallFromThread(func, arg1, arg2, ...) Than: q = Queue.Queue() reactor.callFromThread(func, q, arg1, arg2) result = q.get() Especially seeing as the latter requires cluttering 'func' with details of how to get the result back (i.e. a call to q.put), when the solution I proposed in my previous email would make it transparent. -Andrew.

On Mon, Oct 06, 2003 at 10:30:06PM -0400, Itamar Shtull-Trauring wrote:

t.i,threads should indeed have a utility function that does this for you. It would need to turn Failures into exceptions and deal transparently with Deferreds, of course.

I posted something like this to this thread earlier: ----- import threading, sys from twisted.internet import reactor def blockingCallFromThread(func): e = threading.Event() l = [] def wrapped_func(): try: l.append(func) except: l.append(sys.exc_info()) l.append(0) else: l.append(1) e.set() reactor.callFromThread(wrapped_func) e.wait() result, ok = l if ok: return result else: # Whee! Cross-thread exceptions! raise result[0], result[1], result[2] ---- It doesn't deal with Failures or Deferreds, but it does everything else. Oh, and it's not tested. If I had already written test cases, I probably would've checked it in by now ;) (And I'm sure there's a better name for it than 'blockingCallFromThread'...) -Andrew.

On Mon, 06 Oct 2003 23:05:11 -0400 Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:

Andrew, can you add a bug to the tracker for this, and assign it to someone? :) It would also be good if it did handle Deferreds, but I'm a little sketchy on where the callback would go (I mean, you're in a thread because you're blocking already, right? but deferreds are to avoid blocking, so does that mean the callback runs in a thread or in the main loop?)

The callbacks all run the main thread, but the last one adds the result to a object that feeds it to the waiting thread. That is, the thread blocks until the Deferred has a result. -- Itamar Shtull-Trauring http://itamarst.org/ Available for Python & Twisted consulting