[Twisted-Python] twisted and threading
I wrapped a c library using swig and was hoping to make use of it in a twisted server I am writing. The issue is that the c library fundamentally blocks waiting for input that may never come. Is there a way from twisted to deal with this cleanly within the framework? I.e. some combination of adding a reactor.addSystemEventTrigger event and calling reactor.threadpool.stop seem like it should work but I haven't had any luck yet. (I saw all the previous posts on this topic, but the recommendations don't seem applicable in my case since the blocking is occurring outside my control in a c library and the blocking call may never return.) Thanks, Steve Bauer
On Tue, 17 Aug 2004 20:34:44 -0400, twisted-10152003@atlas.lcs.mit.edu <twisted-10152003@atlas.lcs.mit.edu> wrote:
I wrapped a c library using swig and was hoping to make use of it in a twisted server I am writing. The issue is that the c library fundamentally blocks waiting for input that may never come. Is there a way from twisted to deal with this cleanly within the framework?
I.e. some combination of adding a reactor.addSystemEventTrigger event and calling reactor.threadpool.stop seem like it should work but I haven't had any luck yet.
Geeze, where'd you come up with that stuff? Accessing reactor.threadpool directly is Really Bad. I hope there's no example code or documentation that refers to that. Try twisted.internet.threads.deferToThread(myBlockingCall) --> Deferred. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com
Christopher Armstrong [radeex@gmail.com] wrote:
On Tue, 17 Aug 2004 20:34:44 -0400, twisted-10152003@atlas.lcs.mit.edu <twisted-10152003@atlas.lcs.mit.edu> wrote:
I wrapped a c library using swig and was hoping to make use of it in a twisted server I am writing. The issue is that the c library fundamentally blocks waiting for input that may never come. Is there a way from twisted to deal with this cleanly within the framework?
I.e. some combination of adding a reactor.addSystemEventTrigger event and calling reactor.threadpool.stop seem like it should work but I haven't had any luck yet.
Try twisted.internet.threads.deferToThread(myBlockingCall) --> Deferred.
Ah, but that doesn't work. As I stated the c library (not mine) calls potentially block forever (or at least longer than I want to wait to shutdown the twisted server). The thread created by deferTothread may never complete which makes shutting down the server gracefully impossible (or so I thought?).
Geeze, where'd you come up with that stuff? Accessing reactor.threadpool directly is Really Bad. I hope there's no example code or documentation that refers to that.
:) Ah the hazards of attempting to read documentation and mailing-lists before posting... The threadpool and a link to its interface are mentioned on the primary threading howto. No mention of "Really Bad" there. http://www.twistedmatrix.com/documents/current/howto/threading And other posts about threading on this mailing list mentioned looking at twisted.enterprise.adbapi which is where reactor.addSystemEventTrigger ideas came from. Incidentally, I am happy to help update documentation if I can figure out the right ways to do things. :) Thanks for your rapid response. Sincerely, Steve Bauer
On Tue, 17 Aug 2004 22:18:51 -0400, twisted-10152003@atlas.lcs.mit.edu <twisted-10152003@atlas.lcs.mit.edu> wrote:
Christopher Armstrong [radeex@gmail.com] wrote:
On Tue, 17 Aug 2004 20:34:44 -0400, twisted-10152003@atlas.lcs.mit.edu <twisted-10152003@atlas.lcs.mit.edu> wrote:
I.e. some combination of adding a reactor.addSystemEventTrigger event and calling reactor.threadpool.stop seem like it should work but I haven't had any luck yet.
Try twisted.internet.threads.deferToThread(myBlockingCall) --> Deferred.
Ah, but that doesn't work. As I stated the c library (not mine) calls potentially block forever (or at least longer than I want to wait to shutdown the twisted server). The thread created by deferTothread may never complete which makes shutting down the server gracefully impossible (or so I thought?).
So you want to be able to murder that thread while you're shutting down? I don't the solution to that, but I just want to be clear. Indeed, maybe something involving a shutdown event.. I don't know.
Geeze, where'd you come up with that stuff? Accessing reactor.threadpool directly is Really Bad. I hope there's no example code or documentation that refers to that.
The threadpool and a link to its interface are mentioned on the primary threading howto. No mention of "Really Bad" there.
http://www.twistedmatrix.com/documents/current/howto/threading
Ahh. That doesn't mention anything about reactor.threadpool, though, which is the evil thing to touch. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com
twisted-10152003@atlas.lcs.mit.edu wrote:
Christopher Armstrong [radeex@gmail.com] wrote:
On Tue, 17 Aug 2004 20:34:44 -0400, twisted-10152003@atlas.lcs.mit.edu <twisted-10152003@atlas.lcs.mit.edu> wrote:
I wrapped a c library using swig and was hoping to make use of it in a twisted server I am writing. The issue is that the c library fundamentally blocks waiting for input that may never come. Is there a way from twisted to deal with this cleanly within the framework?
I.e. some combination of adding a reactor.addSystemEventTrigger event and calling reactor.threadpool.stop seem like it should work but I haven't had any luck yet.
Try twisted.internet.threads.deferToThread(myBlockingCall) --> Deferred.
Ah, but that doesn't work. As I stated the c library (not mine) calls potentially block forever (or at least longer than I want to wait to shutdown the twisted server). The thread created by deferTothread may never complete which makes shutting down the server gracefully impossible (or so I thought?).
There is no portable way to terminate a thread without its assistance. Python supports no API for this - save one, the ridiculously named "setDaemon" Thread method. Twisted doesn't expose this, nor call it internally, as it can lead to segfaults. Perhaps this should be parameterizable (defaulting to the current behavior, of course), so that poorly behaved libraries can be dealt with? Alternatively, since daemonized threads might lead to this anyway, perhaps you should just add a shutdown event os.kill(os.getpid(), 9) <wwinkink>. Jp
On Tue, 17 Aug 2004 23:18:59 -0400, Jp Calderone <exarkun@divmod.com> wrote:
twisted-10152003@atlas.lcs.mit.edu wrote:
Ah, but that doesn't work. As I stated the c library (not mine) calls potentially block forever (or at least longer than I want to wait to shutdown the twisted server). The thread created by deferTothread may never complete which makes shutting down the server gracefully impossible (or so I thought?).
There is no portable way to terminate a thread without its assistance. Python supports no API for this - save one, the ridiculously named "setDaemon" Thread method. Twisted doesn't expose this, nor call it internally, as it can lead to segfaults.
Perhaps this should be parameterizable (defaulting to the current behavior, of course), so that poorly behaved libraries can be dealt with?
Alternatively, since daemonized threads might lead to this anyway, perhaps you should just add a shutdown event os.kill(os.getpid(), 9) <wwinkink>.
Well, since he only wants to murder the thread at shutdown, a regular "thread killing" API isn't necessary, just a way to have the thread not block shutting down, by dying. Your os.kill suggestion is along the lines of what I was thinking of, except maybe there's a less horrible option :) -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com
Jp Calderone <exarkun@divmod.com> writes:
There is no portable way to terminate a thread without its assistance. Python supports no API for this - save one, the ridiculously named "setDaemon" Thread method. Twisted doesn't expose this, nor call it internally, as it can lead to segfaults.
Well, all setDaemon does is prevent the threading module from joining that thread during a shutdown - so the thread is left alone in peace to be reclaimed by the operating system rather than by Python itself. Do you have instances where this has actually provoked a segfault? If the thread is executing in Python code, the GIL should prevent it from actually running during the Python portion of the shutdown, and the OS will just clean up. If the thread is truly blocked outside of Python then nothing Python does will be a problem (unless the external entity is writing directly to Python structures, but without grabbing the GIL first which is a buggy extension), and the OS will again clean up. Anyway, I've definitely used daemon threads before where it was infeasible to guarantee the ability to signal the thread to shut itself down before I wanted to exit and at least in my experience I've never run into a problem. -- David
Jp Calderone <exarkun <at> divmod.com> writes:
twisted-10152003 <at> atlas.lcs.mit.edu wrote:
Ah, but that doesn't work. As I stated the c library (not mine) calls potentially block forever (or at least longer than I want to wait to shutdown the twisted server). The thread created by deferTothread may never complete which makes shutting down the server gracefully impossible (or so I thought?).
There is no portable way to terminate a thread without its assistance. Python supports no API for this - save one, the ridiculously named "setDaemon" Thread method. Twisted doesn't expose this, nor call it internally, as it can lead to segfaults.
Perhaps this should be parameterizable (defaulting to the current behavior, of course), so that poorly behaved libraries can be dealt with?
Alternatively, since daemonized threads might lead to this anyway, perhaps you should just add a shutdown event os.kill(os.getpid(), 9) <wwinkink>.
Sorry this is from long ago, possibly the OP has found a solution by now, but I thought I'd share my approach in case it was of more general interest... I have hit the same problem recently (C library calls that might hang, and I want to give up and quit regardless in the end). What I came up with was a function deferToDaemonThread, which put the call in a daemon thread (one that the Python main thread will not wait for). It steals a lot of code from deferToThread, but creates a dedicated daemon thread for the call rather than using the thread pool. Hope it's of interest, Paul. ------------------------------------------------------------------------------ from twisted.internet import reactor from twisted.python import failure from twisted.internet import defer from threading import Thread def deferToDaemonThread(f, *args, **kw): """Run function in thread and return result as Deferred.""" def putResultInDeferred(d, f, args, kw): """Run a function and give results to a Deferred.""" try: result = f(*args, **kw) except: f = failure.Failure() reactor.callFromThread(d.errback, f) else: reactor.callFromThread(d.callback, result) d = defer.Deferred() thread = Thread(target=putResultInDeferred, args=(d, f, args, kw)) thread.setDaemon(1) thread.start() return d
On Thu, 25 Nov 2004 15:24:57 +0000 (UTC), Paul Moore <paul.moore@atosorigin.com> wrote:
Jp Calderone <exarkun <at> divmod.com> writes:
There is no portable way to terminate a thread without its assistance. Python supports no API for this - save one, the ridiculously named "setDaemon" Thread method. Twisted doesn't expose this, nor call it internally, as it can lead to segfaults.
I have hit the same problem recently (C library calls that might hang, and I want to give up and quit regardless in the end). What I came up with was a function deferToDaemonThread, which put the call in a daemon thread (one that the Python main thread will not wait for). It steals a lot of code from deferToThread, but creates a dedicated daemon thread for the call rather than using the thread pool.
IMHO, if a library is _that_ poorly behaved, you ought to be communicating with it over a socket or pipe. Anybody up for writing a 'deferToPythonSubprocess'?
glyph@divmod.com writes:
IMHO, if a library is _that_ poorly behaved, you ought to be communicating with it over a socket or pipe. Anybody up for writing a 'deferToPythonSubprocess'?
Not a bad idea - I might have a go (although as I work on Windows, I don't have a particularly good reactor with process support...) Paul -- "Bother," said the Borg, "We've assimilated Pooh."
Sorry this is from long ago, possibly the OP has found a solution by now, but I thought I'd share my approach in case it was of more general interest...
I have hit the same problem recently (C library calls that might hang, and I want to give up and quit regardless in the end). What I came up with was a function deferToDaemonThread, which put the call in a daemon thread (one that the Python main thread will not wait for). It steals a lot of code from deferToThread, but creates a dedicated daemon thread for the call rather than using the thread pool.
Have you no way to terminate the blocking call from the C library itself ? For instance, if your working python thread is waiting on a mutex in the C library, destroying the mutex (or something else depending on the threading API) may release it. In this case, you can add a cleanup function to the library cancelling all blocking calls. And call it during the reactor shutdown sequence (I do not know where exactly, I am not an expert in this area yet, but I remember there are tons of handlers everywhere to perform cleanup. I hope there are some called before threads shutdown attempts). It means that you actually need a thread-pool but can terminate the blocking operations safely. Or am I missing something ? Patrick Mézard
participants (8)
-
Christopher Armstrong -
David Bolen -
glyph＠divmod.com -
Jp Calderone -
Patrick Mézard -
Paul Moore -
Paul Moore -
twisted-10152003＠atlas.lcs.mit.edu