[Twisted-Python] Exit all threads upon KeyboardInterrupt
Hi, I want to exit my application immediately when CTRL+C is pressed, however reactor hangs when there are running threads. Some of these threads have blocking I/O, so I can't simply set a variable or wait for them to terminate. An example application would be: from twisted.internet import reactor, threads from time import sleep import signal def do(): try: sleep(10) except KeyboardInterrupt: print "interrupt!" #def cleanup(signum, stackframe): def cleanup(): d.cancel() reactor.callFromThread(reactor._stopThreadPool) reactor.callFromThread(reactor.stop) if __name__ == '__main__': d = threads.deferToThread(do) #signal.signal(signal.SIGINT, cleanup) reactor.addSystemEventTrigger('before', 'shutdown', cleanup) reactor.run() How can I achieve this? - John
On 09:01 pm, johnnadre@zoho.com wrote:
Hi,
I want to exit my application immediately when CTRL+C is pressed, however reactor hangs when there are running threads. Some of these threads have blocking I/O, so I can't simply set a variable or wait for them to terminate. An example application would be:
Python threads (being plain old operating systems; for example, POSIX threads) aren't generally interruptable. You could try exiting the entire process using `os._exit`. This is the case whether you're using Twisted or not. Jean-Paul
from twisted.internet import reactor, threads from time import sleep import signal
def do(): try: sleep(10) except KeyboardInterrupt: print "interrupt!"
#def cleanup(signum, stackframe): def cleanup(): d.cancel() reactor.callFromThread(reactor._stopThreadPool) reactor.callFromThread(reactor.stop)
if __name__ == '__main__': d = threads.deferToThread(do) #signal.signal(signal.SIGINT, cleanup) reactor.addSystemEventTrigger('before', 'shutdown', cleanup) reactor.run()
How can I achieve this?
- John
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
---- On Sun, 12 Jan 2014 14:03:24 -0800 <exarkun@twistedmatrix.com> wrote ----
On 09:01 pm, johnnadre@zoho.com wrote:
Hi,
I want to exit my application immediately when CTRL+C is pressed, however reactor hangs when there are running threads. Some of these threads have blocking I/O, so I can't simply set a variable or wait for them to terminate. An example application would be:
Python threads (being plain old operating systems; for example, POSIX threads) aren't generally interruptable. You could try exiting the entire process using `os._exit`.
This is the case whether you're using Twisted or not.
Thanks! That's exactly what I was looking for.
On Jan 12, 2014, at 4:42 PM, johnnadre <johnnadre@zoho.com> wrote:
---- On Sun, 12 Jan 2014 14:03:24 -0800 <exarkun@twistedmatrix.com> wrote ----
On 09:01 pm, johnnadre@zoho.com wrote:
Hi,
I want to exit my application immediately when CTRL+C is pressed, however reactor hangs when there are running threads. Some of these threads have blocking I/O, so I can't simply set a variable or wait for them to terminate. An example application would be:
Python threads (being plain old operating systems; for example, POSIX threads) aren't generally interruptable. You could try exiting the entire process using `os._exit`.
Well, there's always pthread_kill, now exposed as os.pthread_kill on Python 3.3, but ... don't do that. Nothing good will come of it.
This is the case whether you're using Twisted or not.
Thanks! That's exactly what I was looking for.
If you need to use os._exit, your application probably has a bug :-). (Not to say you should never use it, but any time you use it you should be simultaneously looking into why you need to.) -glyph
On 13 Jan, 11:31 pm, glyph@twistedmatrix.com wrote:
On Jan 12, 2014, at 4:42 PM, johnnadre <johnnadre@zoho.com> wrote:
---- On Sun, 12 Jan 2014 14:03:24 -0800 <exarkun@twistedmatrix.com> wrote ----
On 09:01 pm, johnnadre@zoho.com wrote:
Hi,
I want to exit my application immediately when CTRL+C is pressed, however reactor hangs when there are running threads. Some of these threads have blocking I/O, so I can't simply set a variable or wait for them to terminate. An example application would be:
Python threads (being plain old operating systems; for example, POSIX threads) aren't generally interruptable. You could try exiting the entire process using `os._exit`.
Well, there's always pthread_kill, now exposed as os.pthread_kill on Python 3.3, but ... don't do that. Nothing good will come of it.
This is the case whether you're using Twisted or not.
Thanks! That's exactly what I was looking for.
If you need to use os._exit, your application probably has a bug :-).
(Not to say you should never use it, but any time you use it you should be simultaneously looking into why you need to.)
-glyph
I think you may as well say the same about needing to use threads. The needs are pretty closely related, anyway. Consider an application that uses twisted.enterprise.adbapi to talk to a SQL server, is running when the network suffers a failure that leaves a cursor waiting for data from the server, and then the user decides they want to exit the application. Most likely the cursor will block forever on a recv() call, the thread it runs in will never proceed, and the process will never exit. Whether this is a bug or an unavoidable consequence of doing blocking I/O using POSIX threads or both... Jean-Paul
There are two things wrong with your scenario. First, in any production or production-ready environment, your software needs to correctly handle stuck TCP connections. With the proliferation of "smart" network equipment (session-tracking firewalls, etc.), this is much more common than it was five or ten years ago. Some DBAPI backends are better about this than others - for example, MySQL-Python gets it right, but PyMySQL does not. Second, this has nothing whatsoever to do with whether the I/O is blocking or not. Twisted apps can and, believe me from experience, do suffer from stuck TCP connections. Sure, the wasted resources in this case are smaller (just a file descriptor, not a thread), but the user experience is the same (part of the application hangs irreversibly). In fact, this is about the only thing the manhole is used for at Mozilla: running `os.close` on stuck fd's. Dustin
On 14/01/14 14:27, Dustin J. Mitchell wrote:
There are two things wrong with your scenario.
First, in any production or production-ready environment, your software needs to correctly handle stuck TCP connections. With the proliferation of "smart" network equipment (session-tracking firewalls, etc.), this is much more common than it was five or ten years ago. Some DBAPI backends are better about this than others - for example, MySQL-Python gets it right, but PyMySQL does not.
That is certainly a fine sentiment with which I agree in principle, but an ENORMOUSLY LARGE amount of (really very expensive and business critical) software doesn't handle this *at all* as anyone who runs a firewall in front of an e-Business app can tell you. [A well known vendor whose name rhymes with a type of traditional small boat used extensively in the British Isles is a particular offender] So if you're handling this correctly, give yourself a pat on the back - there is software costing millions a year in licence fees that assumes TCP is as reliable as CPU->CPU cache communications :o(
Second, this has nothing whatsoever to do with whether the I/O is blocking or not. Twisted apps can and, believe me from experience, do suffer from stuck TCP connections. Sure, the wasted resources in this case are smaller (just a file descriptor, not a thread), but the user experience is the same (part of the application hangs irreversibly). In fact, this is about the only thing the manhole is used for at Mozilla: running `os.close` on stuck fd's.
I'm wondering if TCP keepalive isn't any use to you in this regard? (and what do Mozilla do with Twisted? :o)
On Tue, Jan 14, 2014 at 10:05 AM, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
I'm wondering if TCP keepalive isn't any use to you in this regard?
It is, but only in cases where we can tweak kernel parameters appropriately so that the timeout is not >2h.
(and what do Mozilla do with Twisted? :o)
Buildbot, among other things. Dustin
On 14/01/14 15:10, Dustin J. Mitchell wrote:
On Tue, Jan 14, 2014 at 10:05 AM, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
I'm wondering if TCP keepalive isn't any use to you in this regard?
It is, but only in cases where we can tweak kernel parameters appropriately so that the timeout is not >2h.
Hmm. I see Twisted has no API to set this using the socket options, which is unfortunate. I've opened #6897 for that. Presumably there should be endpoint syntax for it too...
(and what do Mozilla do with Twisted? :o)
Buildbot, among other things.
Doh yes of course.
On Jan 14, 2014, at 6:27 AM, Dustin J. Mitchell <dustin@v.igoro.us> wrote:
Second, this has nothing whatsoever to do with whether the I/O is blocking or not. Twisted apps can and, believe me from experience, do suffer from stuck TCP connections. Sure, the wasted resources in this case are smaller (just a file descriptor, not a thread), but the user experience is the same (part of the application hangs irreversibly). In fact, this is about the only thing the manhole is used for at Mozilla: running `os.close` on stuck fd's.
The lack of 'abortConnection' was a pretty serious design flaw in Twisted's transport interface initially, but that's been resolved for quite some time now. It should be possible to handle this at the application level on a per-connection basis quite gracefully now. I mean, your point is correct; you can certainly have "stuck" event sources in Twisted too, without any threads. And without any intervention it results in exactly the same bad user experience. I've certainly seen it myself. However, intervention is easier than with a stuck thread. You can't interrupt an arbitrary thread; you don't know and can't tell what it is doing without pretty invasive debugging instrumentation. And, as you note, even before the advent of abortConnection, an os.close() could easily deal with a stuck async FD on a per-connection basis, without need to bring the entire process down. Finally, we have abortConnection now, so my main point stands; the point was not "never call os._exit" but rather "if you have to call os._exit, there's a bug". The same is true here: if you have to manhole into your server to call os.close, there's a bug in your application where it's not timing something out and freeing a resource it otherwise could be freeing. Unless your contention is that there are no bugs in buildbot? ;-) -glyph
You're absolutely right. I wasn't arguing with you, but with JP :) Dustin On Tue, Jan 14, 2014 at 2:06 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
On Jan 14, 2014, at 6:27 AM, Dustin J. Mitchell <dustin@v.igoro.us> wrote:
Second, this has nothing whatsoever to do with whether the I/O is blocking or not. Twisted apps can and, believe me from experience, do suffer from stuck TCP connections. Sure, the wasted resources in this case are smaller (just a file descriptor, not a thread), but the user experience is the same (part of the application hangs irreversibly). In fact, this is about the only thing the manhole is used for at Mozilla: running `os.close` on stuck fd's.
The lack of 'abortConnection' was a pretty serious design flaw in Twisted's transport interface initially, but that's been resolved for quite some time now. It should be possible to handle this at the application level on a per-connection basis quite gracefully now.
I mean, your point is correct; you can certainly have "stuck" event sources in Twisted too, without any threads. And without any intervention it results in exactly the same bad user experience. I've certainly seen it myself.
However, intervention is easier than with a stuck thread. You can't interrupt an arbitrary thread; you don't know and can't tell what it is doing without pretty invasive debugging instrumentation. And, as you note, even before the advent of abortConnection, an os.close() could easily deal with a stuck async FD on a per-connection basis, without need to bring the entire process down.
Finally, we have abortConnection now, so my main point stands; the point was not "never call os._exit" but rather "if you have to call os._exit, there's a bug". The same is true here: if you have to manhole into your server to call os.close, there's a bug in your application where it's not timing something out and freeing a resource it otherwise could be freeing.
Unless your contention is that there are no bugs in buildbot? ;-)
-glyph
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
participants (5)
-
Dustin J. Mitchell -
exarkun＠twistedmatrix.com -
Glyph Lefkowitz -
johnnadre -
Phil Mayers