Responsive signal handling
On the topic of obscure concurrency trivia, signal handling in Python is not very friendly, and I'm wondering if anyone is interested in reviewing changes to fix it. The world today: handlers only run in the main thread, but if the main thread is running a bytecode (e.g. a call to a C function), it will wait for that first. For example, signal handlers don't get run if you are in the middle of a lock acquisition, thread join, or (sometimes) a select call, until after the call returns (which may take a very long time). This makes it difficult to be responsive to signals without acrobatics, or without using a library that does those acrobatics for you (such as Twisted.) Being responsive to SIGTERM and SIGINT is, IMO, important for running programs in the cloud, since otherwise they may be forcefully killed by the job manager, causing user-facing errors. (It's also annoying as a command line user when you can't kill a process with anything less than SIGKILL.) I would stress that, by and large, the signal module is a trap that most people get wrong, and I don't think the stdlib solution should be the way it is. (e.g. it looks to me like gunicorn gets it wrong, and certainly asyncio did in early versions.) One could implement one or both of the following: - Keep only running signal handlers in the main thread, but allow them to run even in the middle of a call to a C function for as many C functions as we can. This is not possible in general, but it can be made to work for all blocking operations in the stdlib. Operations that run in C but just take a long time, or that are part of third-party code, will continue to inhibit responsiveness. - Run signal handlers in a dedicated separate thread. IMO this is generally better than running signal handlers in the main thread, because it eliminates the separate concept of "async-safe" and just requires "thread-safe". So you can use regular threading synchronization primitives for safety, instead of relying on luck / memorized lists of atomic/re-entrant operations. Something still needs to run in the main thread though, for e.g. KeyboardInterrupt, so this is not super straightforward. Also, it could break any code that really relies on signal handlers running in the main thread. Either approach can be turned into a library, albeit potentially hackily in the first case. -- Devin
On 21Jun2015 19:16, Devin Jeanpierre <jeanpierreda@gmail.com> wrote:
I agree with all of this, but I do think that handling signals in the main program by default is a sensible default: it gives very predictable behaviour. [...]
This feels fragile: this means that former one could expect C calls to be "atomic" from the main thread's point of view and conversely the C functions can expect the main thread (or whatever calling thread called them) is paused during their execution. As soon as the calling thread can reactivate these guarrentees are broken. Supposing the C call is doing things to thread local Python variables, for just one scenario. So I'm -1 on this on the face of it.
This is not possible in general, but it can be made to work for all blocking operations in the stdlib.
Hmm. I'm not sure that you will find this universally so. No, I have no examples proving my intuition here.
Yes, I am in favour of this or something like it. Personally I would go for either or both of: - a stdlib function to specify the thread to handle signals instead of main - a stdlib function to declare that signals should immediately place a nice descriptive "signal" object on a Queue, and leaves it to the user to handle the queue (for example, by spawning a thread to consume it)
Something still needs to run in the main thread though, for e.g. KeyboardInterrupt, so this is not super straightforward.
Is this necessarily true?
Which is why it should never be the default; I am firmly of the opinion that that changed handling should be requested by the program. Cheers, Cameron Simpson <cs@zip.com.au> Facts do not discourage the conspiracy-minded. - Robert Crawford <rawford@iac.net>
On Mon, Jun 22, 2015 at 12:52 AM, Cameron Simpson <cs@zip.com.au> wrote:
If you fix select, everything else can hypothetically follow, as you can run all C functions in another thread and signal the result is ready using an fd. (This is my idea for a hacky third-party library. It only ruins stack traces!)
This just moves the problem to another thread. One can already today try to keep the main thread free to handle signals, it's just hard.
I like this. It mirror's Linux's selectfd, too. One small correction, it can't literally be a Queue, because those aren't safe to use in signal handlers. (It can be a pipe that is wrapped in a Queue-like interface, though, and if we do that, we can even use native signalfd if we want.) It also resolves an unspoken concern I had, which is that silently starting threads for the user feels icky.
What I mean is that there needs to be a way to raise KeyboardInterrupt in the main thread from a signal handler. If, as you suggest, the old behavior stays around, then that's enough. Another option, if we went with a dedicated signal handling thread, would be that uncaught exceptions propagate to the main thread when it gets around to it. -- Devin
On 22 June 2015 at 09:42, Devin Jeanpierre <jeanpierreda@gmail.com> wrote:
This particular approach presumably only works on Unix? (On Windows, select is not a general signalling operation, it only works for sockets). Presumably a cross-platform solution would need to use appropriate OS-native signalling based on the platform? Paul
On 22Jun2015 01:42, Devin Jeanpierre <jeanpierreda@gmail.com> wrote:
Yes. But it is very easy to ensure that a specifial purpose Thread is free to handle signals. And it is arguably the minimalist change.
I wasn't proposing silently starting threads. I imagined the former suggestion would be handed a thread as the signal target.
I was imagining the old behaviour stayed around by default, not necessarily as fixed behaviour. But "KeyboardInterrupt occurs in the main thread" is handy. Perhaps a better solution here is not to keep KeyboardInterrupt special (i.e. always going to the main thread) but to extend "raise" to accept a thread argument: raise blah in thread Given that signals are already presented as occuring between Python opcodes, it seems reasonable to me that the signal situation could be addressed with a common mechanism extended to exceptions. How often is the question "how do I terminate another thread?" raised on python-list? Often. The standard answer is "set a flag and have the thread consult it". That is very sensitive to how often the flag is polled: too often and it infuses the code with noise (not to mention ungainly loop termination logic etc), too infrequently and the response is much like your issue here with signals: it can be arbitrarily delayed. Suppose one could raise signals in another thread? Then the answer becomes "raise exception in other_thread". And the other thread will abort as soon as the next python opcode would fire. It has several advantages: it removes any need to poll some shared state, or the set up shared state it lets the target thread remain nice and pythonic, letting unhandled exceptions simply abort the thread automatically as they would anyway it lets the target thread catch the exception and handle it if desired it dovetails neatly with our hypothetical special signal handling thread: the handling thread has merely to "raise KeyboardInterrupt in main_thread" to get the behaviour you seek to preserve, _without_ making SIGINT specially handled - the specialness is not an aspect of the handling thread's code, not hardwired
Perhaps. But I'd rather not; you _can_ always catch every exception and if we have "raise exception in thread" we can implement the above trivially for programs which want it. Cheers, Cameron Simpson <cs@zip.com.au> Nothing is impossible for the man who doesn't have to do it.
On Jun 22, 2015, at 16:00, Cameron Simpson <cs@zip.com.au> wrote:
Perhaps a better solution here is not to keep KeyboardInterrupt special (i.e. always going to the main thread) but to extend "raise" to accept a thread argument:
raise blah in thread
Does this need to be syntax? Why not just: mythread.throw(blah) This could even use the same mechanism as signals in 3.6, while possibly being backportable to something hackier in a C extension module for older versions.
On 22Jun2015 22:42, Andrew Barnert <abarnert@yahoo.com> wrote:
Indeed. I think that extending raise's syntax is a little easier on the eye, but the advantage is small. Certainly giving threads a throw method would function as well. I was indeed hoping that signals and exceptions could be delivered the same way via such a mechanism, which would also allow signals to be delivered to a chosen thread. Cheers, Cameron Simpson <cs@zip.com.au> Thus spake the master programmer: "A well written program is its own heaven; a poorly-written program its own hell."
On 22.06.2015 04:16, Devin Jeanpierre wrote:
IMO, the above can easily be solved by going with an application design which doesn't use the main thread for any long running tasks, but instead runs these in separate threads. I don't know what the overall situation is today, but at least in the past, signal handling only worked reliably across platforms in the main thread of the application. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 22 2015)
2015-06-16: Released eGenix pyOpenSSL 0.13.10 ... http://egenix.com/go78 2015-06-10: Released mxODBC Plone/Zope DA 2.2.2 http://egenix.com/go76 2015-07-20: EuroPython 2015, Bilbao, Spain ... 28 days to go eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
I would regret losing the behavior where just raising an exception in a signal handler causes the main thread to be interrupted by that exception. I agree it would be nice if handlers ran when the main thread is waiting for I/O. -- --Guido van Rossum (python.org/~guido)
On 22Jun2015 10:28, Guido van Rossum <guido@python.org> wrote:
I would regret losing the behavior where just raising an exception in a signal handler causes the main thread to be interrupted by that exception.
I don't think any of us is sguuesting losing that as the default situation. I can see that losing this could be a side effect of a program choosing one of these alternatives.
I agree it would be nice if handlers ran when the main thread is waiting for I/O.
Hmm. That sounds doable (I speak as one totally unfamiliar with CPython's internals:-) Is this affected or improved by the recent discussions about I/O restarting over a signal? Cheers, Cameron Simpson <cs@zip.com.au> No system, regardless of how sophisticated, can repeal the laws of physics or overcome careless driving actions. - Mercedes Benz
On Sun, 21 Jun 2015 19:16:32 -0700 Devin Jeanpierre <jeanpierreda@gmail.com> wrote:
Are you aware that it is already the case today (perhaps not for "all blocking operations", but at least for those that return EINTR when interrupted)? By the way, have you read https://www.python.org/dev/peps/pep-0475/ ? Regards Antoine.
On Mon, Jun 22, 2015 at 1:41 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
That only applies when the signal arrives during the system call. For example, if you call Python select.select() and a signal is received during argument parsing, EINTR is not returned because C select() is not running yet. See https://bugs.python.org/issue5315 . The only cross-platform fix for this that I am aware of is to make select() use the self-pipe trick: - set up a pipe, and use set_wakeup_fd to make signals write to that pipe - check for signals in case any arrived before you called set_wakeup_fd - call select() as before, but also select on the pipe Without something like this, where a signal is handled no matter when it comes in, even a call which returns EINTR usually can miss signals, resulting in potentially drastically reduced responsiveness. This exact trick doesn't work for every blocking call, just for the ones in the select module. I provided a patch on that issue which does this. It is atrocious. :( If I rewrote it, I'd prefer to write it as a pure-python wrapper around select().
By the way, have you read https://www.python.org/dev/peps/pep-0475/ ?
I did once, but I reread it now. I think the PEP is focused not on making signal handling more responsive, but on making EINTR less of a trap. Although it does mention responsiveness in use case 2, it doesn't go far enough. I think the following cases matter: - Signals that arrive before the system call starts, but after the Python function call begins - Signals that arrive during a call to a blocking function which doesn't return EINTR - Signals that arrive during a call to a C function which doesn't block at all, but is just slow -- Devin
IIRC signal handler may be blocked by threading synchronization primitives etc. on Python 2.7 but it's not an issue for Python 3. I don't recall exact version -- it's 3.2 likely. Maybe Benjamin Peterson would provide more info -- hg blame says he is author of EINTR processing in _threadmodule.c On Mon, Jun 22, 2015 at 11:56 AM, Devin Jeanpierre <jeanpierreda@gmail.com> wrote:
-- Thanks, Andrew Svetlov
On 21Jun2015 19:16, Devin Jeanpierre <jeanpierreda@gmail.com> wrote:
I agree with all of this, but I do think that handling signals in the main program by default is a sensible default: it gives very predictable behaviour. [...]
This feels fragile: this means that former one could expect C calls to be "atomic" from the main thread's point of view and conversely the C functions can expect the main thread (or whatever calling thread called them) is paused during their execution. As soon as the calling thread can reactivate these guarrentees are broken. Supposing the C call is doing things to thread local Python variables, for just one scenario. So I'm -1 on this on the face of it.
This is not possible in general, but it can be made to work for all blocking operations in the stdlib.
Hmm. I'm not sure that you will find this universally so. No, I have no examples proving my intuition here.
Yes, I am in favour of this or something like it. Personally I would go for either or both of: - a stdlib function to specify the thread to handle signals instead of main - a stdlib function to declare that signals should immediately place a nice descriptive "signal" object on a Queue, and leaves it to the user to handle the queue (for example, by spawning a thread to consume it)
Something still needs to run in the main thread though, for e.g. KeyboardInterrupt, so this is not super straightforward.
Is this necessarily true?
Which is why it should never be the default; I am firmly of the opinion that that changed handling should be requested by the program. Cheers, Cameron Simpson <cs@zip.com.au> Facts do not discourage the conspiracy-minded. - Robert Crawford <rawford@iac.net>
On Mon, Jun 22, 2015 at 12:52 AM, Cameron Simpson <cs@zip.com.au> wrote:
If you fix select, everything else can hypothetically follow, as you can run all C functions in another thread and signal the result is ready using an fd. (This is my idea for a hacky third-party library. It only ruins stack traces!)
This just moves the problem to another thread. One can already today try to keep the main thread free to handle signals, it's just hard.
I like this. It mirror's Linux's selectfd, too. One small correction, it can't literally be a Queue, because those aren't safe to use in signal handlers. (It can be a pipe that is wrapped in a Queue-like interface, though, and if we do that, we can even use native signalfd if we want.) It also resolves an unspoken concern I had, which is that silently starting threads for the user feels icky.
What I mean is that there needs to be a way to raise KeyboardInterrupt in the main thread from a signal handler. If, as you suggest, the old behavior stays around, then that's enough. Another option, if we went with a dedicated signal handling thread, would be that uncaught exceptions propagate to the main thread when it gets around to it. -- Devin
On 22 June 2015 at 09:42, Devin Jeanpierre <jeanpierreda@gmail.com> wrote:
This particular approach presumably only works on Unix? (On Windows, select is not a general signalling operation, it only works for sockets). Presumably a cross-platform solution would need to use appropriate OS-native signalling based on the platform? Paul
On 22Jun2015 01:42, Devin Jeanpierre <jeanpierreda@gmail.com> wrote:
Yes. But it is very easy to ensure that a specifial purpose Thread is free to handle signals. And it is arguably the minimalist change.
I wasn't proposing silently starting threads. I imagined the former suggestion would be handed a thread as the signal target.
I was imagining the old behaviour stayed around by default, not necessarily as fixed behaviour. But "KeyboardInterrupt occurs in the main thread" is handy. Perhaps a better solution here is not to keep KeyboardInterrupt special (i.e. always going to the main thread) but to extend "raise" to accept a thread argument: raise blah in thread Given that signals are already presented as occuring between Python opcodes, it seems reasonable to me that the signal situation could be addressed with a common mechanism extended to exceptions. How often is the question "how do I terminate another thread?" raised on python-list? Often. The standard answer is "set a flag and have the thread consult it". That is very sensitive to how often the flag is polled: too often and it infuses the code with noise (not to mention ungainly loop termination logic etc), too infrequently and the response is much like your issue here with signals: it can be arbitrarily delayed. Suppose one could raise signals in another thread? Then the answer becomes "raise exception in other_thread". And the other thread will abort as soon as the next python opcode would fire. It has several advantages: it removes any need to poll some shared state, or the set up shared state it lets the target thread remain nice and pythonic, letting unhandled exceptions simply abort the thread automatically as they would anyway it lets the target thread catch the exception and handle it if desired it dovetails neatly with our hypothetical special signal handling thread: the handling thread has merely to "raise KeyboardInterrupt in main_thread" to get the behaviour you seek to preserve, _without_ making SIGINT specially handled - the specialness is not an aspect of the handling thread's code, not hardwired
Perhaps. But I'd rather not; you _can_ always catch every exception and if we have "raise exception in thread" we can implement the above trivially for programs which want it. Cheers, Cameron Simpson <cs@zip.com.au> Nothing is impossible for the man who doesn't have to do it.
On Jun 22, 2015, at 16:00, Cameron Simpson <cs@zip.com.au> wrote:
Perhaps a better solution here is not to keep KeyboardInterrupt special (i.e. always going to the main thread) but to extend "raise" to accept a thread argument:
raise blah in thread
Does this need to be syntax? Why not just: mythread.throw(blah) This could even use the same mechanism as signals in 3.6, while possibly being backportable to something hackier in a C extension module for older versions.
On 22Jun2015 22:42, Andrew Barnert <abarnert@yahoo.com> wrote:
Indeed. I think that extending raise's syntax is a little easier on the eye, but the advantage is small. Certainly giving threads a throw method would function as well. I was indeed hoping that signals and exceptions could be delivered the same way via such a mechanism, which would also allow signals to be delivered to a chosen thread. Cheers, Cameron Simpson <cs@zip.com.au> Thus spake the master programmer: "A well written program is its own heaven; a poorly-written program its own hell."
On 22.06.2015 04:16, Devin Jeanpierre wrote:
IMO, the above can easily be solved by going with an application design which doesn't use the main thread for any long running tasks, but instead runs these in separate threads. I don't know what the overall situation is today, but at least in the past, signal handling only worked reliably across platforms in the main thread of the application. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 22 2015)
2015-06-16: Released eGenix pyOpenSSL 0.13.10 ... http://egenix.com/go78 2015-06-10: Released mxODBC Plone/Zope DA 2.2.2 http://egenix.com/go76 2015-07-20: EuroPython 2015, Bilbao, Spain ... 28 days to go eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
I would regret losing the behavior where just raising an exception in a signal handler causes the main thread to be interrupted by that exception. I agree it would be nice if handlers ran when the main thread is waiting for I/O. -- --Guido van Rossum (python.org/~guido)
On 22Jun2015 10:28, Guido van Rossum <guido@python.org> wrote:
I would regret losing the behavior where just raising an exception in a signal handler causes the main thread to be interrupted by that exception.
I don't think any of us is sguuesting losing that as the default situation. I can see that losing this could be a side effect of a program choosing one of these alternatives.
I agree it would be nice if handlers ran when the main thread is waiting for I/O.
Hmm. That sounds doable (I speak as one totally unfamiliar with CPython's internals:-) Is this affected or improved by the recent discussions about I/O restarting over a signal? Cheers, Cameron Simpson <cs@zip.com.au> No system, regardless of how sophisticated, can repeal the laws of physics or overcome careless driving actions. - Mercedes Benz
On Sun, 21 Jun 2015 19:16:32 -0700 Devin Jeanpierre <jeanpierreda@gmail.com> wrote:
Are you aware that it is already the case today (perhaps not for "all blocking operations", but at least for those that return EINTR when interrupted)? By the way, have you read https://www.python.org/dev/peps/pep-0475/ ? Regards Antoine.
On Mon, Jun 22, 2015 at 1:41 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
That only applies when the signal arrives during the system call. For example, if you call Python select.select() and a signal is received during argument parsing, EINTR is not returned because C select() is not running yet. See https://bugs.python.org/issue5315 . The only cross-platform fix for this that I am aware of is to make select() use the self-pipe trick: - set up a pipe, and use set_wakeup_fd to make signals write to that pipe - check for signals in case any arrived before you called set_wakeup_fd - call select() as before, but also select on the pipe Without something like this, where a signal is handled no matter when it comes in, even a call which returns EINTR usually can miss signals, resulting in potentially drastically reduced responsiveness. This exact trick doesn't work for every blocking call, just for the ones in the select module. I provided a patch on that issue which does this. It is atrocious. :( If I rewrote it, I'd prefer to write it as a pure-python wrapper around select().
By the way, have you read https://www.python.org/dev/peps/pep-0475/ ?
I did once, but I reread it now. I think the PEP is focused not on making signal handling more responsive, but on making EINTR less of a trap. Although it does mention responsiveness in use case 2, it doesn't go far enough. I think the following cases matter: - Signals that arrive before the system call starts, but after the Python function call begins - Signals that arrive during a call to a blocking function which doesn't return EINTR - Signals that arrive during a call to a C function which doesn't block at all, but is just slow -- Devin
IIRC signal handler may be blocked by threading synchronization primitives etc. on Python 2.7 but it's not an issue for Python 3. I don't recall exact version -- it's 3.2 likely. Maybe Benjamin Peterson would provide more info -- hg blame says he is author of EINTR processing in _threadmodule.c On Mon, Jun 22, 2015 at 11:56 AM, Devin Jeanpierre <jeanpierreda@gmail.com> wrote:
-- Thanks, Andrew Svetlov
participants (8)
-
Andrew Barnert -
Andrew Svetlov -
Antoine Pitrou -
Cameron Simpson -
Devin Jeanpierre -
Guido van Rossum -
M.-A. Lemburg -
Paul Moore