[Python-ideas] Responsive signal handling

[Devin Jeanpierre]

On Mon, Jun 22, 2015 at 1:41 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Sun, 21 Jun 2015 19:16:32 -0700
> Devin Jeanpierre <jeanpierreda at gmail.com>
> wrote:
>>
>> One could implement one or both of the following:
>>
>> - Keep only running signal handlers in the main thread, but allow them
>> to run even in the middle of a call to a C function for as many C
>> functions as we can.
>> This is not possible in general, but it can be made to work for all
>> blocking operations in the stdlib.
>
> Are you aware that it is already the case today (perhaps not for "all
> blocking operations", but at least for those that return EINTR when
> interrupted)?

That only applies when the signal arrives during the system call. For
example, if you call Python select.select() and a signal is received
during argument parsing, EINTR is not returned because C select() is
not running yet. See https://bugs.python.org/issue5315 .
The only cross-platform fix for this that I am aware of is to make
select() use the self-pipe trick:

 - set up a pipe, and use set_wakeup_fd to make signals write to that pipe
 - check for signals in case any arrived before you called set_wakeup_fd
 - call select() as before, but also select on the pipe

Without something like this, where a signal is handled no matter when
it comes in, even a call which returns EINTR usually can miss signals,
resulting in potentially drastically reduced responsiveness. This
exact trick doesn't work for every blocking call, just for the ones in
the select module.

I provided a patch on that issue which does this. It is atrocious. :(
If I rewrote it, I'd prefer to write it as a pure-python wrapper
around select().

> By the way, have you read https://www.python.org/dev/peps/pep-0475/ ?

I did once, but I reread it now. I think the PEP is focused not on
making signal handling more responsive, but on making EINTR less of a
trap. Although it does mention responsiveness in use case 2, it
doesn't go far enough.

I think the following cases matter:

- Signals that arrive before the system call starts, but after the
  Python function call begins
- Signals that arrive during a call to a blocking function which
  doesn't return EINTR
- Signals that arrive during a call to a C function which doesn't
  block at all, but is just slow

-- Devin