[Python-ideas] Responsive signal handling

[Devin Jeanpierre]

On Mon, Jun 22, 2015 at 12:52 AM, Cameron Simpson <cs at zip.com.au> wrote:
> On 21Jun2015 19:16, Devin Jeanpierre <jeanpierreda at gmail.com> wrote:
>> This is not possible in general, but it can be made to work for all
>> blocking operations in the stdlib.
>
>
> Hmm. I'm not sure that you will find this universally so. No, I have no
> examples proving my intuition here.

If you fix select, everything else can hypothetically follow, as you
can run all C functions in another thread and signal the result is
ready using an fd. (This is my idea for a hacky third-party library.
It only ruins stack traces!)

>> Operations that run in C but just
>> take a long time, or that are part of third-party code, will continue
>> to inhibit responsiveness.
>>
>> - Run signal handlers in a dedicated separate thread.
>>
>> IMO this is generally better than running signal handlers in the main
>> thread, because it eliminates the separate concept of "async-safe" and
>> just requires "thread-safe". So you can use regular threading
>> synchronization primitives for safety, instead of relying on luck /
>> memorized lists of atomic/re-entrant operations.
>
>
> Yes, I am in favour of this or something like it. Personally I would go for
> either or both of:
>
>  - a stdlib function to specify the thread to handle signals instead of main

This just moves the problem to another thread. One can already today
try to keep the main thread free to handle signals, it's just hard.

>  - a stdlib function to declare that signals should immediately place a nice
> descriptive "signal" object on a Queue, and leaves it to the user to handle
> the queue (for example, by spawning a thread to consume it)

I like this. It mirror's Linux's selectfd, too. One small correction,
it can't literally be a Queue, because those aren't safe to use in
signal handlers. (It can be a pipe that is wrapped in a Queue-like
interface, though, and if we do that, we can even use native signalfd
if we want.)

It also resolves an unspoken concern I had, which is that silently
starting threads for the user feels icky.

>> Something still needs to run in the main thread though, for e.g.
>> KeyboardInterrupt, so this is not super straightforward.
>
>
> Is this necessarily true?

What I mean is that there needs to be a way to raise KeyboardInterrupt
in the main thread from a signal handler. If, as you suggest, the old
behavior stays around, then that's enough.

Another option, if we went with a dedicated signal handling thread,
would be that uncaught exceptions propagate to the main thread when it
gets around to it.

-- Devin