[Python-Dev] Signals, threads, blocking C functions

[Gustavo Carneiro]

On 9/4/06, Nick Maclaren <nmm1 at cus.cam.ac.uk> wrote:
> "Gustavo Carneiro" <gjcarneiro at gmail.com> wrote:
> >   I am now thinking of something along these lines:
> > typedef void (*PyPendingCallNotify)(void *user_data);
> > PyAPI_FUNC(void) Py_AddPendingCallNotify(PyPendingCallNotify callback,
> >     void *user_data);
> > PyAPI_FUNC(void) Py_RemovePendingCallNotify(PyPendingCallNotify
> >     callback, void *user_data);
>
> Why would that help?  The problems are semantic, not syntactic.
>
> Anthony Baxter isn't exaggerating the problem, despite what you may
> think from his posting.

  You guys are tough customers to please.  I am just trying to solve a
problem here, not create a new one; you have to believe me.

  OK, let's review what we know about current python, signals, and threads:

     1. Python launches threads without touching sigprocmask;
     2. Python installs signal handlers for all signals;
     3. Signals can be delivered to any thread, let's assume (because
of point #1 and not others not mentioned) that we have no control over
which threads receive which signals, might as well be random for all
we know;
     4. Python signal handlers do almost nothing: just sets a flag,
and calls Py_AddPendingCall, to postpone the job of handling a signal
until a "safer" time.
     5. The function Py_MakePendingCalls() should eventually get
called at a "safer" time by user or python code.
     6. It follows that until Py_MakePendingCalls() is called, the
signal will not be handled at all!

  Now, back to explaining the problem.

     1. In PyGTK we have a gobject.MainLoop.run() method, which blocks
essentially forever in a poll() system call, and only wakes if/when it
has to process timeout or IO event;
     2. When we only have one thread, we can guarantee that e.g.
SIGINT will always be caught by the thread running the
g_main_loop_run(), so we know poll() will be interrupted and a EINTR
will be generated, giving us control temporarily back to check for
python signals;
     3. When we have multiple thread, we cannot make this assumption,
so instead we install a timeout to periodically check for signals.

  We want to get rid of timeouts.  Now my idea: add a Python API to say:
     "dear Python, please call me when you start having pending calls,
even if from a signal handler context, ok?"

>From that point on, signals will get handled by Python, python calls
PyGTK, PyGTK calls a special API to safely wake up the main loop even
from a thread or signal handler, then main loop checks for signal by
calling PyErr_CheckSignals(), it is handled by Python, and the process
lives happily ever after, or die trying.

  I sincerely hope my explanation was satisfactory this time.

  Best regards.


PS: there's a "funny" comment in Py_AddPendingCall that suggests it is
not very safe against reentrancy problems:

	/* XXX Begin critical section */
	/* XXX If you want this to be safe against nested
	   XXX asynchronous calls, you'll have to work harder! */

Are signal handlers guaranteed to not be interrupted by another
signal, at least?  What about threads?