[Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).

[Guido van Rossum]

On Dec 8, 2007 9:57 AM, Gustavo Carneiro <gjcarneiro at gmail.com> wrote:
> On 08/12/2007, Guido van Rossum <guido at python.org> wrote:
>
> > On Dec 8, 2007 3:58 AM, Gustavo Carneiro <gjcarneiro at gmail.com> wrote:
> > > Not only that, but pygtk is a generic module;
> >
> > What does "generic" mean in this context? Surely not that it applies
> > to other libraries than GTK?
>
> Well, actually this is also a PyGObject issue, not only PyGtk.  PyGObject
> wraps GLib.  GLib was created to serve the needs of Gtk+, but is useful by
> itself for writing portable programs.  Among other things, GLib offers a
> generic "main loop", which programs can use to do generic event-driven
> programming, such as timeouts, polling file descriptors, and doing other
> work when the main loop becomes idle.
>
> You could argue that non-gtk programs are rare and we shouldn't worry too
> much about them.  Maybe it's true, I don't know.
>
>
> > > who are we to forbid the usage
> > > of signals if python itself allows it?  If we were to ignore signals
> then
> > > sooner or later someone would come along and shout "hey, signals work
> just
> > > fine in pure python, so why did pygtk have to break my signals?".
> >
> > Um, signals don't "work just fine" in pure Python. And I would argue
> > they don't either in C. They are extremely subtle, and most code using
> > signals is broken in some way. Just try to read the sigaction() man
> > page and claim you understand it.
> >
> > Unfortunately, in Unix there are some conditions that can only be
> > delivered using signals (e.g. SIGINTR, SIGTERM) and others for which
> > your choices are either polling or signals (SIGCHILD, SIGWINCH).
> > Traditionally, solutions based on select() or poll() with a short
> > timeout (e.g. 20 or 100 ms) have worked well, as the number of
> > instructions executed each time is really negligeable, and the
> > response time is still reasonable on a human time scale. Unfortunately
> > it seems recent developments in power management for ultra-low power
> > devices have made this an issue again.
> >
> > Windows solves this more elegantly by having a unified "wait for
> > multiple conditions" system call which can wait on I/O, semaphores,
> > and other events (within the same process or coming from other
> > processes). Unfortunately, in Unix, some events don't have a file
> > descriptor associated with them, and for those select()/poll() cannot
> > be used.
> >
> > The best solution I can think of is to add a new API that takes a
> > signal and a file descriptor and registers a C-level handler for that
> > signal which writes a byte to the file descriptor. You can then create
> > a pipe, connect the signal handler to the write end, and add the read
> > end to your list of file descriptors passed to select() or poll(). The
> > handler must be written in C in order to avoid the race condition
> > referred to by Glyph (signals arriving after the signal check in the
> > VM main loop but before the select()/poll() system call is entered
> > will not be noticed until the select()/poll() call completes).
>
> Funny that everyone mentions this solution, as it is the solution
> implemented by my patch :-)

Mind pointing me to it again? I can't find it in the Python bug tracker.

> Well, to be fair, it might not be _exactly_ what is implemented by the
> patch.  Reading between the lines, I think what you mean is to have python's
> C signal handler mostly untouched; it would only write a byte to a pipe _in
> addition to_ the normal setting the flag and Py_AddPendingCall.

Well, in many cases I see no problems with the current signal handler,
and people are used to it, so I'm not sure what is gained by getting
rid of it.

> The patch I submitted, on the other hand, completely removes the vector of
> flags and Py_AddPendingCall, and instead writes the number of the signal
> that was raised into the pipe, and reads it back from the pipe in the Python
> main loop.

I believe Py_AddPendingCall was meant to be used by other places
besides the signal handling.

> Which is the best solution?  I think my patch fixes two problems: 1. the
> need to have a FD to wake up poll() (t o fix the problem with what we are
> discussing in this thread), and 2. make Python's signal handling more
> reliable (not 100% reliable because it doesn't handle longer bursts of
> signals than the pipe buffer can take, but at least is race free).

I think it's okay to drop signals if too many come. The FD should be
put in non-blocking mode so the signal handler won't block forever.
Does Unix even promise that a signal gets delivered twice if it gets
sent quickly twice in a row?

> My solution is being reject because people are afraid to touch the signal
> handling code, which has its faults, but well know faults.  But if I
> refactor the patch to keep the crappy-but-sort-of-working signal code, and
> only enhance it with the pipe, maybe it will more likely get accepted.
>
> Do I understand correctly? :-)

Not really; I don't recall seeing your patch. And I object to your
subjective description of the existing signal handling; it has served
us well enough.

I have a worry though -- if signal handlers *always* and *only*
communicate by writing a byte to a FD, does that mean that the VM main
loop will have to attempt to read a byte from a pipe every time it
checks for signals? That sounds very expensive for something that's
not needed by the vast majority of Python applications. Plus, you will
have to expose the FD to the user so that it can be included in
select() and poll() calls.

Anyway, let's see the patch first.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)