[Python-Dev] Signals+Threads (PyGTK waking up 10x/sec).

[Gustavo Carneiro]

On 08/12/2007, Guido van Rossum <guido at python.org> wrote:
>
> On Dec 8, 2007 9:57 AM, Gustavo Carneiro <gjcarneiro at gmail.com> wrote:
> > On 08/12/2007, Guido van Rossum <guido at python.org> wrote:
> >
> > > On Dec 8, 2007 3:58 AM, Gustavo Carneiro <gjcarneiro at gmail.com> wrote:
> > > > Not only that, but pygtk is a generic module;
> > >
> > > What does "generic" mean in this context? Surely not that it applies
> > > to other libraries than GTK?
> >
> > Well, actually this is also a PyGObject issue, not only
> PyGtk.  PyGObject
> > wraps GLib.  GLib was created to serve the needs of Gtk+, but is useful
> by
> > itself for writing portable programs.  Among other things, GLib offers a
> > generic "main loop", which programs can use to do generic event-driven
> > programming, such as timeouts, polling file descriptors, and doing other
> > work when the main loop becomes idle.
> >
> > You could argue that non-gtk programs are rare and we shouldn't worry
> too
> > much about them.  Maybe it's true, I don't know.
> >
> >
> > > > who are we to forbid the usage
> > > > of signals if python itself allows it?  If we were to ignore signals
> > then
> > > > sooner or later someone would come along and shout "hey, signals
> work
> > just
> > > > fine in pure python, so why did pygtk have to break my signals?".
> > >
> > > Um, signals don't "work just fine" in pure Python. And I would argue
> > > they don't either in C. They are extremely subtle, and most code using
> > > signals is broken in some way. Just try to read the sigaction() man
> > > page and claim you understand it.
> > >
> > > Unfortunately, in Unix there are some conditions that can only be
> > > delivered using signals (e.g. SIGINTR, SIGTERM) and others for which
> > > your choices are either polling or signals (SIGCHILD, SIGWINCH).
> > > Traditionally, solutions based on select() or poll() with a short
> > > timeout (e.g. 20 or 100 ms) have worked well, as the number of
> > > instructions executed each time is really negligeable, and the
> > > response time is still reasonable on a human time scale. Unfortunately
> > > it seems recent developments in power management for ultra-low power
> > > devices have made this an issue again.
> > >
> > > Windows solves this more elegantly by having a unified "wait for
> > > multiple conditions" system call which can wait on I/O, semaphores,
> > > and other events (within the same process or coming from other
> > > processes). Unfortunately, in Unix, some events don't have a file
> > > descriptor associated with them, and for those select()/poll() cannot
> > > be used.
> > >
> > > The best solution I can think of is to add a new API that takes a
> > > signal and a file descriptor and registers a C-level handler for that
> > > signal which writes a byte to the file descriptor. You can then create
> > > a pipe, connect the signal handler to the write end, and add the read
> > > end to your list of file descriptors passed to select() or poll(). The
> > > handler must be written in C in order to avoid the race condition
> > > referred to by Glyph (signals arriving after the signal check in the
> > > VM main loop but before the select()/poll() system call is entered
> > > will not be noticed until the select()/poll() call completes).
> >
> > Funny that everyone mentions this solution, as it is the solution
> > implemented by my patch :-)
>
> Mind pointing me to it again? I can't find it in the Python bug tracker.
>
> > Well, to be fair, it might not be _exactly_ what is implemented by the
> > patch.  Reading between the lines, I think what you mean is to have
> python's
> > C signal handler mostly untouched; it would only write a byte to a pipe
> _in
> > addition to_ the normal setting the flag and Py_AddPendingCall.
>
> Well, in many cases I see no problems with the current signal handler,
> and people are used to it, so I'm not sure what is gained by getting
> rid of it.
>
> > The patch I submitted, on the other hand, completely removes the vector
> of
> > flags and Py_AddPendingCall, and instead writes the number of the signal
> > that was raised into the pipe, and reads it back from the pipe in the
> Python
> > main loop.
>
> I believe Py_AddPendingCall was meant to be used by other places
> besides the signal handling.


True.  The patch does not remove Py_AddPendingCall, only stops using it for
signals.

> Which is the best solution?  I think my patch fixes two problems: 1. the
> > need to have a FD to wake up poll() (t o fix the problem with what we
> are
> > discussing in this thread), and 2. make Python's signal handling more
> > reliable (not 100% reliable because it doesn't handle longer bursts of
> > signals than the pipe buffer can take, but at least is race free).
>
> I think it's okay to drop signals if too many come. The FD should be
> put in non-blocking mode so the signal handler won't block forever.
> Does Unix even promise that a signal gets delivered twice if it gets
> sent quickly twice in a row?


Good point.

> My solution is being reject because people are afraid to touch the signal
> > handling code, which has its faults, but well know faults.  But if I
> > refactor the patch to keep the crappy-but-sort-of-working signal code,
> and
> > only enhance it with the pipe, maybe it will more likely get accepted.
> >
> > Do I understand correctly? :-)
>
> Not really; I don't recall seeing your patch. And I object to your
> subjective description of the existing signal handling; it has served
> us well enough.


Sorry.  I have to say existing code is very bad in order to convince people
it is worth changing... :P

Anyway, the approach of writing to a pipe and reading it back later appears
(to me) simpler to read and understand.  That should also count for
something, right?

I have a worry though -- if signal handlers *always* and *only*
> communicate by writing a byte to a FD, does that mean that the VM main
> loop will have to attempt to read a byte from a pipe every time it
> checks for signals? That sounds very expensive for something that's
> not needed by the vast majority of Python applications.


Yes, it would be expensive.  Instead the patch uses a simple global boolean
value, set to true when a signal is received, after writing to the pipe.
PyErr_CheckSignals() does not try to read from the pipe unless that value is
true.

Plus, you will
> have to expose the FD to the user so that it can be included in
> select() and poll() calls.


Correct.

Anyway, let's see the patch first.


http://bugs.python.org/issue1564547

-- 
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
"The universe is always one step beyond logic." -- Frank Herbert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20071208/94a9727d/attachment-0001.htm