[Python-Dev] A proposed solution for Issue 502236: Asyncrhonousexceptions between threads

[Robert Brewer]

Josiah Carlson wrote:
> This doesn't need to be an interpreter thing; it's easy to implement
> by the user (I've done it about a dozen times using a single global
> flag).  If you want it to be automatic, it's even possible to make it
> happen automatically using sys.settrace() and friends (you can even
> make it reasonably fast if you use a C callback).

Agreed. If someone wants a small library to help do this, especially in
web servers, the latest version of Cherrpy includes a 'process'
subpackage under a generous license. It does all the things Andy
describes via a Bus object:

> Andy Scott wrote:
> > 1. Put in place a new function call sys.exitapplication, what this
> > would do is:
> >      a. Mark a flag in t0's data structure saying a request to
> >         shutdown has been made

This is bus.exit(), which publishes a 'stop' message to all subscribed
'stop' listeners, and then an 'exit' message to any 'exit' listeners.

> >      b. Raise a new exception, SystemShuttingDown, in t1.

That's up to the listener.

> >  2. As the main interpreter executes it checks the "shutting down
> >     flag" in the per thread data and follows one of two paths:
> >     If it is t0:
> >      a. Stops execution of the current code sequence
> >      b. Iterates over all extant threads ...
> >      c. Enters a timed wait loop where it will allow the other
> >         threads time to see the signal. It will iterate this loop
> >         a set number of times to avoid being blocked on any given
> >         thread.

This is implemented as [t.join() for t in threading.enumerate()] in the
main thread.

> >      d. When all threads have exited, or been forcefully closed,
> >         raise the SystemShuttingDown exception

The bus just lets the main thread exit at this point.

> > P1. If the thread is in a tight loop will it see the exception? Or
> > more generally: when should the exception be raised?

That's dependent enough on what work the thread is doing that a
completely generic approach is generally not sufficient. Therefore, the
process.bus sends a 'stop' message, and leaves the implementation of the
receiver up to the author of that thread's logic. Presumably, one
wouldn't register a listener for the 'stop' message unless one knew how
to actually stop.

> > P2. When should the interpreter check this flag?
> >
> > I think the answer to both of these problems is to check the flag,
> > and hence raise the exception, in the following circumstances:
> >   - When the interpreter executes a back loop. So this should catch
> >     the jump back to the top of a "while True:" loop
> >   - Just before the interpreter makes a call to a hooked in non-
> >     Python system function, e.g. file I/O, networking &c.

This is indeed how most well-written apps do it already.

> > Checking at these points should be the minimal required, I think, to
> > ensure that a given thread can not ignore the exception. It may be
> > possible, or even required, to perform the check every time a Python
> > function call is made.

PLEASE don't make Python function calls slower.

> >  1. The Python interpreter has per thread information.
> >  2. The Python interpreter can tell if the system, t0, thread is
> >     running.
> >  3. The Python engine has (or can easily obtain) a list of all
> >     threads it created.
> >  4. It is possible to raise exceptions as the byte code is
executing.

Replace 'Python interpreter' with 'your application' and those become
relatively simple architectural issues: maintain a list of threads, have
them expose an interface to determine if they're running, and make them
monitor a flag to know when another thread is asking them to stop.


Robert Brewer
fumanchu at aminus.org