Exceptions from callbacks

[Randall Hopper]

Eric Dorland:
 |Jonathan Giddy wrote:
 |> Eric Dorland <dorland at lords.com> writes:
 |> 
 |> >I'm writing a Python wrapper for a library. The library stores some
 |> >callbacks for some events, and with some glue code, get these to call
 |> >python functions. That's the easy part. Now since its the library these
 |> >callback and not Python per se, how do you pass an exception back to the
 |> >interpreter, because you can't return NULL from the callback. Do you
 |> >just set the exception, and Python will pick up on it?
...
 |You're not understanding exactly. It is synchronous (ie one thread), but
 |since the library is calling the callback, and not the python
 |interpreter, I can't return NULL to the interpreter. Basically, is there
 |a way from inside the callback to say "Wake up python, an exception!",
 |without having to return NULL to the interpreter, which is impossible
 |from the callback.

I've done this, and I got the impression that Jonathan understood your
question.  What he described works and it's what I'm using.

To extend your description, you don't say "Wake up, Python!" from the
callback.  You say it from the C function(s) from which you entered the C
library that invoked the callback.  Return NULL, and the exception is
restarted with the Python stack frames saved in the callback in-tow.

To make this concrete, for a C library API that might potentially invoke a
callback (which in-turn would relay to Python) would look like this:

  (Python) -> HandlePendingMessages -> MYLIB -> Callback -> (Python)

And your wrapper code might look like this:

------------------------------------------------------------------------------
  PyObject *_wrap_MYLIBHandlePendingMessages( PyObject *self, PyObject *args )
  {
    PyErr_Clear();

    <<< call C function here >>>

    /*  If a Python callback threw an exception, restart its propagation  */
    /*    in the controlling Python.                                      */
    if ( PyErr_Occurred() )
      return NULL;
    
    ...
  }

------------------------------------------------------------------------------
  static void _wrap_MYLIBCallback( void )
    /*  C callback used to relay to Python callable objects.  */
  {
    ...

    /*  Pass the buck onto Python  */
    pyresult = PyEval_CallObject( pyfunc, pyargs );     /*  Call Python     */
    Py_DECREF( pyargs );                                /*  Trash arg list  */
    if ( pyresult )
      Py_DECREF( pyresult );                            /*  Trash any result*/

    /*  If the callback threw an exception, it will be propagated across    */
    /*    the C layer ( we check PyErr_Occurred() on the other side).       */
  }
------------------------------------------------------------------------------

If the Python callback tossed an exception, then it's still lying around
when _wrap_MYLIBCallback() returns.  We then return through the C library,
which eventually returns to _wrap_MYLIBHandlePendingMessages, who
recognizes that a Python exception is pending and restarts its propagation
in the controlling Python stack frames.

Note that if you have a C library layer that isn't Python-exception-aware,
then it could go on to perform useless or invalid work after a Python
callback throws an exception.  In this case you'll need to "make" it aware.
That is, have it call PyErr_Occurred(), or turn Python exceptions into C
library errors inside of your callback wrappers.

Hope this clarifies things a bit.

Randall