[Cython] 'with gil:' statement

Stefan Behnel stefan_ml at behnel.de
Thu Mar 17 09:27:00 CET 2011

Dag Sverre Seljebotn, 17.03.2011 08:38:
> On 03/17/2011 12:24 AM, Greg Ewing wrote:
>> Stefan Behnel wrote:
>>> I'm not sure if this is a good idea. "nogil" blocks don't have a way to
>>> handle exceptions, so simply jumping out of them because an inner 'with
>>> gil' block raised an exception can have unexpected side effects.
>> Seems to me that the __Pyx_WriteUnraisable should be done at
>> the end of the 'with gil' block, and execution should then
>> continue from there.
>> In other words, the effect on exception handling should be
>> the same as if the 'with gil' block had been factored out into
>> a separate function having no exception return value.
> -1.
> I consider the fact that exceptions don't propagate from some functions a
> "currently unfixable bug". We should plan for it being fixed some day.

It can't be fixed in general, because there are cases where exceptions 
simply cannot be propagated. Think of C callbacks, for example. C doesn't 
have a "normal" way of dealing with exceptions, so if an exception that 
originated from a callback simply leads to returning from the function, it 
may mean that the outer C code will simply continue to execute normally. 
Nothing's won in that case.

In code:

     cdef void c_callback(...) nogil:
         ... do some C stuff ...
         with gil:
             ... do some Python stuff ...
         ... do some more C stuff ...

So far, there are two proposed ways of doing this.

1) acquire the GIL on entry and exit, handling unraisable exceptions right 
before exiting.

2) keep all GIL requiring code inside of the "with gil" block, including 
unraisable exceptions.

I find (2) a *lot* more intuitive, as well as much safer. We can't know 
what effects the surrounding "do C stuff" code has. It may contain 
thread-safe C level cleanup code for the "with gil" block, for example, or 
preparation code that enables returning into the calling C code. Simply 
jumping out of the GIL block without executing the trailing code may simply 
not work at all.

> We could perhaps fix exception propagation from nogil functions by using
> some conventions + setjmp/longjmp. Mono does this when calling into native
> code, and I recently did it manually in Cython to propagate exceptions
> through the Fortran wrappers in SciPy.

Regardless of the topic of this thread, it would be nice to have longjmp 
support in Cython. Lupa, my Cython wrapper for LuaJIT, currently has to 
work around several quirks in that area.

> Also, the GIL may not be around
> forever even in CPython? (All arguments I've seen for keeping it has been
> along the lines of "it slows down serial code", not that it is considered a
> good thing.)

If it ever gets removed, there will surely have to be an emulation layer 
for C modules. Many of them simply use it as thread-lock, and that's 
totally reasonable IMHO.

> Designing a language around the GIL feels like a dead-end to me.

We keep having diverging opinions about the GIL. I like it, and I keep 
repeating myself by saying that "threading should be explicit". Having a 
way to lock the whole interpreter and to keep parallel execution and 
reentry points to well defined places in your code is a great feature.

> I'm OK
> with being practical in the face of the limitations of today; but let's
> keep "with gil" and "with nogil" something that can become noops in the
> future without too much pain. Yes, I know that if the GIL goes it will
> break Stefan's lxml code, and I'm sure other code -- I'm just saying that
> we shouldn't make the language design even more GIL-centric than it already
> is.

It's not. Even a removal of the GIL won't remove the fact that C can't 
propagate exceptions.


More information about the cython-devel mailing list