[Cython] 'with gil:' statement

mark florisson markflorisson88 at gmail.com
Thu Mar 17 11:16:10 CET 2011

On 17 March 2011 10:08, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
> On 03/17/2011 09:27 AM, Stefan Behnel wrote:
>> Dag Sverre Seljebotn, 17.03.2011 08:38:
>>> On 03/17/2011 12:24 AM, Greg Ewing wrote:
>>>> Stefan Behnel wrote:
>>>>> I'm not sure if this is a good idea. "nogil" blocks don't have a way to
>>>>> handle exceptions, so simply jumping out of them because an inner 'with
>>>>> gil' block raised an exception can have unexpected side effects.
>>>> Seems to me that the __Pyx_WriteUnraisable should be done at
>>>> the end of the 'with gil' block, and execution should then
>>>> continue from there.
>>>> In other words, the effect on exception handling should be
>>>> the same as if the 'with gil' block had been factored out into
>>>> a separate function having no exception return value.
>>> -1.
>>> I consider the fact that exceptions don't propagate from some functions a
>>> "currently unfixable bug". We should plan for it being fixed some day.
>> It can't be fixed in general, because there are cases where exceptions
>> simply cannot be propagated. Think of C callbacks, for example. C doesn't
>> have a "normal" way of dealing with exceptions, so if an exception that
>> originated from a callback simply leads to returning from the function, it
>> may mean that the outer C code will simply continue to execute normally.
>> Nothing's won in that case.
> Yes, that's a good point. (This is what I used setjmp/longjmp to work around
> BTW, to longjmp across the calling Fortran code. I knew it wasn't doing any
> mallocs/frees, let alone any file handling etc., so this was safe.)
> I'll admit that I'm mostly focused on code like
> def f():
>    with nogil:
>        for ...:
>            A
>            if something_exceptional:
>                with gil:
>                    raise Exception(...)
>            B
>        C
> where I'd say it's up to me to make sure that B and C can safely be skipped.
> It would be a major pain to have my raised exception here be "trapped" -- in
> fact, it would make the "with gil" statement unusable for my purposes.
>> In code:
>>    cdef void c_callback(...) nogil:
>>        ... do some C stuff ...
>>        with gil:
>>            ... do some Python stuff ...
>>        ... do some more C stuff ...
>> So far, there are two proposed ways of doing this.
>> 1) acquire the GIL on entry and exit, handling unraisable exceptions right
>> before exiting.
>> 2) keep all GIL requiring code inside of the "with gil" block, including
>> unraisable exceptions.
>> I find (2) a *lot* more intuitive, as well as much safer. We can't know
>> what effects the surrounding "do C stuff" code has. It may contain
>> thread-safe C level cleanup code for the "with gil" block, for example, or
>> preparation code that enables returning into the calling C code. Simply
>> jumping out of the GIL block without executing the trailing code may simply
>> not work at all.
> I think you find (2) more intuitive because you have a very detailed
> knowledge of Cython and CPython, but that somebody new to Cython would
> expect a "with" statement to have the same control flow logic as the Python
> with statement. Of course, I don't have any data for that.
> How about this compromise: We balk on the code you wrote with:
> Error line 345: Exceptions propagating from "with gil" block cannot be
> propagated out of function, please insert try/except and handle exception
> So that we require this:
> with gil:
>    try:
>        ...
>    except:
>        warnings.warning(...) # or even cython.unraisable(e)
> This keeps me happy about not abusing the with statement for strange control
> flow, and makes the "with gil" useful for raising exceptions inside regular
> def functions with nogil blocks.

I agree with your previous statement, but not with your compromise :).
We have to differentiate between two cases, similar to Stefan's cases,
but different in a very important way that matter for nested GIL

1) Exceptions can propagate to some outer GIL section (in or outside
the current function)
2) Exceptions can't propagate, because there is no outer GIL section
and the function has a non-object return type

With your compromise, with 1) exceptions cannot propagate, but with 2)
you win forcing the user to be explicit. But then you still need to
write to some variable indicating that an exception occurred and
adjust control flow accordingly in your nogil section (unless you want
to clean up and return immediately).

If you have Python with-statement semantics, you can do the following,
for instance:

cdef void func() nogil:
    with gil:

            with nogil:
                with gil:
                    code that may raise an exception

                this is not executed

        except ExceptionRaisedFromInnerWithGilBlock:
            handle exception here

The point is, if you have case 2), and you want to use GIL code, you
need to handle exceptions in some way. Forcing the user to not
propagate anything doesn't sound right, unless this holds only for the
outermost 'with gil' block. I would be OK with that, although it would
be inconsistent with how exceptions in normal cdef functions with
non-object return work, so I would say that we'd have to force it in
the same manner there.

>>> We could perhaps fix exception propagation from nogil functions by using
>>> some conventions + setjmp/longjmp. Mono does this when calling into
>>> native
>>> code, and I recently did it manually in Cython to propagate exceptions
>>> through the Fortran wrappers in SciPy.
>> Regardless of the topic of this thread, it would be nice to have longjmp
>> support in Cython. Lupa, my Cython wrapper for LuaJIT, currently has to work
>> around several quirks in that area.
> Not sure what you mean here, I used longjmp (in a function without any
> Python objects) and it seems to work just fine. Did I miss anything?
>>> Also, the GIL may not be around
>>> forever even in CPython? (All arguments I've seen for keeping it has been
>>> along the lines of "it slows down serial code", not that it is considered
>>> a
>>> good thing.)
>> If it ever gets removed, there will surely have to be an emulation layer
>> for C modules. Many of them simply use it as thread-lock, and that's totally
>> reasonable IMHO.
> Good point. But there may be an option to disable said emulation layer that
> we want to make use of in Cython...
> (This is relevant today for Cython-on-.NET, for instance.)
>>> Designing a language around the GIL feels like a dead-end to me.
>> We keep having diverging opinions about the GIL. I like it, and I keep
>> repeating myself by saying that "threading should be explicit". Having a way
>> to lock the whole interpreter and to keep parallel execution and reentry
>> points to well defined places in your code is a great feature.
> I think all of this just comes from using Cython for totally different
> things, which gives us different perspectives on the GIL. I guess we should
> just sit down in Munich and look at each other's codes and see if we can
> understand one another that way. A big discussion on whether "the GIL is
> good or not" is not very constructive.
> Dag Sverre
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

More information about the cython-devel mailing list