[Cython] 'with gil:' statement

[mark florisson]

On 18 March 2011 13:36, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Dag Sverre Seljebotn, 18.03.2011 12:07:
>>
>> On 03/18/2011 11:10 AM, Stefan Behnel wrote:
>>>
>>> Actually, I think I still find it more convenient to not provide *any*
>>> special exception paths through nogil code, i.e. to not let exceptions in
>>> "with gil" blocks exit from outer "nogil" blocks. That would avoid all of
>>> the semantic difficulties above.
>>
>> Well, of course not supporting something is easier.
>
> That's a bit exaggerated, IMHO. I would expect that there'd be many cases
> where the right thing to do in case of an exception is to return from a
> function, with a bit of explicit cleanup. So many exceptions won't leave the
> "with gil" block anyways. And few users will actually use nested "with gil"
> and "nogil" blocks.

This also holds for cdef nogil functions with 'with gil' blocks.

> But still, there's the question how code that re-acquires the GIL should
> behave (still thinking of a callback case here). Should it check for a
> raised exception and jump into the exception handling path? Or should it
> ignore it and continue until something tests for the exception explicitly?

I'm not sure what you mean, are you talking about exceptions raised in
the with gil block, or about exceptions that were pending before the
function was called? In the latter, case, my answer would be, "in the
same way that functions declared 'with gil' work": you assume there is
no pending exception. It doesn't make sense to call Python-related
code with an exception set. And indeed, the finally clause works by
first clearing and storing the exception on the stack, and then
restoring it afterwards if no other exception propagated. This is not
a case we should strife to handle, it's incorrect code.

> And what if intermediately running nogil code changed some state somewhere
> that breaks the exception handling path?

If users call into C code that raises an exception, it is their task
to take care of any exceptions manually, or by using the right Cython
declarations (such as the 'except' declaration clause for cdef
functions). This is not any different from any other semantics we
currently have for gil sections.

> Or what if the code block that
> temporarily re-acquires the GIL (e.g. a logging handler) is completely
> unrelated to the one that actually raised the exception or the one that
> called into the whole machinery (and that would be the right one to handle
> it)? These things are not trivial, neither for Cython's language semantics
> nor for users.

If this happens in a with gil section in the finally clause of a nogil
section, then I think they should be chained (in python 3). In python
2 the exception should simply be replaced, as per the semantics of
finally.

> We shouldn't forget that basically all Python operations can at least raise
> a MemoryError or a KeyboardInterrupt etc. Most users won't think of these
> cases. I think it would help users in writing safer code if the need to
> handle exceptions in nogil blocks was always made explicit by the compiler.
> That's different from "not supporting something".
>
>> Relying on boolean flags to signal errors states is a real pain
>> when one is used to using exceptions.
>
> Well, if those exception do not take safe and obvious paths, it may still be
> better to use explicit boolean flags instead.

Again, I think "Python semantics" are quite obvious for exception
paths from with gil statements. If the user really needs to do
cleanup, exceptions may be caught in the with gil block and an error
flag used, or the cleanup can be moved to the finally clause in the
gil section. Without compiler enforcement, the user will forget the
try/finally around the gil section as easily as the try/except or
try/finally around the body of the with gil section. I don't find
enforcing it useful at all, though. The only thing that would be
useful is enforcing the user to handle exceptions when they might be
unraisable due to the function's return type.

> Specifically, I'm fine with letting exceptions flow through nogil code iff
> there is a way to react on exceptional exit paths inside of nogil blocks.
> I'm against enabling "except" clauses in nogil blocks, but I think a
> try-finally would catch the most important use cases iff we can figure out
> clean semantics for this, including the cases I mentioned previously. That's
> the condition I see.

To summarize, I think the semantics should be like this:
- propagate exceptions from with gil blocks through nogil sections
- if there is a finally clause in a nogil section corresponding to a
try surrounding a with gil block, execute it
- if the finally clause has with gil blocks, allow them to raise
exceptions, thereby overriding any pending exception, using the
semantics just mentioned (applied recursively)

> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>