[Cython] Recent bugs in generators
vitja.makarov at gmail.com
Mon Apr 18 15:19:18 CEST 2011
2011/4/18 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 18.04.2011 06:38:
>> 2011/4/18 Stefan Behnel<stefan_ml at behnel.de>:
>>> Vitja Makarov, 17.04.2011 17:57:
>>>> 3. check_yield_in_exception()
>>> I added this because I found a failing pyregr test that uses it (testing
>>> @contextmanager decorator).
>>>> Cython calls __Pyx_ExceptionReset when except block is done, so when
>>>> yield is there no exception reset is called.
>>>> I'm not sure how to fix this.
>>> I'm not completely sure either.
>>>> import sys
>>>> def foo():
>>>> >>> list(foo())
>>>> [<type 'exceptions.ValueError'>, None]
>>>> raise ValueError
>>>> except ValueError:
>>>> yield sys.exc_info()
>>>> yield sys.exc_info() # exc_info is lost here
>>> I think (!), the difference here is that CPython actually keeps the
>>> exception in the generator frame. We don't have a frame, so we have to
>>> emulate it using the closure class. I guess we'll have to store away the
>>> exception into the closure when we yield while an exception is being
>>> handled, and restore it afterwards. Note: this is not the exception that
>>> freshly *being* raised (the "_cur*" fields in the thread state), it's the
>>> exception that *was* raised and is now being handled, i.e. the thread
>>> fields without the "_cur", that are reflected by sys.exc_info().
>> Interesting difference between py2 and py3:
>> def foo():
>> raise ValueError
>> except ValueError:
>> File "xxx.py", line 7, in<module>
>> File "xxx.py", line 6, in foo
>> TypeError: exceptions must be old-style classes or derived from
>> BaseException, not NoneType
>> It seems that exception info is completely lost (tried 2.6, 2.7) and
>> seems to be fixed in python3.
> Not surprising. The implementation is completely different in Py2 and Py3,
> both in CPython and in Cython. It's actually much simpler in Cython under
> Py3, due to better semantics and C-API support. That also implies that
> there's much less Cython can do wrong in that environment. ;-)
>> Btw exception info temps are already saved and restored between yields.
> Right, but the exc_info itself is not reset and recovered around the yield.
> As I said above, generators have their own lifetime frame in CPython, and
> exceptions don't leak from that. So, whenever it's the generator (or code
> called by it) that raises an exception, that must be kept local to the
There is one more interesting thing:
>>> o = test_yield_inside_genexp()
[0, None, 1, None, 2, None]
return ((yield i) for i in range(3))
More information about the cython-devel