[Python-Dev] Debugging opportunity :-)

[Phillip J. Eby]

At 01:11 AM 4/15/2006 +0200, Thomas Wouters wrote:
>On 4/15/06, Thomas Wouters <<mailto:thomas at python.org>thomas at python.org> 
>wrote:
>>(I first thought the problem was caused by gen_dealloc doing 
>>'Py_DECREF(gen->gen_frame)' instead of 'frame = gen->gen_frame; 
>>gen->gen_frame = NULL; Py_DECREF(frame)', but that isn't the case. It 
>>should do it that way, I believe, but it's not the cause of this crash.)
>
>fixes the crash.Ah, found the problem. After I hit 'send', I realized I 
>hadn't checked frameobject's tp_clear, and sure enough, it calls 
>Py_XDECREF in-place. That explains why the first generator object gets 
>dealloced twice, although it doesn't explain why it doesn't blow up when 
>it reaches a negative refcount.

You're missing another piece of the puzzle: the problem is that since 
generators don't have a tp_clear, they don't know they're pointing to an 
invalid frame; it appears to still be running.  So when the frame object 
releases items off the stack, it releases its reference to the generator, 
which is released normally, dropping out the local variable reference back 
to the generator whose frame is being cleared.  That generator doesn't know 
it's holding a garbage frame, and thus proceeds to finalize it by resuming 
it...  which then pops the blockstack, and f_stacktop is still valid, so 
popping the blockstack decrefs the frame a second time.


>  Fixing frameobject and genobject to both use Py_CLEAR() makes both the 
> 'minimal' testcase and test_generators work.

It seems to me that frame_clear() should also set f_stacktop to NULL before 
doing any clearing; otherwise it's possible for a generator to think that 
the frame is still executable, and the double-decref could thus be replaced 
by a null pointer dereference.