[ python-Bugs-1470508 ] Error in PyGen_NeedsFinalizing()

[SourceForge.net]

Bugs item #1470508, was opened at 2006-04-14 15:31
Message generated for change (Tracker Item Submitted) made by Item Submitter
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1470508&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Interpreter Core
Group: Python 2.5
Status: Open
Resolution: None
Priority: 7
Submitted By: Tim Peters (tim_one)
Assigned to: Phillip J. Eby (pje)
Summary: Error in PyGen_NeedsFinalizing()

Initial Comment:
There's an off-by-one error in PyGen_NeedsFinalizing():
 the loop starts by looking at
f->f_blockstack[f->f_iblock], but that's one beyond the
largest legitimate f_blockstack index.  As a result,
any generator with an active loop is very likely to
claim it needs finalizing (to the extent that it's
unlikely that reading up trash for b_type will match
SETUP_LOOP).

The attached patch repairs that, but also tries to
remove some of the goofy .close() gimmicks added to old
tests in test_generators to worm around "leaks" caused
by that all generators said they needed finalizers in
2.5 at first.  (That's how I bumped into this:  I tried
removing the .close() gimmicks first, but -R:: runs
said they still leaked -- that started the trail
leading to PyGen_NeedsFinalizing()).

Alas, after applying the patch, test_generators craps
out in a debug build with a NULL-pointer dereference in
_Py_ForgetReference(), in the guts of the `if` test
deciding whether to produce an "UNREF invalid object"
error.  At the time, op_type is _PyGen_Type and
op->_ob_next = _ob_prev = NULL, probably meaning that
op was already passed to _Py_ForgetReference() previously.

We get into _Py_ForgetReference() via a chain starting
at the the decref in ceval.c's

	while (STACK_LEVEL() > b->b_level) {
		v = POP();
		Py_XDECREF(v);

and we got into PyEval_EvalFrameEx via gen_send_ex via
gen_close via gen_dealloc via (eventually) a dict on
some instance going away ... and then I got lost. 
Looks like there are several more layers of generator
deallocs on the call stack too.

I'm out of time for looking at this now, so recording
it here.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1470508&group_id=5470