[Python-Dev] Segfault in python 2.5

[Mike Klaas]

On 10/18/06, Tim Peters <tim.peters at gmail.com> wrote:
> [Mike Klaas]
> > Indeed.
>
> Note that I just attached a much simpler pure-Python script that fails
> very quickly, on Windows, using a debug build.  Read the new comment
> to learn why both "Windows" and "debug build" are essential to it
> failing reliably and quickly ;-)

Thanks!  Next time I find a bug, installing Windows will  certainly be
my first step <g>.

<>
> Yes, but you did good!  This is still just an educated guess on my
> part, but my education here is hard to match ;-):  this new business
> of generators deciding to "clean up after themselves" if they're left
> hanging appears to have made it possible for a generator to hold on to
> a frame whose thread state has been free()'d, after the thread that
> created the generator has gone away.  Then when the generator gets
> collected as trash, the new exception-based "clean up abandoned
> generator" gimmick tries to access the generator's frame's thread
> state, but that's just a raw C struct (not a Python object with
> reachability-based lifetime), and the thread free()'d that struct when
> the thread went away.  The important timing-based vagary here is
> whether dead-thread cleanup gets performed before the main thread
> tries to clean up the trash generator.

Indeed--and normally it doesn't happen that way.  My/your script never
crashes on the first iteration because the thread's target is the
generator and thus it gets DECREF'd before the thread terminates.  But
the exception from the first iteration holds on to a reference to the
frame/generator so when it gets cleaned up (in the second iteration,
due to a new exception overwriting it) the generator is freed after
the thread is destroyed.  At least, I think...

<>
> Offhand I don't know how to repair it.  Thread states /aren't/ Python
> objects, and there's no provision for a thread state to outlive the
> thread it represents.

Take this with a grain of salt, but ISTM that the problem can be
repaired by resetting the generator's frame threadstate to the current
threadstate:

(in genobject.c:gen_send_ex():80)
        Py_XINCREF(tstate->frame);
        assert(f->f_back == NULL);
        f->f_back = tstate->frame;
+        f->f_tstate = tstate;

        gen->gi_running = 1;
        result = PyEval_EvalFrameEx(f, exc);
        gen->gi_running = 0;

Shouldn't the thread state generally be the same anyway? (I seem to
recall some gloomy warning against resuming generators in separate
threads).

This solution is surely wrong--if f_tstate != tstate, then the
generator _is_ being resumed in another thread and so the generated
traceback will be wrong (among other issues which surely occur by
fudging a frame's threadstate).  Perhaps it could be set conditionally
by gen_close before signalling the exception?  A lie, but a smaller
lie than a segfault.  We could advertise that the exception ocurring
from generator .close() isn't guaranteed to have an accurate traceback
in this case.

Take all this with a grain of un-core-savvy salt.

Thanks again for investigating this, Tim,
-Mike