[Python-Dev] Segfault in python 2.5

Tim Peters tim.peters at gmail.com
Thu Oct 19 03:02:31 CEST 2006

[Michael Hudson]
>> I've been reading the bug report with interest, but unless I can
>> reproduce it it's mighty hard for me to debug, as I'm sure you know.

[Mike Klaas]
> Indeed.

Note that I just attached a much simpler pure-Python script that fails
very quickly, on Windows, using a debug build.  Read the new comment
to learn why both "Windows" and "debug build" are essential to it
failing reliably and quickly ;-)

>>> Unfortunately, I've been attempting for hours to reduce the problem to a
>>> completely self-contained script, but it is resisting my efforts
due to timing
>>> problems.

Yes, but you did good!  This is still just an educated guess on my
part, but my education here is hard to match ;-):  this new business
of generators deciding to "clean up after themselves" if they're left
hanging appears to have made it possible for a generator to hold on to
a frame whose thread state has been free()'d, after the thread that
created the generator has gone away.  Then when the generator gets
collected as trash, the new exception-based "clean up abandoned
generator" gimmick tries to access the generator's frame's thread
state, but that's just a raw C struct (not a Python object with
reachability-based lifetime), and the thread free()'d that struct when
the thread went away.  The important timing-based vagary here is
whether dead-thread cleanup gets performed before the main thread
tries to clean up the trash generator.

> I've peered at the code, but my knowledge of the python core is
> superficial at best.  The fact that it is occuring as a result of a
> long string of garbage collection/dealloc/etc. and involves threading
> lowers my confidence further.   That said, I'm beginning to think that
> to reproduce this in a standalone script will require understanding
> the problem in greater depth regardless...

Or upgrade to Windows ;-)

>> Are you absolutely sure that the fault does not lie with any extension
>> modules you may be using?  Memory scribbling bugs have been known to
>> cause arbitrarily confusing problems...

Unless I've changed the symptom, it's been reduced to minimal pure
Python.  It does require a thread T, and creating a generator in T,
where the generator object's lifetime is controlled by the main
thread, and where T vanishes before the generator has exited of its
own accord.

Offhand I don't know how to repair it.  Thread states /aren't/ Python
objects, and there's no provision for a thread state to outlive the
thread it represents.

> I've had sufficient experience being arbitrarily confused to never be
> sure about such things, but I am quite confident.  The script I posted
> in the bug report is all stock python save for the operation in <>'s.
> That operation is pickling and unpickling (using pickle, not cPickle)
> a somewhat complicated pure-python instance several times.

FYI, in my whittled script, your `getdocs()` became simply:

def getdocs():
    while True:
        yield None

and it's called only once, via self.docIter.next().  In fact, the
"while True:" isn't needed there either (given that it's only resumed
once now).

More information about the Python-Dev mailing list