[Python-Dev] Re: More fun with Python shutdown

Jim Fulton jim at zope.com
Tue Nov 11 05:32:01 EST 2003

Tim Peters wrote:
 > Jim (Fulton) refactored oodles of Zope3 to make heavier use of weak
 > references.  Now Zope3 dies with a segfault when it's shut down, which makes
 > its adoption of Python 2.3.2 a bit less attractive <wink>.

My main concern at this point is getting a 2.3.3 that doesn't have this
behavior. In the worst case, I think I could create a version of weak dicts
that avoided the symptom, by avoiding attribute accesses in weakref callbacks.

> The problem isn't really understood.  I hope that once it is, there will be
> a simple way to avoid it under 2.3.2.  Jim filed a bug report with a fix to
> the symptom here:
>     http://www.python.org/sf/839548

The theory is that it occurs when a cycle involving a class is broken by
calling the tp_clear slot on a heap type.  I verified this by setting
a gdb break point in Zope 3 and verifying that type_clear was called while a
type still had a ref count much higher than 1.

 From a purely theoretical point of view, the current behavior is wrong.
There is clearly an invariant that tp_mro is not None and type_clear violates
this. The fix (setting the mro to () in type_clear, is pretty straightforward.

My assumption is that it's possible for this to occur at times other than
shutdown, although, perhaps, wildly unlikely.

What's especially poorly understood is how to make it happen in a smallter
test program.

> It's another case where things go crazy during the second call of
> PyGC_Collect in Py_Finalize.  Alas, we haven't found a simpler failing test
> case than "Zope3" yet.
> For bafflement value, I'll give a cmdline-parameterized snippet here that
> displays at least 4 distinct behaviors at shutdown, although a segfault
> isn't one of them:

BTW, with a debug build, I get an assertion error rather than a segfault.

> """
> import weakref
> import os
> class C(object):
>     def hi(self, w=os.write):
>         w(1, 'hi 1\n')
>         print 'hi 2'
> def pp(c=C()):
>     c.hi()
> import sys
> exec "import %s as somemodule" % sys.argv[1] in globals()
> del sys
> somemodule.c1 = C()
> somemodule.awr = weakref.ref(somemodule.c1, lambda ignore, pp=pp: pp())
> del C, pp
> """
> Here are the ways it behaves (on Windows, anyway):
> C:\Code\python\PCbuild>python temp4.py tempfile
> hi 1
> hi 2
> C:\Code\python\PCbuild>python temp4.py math # curiously, __main__ the same
> C:\Code\python\PCbuild>python temp4.py __builtin__
> hi 1
> C:\Code\python\PCbuild>python temp4.py sys
> hi 1
> Exception exceptions.AttributeError: "'NoneType' object has no attribute
>     'write'" in <function <lambda> at 0x006B6C70> ignored
> C:\Code\python\PCbuild>
> The only one I can't make any sense of is __builtin__:  the weakref callback
> is certainly invoked then, but its print statement neither produces output
> nor raises an exception.

When trying to debug this in Zope 3, I similarly noticed that prints in the
weakref callback produced no output.


Jim Fulton           mailto:jim at zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org

More information about the Python-Dev mailing list