[Python-Dev] patch-finalizer vs Zope3

[Tim Peters]

[Tim, on a patch-finalizer consequence]
> OK, that actually wasn't the first test to produce garbage, it was
> just the first time test.py *noticed* gc.garbage wasn't empty.  The
> first test to produce garbage was BuddyCityState, and it produces
> garbage in isolation:
> ...

Man, this sure brings back miserable memories of how hard it could be
to debug leaks due to cycles before Neil added gcmodule.c!  The best
tool we had then was my Cyclops.py, but I stopped keeping that up to
date because there was obviously no more need for it <wink>.

Desktop boxes have gotten a lot more capable since then, and apps have
grown in ambition to match.  Here we ran one measly test from the huge
Zope3 test suite; it ran in an eye blink (maybe two).  From the first
weakref in gc.garbage alone, it turned out that more than 42,000
distinct cycles were reachable, breaking into a bit fewer than 4,000
strongly connected components, the latter ranging in size from 2
objects (almost all a new-style class and its mro) to about 40,000.  A
text file account of these SCCs consumed about 20MB.

So what you do?  I don't know.  These are always solvable, but they're
rarely easy.  I always end up writing a small pile of new graph
analysis code specifically exploiting peculiarities of the case at
hand.  In this case I also changed gcmodule.c a little, to put trash
reachable from resurrected trash weakrefs into gc.garbage too, but
unlike gc.DEBUG_SAVEALL not to also put *all* trash in gc.garbage. 
That was key to figuring out what the original trash cycles were.

"The answer" in this case appears to be in Zope3's
zope/interface/adapter.py's AdapterRegistry.__init__:

class AdapterRegistry(object):
   """Adapter registry
   """

   # Implementation note:
   # We are like a weakref dict ourselves. We can't use a weakref
   # dict because we have to use spec.weakref() rather than
   # weakref.ref(spec) to get weak refs to specs.

   _surrogateClass = Surrogate

   def __init__(self):
       default = self._surrogateClass(Default, self)
       self._default = default
       null = self._surrogateClass(Null, self)
       self._null = null

       # Create separate lookup object and copy it's methods
       surrogates = {Default.weakref(): default, Null.weakref(): null}
       def _remove(k):
           try:
               del surrogates[k]
           except KeyError:
               pass
       lookup = AdapterLookup(self, surrogates, _remove)

The rest doesn't appear to matter.  As the attached smoking gun shows,
a weakref W has an instance of the nested _remove function as its
callback(*).  The _remove function has a cell object, pointing to the
lexically enclosing `surrogates` dict.  That dict in turn has W as a
key.  So there's a cycle, and refcounting alone can't clean it up. 
All of this stuff *was* trash, but because patch-finalizer decides to
call W reachable, the cell and the `surrogates` dict became reachable
too, and none of it got cleaned up (nor, of course, did anything else
reachable from this cycle).

When 2.4b2 looks at this case, it clears W because its referent is
trash, and ignore's W's callback because W is also trash.  Since, had
it been called, the only thing the callback would have done is mutate
other trash, it didn't hurt not to call it.  But it remains a stretch
to believe that "ignoring callbacks on trash weakrefs is harmless" is
a necessary outcome.

Still, whatever else this may or may not imply, it convinces me we
can't adopt a patch-finalizer-ish approach without serious effort at
making leaks "due to it" more easily analyzable.

(*) A weakref W's callback is discoverable after all in Python, just
by calling gc.get_referents(W).  This is something I've rediscovered
about 6 times by now, and it always catches me by surprise <wink>.
-------------- next part --------------
cycle
obj 0:
cycle starts here
    0x3c84f20
    <type 'weakref'>
<weakref at 03C84F20; to 'InterfaceClass' at 012B8118 (IWriteContainer)>
    weakref to <InterfaceClass zope.app.container.interfaces.IWriteContainer>
obj 1:
    0x3c89a30
    <type 'function'>
<function _remove at 0x03C89A30>
obj 2:
    0x3c8b658
    <type 'tuple'>
(<cell at 0x03C8B968: dict object at 0x03C6A620>,)
obj 3:
    0x3c8b968
    <type 'cell'>
<cell at 0x03C8B968: dict object at 0x03C6A620>
obj 4:
    0x3c6a620
    <type 'dict'>
{<weakref at 020628F0; to 'InterfaceClass' at 015748F8 (Default)>: <Surrogate(<InterfaceClass zope.interface.adapter.Default>)>,
 <weakref at 020629C8; to 'InterfaceClass' at 01574A10 (Null)>: <Surrogate(<InterfaceClass zope.interface.adapter.Null>)>,
 <weakref at 03C84F20; to 'InterfaceClass' at 012B8118 (IWriteContainer)>: <Surrogate(<InterfaceClass zope.app.container.interfaces.IWriteContainer>)>,
 <weakref at 03C8C080; to 'InterfaceClass' at 00BD0188 (Interface)>: <Surrogate(<InterfaceClass zope.interface.Interface>)>,
 <weakref at 03C8C278; to 'InterfaceClass' at 012B3D20 (IObjectEvent)>: <Surrogate(<InterfaceClass zope.app.event.interfaces.IObjectEvent>)>,
 <weakref at 03C8C3E0; to 'InterfaceClass' at 011909D8 (IHTTPRequest)>: <Surrogate(<InterfaceClass zope.publisher.interfaces.http.IHTTPRequest>)>,
 <weakref at 03C8C500; to 'InterfaceClass' at 011904D0 (IRequest)>: <Surrogate(<InterfaceClass zope.publisher.interfaces.IRequest>)>,
 <weakref at 03C8C620; to 'InterfaceClass' at 01185FC0 (IPublisherRequest)>: <Surrogate(<InterfaceClass zope.publisher.interfaces.IPublisherRequest>)>,
 <weakref at 03C8C740; to 'InterfaceClass' at 01185D90 (IPublicationRequest)>: <Surrogate(<InterfaceClass zope.publisher.interfaces.IPublicationRequest>)>,
 <weakref at 03C8C860; to 'InterfaceClass' at 00DB5D20 (IPresentationRequest)>: <Surrogate(<InterfaceClass zope.component.interfaces.IPresentationRequest>)>,
 <weakref at 03C8C9C8; to 'InterfaceClass' at 00D99818 (IParticipation)>: <Surrogate(<InterfaceClass zope.security.interfaces.IParticipation>)>,
 <weakref at 03C8CC08; to 'InterfaceClass' at 011903F0 (IApplicationRequest)>: <Surrogate(<InterfaceClass zope.publisher.interfaces.IApplicationRequest>)>,
 <weakref at 03C8CD28; to 'InterfaceClass' at 0117A700 (IEnumerableMapping)>: <Surrogate(<InterfaceClass zope.interface.common.mapping.IEnumerableMapping>)>,
 <weakref at 03C8CE48; to 'InterfaceClass' at 0117A540 (IReadMapping)>: <Surrogate(<InterfaceClass zope.interface.common.mapping.IReadMapping>)>,
 <weakref at 03C8CF68; to 'InterfaceClass' at 0117A428 (IItemMapping)>: <Surrogate(<InterfaceClass zope.interface.common.mapping.IItemMapping>)>}
cycle ends here
    0x3c84f20
    <type 'weakref'>
<weakref at 03C84F20; to 'InterfaceClass' at 012B8118 (IWriteContainer)>
    weakref to <InterfaceClass zope.app.container.interfaces.IWriteContainer>