Possible to assure no "cyclic"/"uncollectible" memory leaks?

[Klaas]

Joe Peterson wrote:
> I've been doing a lot of searching on the topic of one of Python's more
> disturbing issues (at least to me): the fact that if a __del__ finalizer
> is defined and a cyclic (circular) reference is made, the garbage
> collector cannot clean it up.

It is a somewhat fundamental limitation of GCs, if you want to support:

1. __del__ that can resurrect objects and is deterministically called
when objects are destroyed
2. the "view" of alive objects by __del__ methods is consistent
3. no crashing

If there is a cycle of objects containing __del__ methods, there is
clearly no way of knowing a safe order of invoking them.

> First of all, it seems that it's best to avoid using __del__.  So far, I
> have never used it in my Python programming.  So I am safe there.  Or am
> I?  Also, to my knowledge, I have never created a cyclic reference, but
> we do not typically create bugs intentionally either (and there are
> certainly times when it is an OK thing to do).

It is good practice to avoid __del__ unless there is a compelling
reason to do so.  weakref resource management is much safer.  Note that
it is pretty much impossible to avoid creating reference cycles--they
have a tendency to sneak into unsuspecting places (for instance, bound
methods can be a subtle source of cycles).

> Still, it's not comforting to know that it is possible to create a
> situation that would create a memory leak using a language that is
> supposed to relieve us of that worry.  I understand the problem, but it
> would be nice to know that as a programmer, I could be assured that
> Python would always deal with memory management and that memory leaks
> were not something I had to think about.

It is unrealistic to ever be completely relieved of such worry, since
it is always possible to accidently hold on to a strong reference to
data that should actually be "garbage".  But your question is perhaps
precluding these kinds of memory leak.  In that case, it is a matter of
providing to the programmer sufficiently-fine-grained abstractions such
that the compiler can reason about their safety.  For instance, an
included weakref-based resource cleanup scheme has been discussed and
would cover many of the current uses of __del__.  It would also be nice
to remove some of the hidden "gotchas" that are inherent in CPython,
like the integer and float object freelist (not necessarily removing
those features, but providing some mechanism for reclaiming them when
they get out of hand).

These things can reduce the possibility of a problem, but (IMO) can
never completely obviate it.

> So here's a question: if I write Python software and never use __del__,
> can I guarantee that there is no way to create a memory leak?  What
> about system libraries - do any of them use __del__, and if so, are they
> written in such a way that it is not possible to create a cyclic reference?

It is always possible to create a cyclic reference by monkeypatching a
class.  Here are the stdlib modules which use __del__:
$ find -name \*.py | xargs grep __del__ | grep -v test
./Mac/Demo/sound/morselib.py:    def __del__(self):
./Lib/telnetlib.py:    def __del__(self):
./Lib/plat-mac/EasyDialogs.py:    def __del__(self):
./Lib/plat-mac/FrameWork.py:    def __del__(self):
./Lib/plat-mac/MiniAEFrame.py:    def __del__(self):
./Lib/plat-mac/Audio_mac.py:    def __del__(self):
./Lib/plat-mac/videoreader.py:    def __del__(self):
./Lib/fileinput.py:    def __del__(self):
./Lib/subprocess.py:    def __del__(self):
./Lib/gzip.py:    def __del__(self):
./Lib/wave.py:    def __del__(self):
./Lib/wave.py:    def __del__(self):
./Lib/popen2.py:    def __del__(self):
./Lib/lib-tk/Tkdnd.py:    def __del__(self):
./Lib/lib-tk/tkFont.py:    def __del__(self):
./Lib/lib-tk/Tkinter.py:    def __del__(self):
./Lib/lib-tk/Tkinter.py:    def __del__(self):
./Lib/urllib.py:    def __del__(self):
./Lib/tempfile.py:        # __del__ is called.
./Lib/tempfile.py:        def __del__(self):
./Lib/tarfile.py:    def __del__(self):
./Lib/socket.py:    def __del__(self):
./Lib/zipfile.py:    fp = None                   # Set here since
__del__ checks it
./Lib/zipfile.py:    def __del__(self):
./Lib/httplib.py:    def __del__(self):
./Lib/bsddb/dbshelve.py:    def __del__(self):
./Lib/bsddb/dbshelve.py:    def __del__(self):
./Lib/bsddb/__init__.py:    def __del__(self):
./Lib/bsddb/dbtables.py:    def __del__(self):
./Lib/idlelib/MultiCall.py:    def __del__(self):
./Lib/idlelib/MultiCall.py:    def __del__(self):
./Lib/idlelib/MultiCall.py:        def __del__(self):
./Lib/sunau.py:    def __del__(self):
./Lib/sunau.py:    def __del__(self):
./Lib/poplib.py:    #__del__ = quit
./Lib/_threading_local.py:    def __del__(self):
./Lib/aifc.py:    def __del__(self):
./Lib/dumbdbm.py:    # gets called.  One place _commit() gets called is
from __del__(),
./Lib/dumbdbm.py:        # be called from __del__().  Therefore we must
never reference a
./Lib/dumbdbm.py:    __del__ = close
./Lib/wsgiref/validate.py:    def __del__(self):
./Lib/shelve.py:    def __del__(self):
./Lib/cgi.py:        terminates, try defining a __del__ method in a
derived class
./Lib/platform.py:    __del__ = close
./Lib/audiodev.py:    def __del__(self):
./Lib/audiodev.py:    def __del__(self):

-Mike