[Python-Dev] Fun with ExitStack

[Barry Warsaw]

I was trying to debug a problem in some work code and I ran into some
interesting oddities with contextlib.ExitStack and other context managers in
Python 3.5.

This program creates a temporary directory, and I wanted to give it a --keep
flag to not automatically delete the tempdir at program exit.  I was using an
ExitStack to manage a bunch of resources, and the temporary directory is the
first thing pushed into the ExitStack.  At that point in the program, I check
the value of --keep and if it's set, I use ExitStack.pop_all() to clear the
stack, and thus, presumably, prevent the temporary directory from being
deleted.

There's this relevant quote in the contextlib documentation:

"""
Each instance [of an ExitStack] maintains a stack of registered callbacks that
are called in reverse order when the instance is closed (either explicitly or
implicitly at the end of a with statement). Note that callbacks are not
invoked implicitly when the context stack instance is garbage collected.
"""

However if I didn't save the reference to the pop_all'd ExitStack, the tempdir
would be immediately deleted.  If I did save a reference to the pop_all'd
ExitStack, the tempdir would live until the saved reference went out of scope
and got refcounted away.

As best I can tell this happens because TemporaryDirectory.__init__() creates
a weakref finalizer which ends up calling the _cleanup() function.  Although
it's rather difficult to trace, it does appear that when the ExitStack is
gc'd, this finalizer gets triggered (via weakref.detach()), thus causing the
_cleanup() method to be called and the tmpdir to get deleted.  I "fix" this by
doing:

    def __init__(self):
        tmpdir = TemporaryDirectory()
        self._tmpdir = (tmpdir.name if keep
                        else self.resources.enter_context(tmpdir))

There must be more to the story because when __init__() exits in the --keep
case, tmpdir should have gotten refcounted away and the directory deleted, but
it doesn't.  I haven't dug down deep enough to figure that out.

Now, while I was debugging that behavior, I ran across more interesting bits.
I put this in a file to drive some tests:

------snip snip-----
with ExitStack() as resources:
    print('enter context')
    tmpdir = resources.enter_context(X())
    resources.pop_all()
    print('exit context')
------snip snip-----

Let's say X is:

class X:
    def __enter__(self):
        print('enter Foo')
        return self

    def __exit__(self, *args, **kws):
        print('exit Foo')
        return False

the output is:

enter context
enter Foo
exit context

So far so good.  A fairly standard context manager class doesn't get its
__exit__() called even when the program exits.  Let's try this:

@contextmanager
def X():
    print('enter bar')
    yield
    print('exit bar')

still good:

enter context
enter bar
exit context

Let's modify X a little bit to be a more common idiom:

@contextmanager
def X():
    print('enter foo')
    try:
        yield
    finally:
        print('exit foo')

enter context
enter foo
exit foo
exit context

Ah, the try-finally changes the behavior!  There's probably some documentation
somewhere that defines how a generator gets finalized, and that triggers the
finally clause, whereas in the previous example, nothing after the yield gets
run.  I just can't find that anything that would describe the observed
behavior.

It's all very twisty, and I'm not sure Python is doing anything wrong, but I'm
also not sure it's *not* doing anything wrong. ;)

In any case, the contextlib documentation quoted above should probably be more
liberally sprinkled with salty caveats.  Just calling .pop_all() isn't
necessarily enough to ensure that resources managed by an ExitStack will
survive its garbage collection.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160718/701ddf41/attachment.sig>