Re: [Python-ideas] ExitStack: Allow exiting individual context managers

8 Dec 2015

      On 8 December 2015 at 06:37, Ram Rachum <ram@rachum.com> wrote:
...
I would actually want a method that exits not just the last context manager,
but any context manager in the stack according to my choosing. Maybe it
clashes with the fact that you're using `deque`, but I'm not sure that you
have a compelling reason to use `deque`.
deque itself is an implementation detail, but from a design
perspective, ExitStack is intended to recreate the semantics of
lexical context management using with statements, without having to
actually use that layout in your code. In other words, if the exit
semantics can't be expressed in terms of with statements, then I'm not
interested in allowing it in ExitStack specifically (see the ExitPool
discussion below for more on that qualifier).

That means the semantic structures I'm open to ExitStack supporting are:

* a nested context stack (which it already does)
* a tree structure (which exit_last_context() would allow)

The first structure corresponds to passing multiple contexts to the
with statement:

    with cm1(), cm2(), cm3():
        ...

Which in turn corresponds to nested with statements:

    with cm1():
        with cm2():
            with cm3():
                ...

The ExitStack equivalent is:

    with ExitStack() as stack:
        stack.enter_context(cm1())
        stack.enter_context(cm2())
        stack.enter_context(cm3())
        ...

Adding an exit_last_context() method would make it possible to
replicate the following kind of structure:

    with cm1():
        with cm2():
            ...
        with cm3():
            ...

Given exit_last_context(), replicating that dynamically would look like:

    with ExitStack() as stack:
        stack.enter_context(cm1())
        stack.enter_context(cm2())
        ...
        stack.exit_last_context()
        stack.enter_context(cm3())
        ...

I'm not aware of any specific use cases for the latter behaviour
though, which is why that feature doesn't exist yet.
...
If you're asking about my use case: It's pretty boring. I have a sysadmin
script with a long function that does remote actions on a few servers. I
wrapped it all in an `ExitStack` since I use file-based locks and I want to
ensure they get released eventually. Now, at some point I want to release
the file-based lock manually, but I can't use a with statement, because
there's a condition around the place where I acquire the lock. It's
something like this:
if condition:
exit_stack.enter_context(get_lock_1())
else:
exit_stack.enter_context(get_lock_2())
So ideally I would want a method that takes a context manager and just exits
it. Maybe even add an optional argument `context_manager` to the existing
`close` method. Personally I don't care about exception-handling in this
case, and while I think it would be nice to include exception-handling, I
see that the existing close method doesn't provide exception-handling
either, so I wouldn't feel bad about it.
OK, thanks for the clarification. The additional details show that
this is a slightly different use case from those that ExitStack is
designed to support, as ExitStack aims to precisely replicate the
semantics of nested with statements (as described above). That
includes both the order in which the __exit__ methods get called, and
which context managers can suppress exceptions from which other
context managers.

That's not the only way to manage cleanup logic though, and one
relevant alternative is the way the atexit module works:
https://docs.python.org/3/library/atexit.html

In that model, the cleanup handlers are all considered peer
operations, and while they're defined to be run in last-in-first-out
order, the assumption is that there aren't any direct dependencies
between them the way there can be with lexically nested context
managers. That then makes it reasonable to offer the ability to
unregister arbitrary callbacks without worrying about the potential
impact on other callbacks that were registered later.

While I'm not open to adding atexit style logic to ExitStack, I'm *am*
amenable to the idea of adding a separate ExitPool context manager
that doesn't try to replicate with statement semantics the way
ExitStack does, and instead offers atexit style logic where each exit
function receives the original exception state passed in to
ExitPool.__exit__. One key difference from atexit would be that if any
of the exit methods raised an exception, then I'd have ExitPool raise
a subclass of RuntimeError (PoolExitError perhaps?) containing a list
of all of the cleanup operations that failed.

The API for that would probably look something like:

    class ExitPool:
        def enter_context(cm):
            # Call cm.__enter__ and register cm
        def register(cm):
            # Register cm.__exit__ to be called on pool exit
        def callback(func, *args, **kwds):
            # Register func to be called on pool exit
        def unregister(cm_or_func):
            # Unregister a registered CM or callback function
        def unregister_all():
            # Empty the pool without calling anything
        def close():
            # Empty the pool, calling all registered callbacks in LIFO
order (via self.__exit__)

Internally, the main data structure would be an OrderedDict instance
mapping from cm's or functions to their registered callbacks (for ease
of unregistration).

At this point, if you're open to filing one, we should probably move
further discussion over to a new RFE on the contextlib2 issue tracker:
https://bitbucket.org/ncoghlan/contextlib2/

That's still pending a rebase on Python 3.5 standard library version
of contextlib though...

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia