[Python-ideas] ExitStack: Allow exiting individual context managers

[Ram Rachum]

Thanks for the detailed reply Nick.

I would like an `ExitPool` class but I'm not invested enough in this
feature to champion it, so I'll let it go at this point. If you'll open the
ticket and want any feedback from me about the API I'll be happy to give
it, just email me. I think I can solve my personal problem with a context
manager wrapper that doesn't complain when you try to close the context
manager twice.

Thanks,
Ram.

On Tue, Dec 8, 2015 at 7:56 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 8 December 2015 at 06:37, Ram Rachum <ram at rachum.com> wrote:
> > I would actually want a method that exits not just the last context
> manager,
> > but any context manager in the stack according to my choosing. Maybe it
> > clashes with the fact that you're using `deque`, but I'm not sure that
> you
> > have a compelling reason to use `deque`.
>
> deque itself is an implementation detail, but from a design
> perspective, ExitStack is intended to recreate the semantics of
> lexical context management using with statements, without having to
> actually use that layout in your code. In other words, if the exit
> semantics can't be expressed in terms of with statements, then I'm not
> interested in allowing it in ExitStack specifically (see the ExitPool
> discussion below for more on that qualifier).
>
> That means the semantic structures I'm open to ExitStack supporting are:
>
> * a nested context stack (which it already does)
> * a tree structure (which exit_last_context() would allow)
>
> The first structure corresponds to passing multiple contexts to the
> with statement:
>
>     with cm1(), cm2(), cm3():
>         ...
>
> Which in turn corresponds to nested with statements:
>
>     with cm1():
>         with cm2():
>             with cm3():
>                 ...
>
> The ExitStack equivalent is:
>
>     with ExitStack() as stack:
>         stack.enter_context(cm1())
>         stack.enter_context(cm2())
>         stack.enter_context(cm3())
>         ...
>
> Adding an exit_last_context() method would make it possible to
> replicate the following kind of structure:
>
>     with cm1():
>         with cm2():
>             ...
>         with cm3():
>             ...
>
> Given exit_last_context(), replicating that dynamically would look like:
>
>     with ExitStack() as stack:
>         stack.enter_context(cm1())
>         stack.enter_context(cm2())
>         ...
>         stack.exit_last_context()
>         stack.enter_context(cm3())
>         ...
>
> I'm not aware of any specific use cases for the latter behaviour
> though, which is why that feature doesn't exist yet.
>
> > If you're asking about my use case: It's pretty boring. I have a sysadmin
> > script with a long function that does remote actions on a few servers. I
> > wrapped it all in an `ExitStack` since I use file-based locks and I want
> to
> > ensure they get released eventually. Now, at some point I want to release
> > the file-based lock manually, but I can't use a with statement, because
> > there's a condition around the place where I acquire the lock. It's
> > something like this:
> >
> > if condition:
> >
> >     exit_stack.enter_context(get_lock_1())
> >
> > else:
> >
> >     exit_stack.enter_context(get_lock_2())
> >
> > So ideally I would want a method that takes a context manager and just
> exits
> > it. Maybe even add an optional argument `context_manager` to the existing
> > `close` method. Personally I don't care about exception-handling in this
> > case, and while I think it would be nice to include exception-handling, I
> > see that the existing close method doesn't provide exception-handling
> > either, so I wouldn't feel bad about it.
>
> OK, thanks for the clarification. The additional details show that
> this is a slightly different use case from those that ExitStack is
> designed to support, as ExitStack aims to precisely replicate the
> semantics of nested with statements (as described above). That
> includes both the order in which the __exit__ methods get called, and
> which context managers can suppress exceptions from which other
> context managers.
>
> That's not the only way to manage cleanup logic though, and one
> relevant alternative is the way the atexit module works:
> https://docs.python.org/3/library/atexit.html
>
> In that model, the cleanup handlers are all considered peer
> operations, and while they're defined to be run in last-in-first-out
> order, the assumption is that there aren't any direct dependencies
> between them the way there can be with lexically nested context
> managers. That then makes it reasonable to offer the ability to
> unregister arbitrary callbacks without worrying about the potential
> impact on other callbacks that were registered later.
>
> While I'm not open to adding atexit style logic to ExitStack, I'm *am*
> amenable to the idea of adding a separate ExitPool context manager
> that doesn't try to replicate with statement semantics the way
> ExitStack does, and instead offers atexit style logic where each exit
> function receives the original exception state passed in to
> ExitPool.__exit__. One key difference from atexit would be that if any
> of the exit methods raised an exception, then I'd have ExitPool raise
> a subclass of RuntimeError (PoolExitError perhaps?) containing a list
> of all of the cleanup operations that failed.
>
> The API for that would probably look something like:
>
>     class ExitPool:
>         def enter_context(cm):
>             # Call cm.__enter__ and register cm
>         def register(cm):
>             # Register cm.__exit__ to be called on pool exit
>         def callback(func, *args, **kwds):
>             # Register func to be called on pool exit
>         def unregister(cm_or_func):
>             # Unregister a registered CM or callback function
>         def unregister_all():
>             # Empty the pool without calling anything
>         def close():
>             # Empty the pool, calling all registered callbacks in LIFO
> order (via self.__exit__)
>
> Internally, the main data structure would be an OrderedDict instance
> mapping from cm's or functions to their registered callbacks (for ease
> of unregistration).
>
> At this point, if you're open to filing one, we should probably move
> further discussion over to a new RFE on the contextlib2 issue tracker:
> https://bitbucket.org/ncoghlan/contextlib2/
>
> That's still pending a rebase on Python 3.5 standard library version
> of contextlib though...
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151208/604e84fe/attachment.html>