ExitStack: Allow exiting individual context managers
Hi guys, I'm using `contextlib.ExitStack` today, and pushing context managers into it. I find myself wanting to exit specific context managers that I've pushed into it, while still inside the `with` suite of the `ExitStack`. In other words, I want to exit one of the context managers but still keep the `ExitStack`, and all other context managers, acquired. This isn't currently possible, right? What do you think about implementing this? Thanks, Ram.
On Sun, Dec 06, 2015 at 11:48:41PM +0200, Ram Rachum wrote:
I'm not entirely sure what you mean. Can you give an example? Some of the examples given here: https://docs.python.org/3/library/contextlib.html#examples-and-recipes sound like they might be related to what you are trying to do. Otherwise, if I have understood your requirement correctly, I think you might have a good case for *not* using ExitStack. If you have one context manager that you want to treat differently from the others, perhaps you should write it differently from the others: with ExitStack() as stack: files = [stack.enter_context(open(fname)) for fname in filenames] do_things() with open(special_file) as sf: do_other_things() do_more_things() -- Steven
On 7 December 2015 at 10:32, Steven D'Aprano <steve@pearwood.info> wrote:
It's a concept I considered implementing, but decided to hold off on it because there are a few different design options here and I didn't have any use cases to guide the API design, nor the capacity to do usability testing to see if the additional API complexity actually made ExitStack easier to use overall. The first design option is the status quo: using multiple with statement blocks, potentially in conjunction with multiple ExitStack instances. The virtue of this approach is that it means that once a context manager is added to an ExitStack instance, that's it - its lifecycle is now coupled to that of the other context managers in the stack. You can postpone cleaning up all of them with "pop_all()" (transferring responsibility for the cleanup to a fresh ExitStack instance), but you can't selectively clean them up from the end. The usage guidelines are thus relatively simple: if you don't want to couple the lifecycles of two context managers together, then don't add them to the same ExitStack instance. However, there is also that "Stack" in the name, so it's natural for users to expect to be able to both push() *and* pop() individual context managers on the stack. The (on the surface) simplest design option to *add* to the API would be a single "pop()" operation that returned a new ExitStack instance (for return type consistency with pop_all()) that contained the last context manager pushed onto the stack. However, this API is problematic, as you've now decoupled the nesting of the context manager stack - the popped context manager may now survive beyond the closure of the original ExitStack. Since this kind a pop_all() inspired selective clean-up API would result in two different ExitStack instances anyway, the status quo seems cleaner to me than this option, as it's obvious from the start that there are seperate ExitStack instances with potentially distinct lifecycles. The next option would then be to offer a separate "exit_last_context()" method, that exited the last context manager pushed onto the stack. This would be a single-stepping counterpart to the existing close() method, that allowed you to dynamically descend and ascend the context management stack during normal operation, while still preserving the property that the entire stack will be cleaned up when encountering an exception. Assuming we went with that simpler in-place API, there would still be a number of further design questions to be answered: * Do we need to try to manage the reported exception context the way ExitStack.__exit__ does? * Does "exit_last_context()" need to accept exception details like __exit__ does? * Does "exit_last_context()" need to support the ability to suppress exceptions? * What, if anything, should the return value be? * What happens if there are no contexts on the stack to pop? * Should it become possible to query the number of registered callbacks? Taking them in order, as a matter of implementation feasibility, the answer to the first question likely needs to be "No". For consistency with calling __exit__ methods directly, the answers to the next three questions likely need to be "support the same thing __exit__ supports". For the second last question, while it's reasonable to call close(), pop_all() or __exit__() on an empty stack and have it silently do nothing, if someone has taken it on themselves to manage the stack depth manually, then it's likely more helpful to complain that the stack is empty than it is to silently do nothing. Since exit_last_context() may behave differently depending on whether or not there are items on the stack, and the number of items on the stack would be useful for diagnostic purposes, then it likely also makes sense to implement a __len__ method that delegated to "len(self._exit_callbacks)". That all suggests a possible implementation along the lines of the following: def exit_last_context(self, *exc_details): if not self._exit_callbacks: raise RuntimeError("Attempted to exit last context on empty ExitStack instance") cb = self._exit_callbacks.pop() return cb(*exc_details) def __len__(self): return len(self._exit_callbacks) What I don't know is whether or not that's actually a useful enough improvement over the status quo to justify the additional cognitive burden when learning the ExitStack API - the current API was designed around the ExitStack recipes in the documentation, which were all fairly common code patterns, but most cases where I might consider using an "exit_last_context()" method, I'd be more inclined to follow Steven's advice and use a separate context manager for the resource with an independent lifecycle. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
I would actually want a method that exits not just the last context manager, but any context manager in the stack according to my choosing. Maybe it clashes with the fact that you're using `deque`, but I'm not sure that you have a compelling reason to use `deque`. If you're asking about my use case: It's pretty boring. I have a sysadmin script with a long function that does remote actions on a few servers. I wrapped it all in an `ExitStack` since I use file-based locks and I want to ensure they get released eventually. Now, at some point I want to release the file-based lock manually, but I can't use a with statement, because there's a condition around the place where I acquire the lock. It's something like this: if condition: exit_stack.enter_context(get_lock_1()) else: exit_stack.enter_context(get_lock_2()) So ideally I would want a method that takes a context manager and just exits it. Maybe even add an optional argument `context_manager` to the existing `close` method. Personally I don't care about exception-handling in this case, and while I think it would be nice to include exception-handling, I see that the existing close method doesn't provide exception-handling either, so I wouldn't feel bad about it. So maybe something like this: def close(self, context_manager=None): """Immediately unwind the context stack""" if context_manager is None: self.__exit__(None, None, None) else: for _exit_wrapper in reversed(self._exit_callbacks): if _exit_wrapper.__self__ is context_manager: _exit_wrapper(None, None, None) self._exit_callbacks.remove(_exit_wrapper) Maybe also support accepting a tuple of context managers. On Mon, Dec 7, 2015 at 5:40 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Mon, Dec 7, 2015 at 3:37 PM Ram Rachum <ram@rachum.com> wrote:
You can't put the condition in a context manager's __init__ or __enter__? class CM: def __init__(self, condition): self.lock = lock1 if condition else lock2 def __enter__(self): self.lock.acquire() delf __exit__(self, *info): self.lock.release()
On 8 December 2015 at 06:37, Ram Rachum <ram@rachum.com> wrote:
deque itself is an implementation detail, but from a design perspective, ExitStack is intended to recreate the semantics of lexical context management using with statements, without having to actually use that layout in your code. In other words, if the exit semantics can't be expressed in terms of with statements, then I'm not interested in allowing it in ExitStack specifically (see the ExitPool discussion below for more on that qualifier). That means the semantic structures I'm open to ExitStack supporting are: * a nested context stack (which it already does) * a tree structure (which exit_last_context() would allow) The first structure corresponds to passing multiple contexts to the with statement: with cm1(), cm2(), cm3(): ... Which in turn corresponds to nested with statements: with cm1(): with cm2(): with cm3(): ... The ExitStack equivalent is: with ExitStack() as stack: stack.enter_context(cm1()) stack.enter_context(cm2()) stack.enter_context(cm3()) ... Adding an exit_last_context() method would make it possible to replicate the following kind of structure: with cm1(): with cm2(): ... with cm3(): ... Given exit_last_context(), replicating that dynamically would look like: with ExitStack() as stack: stack.enter_context(cm1()) stack.enter_context(cm2()) ... stack.exit_last_context() stack.enter_context(cm3()) ... I'm not aware of any specific use cases for the latter behaviour though, which is why that feature doesn't exist yet.
OK, thanks for the clarification. The additional details show that this is a slightly different use case from those that ExitStack is designed to support, as ExitStack aims to precisely replicate the semantics of nested with statements (as described above). That includes both the order in which the __exit__ methods get called, and which context managers can suppress exceptions from which other context managers. That's not the only way to manage cleanup logic though, and one relevant alternative is the way the atexit module works: https://docs.python.org/3/library/atexit.html In that model, the cleanup handlers are all considered peer operations, and while they're defined to be run in last-in-first-out order, the assumption is that there aren't any direct dependencies between them the way there can be with lexically nested context managers. That then makes it reasonable to offer the ability to unregister arbitrary callbacks without worrying about the potential impact on other callbacks that were registered later. While I'm not open to adding atexit style logic to ExitStack, I'm *am* amenable to the idea of adding a separate ExitPool context manager that doesn't try to replicate with statement semantics the way ExitStack does, and instead offers atexit style logic where each exit function receives the original exception state passed in to ExitPool.__exit__. One key difference from atexit would be that if any of the exit methods raised an exception, then I'd have ExitPool raise a subclass of RuntimeError (PoolExitError perhaps?) containing a list of all of the cleanup operations that failed. The API for that would probably look something like: class ExitPool: def enter_context(cm): # Call cm.__enter__ and register cm def register(cm): # Register cm.__exit__ to be called on pool exit def callback(func, *args, **kwds): # Register func to be called on pool exit def unregister(cm_or_func): # Unregister a registered CM or callback function def unregister_all(): # Empty the pool without calling anything def close(): # Empty the pool, calling all registered callbacks in LIFO order (via self.__exit__) Internally, the main data structure would be an OrderedDict instance mapping from cm's or functions to their registered callbacks (for ease of unregistration). At this point, if you're open to filing one, we should probably move further discussion over to a new RFE on the contextlib2 issue tracker: https://bitbucket.org/ncoghlan/contextlib2/ That's still pending a rebase on Python 3.5 standard library version of contextlib though... Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Thanks for the detailed reply Nick. I would like an `ExitPool` class but I'm not invested enough in this feature to champion it, so I'll let it go at this point. If you'll open the ticket and want any feedback from me about the API I'll be happy to give it, just email me. I think I can solve my personal problem with a context manager wrapper that doesn't complain when you try to close the context manager twice. Thanks, Ram. On Tue, Dec 8, 2015 at 7:56 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 9 December 2015 at 03:21, Ram Rachum <ram@rachum.com> wrote:
Unfortunately, I haven't even found the time to bring contextlib2 up to date with the current standard library version, let alone consider using it to evaluate new APIs like ExitPool :( Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, Dec 06, 2015 at 11:48:41PM +0200, Ram Rachum wrote:
I'm not entirely sure what you mean. Can you give an example? Some of the examples given here: https://docs.python.org/3/library/contextlib.html#examples-and-recipes sound like they might be related to what you are trying to do. Otherwise, if I have understood your requirement correctly, I think you might have a good case for *not* using ExitStack. If you have one context manager that you want to treat differently from the others, perhaps you should write it differently from the others: with ExitStack() as stack: files = [stack.enter_context(open(fname)) for fname in filenames] do_things() with open(special_file) as sf: do_other_things() do_more_things() -- Steven
On 7 December 2015 at 10:32, Steven D'Aprano <steve@pearwood.info> wrote:
It's a concept I considered implementing, but decided to hold off on it because there are a few different design options here and I didn't have any use cases to guide the API design, nor the capacity to do usability testing to see if the additional API complexity actually made ExitStack easier to use overall. The first design option is the status quo: using multiple with statement blocks, potentially in conjunction with multiple ExitStack instances. The virtue of this approach is that it means that once a context manager is added to an ExitStack instance, that's it - its lifecycle is now coupled to that of the other context managers in the stack. You can postpone cleaning up all of them with "pop_all()" (transferring responsibility for the cleanup to a fresh ExitStack instance), but you can't selectively clean them up from the end. The usage guidelines are thus relatively simple: if you don't want to couple the lifecycles of two context managers together, then don't add them to the same ExitStack instance. However, there is also that "Stack" in the name, so it's natural for users to expect to be able to both push() *and* pop() individual context managers on the stack. The (on the surface) simplest design option to *add* to the API would be a single "pop()" operation that returned a new ExitStack instance (for return type consistency with pop_all()) that contained the last context manager pushed onto the stack. However, this API is problematic, as you've now decoupled the nesting of the context manager stack - the popped context manager may now survive beyond the closure of the original ExitStack. Since this kind a pop_all() inspired selective clean-up API would result in two different ExitStack instances anyway, the status quo seems cleaner to me than this option, as it's obvious from the start that there are seperate ExitStack instances with potentially distinct lifecycles. The next option would then be to offer a separate "exit_last_context()" method, that exited the last context manager pushed onto the stack. This would be a single-stepping counterpart to the existing close() method, that allowed you to dynamically descend and ascend the context management stack during normal operation, while still preserving the property that the entire stack will be cleaned up when encountering an exception. Assuming we went with that simpler in-place API, there would still be a number of further design questions to be answered: * Do we need to try to manage the reported exception context the way ExitStack.__exit__ does? * Does "exit_last_context()" need to accept exception details like __exit__ does? * Does "exit_last_context()" need to support the ability to suppress exceptions? * What, if anything, should the return value be? * What happens if there are no contexts on the stack to pop? * Should it become possible to query the number of registered callbacks? Taking them in order, as a matter of implementation feasibility, the answer to the first question likely needs to be "No". For consistency with calling __exit__ methods directly, the answers to the next three questions likely need to be "support the same thing __exit__ supports". For the second last question, while it's reasonable to call close(), pop_all() or __exit__() on an empty stack and have it silently do nothing, if someone has taken it on themselves to manage the stack depth manually, then it's likely more helpful to complain that the stack is empty than it is to silently do nothing. Since exit_last_context() may behave differently depending on whether or not there are items on the stack, and the number of items on the stack would be useful for diagnostic purposes, then it likely also makes sense to implement a __len__ method that delegated to "len(self._exit_callbacks)". That all suggests a possible implementation along the lines of the following: def exit_last_context(self, *exc_details): if not self._exit_callbacks: raise RuntimeError("Attempted to exit last context on empty ExitStack instance") cb = self._exit_callbacks.pop() return cb(*exc_details) def __len__(self): return len(self._exit_callbacks) What I don't know is whether or not that's actually a useful enough improvement over the status quo to justify the additional cognitive burden when learning the ExitStack API - the current API was designed around the ExitStack recipes in the documentation, which were all fairly common code patterns, but most cases where I might consider using an "exit_last_context()" method, I'd be more inclined to follow Steven's advice and use a separate context manager for the resource with an independent lifecycle. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
I would actually want a method that exits not just the last context manager, but any context manager in the stack according to my choosing. Maybe it clashes with the fact that you're using `deque`, but I'm not sure that you have a compelling reason to use `deque`. If you're asking about my use case: It's pretty boring. I have a sysadmin script with a long function that does remote actions on a few servers. I wrapped it all in an `ExitStack` since I use file-based locks and I want to ensure they get released eventually. Now, at some point I want to release the file-based lock manually, but I can't use a with statement, because there's a condition around the place where I acquire the lock. It's something like this: if condition: exit_stack.enter_context(get_lock_1()) else: exit_stack.enter_context(get_lock_2()) So ideally I would want a method that takes a context manager and just exits it. Maybe even add an optional argument `context_manager` to the existing `close` method. Personally I don't care about exception-handling in this case, and while I think it would be nice to include exception-handling, I see that the existing close method doesn't provide exception-handling either, so I wouldn't feel bad about it. So maybe something like this: def close(self, context_manager=None): """Immediately unwind the context stack""" if context_manager is None: self.__exit__(None, None, None) else: for _exit_wrapper in reversed(self._exit_callbacks): if _exit_wrapper.__self__ is context_manager: _exit_wrapper(None, None, None) self._exit_callbacks.remove(_exit_wrapper) Maybe also support accepting a tuple of context managers. On Mon, Dec 7, 2015 at 5:40 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Mon, Dec 7, 2015 at 3:37 PM Ram Rachum <ram@rachum.com> wrote:
You can't put the condition in a context manager's __init__ or __enter__? class CM: def __init__(self, condition): self.lock = lock1 if condition else lock2 def __enter__(self): self.lock.acquire() delf __exit__(self, *info): self.lock.release()
On 8 December 2015 at 06:37, Ram Rachum <ram@rachum.com> wrote:
deque itself is an implementation detail, but from a design perspective, ExitStack is intended to recreate the semantics of lexical context management using with statements, without having to actually use that layout in your code. In other words, if the exit semantics can't be expressed in terms of with statements, then I'm not interested in allowing it in ExitStack specifically (see the ExitPool discussion below for more on that qualifier). That means the semantic structures I'm open to ExitStack supporting are: * a nested context stack (which it already does) * a tree structure (which exit_last_context() would allow) The first structure corresponds to passing multiple contexts to the with statement: with cm1(), cm2(), cm3(): ... Which in turn corresponds to nested with statements: with cm1(): with cm2(): with cm3(): ... The ExitStack equivalent is: with ExitStack() as stack: stack.enter_context(cm1()) stack.enter_context(cm2()) stack.enter_context(cm3()) ... Adding an exit_last_context() method would make it possible to replicate the following kind of structure: with cm1(): with cm2(): ... with cm3(): ... Given exit_last_context(), replicating that dynamically would look like: with ExitStack() as stack: stack.enter_context(cm1()) stack.enter_context(cm2()) ... stack.exit_last_context() stack.enter_context(cm3()) ... I'm not aware of any specific use cases for the latter behaviour though, which is why that feature doesn't exist yet.
OK, thanks for the clarification. The additional details show that this is a slightly different use case from those that ExitStack is designed to support, as ExitStack aims to precisely replicate the semantics of nested with statements (as described above). That includes both the order in which the __exit__ methods get called, and which context managers can suppress exceptions from which other context managers. That's not the only way to manage cleanup logic though, and one relevant alternative is the way the atexit module works: https://docs.python.org/3/library/atexit.html In that model, the cleanup handlers are all considered peer operations, and while they're defined to be run in last-in-first-out order, the assumption is that there aren't any direct dependencies between them the way there can be with lexically nested context managers. That then makes it reasonable to offer the ability to unregister arbitrary callbacks without worrying about the potential impact on other callbacks that were registered later. While I'm not open to adding atexit style logic to ExitStack, I'm *am* amenable to the idea of adding a separate ExitPool context manager that doesn't try to replicate with statement semantics the way ExitStack does, and instead offers atexit style logic where each exit function receives the original exception state passed in to ExitPool.__exit__. One key difference from atexit would be that if any of the exit methods raised an exception, then I'd have ExitPool raise a subclass of RuntimeError (PoolExitError perhaps?) containing a list of all of the cleanup operations that failed. The API for that would probably look something like: class ExitPool: def enter_context(cm): # Call cm.__enter__ and register cm def register(cm): # Register cm.__exit__ to be called on pool exit def callback(func, *args, **kwds): # Register func to be called on pool exit def unregister(cm_or_func): # Unregister a registered CM or callback function def unregister_all(): # Empty the pool without calling anything def close(): # Empty the pool, calling all registered callbacks in LIFO order (via self.__exit__) Internally, the main data structure would be an OrderedDict instance mapping from cm's or functions to their registered callbacks (for ease of unregistration). At this point, if you're open to filing one, we should probably move further discussion over to a new RFE on the contextlib2 issue tracker: https://bitbucket.org/ncoghlan/contextlib2/ That's still pending a rebase on Python 3.5 standard library version of contextlib though... Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Thanks for the detailed reply Nick. I would like an `ExitPool` class but I'm not invested enough in this feature to champion it, so I'll let it go at this point. If you'll open the ticket and want any feedback from me about the API I'll be happy to give it, just email me. I think I can solve my personal problem with a context manager wrapper that doesn't complain when you try to close the context manager twice. Thanks, Ram. On Tue, Dec 8, 2015 at 7:56 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 9 December 2015 at 03:21, Ram Rachum <ram@rachum.com> wrote:
Unfortunately, I haven't even found the time to bring contextlib2 up to date with the current standard library version, let alone consider using it to evaluate new APIs like ExitPool :( Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (4)
-
Michael Selik -
Nick Coghlan -
Ram Rachum -
Steven D'Aprano