Iterator version of contextlib.nested
contextlib.nested has recently been deprecated on grounds of being unnecessary now that the with statement accepts multiple context managers. However, as has been mentioned before (http://mail.python.org/pipermail/python-dev/2009-May/089359.html), that doesn't cover the case of a variable number of context managers, i.e. with contextlib.nested(*list_of_managers) as list_of_results: or with contexlib.nested(*iterator_of_managers): It was suggested that in these use cases a custom context manager should be implemented. However, it seems that such an implementation would be an almost exact copy of the present code for "nested". I'm proposing to add an iterator version of "nested" to contextlib (possibly called "inested"), which takes an iterable of context managers instead of a variable number of parameters. The implementation could be taken over from the present "nested", only changing "def nested(*managers)" to "def inested(managers)". This has the advantage that an iterator can be passed to "inested", so that each context managers is created in the context of all previous ones, which was one of the reasons for introducing the multi-with statement in the first place. "contextlib.inested" would therefore be the generalization of the multi-with statement to a variable number of managers (and "contextlib.nested" would stay deprecated). - Hagen
Hagen Fürstenau wrote:
I'm proposing to add an iterator version of "nested" to contextlib (possibly called "inested"), which takes an iterable of context managers instead of a variable number of parameters. The implementation could be taken over from the present "nested", only changing "def nested(*managers)" to "def inested(managers)".
This has the advantage that an iterator can be passed to "inested", so that each context managers is created in the context of all previous ones, which was one of the reasons for introducing the multi-with statement in the first place. "contextlib.inested" would therefore be the generalization of the multi-with statement to a variable number of managers (and "contextlib.nested" would stay deprecated).
The semantic change actually needed to make nested() more equivalent to the multi-with statement is for it to accept zero-argument callables that create context managers as arguments rather than pre-created context managers. Rather than changing the name of the function, this could be done by inspecting the first argument for an "__enter__" method. If it has one, use the old semantics (and issue a DeprecationWarning as in 3.1). Otherwise, use the proposed new semantics. However, the semantic equivalence still won't be complete as nested() currently has no way of matching the multi-with behaviour of allowing outer context managers to suppress exceptions raised by the constructors or __enter__ methods of inner context managers. Attempting to do so will result in a RuntimeError as the contextlib.contextmanager wrapper complains that the generator implementing nested() didn't yield. The further enhancement needed to address that would be to tweak nested() to raise a custom exception in that case (e.g. ContextSkipped) and provide an "allowskip" context manager that just catches and suppresses that specific exception: i.e. @contextmanager def allowskip(): try: yield except ContextSkipped: pass with allowskip(), nested(*cm_factories): # Do something I suggest putting an RFE on the tracker for this. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
The semantic change actually needed to make nested() more equivalent to the multi-with statement is for it to accept zero-argument callables that create context managers as arguments rather than pre-created context managers.
It seems to me that both passing callables which return managers and passing a generator which yields managers achieve about the same thing. Are you proposing the former just to avoid introducing a new interface?
Rather than changing the name of the function, this could be done by inspecting the first argument for an "__enter__" method. If it has one, use the old semantics (and issue a DeprecationWarning as in 3.1). Otherwise, use the proposed new semantics.
I guess this is much too late for 3.1, but could we then at least un-deprecate "contextlib.nested" for now? As it is, you get a DeprecationWarning for something like with contextlib.nested(*my_managers): without any good way to get rid of it. - Hagen
Hagen Fürstenau wrote:
I guess this is much too late for 3.1, but could we then at least un-deprecate "contextlib.nested" for now? As it is, you get a DeprecationWarning for something like
with contextlib.nested(*my_managers):
without any good way to get rid of it.
I actually almost asked for that to be changed to a PendingDeprecationWarning when it was first added - Benjamin, do you mind if I downgrade this warning to a pending one post rc2? Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
I actually almost asked for that to be changed to a PendingDeprecationWarning when it was first added - Benjamin, do you mind if I downgrade this warning to a pending one post rc2?
I'm not sure what that would buy us. For the use case I mentioned it would be just as annoying to get a PendingDeprecationWarning. But if the warning was completely removed now, nested could still get deprecated in 3.2 as soon as some better mechanism for a variable number of managers has been provided. - Hagen
FWIW, I think resurrecting contextlib.nested() is a bad idea. Part of the justification for the new with-statement syntax was that nested() doesn't have a way to finalize the constructors if one of them fails. It is a pitfall for the unwary. And now that we have the new with-statement syntax, it mostly just represents a second-way-to-do-it (a second way that has has the stated pitfall). The new statement was not designed to support passing in tuples of context-managers. This issue was raised while the new with-statement was being designed and it was intentionally left-out (in part, because the use cases were questionable and in-part because there were other ways to do it such as adding __enter__ and __exit__ to tuple). I suggest a PEP for 2.7 and 3.2 for building-out the with-statement to support tuples of context managers (perhaps modeled after the syntax for except-statements which allows either individual exceptions or tuples of exceptions). The reason I suggest a PEP is that use cases need to be fully thought out and the failing constructor problem needs to be dealt with. IMO, this represents doing-it-the-right-way instead of preserving a construct that is known to be problematic. Leaving it in will enshrine it. Better to just provide our new syntax that correctly handles the common case and then wait to carefully think through how to add support for passed-in nested cm's if in-fact those turn-out to have reasonable use cases. Raymond
2009/6/14 Raymond Hettinger <python@rcn.com>:
FWIW, I think resurrecting contextlib.nested() is a bad idea. Part of the justification for the new with-statement syntax was that nested() doesn't have a way to finalize the constructors if one of them fails. It is a pitfall for the unwary. And now that we have the new with-statement syntax, it mostly just represents a second-way-to-do-it (a second way that has has the stated pitfall).
I don't consider changing a DeprecationWarning to a PendingDeprecationWarning "resurrecting" its target. Fully deprecating a feature in the same version that we add its replacement will just make more difficulties for cross-version libraries. ....
I suggest a PEP for 2.7 and 3.2 for building-out the with-statement to support tuples of context managers (perhaps modeled after the syntax for except-statements which allows either individual exceptions or tuples of exceptions).
I think the question of extending the syntax later is orthogonal to the issue of the DeprecationWarning. -- Regards, Benjamin
I don't consider changing a DeprecationWarning to a PendingDeprecationWarning "resurrecting" its target.
Seems like resurrection to me. Pending warnings are hidden by default, so someone has to go look for it (and no one does this). The problem with the nested() construct is not so much that it duplicates the new with-statement; the problem is that it is a bug factory when used as advertised. The sole justification for keeping it around is that it handles an obscure use case (one that isn't even shown in its documentation or examples). I'm not opposing the idea to change the DeprecationWarning to a PendingDeprecationWarning, but I don't think we're doing the users any favors by hiding the warning message. Raymond P.S. If you switch to PendingDeprecationWarning, the example in the docs should probably be switched to show the one valid use case (passing in a prepackaged nest of context managers). Right now, the current example just shows the hazardous pattern that is much better served by the new with-statement syntax.
2009/6/15 Raymond Hettinger <python@rcn.com>:
P.S. If you switch to PendingDeprecationWarning, the example in the docs should probably be switched to show the one valid use case (passing in a prepackaged nest of context managers). Right now, the current example just shows the hazardous pattern that is much better served by the new with-statement syntax.
+1 I think this should be done anyway. -- Regards, Benjamin
Raymond Hettinger wrote:
P.S. If you switch to PendingDeprecationWarning, the example in the docs should probably be switched to show the one valid use case (passing in a prepackaged nest of context managers).
It could even suggest that it only be used for this, since it may disappear, and that other uses should use the new syntax. That would give people the best chance of writing future-proof code when they can.
2009/6/15 Terry Reedy <tjreedy@udel.edu>:
Raymond Hettinger wrote:
P.S. If you switch to PendingDeprecationWarning, the example in the docs should probably be switched to show the one valid use case (passing in a prepackaged nest of context managers).
It could even suggest that it only be used for this, since it may disappear, and that other uses should use the new syntax. That would give people the best chance of writing future-proof code when they can.
And if the warning is *not* changed to a PendingDeprecationWarning, the documentation could also note the necessary warnings call needed to let the example code run warning-free. Paul.
Paul Moore wrote:
2009/6/15 Terry Reedy <tjreedy@udel.edu>:
Raymond Hettinger wrote:
P.S. If you switch to PendingDeprecationWarning, the example in the docs should probably be switched to show the one valid use case (passing in a prepackaged nest of context managers). It could even suggest that it only be used for this, since it may disappear, and that other uses should use the new syntax. That would give people the best chance of writing future-proof code when they can.
And if the warning is *not* changed to a PendingDeprecationWarning, the documentation could also note the necessary warnings call needed to let the example code run warning-free.
I think I like that option even better than downgrading the warning. I created http://bugs.python.org/issue6288 to make it clear to Benjamin when I have finished updating the docs. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Part of the justification for the new with-statement syntax was that nested() doesn't have a way to finalize the constructors if one of them fails.
I think the problem was a little bit more subtle: nested() gets passed managers, so their __init__()s should all have run when the first context is entered. The only problem comes up when the __exit__() of an outer manager tries to suppress an exception raised by the __enter__() of an inner one. This is a limited defect in that it doesn't affect the common situation where no __exit__() tries to suppress any exceptions. (In a quick glance over the std library I couldn't find a single instance of an exception-suppressing __exit__().).
And now that we have the new with-statement syntax, it mostly just represents a second-way-to-do-it (a second way that has has the stated pitfall).
So the functionalities of nested() and multi-with overlap in the common use cases, and each has its own limitation in an uncommon one. I agree that this situation is unfortunate, but I think introducing support for one uncommon case and removing it for another is not the way to go in 3.1. That's why I think nested() should stay un-deprecated until there is a replacement which handles a superset of its use cases.
The new statement was not designed to support passing in tuples of context-managers. This issue was raised while the new with-statement was being designed and it was intentionally left-out (in part, because the use cases were questionable
FWIW, my use case (which made me notice the DeprecationWarning in the first place) is in a command dispatch function, which looks at the command to be executed and pre-processes its arguments in a uniform way. Part of that pre-processing is entering contexts of context manager before passing them along (and exiting them when the command finishes or raises an exception).
and in-part because there were other ways to do it such as adding __enter__ and __exit__ to tuple).
Which has not been done for 3.1. Granted, you could subclass tuple and add them yourself, but then you would mostly be copying what's already implemented in nested().
I suggest a PEP for 2.7 and 3.2 for building-out the with-statement to support tuples of context managers
That sounds like a good idea.
IMO, this represents doing-it-the-right-way instead of preserving a construct that is known to be problematic. Leaving it in will enshrine it.
I don't see the problem with deprecating it only after a completely suitable replacement is found. Why would it be any harder to deprecate nested() in 3.2? - Hagen
Raymond Hettinger wrote:
I suggest a PEP for 2.7 and 3.2 for building-out the with-statement to support tuples of context managers (perhaps modeled after the syntax for except-statements which allows either individual exceptions or tuples of exceptions). The reason I suggest a PEP is that use cases need to be fully thought out and the failing constructor problem needs to be dealt with. IMO, this represents doing-it-the-right-way instead of preserving a construct that is known to be problematic. Leaving it in will enshrine it. Better to just provide our new syntax that correctly handles the common case and then wait to carefully think through how to add support for passed-in nested cm's if in-fact those turn-out to have reasonable use cases.
I agree that the current incarnation of contextlib.nested needs to go - it isn't really salvagable in its current form. However, I don't think we should generate a warning for it by default until we provide a new mechanism for handling a variable number of context managers - PendingDeprecationWarning seems a much better fit. A 2.7/3.2 PEP can then address the two main issues with the current approach: 1. The __init__ calls for the inner context managers occur outside the scope of the outer context managers. Some form of lazy evaluation would be needed to deal with that. 2. If an inner __enter__ call raises an exception that is suppressed by an outer __exit__ call then the body of with statement should be skipped rather than raising RuntimeError. This means either new syntax with semantics along the lines of the previously rejected PEP 377 or else a custom exception and a second context manager that suppresses it. Personally, I don't think new syntax for the PEP 377 semantics is warranted for the same reason that PEP 377 itself was rejected - it complicates the statement definition significantly for a really obscure corner case. Better to come up with a new and improved version of nested that eliminates (or better handles) the quirks and leave the statement definition alone. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Hagen Fürstenau wrote:
I actually almost asked for that to be changed to a PendingDeprecationWarning when it was first added - Benjamin, do you mind if I downgrade this warning to a pending one post rc2?
I'm not sure what that would buy us. For the use case I mentioned it would be just as annoying to get a PendingDeprecationWarning. But if the warning was completely removed now, nested could still get deprecated in 3.2 as soon as some better mechanism for a variable number of managers has been provided.
Unlike a full DeprecationWarning, a PendingDeprecationWarning is ignored by default. You have to switch them on explicitly via code or a command line switch in order to see them. Pending warnings give the hint that we intend to get rid of something, but either don't have clear replacements for some legitimate use cases (as in the case of contextlib.nested) or have some other reason for not generating a warning by default (e.g. we may not have cleared all uses out of the standard library yet). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Unlike a full DeprecationWarning, a PendingDeprecationWarning is ignored by default. You have to switch them on explicitly via code or a command line switch in order to see them.
Sorry, I should have made myself more familiar with the warnings mechanism before writing. In that case I'm fine with a PendingDeprecationWarning. :-) - Hagen
2009/6/13 Nick Coghlan <ncoghlan@gmail.com>:
Hagen Fürstenau wrote:
I guess this is much too late for 3.1, but could we then at least un-deprecate "contextlib.nested" for now? As it is, you get a DeprecationWarning for something like
with contextlib.nested(*my_managers):
without any good way to get rid of it.
I actually almost asked for that to be changed to a PendingDeprecationWarning when it was first added - Benjamin, do you mind if I downgrade this warning to a pending one post rc2?
Yes, I think that's a good idea. It will also help people who have to use contextlib.nested() for backwards compatibility. -- Regards, Benjamin
participants (6)
-
Benjamin Peterson -
Hagen Fürstenau -
Nick Coghlan -
Paul Moore -
Raymond Hettinger -
Terry Reedy