Hagen Fürstenau wrote:

I'm proposing to add an iterator version of "nested" to contextlib (possibly called "inested"), which takes an iterable of context managers instead of a variable number of parameters. The implementation could be taken over from the present "nested", only changing "def nested(*managers)" to "def inested(managers)".

This has the advantage that an iterator can be passed to "inested", so that each context managers is created in the context of all previous ones, which was one of the reasons for introducing the multi-with statement in the first place. "contextlib.inested" would therefore be the generalization of the multi-with statement to a variable number of managers (and "contextlib.nested" would stay deprecated).

The semantic change actually needed to make nested() more equivalent to the multi-with statement is for it to accept zero-argument callables that create context managers as arguments rather than pre-created context managers. Rather than changing the name of the function, this could be done by inspecting the first argument for an "__enter__" method. If it has one, use the old semantics (and issue a DeprecationWarning as in 3.1). Otherwise, use the proposed new semantics. However, the semantic equivalence still won't be complete as nested() currently has no way of matching the multi-with behaviour of allowing outer context managers to suppress exceptions raised by the constructors or __enter__ methods of inner context managers. Attempting to do so will result in a RuntimeError as the contextlib.contextmanager wrapper complains that the generator implementing nested() didn't yield. The further enhancement needed to address that would be to tweak nested() to raise a custom exception in that case (e.g. ContextSkipped) and provide an "allowskip" context manager that just catches and suppresses that specific exception: i.e. @contextmanager def allowskip(): try: yield except ContextSkipped: pass with allowskip(), nested(*cm_factories): # Do something I suggest putting an RFE on the tracker for this. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

Hagen Fürstenau wrote:

I guess this is much too late for 3.1, but could we then at least un-deprecate "contextlib.nested" for now? As it is, you get a DeprecationWarning for something like

with contextlib.nested(*my_managers):

without any good way to get rid of it.

I actually almost asked for that to be changed to a PendingDeprecationWarning when it was first added - Benjamin, do you mind if I downgrade this warning to a pending one post rc2? Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

2009/6/13 Nick Coghlan :

Hagen Fürstenau wrote:

...

I guess this is much too late for 3.1, but could we then at least un-deprecate "contextlib.nested" for now? As it is, you get a DeprecationWarning for something like

with contextlib.nested(*my_managers):

without any good way to get rid of it.

I actually almost asked for that to be changed to a PendingDeprecationWarning when it was first added - Benjamin, do you mind if I downgrade this warning to a pending one post rc2?

Yes, I think that's a good idea. It will also help people who have to use contextlib.nested() for backwards compatibility. -- Regards, Benjamin

2009/6/14 Raymond Hettinger :

FWIW, I think resurrecting contextlib.nested() is a bad idea. Part of the justification for the new with-statement syntax was that nested() doesn't have a way to finalize the constructors if one of them fails.   It is a pitfall for the unwary.  And now that we have the new with-statement syntax, it mostly just represents a second-way-to-do-it (a second way that has has the stated pitfall).

I don't consider changing a DeprecationWarning to a PendingDeprecationWarning "resurrecting" its target. Fully deprecating a feature in the same version that we add its replacement will just make more difficulties for cross-version libraries. ....

I suggest a PEP for 2.7 and 3.2 for building-out the with-statement to support tuples of context managers (perhaps modeled after the syntax for except-statements which allows either individual exceptions or tuples of exceptions).

I think the question of extending the syntax later is orthogonal to the issue of the DeprecationWarning. -- Regards, Benjamin

Hagen Fürstenau wrote:

...

I actually almost asked for that to be changed to a PendingDeprecationWarning when it was first added - Benjamin, do you mind if I downgrade this warning to a pending one post rc2?

I'm not sure what that would buy us. For the use case I mentioned it would be just as annoying to get a PendingDeprecationWarning. But if the warning was completely removed now, nested could still get deprecated in 3.2 as soon as some better mechanism for a variable number of managers has been provided.

Unlike a full DeprecationWarning, a PendingDeprecationWarning is ignored by default. You have to switch them on explicitly via code or a command line switch in order to see them. Pending warnings give the hint that we intend to get rid of something, but either don't have clear replacements for some legitimate use cases (as in the case of contextlib.nested) or have some other reason for not generating a warning by default (e.g. we may not have cleared all uses out of the standard library yet). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

Raymond Hettinger wrote:

I suggest a PEP for 2.7 and 3.2 for building-out the with-statement to support tuples of context managers (perhaps modeled after the syntax for except-statements which allows either individual exceptions or tuples of exceptions). The reason I suggest a PEP is that use cases need to be fully thought out and the failing constructor problem needs to be dealt with. IMO, this represents doing-it-the-right-way instead of preserving a construct that is known to be problematic. Leaving it in will enshrine it. Better to just provide our new syntax that correctly handles the common case and then wait to carefully think through how to add support for passed-in nested cm's if in-fact those turn-out to have reasonable use cases.

I agree that the current incarnation of contextlib.nested needs to go - it isn't really salvagable in its current form. However, I don't think we should generate a warning for it by default until we provide a new mechanism for handling a variable number of context managers - PendingDeprecationWarning seems a much better fit. A 2.7/3.2 PEP can then address the two main issues with the current approach: 1. The __init__ calls for the inner context managers occur outside the scope of the outer context managers. Some form of lazy evaluation would be needed to deal with that. 2. If an inner __enter__ call raises an exception that is suppressed by an outer __exit__ call then the body of with statement should be skipped rather than raising RuntimeError. This means either new syntax with semantics along the lines of the previously rejected PEP 377 or else a custom exception and a second context manager that suppresses it. Personally, I don't think new syntax for the PEP 377 semantics is warranted for the same reason that PEP 377 itself was rejected - it complicates the statement definition significantly for a really obscure corner case. Better to come up with a new and improved version of nested that eliminates (or better handles) the quirks and leave the statement definition alone. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

2009/6/15 Raymond Hettinger :

P.S.  If you switch to PendingDeprecationWarning, the example in the docs should probably be switched to show the one valid use case (passing in a prepackaged nest of context managers). Right now, the current example just shows the hazardous pattern that is much better served by the new with-statement syntax.

+1 I think this should be done anyway. -- Regards, Benjamin

2009/6/15 Terry Reedy :

Raymond Hettinger wrote:

...

P.S.  If you switch to PendingDeprecationWarning, the example in the docs should probably be switched to show the one valid use case (passing in a prepackaged nest of context managers).

It could even suggest that it only be used for this, since it may disappear, and that other uses should use the new syntax.  That would give people the best chance of writing future-proof code when they can.

And if the warning is *not* changed to a PendingDeprecationWarning, the documentation could also note the necessary warnings call needed to let the example code run warning-free. Paul.