Allow __enter__() methods to skip the with statement body?
An interesting discrepancy [1] has been noted when comparing contextlib.nested (and contextlib.contextmanager) with the equivalent nested with statements. Specifically, the following examples behave differently if cmB().__enter__() raises an exception which cmA().__exit__() then handles (and suppresses): with cmA(): with cmB(): do_stuff() # This will resume here without executing "Do stuff" @contextlib.contextmanager def combined(): with cmA(): with cmB(): yield with combined(): do_stuff() # This will raise RuntimeError complaining that the underlying # generator didn't yield with contextlib.nested(cmA(), cmB()): do_stuff() # This will raise the same RuntimeError as the contextmanager # example (unsurprising, given the way nested() is implemented) The problem arises any time it is possible to skip over the yield statement in a contextlib.contextmanager based context manager without raising an exception that can be seen by the code calling __enter__(). I think the right way to fix this (as suggested by the original poster of the bug report) is to introduce a new flow control exception along the lines of GeneratorExit (e.g. SkipContext) and tweak the expansion of the with statement [2] to skip the body of the statement if __enter__() throws that specific exception: mgr = (EXPR) exit = mgr.__exit__ # Not calling it yet try: value = mgr.__enter__() except SkipContext: pass # This exception handler is the new part... else: exc = True try: VAR = value # Only if "as VAR" is present BLOCK except: # The exceptional case is handled here exc = False if not exit(*sys.exc_info()): raise # The exception is swallowed if exit() returns true finally: # The normal and non-local-goto cases are handled here if exc: exit(None, None, None) Naturally, contextlib.contextmanager would then be modified to raise SkipContext instead of RuntimeError if the generator doesn't yield. The latter two examples would then correctly resume execution at the first statement after the with block. I don't see any other way to comprehensively fix the problem - without it, there will always be some snippets of code which cannot correctly be converted into context managers, and those snippets won't always be obvious (e.g. the fact that combined() is potentially a broken context manager implementation would surprise most people - it certainly surprised me). Thoughts? Do people hate the idea? Are there any backwards compatibility problems that I'm missing? Should I write a PEP or just add the feature to the with statement in 2.7/3.1? Cheers, Nick. [1] http://bugs.python.org/issue5251 [2] http://www.python.org/dev/peps/pep-0343/ -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
On Wed, Feb 25, 2009 at 04:24, Nick Coghlan
An interesting discrepancy [1] has been noted when comparing contextlib.nested (and contextlib.contextmanager) with the equivalent nested with statements.
Specifically, the following examples behave differently if cmB().__enter__() raises an exception which cmA().__exit__() then handles (and suppresses):
with cmA(): with cmB(): do_stuff() # This will resume here without executing "Do stuff"
@contextlib.contextmanager def combined(): with cmA(): with cmB(): yield
with combined(): do_stuff() # This will raise RuntimeError complaining that the underlying # generator didn't yield
with contextlib.nested(cmA(), cmB()): do_stuff() # This will raise the same RuntimeError as the contextmanager # example (unsurprising, given the way nested() is implemented)
The problem arises any time it is possible to skip over the yield statement in a contextlib.contextmanager based context manager without raising an exception that can be seen by the code calling __enter__().
I think the right way to fix this (as suggested by the original poster of the bug report) is to introduce a new flow control exception along the lines of GeneratorExit (e.g. SkipContext) and tweak the expansion of the with statement [2] to skip the body of the statement if __enter__() throws that specific exception:
mgr = (EXPR) exit = mgr.__exit__ # Not calling it yet try: value = mgr.__enter__() except SkipContext: pass # This exception handler is the new part... else: exc = True try: VAR = value # Only if "as VAR" is present BLOCK except: # The exceptional case is handled here exc = False if not exit(*sys.exc_info()): raise # The exception is swallowed if exit() returns true finally: # The normal and non-local-goto cases are handled here if exc: exit(None, None, None)
Naturally, contextlib.contextmanager would then be modified to raise SkipContext instead of RuntimeError if the generator doesn't yield. The latter two examples would then correctly resume execution at the first statement after the with block.
I don't see any other way to comprehensively fix the problem - without it, there will always be some snippets of code which cannot correctly be converted into context managers, and those snippets won't always be obvious (e.g. the fact that combined() is potentially a broken context manager implementation would surprise most people - it certainly surprised me).
Thoughts? Do people hate the idea?
No, but I do wonder how useful this truly is.
Are there any backwards compatibility problems that I'm missing?
As long as the exception inherits from BaseException, no.
Should I write a PEP or just add the feature to the with statement in 2.7/3.1?
Sounds PEPpy to me since you are proposing changing the semantics for a syntactic construct.
On Wed, Feb 25, 2009 at 4:24 AM, Nick Coghlan
An interesting discrepancy [1] has been noted when comparing contextlib.nested (and contextlib.contextmanager) with the equivalent nested with statements.
Specifically, the following examples behave differently if cmB().__enter__() raises an exception which cmA().__exit__() then handles (and suppresses):
with cmA(): with cmB(): do_stuff() # This will resume here without executing "Do stuff"
@contextlib.contextmanager def combined(): with cmA(): with cmB(): yield
with combined(): do_stuff() # This will raise RuntimeError complaining that the underlying # generator didn't yield
with contextlib.nested(cmA(), cmB()): do_stuff() # This will raise the same RuntimeError as the contextmanager # example (unsurprising, given the way nested() is implemented)
The problem arises any time it is possible to skip over the yield statement in a contextlib.contextmanager based context manager without raising an exception that can be seen by the code calling __enter__().
If the problem is just the yield, can't this just be fixed by implementing contextlib.nested() as a class rather than as a @contextmanager decorated generator? Or is this a problem with class based context managers too? Steve -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy
Steven Bethard wrote:
If the problem is just the yield, can't this just be fixed by implementing contextlib.nested() as a class rather than as a @contextmanager decorated generator? Or is this a problem with class based context managers too?
It's a problem for class-based context managers as well. Setting aside the difficulties of actually maintaining nested()'s state on a class rather than in a frame (it's definitely possible, but also somewhat painful), you still end up in the situation where nested() knows that cmB().__enter__() threw an exception that was then handled by cmA().__exit__() and hence the body of the with statement should be skipped but no exception should occur from the point of view of the surrounding code. However, indicating that is not currently an option available to nested().__enter__(): it can either raise an exception (thus skipping the body of the with statement, but also propagating the exception into the surrounding code), or it can return a value (which would lead to the execution of the body of the with statement). Returning a value would definitely be wrong, but raising the exception isn't really right either. contextmanager is just a special case - the "skipped yield" inside the generator reflects the body of the with statement being skipped in the original non-context manager code. As to Brett's question of whether or not this is necessary/useful... the problem I really have with the status quo is that it is currently impossible to look at the following code snippets and say whether or not the created CM's are valid: cm = contextlib.nested(cmA(), cmB()) @contextlib.contextmanager def cm(): with cmA(): with cmB(): yield # Not tested, probably have the class version wrong # This should illustrate why nested() wasn't written # as a class-based CM though - this one only nests # two specifically named CMs and look how tricky it gets! class CM(object): def __init__(self): self.cmA = None self.cmB = None def __enter__(self): if self.cmA is not None: raise RuntimeError("Can't re-use this CM") self.cmA = cmA() self.cmA.__enter__() try: self.cmB = cmB() self.cmB.__enter__() except: self.cmA.__exit__(*sys.exc_info()) # Can't suppress in __enter__(), so must raise raise def __exit__(self, *args): suppress = False try: if self.cmB is not None: suppress = self.cmB.__exit__(*args) except: suppress = self.cmA.__exit__(*sys.exc_info()): if not suppress: # Exception has changed, so reraise explicitly raise else: if suppress: # cmB already suppressed the exception, # so don't pass it to cmA suppress = self.cmA.__exit__(None, None, None): else: suppress = self.cmA.__exit__(*args): return suppress With the current with statement semantics, those CM's may raise exceptions where the original multiple with statement code would work fine. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
participants (3)
-
Brett Cannon -
Nick Coghlan -
Steven Bethard