PEP 377 - allow __enter__() methods to skip the statement body
PEP 377 is a proposal to allow context manager __enter__() methods to skip the body of the with statement by raising a specific (new) flow control exception.
Since there is a working reference implementation now, I thought it was time to open it up for broader discussion.
Full PEP attached, or you can find it in the usual place at http://www.python.org/dev/peps/pep-0377
Cheers, Nick.
P.S. I expect a rationale for the StatementSkipped value binding is probably going to be pretty high on the list of questions that aren't currently covered by the PEP. I hope to write more on that some time this week.
Without knowing what StatementSkipped is (just some singleton? If so why not just used SkipStatement instance that was raised?) and wondering if we are just going to continue to adding control flow exceptions that directly inherit from BaseException or some ControlFlowException base class, the basic idea seems fine by me.
On Sun, Mar 15, 2009 at 05:56, Nick Coghlan ncoghlan@gmail.com wrote:
PEP 377 is a proposal to allow context manager __enter__() methods to skip the body of the with statement by raising a specific (new) flow control exception.
Since there is a working reference implementation now, I thought it was time to open it up for broader discussion.
Full PEP attached, or you can find it in the usual place at http://www.python.org/dev/peps/pep-0377
Cheers, Nick.
P.S. I expect a rationale for the StatementSkipped value binding is probably going to be pretty high on the list of questions that aren't currently covered by the PEP. I hope to write more on that some time this week.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
PEP: 377 Title: Allow __enter__() methods to skip the statement body Version: $Revision: 70384 $ Last-Modified: $Date: 2009-03-15 22:48:49 +1000 (Sun, 15 Mar 2009) $ Author: Nick Coghlan ncoghlan@gmail.com Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 8-Mar-2009 Python-Version: 2.7, 3.1 Post-History: 8-Mar-2009
Abstract
This PEP proposes a backwards compatible mechanism that allows ``__enter__()`` methods to skip the body of the associated ``with`` statement. The lack of this ability currently means the ``contextlib.contextmanager`` decorator is unable to fulfil its specification of being able to turn arbitrary code into a context manager by moving it into a generator function with a yield in the appropriate location. One symptom of this is that ``contextlib.nested`` will currently raise ``RuntimeError`` in situations where writing out the corresponding nested ``with`` statements would not [1].
The proposed change is to introduce a new flow control exception ``SkipStatement``, and skip the execution of the ``with`` statement body if ``__enter__()`` raises this exception.
Proposed Change
The semantics of the ``with`` statement will be changed to include a new ``try``/``except``/``else`` block around the call to ``__enter__()``. If ``SkipStatement`` is raised by the ``__enter__()`` method, then the main section of the ``with`` statement (now located in the ``else`` clause) will not be executed. To avoid leaving the names in any ``as`` clause unbound in this case, a new ``StatementSkipped`` singleton (similar to the existing ``NotImplemented`` singleton) will be assigned to all names that appear in the ``as`` clause.
The components of the ``with`` statement remain as described in PEP 343 [2]::
with EXPR as VAR: BLOCK
After the modification, the ``with`` statement semantics would be as follows::
mgr = (EXPR) exit = mgr.__exit__ # Not calling it yet try: value = mgr.__enter__() except SkipStatement: VAR = StatementSkipped # Only if "as VAR" is present and # VAR is a single name # If VAR is a tuple of names, then StatementSkipped # will be assigned to each name in the tuple else: exc = True try: try: VAR = value # Only if "as VAR" is present BLOCK except: # The exceptional case is handled here exc = False if not exit(*sys.exc_info()): raise # The exception is swallowed if exit() returns true finally: # The normal and non-local-goto cases are handled here if exc: exit(None, None, None)
With the above change in place for the ``with`` statement semantics, ``contextlib.contextmanager()`` will then be modified to raise ``SkipStatement`` instead of ``RuntimeError`` when the underlying generator doesn't yield.
Rationale for Change
Currently, some apparently innocuous context managers may raise ``RuntimeError`` when executed. This occurs when the context manager's ``__enter__()`` method encounters a situation where the written out version of the code corresponding to the context manager would skip the code that is now the body of the ``with`` statement. Since the ``__enter__()`` method has no mechanism available to signal this to the interpreter, it is instead forced to raise an exception that not only skips the body of the ``with`` statement, but also jumps over all code until the nearest exception handler. This goes against one of the design goals of the ``with`` statement, which was to be able to factor out arbitrary common exception handling code into a single context manager by putting into a generator function and replacing the variant part of the code with a ``yield`` statement.
Specifically, the following examples behave differently if ``cmB().__enter__()`` raises an exception which ``cmA().__exit__()`` then handles and suppresses::
with cmA(): with cmB(): do_stuff() # This will resume here without executing "do_stuff()"
@contextlib.contextmanager def combined(): with cmA(): with cmB(): yield
with combined(): do_stuff() # This will raise a RuntimeError complaining that the context # manager's underlying generator didn't yield
with contextlib.nested(cmA(), cmB()): do_stuff() # This will raise the same RuntimeError as the contextmanager() # example (unsurprising, given that the nested() implementation # uses contextmanager())
# The following class based version shows that the issue isn't # specific to contextlib.contextmanager() (it also shows how # much simpler it is to write context managers as generators # instead of as classes!) class CM(object): def __init__(self): self.cmA = None self.cmB = None
def __enter__(self): if self.cmA is not None: raise RuntimeError("Can't re-use this CM") self.cmA = cmA() self.cmA.__enter__() try: self.cmB = cmB() self.cmB.__enter__() except: self.cmA.__exit__(*sys.exc_info()) # Can't suppress in __enter__(), so must raise raise
def __exit__(self, *args): suppress = False try: if self.cmB is not None: suppress = self.cmB.__exit__(*args) except: suppress = self.cmA.__exit__(*sys.exc_info()): if not suppress: # Exception has changed, so reraise explicitly raise else: if suppress: # cmB already suppressed the exception, # so don't pass it to cmA suppress = self.cmA.__exit__(None, None, None): else: suppress = self.cmA.__exit__(*args): return suppress
With the proposed semantic change in place, the contextlib based examples above would then "just work", but the class based version would need a small adjustment to take advantage of the new semantics::
class CM(object): def __init__(self): self.cmA = None self.cmB = None
def __enter__(self): if self.cmA is not None: raise RuntimeError("Can't re-use this CM") self.cmA = cmA() self.cmA.__enter__() try: self.cmB = cmB() self.cmB.__enter__() except: if self.cmA.__exit__(*sys.exc_info()): # Suppress the exception, but don't run # the body of the with statement either raise SkipStatement raise
def __exit__(self, *args): suppress = False try: if self.cmB is not None: suppress = self.cmB.__exit__(*args) except: suppress = self.cmA.__exit__(*sys.exc_info()): if not suppress: # Exception has changed, so reraise explicitly raise else: if suppress: # cmB already suppressed the exception, # so don't pass it to cmA suppress = self.cmA.__exit__(None, None, None): else: suppress = self.cmA.__exit__(*args): return suppress
There is currently a tentative suggestion [3] to add import-style syntax to the ``with`` statement to allow multiple context managers to be included in a single ``with`` statement without needing to use ``contextlib.nested``. In that case the compiler has the option of simply emitting multiple ``with`` statements at the AST level, thus allowing the semantics of actual nested ``with`` statements to be reproduced accurately. However, such a change would highlight rather than alleviate the problem the current PEP aims to address: it would not be possible to use ``contextlib.contextmanager`` to reliably factor out such ``with`` statements, as they would exhibit exactly the same semantic differences as are seen with the ``combined()`` context manager in the above example.
Performance Impact
Implementing the new semantics makes it necessary to store the references to the ``__enter__`` and ``__exit__`` methods in temporary variables instead of on the stack. This results in a slight regression in ``with`` statement speed relative to Python 2.6/3.1. However, implementing a custom ``SETUP_WITH`` opcode would negate any differences between the two approaches (as well as dramatically improving speed by eliminating more than a dozen unnecessary trips around the eval loop).
Reference Implementation
Patch attached to Issue 5251 [1]. That patch uses only existing opcodes (i.e. no ``SETUP_WITH``).
Acknowledgements
James William Pye both raised the issue and suggested the basic outline of the solution described in this PEP.
References
.. [1] Issue 5251: contextlib.nested inconsistent with nested with statements (http://bugs.python.org/issue5251)
.. [2] PEP 343: The "with" Statement (http://www.python.org/dev/peps/pep-0343/)
.. [3] Import-style syntax to reduce indentation of nested with statements (http://mail.python.org/pipermail/python-ideas/2009-March/003188.html)
Copyright
This document has been placed in the public domain.
.. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End:
Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org
Brett Cannon wrote:
Without knowing what StatementSkipped is (just some singleton? If so why not just used SkipStatement instance that was raised?) and wondering if we are just going to continue to adding control flow exceptions that directly inherit from BaseException or some ControlFlowException base class, the basic idea seems fine by me.
Note that using exceptions for control flow can be bad for other implementations of Python. For example exceptions on the .NET framework are very expensive. (Although there are workarounds such as not really raising the exception - but they're ugly).
Isn't it better practise for exceptions to be used for exceptional circumstances rather than for control flow?
Michael
On Sun, Mar 15, 2009 at 05:56, Nick Coghlan <ncoghlan@gmail.com mailto:ncoghlan@gmail.com> wrote:
PEP 377 is a proposal to allow context manager __enter__() methods to skip the body of the with statement by raising a specific (new) flow control exception. Since there is a working reference implementation now, I thought it was time to open it up for broader discussion. Full PEP attached, or you can find it in the usual place at http://www.python.org/dev/peps/pep-0377 Cheers, Nick. P.S. I expect a rationale for the StatementSkipped value binding is probably going to be pretty high on the list of questions that aren't currently covered by the PEP. I hope to write more on that some time this week. -- Nick Coghlan | ncoghlan@gmail.com <mailto:ncoghlan@gmail.com> | Brisbane, Australia --------------------------------------------------------------- PEP: 377 Title: Allow __enter__() methods to skip the statement body Version: $Revision: 70384 $ Last-Modified: $Date: 2009-03-15 22:48:49 +1000 (Sun, 15 Mar 2009) $ Author: Nick Coghlan <ncoghlan@gmail.com <mailto:ncoghlan@gmail.com>> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 8-Mar-2009 Python-Version: 2.7, 3.1 Post-History: 8-Mar-2009 Abstract ======== This PEP proposes a backwards compatible mechanism that allows ``__enter__()`` methods to skip the body of the associated ``with`` statement. The lack of this ability currently means the ``contextlib.contextmanager`` decorator is unable to fulfil its specification of being able to turn arbitrary code into a context manager by moving it into a generator function with a yield in the appropriate location. One symptom of this is that ``contextlib.nested`` will currently raise ``RuntimeError`` in situations where writing out the corresponding nested ``with`` statements would not [1]. The proposed change is to introduce a new flow control exception ``SkipStatement``, and skip the execution of the ``with`` statement body if ``__enter__()`` raises this exception. Proposed Change =============== The semantics of the ``with`` statement will be changed to include a new ``try``/``except``/``else`` block around the call to ``__enter__()``. If ``SkipStatement`` is raised by the ``__enter__()`` method, then the main section of the ``with`` statement (now located in the ``else`` clause) will not be executed. To avoid leaving the names in any ``as`` clause unbound in this case, a new ``StatementSkipped`` singleton (similar to the existing ``NotImplemented`` singleton) will be assigned to all names that appear in the ``as`` clause. The components of the ``with`` statement remain as described in PEP 343 [2]:: with EXPR as VAR: BLOCK After the modification, the ``with`` statement semantics would be as follows:: mgr = (EXPR) exit = mgr.__exit__ # Not calling it yet try: value = mgr.__enter__() except SkipStatement: VAR = StatementSkipped # Only if "as VAR" is present and # VAR is a single name # If VAR is a tuple of names, then StatementSkipped # will be assigned to each name in the tuple else: exc = True try: try: VAR = value # Only if "as VAR" is present BLOCK except: # The exceptional case is handled here exc = False if not exit(*sys.exc_info()): raise # The exception is swallowed if exit() returns true finally: # The normal and non-local-goto cases are handled here if exc: exit(None, None, None) With the above change in place for the ``with`` statement semantics, ``contextlib.contextmanager()`` will then be modified to raise ``SkipStatement`` instead of ``RuntimeError`` when the underlying generator doesn't yield. Rationale for Change ==================== Currently, some apparently innocuous context managers may raise ``RuntimeError`` when executed. This occurs when the context manager's ``__enter__()`` method encounters a situation where the written out version of the code corresponding to the context manager would skip the code that is now the body of the ``with`` statement. Since the ``__enter__()`` method has no mechanism available to signal this to the interpreter, it is instead forced to raise an exception that not only skips the body of the ``with`` statement, but also jumps over all code until the nearest exception handler. This goes against one of the design goals of the ``with`` statement, which was to be able to factor out arbitrary common exception handling code into a single context manager by putting into a generator function and replacing the variant part of the code with a ``yield`` statement. Specifically, the following examples behave differently if ``cmB().__enter__()`` raises an exception which ``cmA().__exit__()`` then handles and suppresses:: with cmA(): with cmB(): do_stuff() # This will resume here without executing "do_stuff()" @contextlib.contextmanager def combined(): with cmA(): with cmB(): yield with combined(): do_stuff() # This will raise a RuntimeError complaining that the context # manager's underlying generator didn't yield with contextlib.nested(cmA(), cmB()): do_stuff() # This will raise the same RuntimeError as the contextmanager() # example (unsurprising, given that the nested() implementation # uses contextmanager()) # The following class based version shows that the issue isn't # specific to contextlib.contextmanager() (it also shows how # much simpler it is to write context managers as generators # instead of as classes!) class CM(object): def __init__(self): self.cmA = None self.cmB = None def __enter__(self): if self.cmA is not None: raise RuntimeError("Can't re-use this CM") self.cmA = cmA() self.cmA.__enter__() try: self.cmB = cmB() self.cmB.__enter__() except: self.cmA.__exit__(*sys.exc_info()) # Can't suppress in __enter__(), so must raise raise def __exit__(self, *args): suppress = False try: if self.cmB is not None: suppress = self.cmB.__exit__(*args) except: suppress = self.cmA.__exit__(*sys.exc_info()): if not suppress: # Exception has changed, so reraise explicitly raise else: if suppress: # cmB already suppressed the exception, # so don't pass it to cmA suppress = self.cmA.__exit__(None, None, None): else: suppress = self.cmA.__exit__(*args): return suppress With the proposed semantic change in place, the contextlib based examples above would then "just work", but the class based version would need a small adjustment to take advantage of the new semantics:: class CM(object): def __init__(self): self.cmA = None self.cmB = None def __enter__(self): if self.cmA is not None: raise RuntimeError("Can't re-use this CM") self.cmA = cmA() self.cmA.__enter__() try: self.cmB = cmB() self.cmB.__enter__() except: if self.cmA.__exit__(*sys.exc_info()): # Suppress the exception, but don't run # the body of the with statement either raise SkipStatement raise def __exit__(self, *args): suppress = False try: if self.cmB is not None: suppress = self.cmB.__exit__(*args) except: suppress = self.cmA.__exit__(*sys.exc_info()): if not suppress: # Exception has changed, so reraise explicitly raise else: if suppress: # cmB already suppressed the exception, # so don't pass it to cmA suppress = self.cmA.__exit__(None, None, None): else: suppress = self.cmA.__exit__(*args): return suppress There is currently a tentative suggestion [3] to add import-style syntax to the ``with`` statement to allow multiple context managers to be included in a single ``with`` statement without needing to use ``contextlib.nested``. In that case the compiler has the option of simply emitting multiple ``with`` statements at the AST level, thus allowing the semantics of actual nested ``with`` statements to be reproduced accurately. However, such a change would highlight rather than alleviate the problem the current PEP aims to address: it would not be possible to use ``contextlib.contextmanager`` to reliably factor out such ``with`` statements, as they would exhibit exactly the same semantic differences as are seen with the ``combined()`` context manager in the above example. Performance Impact ================== Implementing the new semantics makes it necessary to store the references to the ``__enter__`` and ``__exit__`` methods in temporary variables instead of on the stack. This results in a slight regression in ``with`` statement speed relative to Python 2.6/3.1. However, implementing a custom ``SETUP_WITH`` opcode would negate any differences between the two approaches (as well as dramatically improving speed by eliminating more than a dozen unnecessary trips around the eval loop). Reference Implementation ======================== Patch attached to Issue 5251 [1]. That patch uses only existing opcodes (i.e. no ``SETUP_WITH``). Acknowledgements ================ James William Pye both raised the issue and suggested the basic outline of the solution described in this PEP. References ========== .. [1] Issue 5251: contextlib.nested inconsistent with nested with statements (http://bugs.python.org/issue5251) .. [2] PEP 343: The "with" Statement (http://www.python.org/dev/peps/pep-0343/) .. [3] Import-style syntax to reduce indentation of nested with statements (http://mail.python.org/pipermail/python-ideas/2009-March/003188.html) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: _______________________________________________ Python-Dev mailing list Python-Dev@python.org <mailto:Python-Dev@python.org> http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org
Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...
On Sun, Mar 15, 2009 at 10:50 AM, Michael Foord fuzzyman@voidspace.org.uk wrote:
Brett Cannon wrote:
Without knowing what StatementSkipped is (just some singleton? If so why not just used SkipStatement instance that was raised?) and wondering if we are just going to continue to adding control flow exceptions that directly inherit from BaseException or some ControlFlowException base class, the basic idea seems fine by me.
Note that using exceptions for control flow can be bad for other implementations of Python. For example exceptions on the .NET framework are very expensive. (Although there are workarounds such as not really raising the exception - but they're ugly).
Isn't it better practise for exceptions to be used for exceptional circumstances rather than for control flow?
If my understanding is correct, the primary use case for this is when an exception is raised by an __enter__() method and caught by an enclosing __exit__() method. So at least in that case, you've already incurred the cost of an exception.
It might be nice to see an example of this being used with only a single context manager. Is that possible?
Steve
Note that using exceptions for control flow can be bad for other implementations of Python. For example exceptions on the .NET framework are very expensive.
Why do you say that? What specific implementation of .NET are you referring to? What do you mean by "very"?
Isn't it better practise for exceptions to be used for exceptional circumstances rather than for control flow?
This is an ongoing debate (in Python, and outside). I'm in the camp that says that exceptions are a control flow mechanism just like loops, conditionals, and recursion. With exceptions, you get essentially multiple alternative outcomes of a function call, rather than just a single result. In principle, it would be possible to eliminate the return statement altogether, but it is useful syntactic sugar.
Regards, Martin
Martin v. Löwis wrote:
Note that using exceptions for control flow can be bad for other implementations of Python. For example exceptions on the .NET framework are very expensive.
Why do you say that? What specific implementation of .NET are you referring to? What do you mean by "very"?
I'm talking about IronPython on the Microsoft .NET framework - although it is likely that the same is true of IronPython on Mono.
On the .NET framework the setup for exception handling is virtually free until an exception is raised. Once an exception is raised it takes a lot longer (expensive in time). This means that in IronPython exception handling code (try... except and try... finally blocks) are much faster than CPython if no exception is raised - but much more expensive if an exception is raised.
You can see this in a comparison of IronPython 2 and Python 2.5 running PyBench:
http://ironpython.codeplex.com/Wiki/View.aspx?title=IP201VsCPy25Perf
TryExcept: 26ms 888ms -97.1% 63ms 890ms -92.9% TryRaiseExcept: 58234ms 1286ms +4428.6% 58474ms 1298ms +4404.6%
Isn't it better practise for exceptions to be used for exceptional circumstances rather than for control flow?
This is an ongoing debate (in Python, and outside). I'm in the camp that says that exceptions are a control flow mechanism just like loops, conditionals, and recursion. With exceptions, you get essentially multiple alternative outcomes of a function call, rather than just a single result. In principle, it would be possible to eliminate the return statement altogether, but it is useful syntactic sugar.
Using exceptions for control flow is akin to goto. Sometimes useful but a dubious practise. :-)
Michael
Regards, Martin
I'm talking about IronPython on the Microsoft .NET framework - although it is likely that the same is true of IronPython on Mono.
I see. It would be interesting to find out why this is so much slower - I cannot believe that it is inherent in the commercial .NET framework, but rather expect that it is some issue in IronPython (*). Also, the test case measured doesn't entirely reflect what is proposed, since it catches the exception in the same function - for a realistic comparison, the raise should occur in a function call (so to measure the overhead of stack unwinding also).
Regards, Martin
(*) My wild guess is that IronPython feels obliged to provide traceback objects, and that this a costly operation - I just can't believe that exceptions are themselves costly on .NET, in the Microsoft implementation. In the specific case, it would be possible to suppress traceback generation.
Martin v. Löwis wrote:
I'm talking about IronPython on the Microsoft .NET framework - although it is likely that the same is true of IronPython on Mono.
I see. It would be interesting to find out why this is so much slower - I cannot believe that it is inherent in the commercial .NET framework, but rather expect that it is some issue in IronPython (*). Also, the test case measured doesn't entirely reflect what is proposed, since it catches the exception in the same function - for a realistic comparison, the raise should occur in a function call (so to measure the overhead of stack unwinding also).
Regards, Martin
(*) My wild guess is that IronPython feels obliged to provide traceback objects, and that this a costly operation - I just can't believe that exceptions are themselves costly on .NET, in the Microsoft implementation. In the specific case, it would be possible to suppress traceback generation.
I have discussed this issue with the IronPython team several times. They say that it is a deliberate design decision in .NET - to minimize the cost of exception handling code in the absence of exceptions at the expense of slower performance when exceptions are raised.
Googling for ".NET exceptions performance" would seem to confirm that.
Apparently this article on the managed exception model was written by one of the core developers:
http://blogs.msdn.com/cbrumme/archive/2003/10/01/51524.aspx
"This is light years away from returning a -1 from your function call. Exceptions are inherently non-local, and if there’s an obvious and enduring trend for today’s architectures, it’s that you must remain local for good performance.
Relative to straight-line local execution, exception performance will keep getting worse. Sure, we might dig into our current behavior and speed it up a little. But the trend will relentlessly make exceptions perform worse.
How do I reconcile the trend to worse performance with our recommendation that managed code should use exceptions to communicate errors? By ensuring that error cases are exceedingly rare. We used to say that exceptions should be used for exceptional cases, but folks pushed back on that as tautological."
Michael
Michael Foord wrote:
Brett Cannon wrote:
Without knowing what StatementSkipped is (just some singleton? If so why not just used SkipStatement instance that was raised?)
It does get described in the full PEP - it is indeed just a singleton like NotImplemented. That whole aspect of the PEP is something I'm not entirely sure of at this stage - it may make more sense to just leave the variables unbound. If a particular program cares to tell whether or not the statement was skipped, that would still be easy enough to do:
x = skipped = object() with cm() as x: do_stuff()
if x is skipped: print "CM aborted with statement!"
Actually, now that I see how easy it is to do something equivalent for yourself, I'm definitely going to drop the StatementSkipped assignment part of the PEP in favour of just skipping over the variable assignment part as well.
and wondering if we are just going to continue to adding control flow exceptions that directly inherit from BaseException or some ControlFlowException base class, the basic idea seems fine by me.
Given how different the control flow exceptions are from each other, I don't think it really makes sense to ever treat them as a cohesive group. There's also the fact that StopIteration is a control flow exception that doesn't inherit directly from BaseException.
Note that using exceptions for control flow can be bad for other implementations of Python. For example exceptions on the .NET framework are very expensive. (Although there are workarounds such as not really raising the exception - but they're ugly).
Is it that exceptions are expensive, or setting up a try/except block is expensive? The reason the SkipStatement idea is tenable at all (even in CPython) is that try/except is fairly cheap when no exception is raised.
(In this specific case, my initial patch does slow things down a bit, since one side effect of the extra try/except block is to disallow a couple of stack based optimisations that are used in the current CPython implementation of the with statement)
Isn't it better practise for exceptions to be used for exceptional circumstances rather than for control flow?
This *is* an exceptional circumstance: a typical __enter__ method will just return or raise some other exception. I suppose you could use some kind of dedicated thread-local state instead of an exception to indicate that the underlying generator didn't yield, but a new control flow exception seemed like the most straightforward option.
I'm somewhat intrigued by Glyph's idea though - if I can figure out a way to make it practical, it does offer very some interesting possibilities (and would, in effect, bring reusable embedded code blocks to Python...).
Cheers, Nick.
Nick Coghlan wrote:
Note that using exceptions for control flow can be bad for other implementations of Python. For example exceptions on the .NET framework are very expensive. (Although there are workarounds such as not really raising the exception - but they're ugly).
Is it that exceptions are expensive, or setting up a try/except block is expensive? The reason the SkipStatement idea is tenable at all (even in CPython) is that try/except is fairly cheap when no exception is raised.
It is the raising of the exception that is expensive.
Michael
(In this specific case, my initial patch does slow things down a bit, since one side effect of the extra try/except block is to disallow a couple of stack based optimisations that are used in the current CPython implementation of the with statement)
Isn't it better practise for exceptions to be used for exceptional circumstances rather than for control flow?
This *is* an exceptional circumstance: a typical __enter__ method will just return or raise some other exception. I suppose you could use some kind of dedicated thread-local state instead of an exception to indicate that the underlying generator didn't yield, but a new control flow exception seemed like the most straightforward option.
I'm somewhat intrigued by Glyph's idea though - if I can figure out a way to make it practical, it does offer very some interesting possibilities (and would, in effect, bring reusable embedded code blocks to Python...).
Cheers, Nick.
Michael Foord wrote:
Nick Coghlan wrote:
Note that using exceptions for control flow can be bad for other implementations of Python. For example exceptions on the .NET framework are very expensive. (Although there are workarounds such as not really raising the exception - but they're ugly).
Is it that exceptions are expensive, or setting up a try/except block is expensive? The reason the SkipStatement idea is tenable at all (even in CPython) is that try/except is fairly cheap when no exception is raised.
It is the raising of the exception that is expensive.
Then that isn't a huge drawback in this case - the SkipStatement exception is only used in situations which would currently probably be handled by raising an exception anyway (e.g. the change to contextlib.contextmanager.__enter__() in the patch is to raise SkipStatement where it currently raises RuntimeError).
Cheers, Nick.
On Sun, Mar 15, 2009, Michael Foord wrote:
Note that using exceptions for control flow can be bad for other implementations of Python. For example exceptions on the .NET framework are very expensive. (Although there are workarounds such as not really raising the exception - but they're ugly).
Isn't it better practise for exceptions to be used for exceptional circumstances rather than for control flow?
It seems to me that we as a development community already made a decision when we switched to StopIteration as the primary mechanism for halting ``for`` loops. (Not that it was really a new decision because parts of the Python community have always advocated using exceptions for control flow, but the ``for`` loop enshrines it.) I doubt that using exceptions for control flow in ``with`` blocks will cause anywhere near so much a performance degradation.
Aahz wrote:
On Sun, Mar 15, 2009, Michael Foord wrote:
Note that using exceptions for control flow can be bad for other implementations of Python. For example exceptions on the .NET framework are very expensive. (Although there are workarounds such as not really raising the exception - but they're ugly).
Isn't it better practise for exceptions to be used for exceptional circumstances rather than for control flow?
It seems to me that we as a development community already made a decision when we switched to StopIteration as the primary mechanism for halting ``for`` loops. (Not that it was really a new decision because parts of the Python community have always advocated using exceptions for control flow, but the ``for`` loop enshrines it.) I doubt that using exceptions for control flow in ``with`` blocks will cause anywhere near so much a performance degradation.
Well, StopIteration is still an implementation detail that only occasionally bleeds through to actual programming. It says nothing about whether using exceptions for non-exceptional circumstances (control flow) is good practise. Personally I think it makes the intent of code less easy to understand - in effect the exceptions *are* being used as a goto.
Michael
On Sun, Mar 15, 2009, Michael Foord wrote:
Aahz wrote:
On Sun, Mar 15, 2009, Michael Foord wrote:
Note that using exceptions for control flow can be bad for other implementations of Python. For example exceptions on the .NET framework are very expensive. (Although there are workarounds such as not really raising the exception - but they're ugly).
Isn't it better practise for exceptions to be used for exceptional circumstances rather than for control flow?
It seems to me that we as a development community already made a decision when we switched to StopIteration as the primary mechanism for halting ``for`` loops. (Not that it was really a new decision because parts of the Python community have always advocated using exceptions for control flow, but the ``for`` loop enshrines it.) I doubt that using exceptions for control flow in ``with`` blocks will cause anywhere near so much a performance degradation.
Well, StopIteration is still an implementation detail that only occasionally bleeds through to actual programming. It says nothing about whether using exceptions for non-exceptional circumstances (control flow) is good practise. Personally I think it makes the intent of code less easy to understand - in effect the exceptions *are* being used as a goto.
Let me know how you'd rewrite this more clearly without a control-flow exception:
try: for field in curr_fields: for item in record[field]: item = item.lower() for filter in excludes: if match(item, filter): raise Excluded except Excluded: continue
This is pretty much the canonical example showing why control-flow exceptions are a Good Thing. They're a *structured* goto.
Aahz wrote:
On Sun, Mar 15, 2009, Michael Foord wrote:
Aahz wrote:
On Sun, Mar 15, 2009, Michael Foord wrote:
Note that using exceptions for control flow can be bad for other implementations of Python. For example exceptions on the .NET framework are very expensive. (Although there are workarounds such as not really raising the exception - but they're ugly).
Isn't it better practise for exceptions to be used for exceptional circumstances rather than for control flow?
It seems to me that we as a development community already made a decision when we switched to StopIteration as the primary mechanism for halting ``for`` loops. (Not that it was really a new decision because parts of the Python community have always advocated using exceptions for control flow, but the ``for`` loop enshrines it.) I doubt that using exceptions for control flow in ``with`` blocks will cause anywhere near so much a performance degradation.
Well, StopIteration is still an implementation detail that only occasionally bleeds through to actual programming. It says nothing about whether using exceptions for non-exceptional circumstances (control flow) is good practise. Personally I think it makes the intent of code less easy to understand - in effect the exceptions *are* being used as a goto.
Let me know how you'd rewrite this more clearly without a control-flow exception:
try: for field in curr_fields: for item in record[field]: item = item.lower() for filter in excludes: if match(item, filter): raise Excluded except Excluded: continueThis is pretty much the canonical example showing why control-flow exceptions are a Good Thing. They're a *structured* goto.
You didn't include all the code - so impossible to match the exact semantics. Breaking out of multiple loops with a return is a cleaner way to handle it IMHO.
def find_excludes(): for field in curr_fields: for item in record[field]: item = item.lower() for filter in excludes: if match(item, filter): return
while something: find_excludes()
Michael
On Mon, Mar 16, 2009 at 1:00 AM, Michael Foord fuzzyman@voidspace.org.uk wrote:
You didn't include all the code - so impossible to match the exact semantics. Breaking out of multiple loops with a return is a cleaner way to handle it IMHO.
I don't really see why this is cleaner; they're both just structured gotos. Heck, the code even looks practically the same.
Aahz wrote:
This is pretty much the canonical example showing why control-flow exceptions are a Good Thing. They're a *structured* goto.
I'm wondering whether what we really want is something that actually *is* a structured goto. Or something like a very light-weight exception that doesn't carry all the expensive baggage of tracebacks, isinstance() tests for matching, etc.
Ruby seems to have something like this. It has a try/rescue/raise mechanism that works like Python's try/except/raise, but it also has try/catch/throw as a less expensive alternative for flow control.
A Python version might look like
Foo = object() # a token to throw
try: do_something() catch Foo: someone_threw_a_foo()
def do_something(): throw Foo
Nothing is instantiated -- the token value itself is thrown -- and the catch clauses compare it by identity with candidate values.
There is also no traceback carried by the thrown token.
(Although if something is thrown but not caught, an exception should be raised at the point of the throw with an appropriate traceback -- implementation of that is left as an exercise for the reader.)
Well, StopIteration is still an implementation detail that only occasionally bleeds through to actual programming. It says nothing about whether using exceptions for non-exceptional circumstances (control flow) is good practise. Personally I think it makes the intent of code less easy to understand - in effect the exceptions *are* being used as a goto.
The same can be said about if statements and while loops - they are also being used "as gotos". The bad thing about the goto statement is that it allows arbitrary, unstructured control flow. This is unlike if statements, while loops, and - yes - exceptions. They all provide for structured control flow. raise, in particular, is no more evil than break, continue, return, or yield.
Regards, Martin
Michael Foord wrote:
Well, StopIteration is still an implementation detail that only occasionally bleeds through to actual programming. It says nothing about whether using exceptions for non-exceptional circumstances (control flow) is good practise. Personally I think it makes the intent of code less easy to understand - in effect the exceptions *are* being used as a goto.
Note that raising SkipStatement manually is likely to be even rarer than raising StopIteration. Catching it should almost never happen other than implicitly inside a with statement (that's the reason I made it a peer of SystemExit and GeneratorExit rather than a peer of StopIteration).
It is primarily proposed as a way for contextlib.contextmanager to tell the interpreter that the underlying generator didn't yield, so the body of the with statement should be skipped completely. It just so happens that manually implemented context managers will also be free to use it if they need to for some reason.
An alternative approach worth considering may be to use NotImplemented as a model instead of StopIteration. With that approach, instead of having SkipStatement be an exception, have it be a singleton that can be returned from __enter__ to indicate that the with statement body would be skipped.
That has a big advantage over using an exception when it comes to execution speed. The statement semantics in that case would become:
mgr = (EXPR) exit = mgr.__exit__ # Not calling it yet value = mgr.__enter__() if value is not SkipStatement: exc = True try: try: VAR = value # Only if "as VAR" is present BLOCK except: # The exceptional case is handled here exc = False if not exit(*sys.exc_info()): raise # The exception is swallowed if exit() returns true finally: # The normal and non-local-goto cases are handled here if exc: exit(None, None, None)
(keeping in mind that I already plan to change PEP 377 to drop the idea of assigning anything to VAR when the statement body is skipped)
The major drawback of that approach is that it becomes a little trickier to write a context manager like nested() correctly - it would need to check all of the __enter__() return values and start unwinding the context manager stack if it encountered SkipStatement. The fix isn't particularly complicated*, but it does contrast with the fact that having SkipStatement as an exception means that the current implementation of nested() will "just work" with the new semantics.
Cheers, Nick.
* For reference, to support a "SkipStatement as return value" approach the main loop in nested() would have to change from this:
for mgr in managers: exit = mgr.__exit__ enter = mgr.__enter__ vars.append(enter()) exits.append(exit) yield vars
To this:
for mgr in managers: exit = mgr.__exit__ enter = mgr.__enter__ var = enter() if var is SkipStatement: break vars.append(var) exits.append(exit) else: yield vars
As mentioned above, if SkipStatement is an exception then nested() works correctly without any changes.
I have no right to speak because I haven't read through all the details of the proposal, but reading this I am very sad that we have to introduce a whole new exception (and one with special status as well) in order to fix such a niggly corner case of the context manager protocol.
Since IIUC the original context manager design was intended to have exactly one yield in the body of the context manager -- can't we just declare fewer (or more) yields an error and rase an appropriate TypeError or something?
--Guido
On Mon, Mar 16, 2009 at 4:43 AM, Nick Coghlan ncoghlan@gmail.com wrote:
Michael Foord wrote:
Well, StopIteration is still an implementation detail that only occasionally bleeds through to actual programming. It says nothing about whether using exceptions for non-exceptional circumstances (control flow) is good practise. Personally I think it makes the intent of code less easy to understand - in effect the exceptions *are* being used as a goto.
Note that raising SkipStatement manually is likely to be even rarer than raising StopIteration. Catching it should almost never happen other than implicitly inside a with statement (that's the reason I made it a peer of SystemExit and GeneratorExit rather than a peer of StopIteration).
It is primarily proposed as a way for contextlib.contextmanager to tell the interpreter that the underlying generator didn't yield, so the body of the with statement should be skipped completely. It just so happens that manually implemented context managers will also be free to use it if they need to for some reason.
An alternative approach worth considering may be to use NotImplemented as a model instead of StopIteration. With that approach, instead of having SkipStatement be an exception, have it be a singleton that can be returned from __enter__ to indicate that the with statement body would be skipped.
That has a big advantage over using an exception when it comes to execution speed. The statement semantics in that case would become:
mgr = (EXPR) exit = mgr.__exit__ # Not calling it yet value = mgr.__enter__() if value is not SkipStatement: exc = True try: try: VAR = value # Only if "as VAR" is present BLOCK except: # The exceptional case is handled here exc = False if not exit(*sys.exc_info()): raise # The exception is swallowed if exit() returns true finally: # The normal and non-local-goto cases are handled here if exc: exit(None, None, None)
(keeping in mind that I already plan to change PEP 377 to drop the idea of assigning anything to VAR when the statement body is skipped)
The major drawback of that approach is that it becomes a little trickier to write a context manager like nested() correctly - it would need to check all of the __enter__() return values and start unwinding the context manager stack if it encountered SkipStatement. The fix isn't particularly complicated*, but it does contrast with the fact that having SkipStatement as an exception means that the current implementation of nested() will "just work" with the new semantics.
Cheers, Nick.
- For reference, to support a "SkipStatement as return value" approach
the main loop in nested() would have to change from this:
for mgr in managers: exit = mgr.__exit__ enter = mgr.__enter__ vars.append(enter()) exits.append(exit) yield vars
To this:
for mgr in managers: exit = mgr.__exit__ enter = mgr.__enter__ var = enter() if var is SkipStatement: break vars.append(var) exits.append(exit) else: yield vars
As mentioned above, if SkipStatement is an exception then nested() works correctly without any changes.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
Moreover, since the main use case seems to be fixing a corner case of the nested() context manager, perhaps the effort towards changing the language would be better directed towards supporting "with a, b:" as a shorthand for "with a: with b:" .
On Mon, Mar 16, 2009 at 10:01 AM, Guido van Rossum guido@python.org wrote:
I have no right to speak because I haven't read through all the details of the proposal, but reading this I am very sad that we have to introduce a whole new exception (and one with special status as well) in order to fix such a niggly corner case of the context manager protocol.
Since IIUC the original context manager design was intended to have exactly one yield in the body of the context manager -- can't we just declare fewer (or more) yields an error and rase an appropriate TypeError or something?
--Guido
On Mon, Mar 16, 2009 at 4:43 AM, Nick Coghlan ncoghlan@gmail.com wrote:
Michael Foord wrote:
Well, StopIteration is still an implementation detail that only occasionally bleeds through to actual programming. It says nothing about whether using exceptions for non-exceptional circumstances (control flow) is good practise. Personally I think it makes the intent of code less easy to understand - in effect the exceptions *are* being used as a goto.
Note that raising SkipStatement manually is likely to be even rarer than raising StopIteration. Catching it should almost never happen other than implicitly inside a with statement (that's the reason I made it a peer of SystemExit and GeneratorExit rather than a peer of StopIteration).
It is primarily proposed as a way for contextlib.contextmanager to tell the interpreter that the underlying generator didn't yield, so the body of the with statement should be skipped completely. It just so happens that manually implemented context managers will also be free to use it if they need to for some reason.
An alternative approach worth considering may be to use NotImplemented as a model instead of StopIteration. With that approach, instead of having SkipStatement be an exception, have it be a singleton that can be returned from __enter__ to indicate that the with statement body would be skipped.
That has a big advantage over using an exception when it comes to execution speed. The statement semantics in that case would become:
mgr = (EXPR) exit = mgr.__exit__ # Not calling it yet value = mgr.__enter__() if value is not SkipStatement: exc = True try: try: VAR = value # Only if "as VAR" is present BLOCK except: # The exceptional case is handled here exc = False if not exit(*sys.exc_info()): raise # The exception is swallowed if exit() returns true finally: # The normal and non-local-goto cases are handled here if exc: exit(None, None, None)
(keeping in mind that I already plan to change PEP 377 to drop the idea of assigning anything to VAR when the statement body is skipped)
The major drawback of that approach is that it becomes a little trickier to write a context manager like nested() correctly - it would need to check all of the __enter__() return values and start unwinding the context manager stack if it encountered SkipStatement. The fix isn't particularly complicated*, but it does contrast with the fact that having SkipStatement as an exception means that the current implementation of nested() will "just work" with the new semantics.
Cheers, Nick.
- For reference, to support a "SkipStatement as return value" approach
the main loop in nested() would have to change from this:
for mgr in managers: exit = mgr.__exit__ enter = mgr.__enter__ vars.append(enter()) exits.append(exit) yield vars
To this:
for mgr in managers: exit = mgr.__exit__ enter = mgr.__enter__ var = enter() if var is SkipStatement: break vars.append(var) exits.append(exit) else: yield vars
As mentioned above, if SkipStatement is an exception then nested() works correctly without any changes.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (home page: http://www.python.org/~guido/)
On Mon, Mar 16, 2009 at 11:06 AM, Guido van Rossum guido@python.org wrote:
Moreover, since the main use case seems to be fixing a corner case of the nested() context manager, perhaps the effort towards changing the language would be better directed towards supporting "with a, b:" as a shorthand for "with a: with b:" .
On Mon, Mar 16, 2009 at 10:01 AM, Guido van Rossum guido@python.org wrote:
I have no right to speak because I haven't read through all the details of the proposal, but reading this I am very sad that we have to introduce a whole new exception (and one with special status as well) in order to fix such a niggly corner case of the context manager protocol.
Since IIUC the original context manager design was intended to have exactly one yield in the body of the context manager -- can't we just declare fewer (or more) yields an error and rase an appropriate TypeError or something?
It's not really a generator specific thing. You can generate similar problems by just defining a class with an __enter__() method that raises an exception.
But I agree that it seems like a big change for a small corner case. Is there anything other than contextlib.nested() which needs this? If there is no other use case, then I'm a strong +1 for Guido's suggestion of providing syntactic support for ``with a, b:`` instead.
BTW, I think the explanation of the problem isn't as clear as it could be. The core problem, if I understand it right, is that contextlib.nested() is not equivalent to a real nested with statement because it calls the nested __enter__() methods too early. A real nested with statement translates into something like::
mgr1.__enter__() try: mgr2.__enter__() try: BLOCK except: ... except: if not mgr1.__exit__(*sys.exc_info()): raise
But contextlib.nested() calls all the __enter__() methods in its own __enter__() so it translates into something like::
mgr1.__enter__() mgr2.__enter__() try: BLOCK except: ...
The key problem here is that ``mgr2.__enter__()`` is outside of the try block, and the context manager has no way to put it inside. So the thing that contextlib.nested() really needs is a way to be able to insert statements into the BLOCK part of the code. (I'm not actually suggesting we go this route, but that seems to be what contextlib.nested() is really after.)
Steve
On Mon, Mar 16, 2009 at 11:26 AM, Steven Bethard steven.bethard@gmail.com wrote:
On Mon, Mar 16, 2009 at 11:06 AM, Guido van Rossum guido@python.org wrote:
Moreover, since the main use case seems to be fixing a corner case of the nested() context manager, perhaps the effort towards changing the language would be better directed towards supporting "with a, b:" as a shorthand for "with a: with b:" .
On Mon, Mar 16, 2009 at 10:01 AM, Guido van Rossum guido@python.org wrote:
I have no right to speak because I haven't read through all the details of the proposal, but reading this I am very sad that we have to introduce a whole new exception (and one with special status as well) in order to fix such a niggly corner case of the context manager protocol.
Since IIUC the original context manager design was intended to have exactly one yield in the body of the context manager -- can't we just declare fewer (or more) yields an error and rase an appropriate TypeError or something?
It's not really a generator specific thing. You can generate similar problems by just defining a class with an __enter__() method that raises an exception.
Huh? According to PEP 343, if __enter__ raises an exception, that's the end of the story. __exit__ shouldn't be called, the exception is not modified, the flow is interrupted right there.
But I agree that it seems like a big change for a small corner case. Is there anything other than contextlib.nested() which needs this? If there is no other use case, then I'm a strong +1 for Guido's suggestion of providing syntactic support for ``with a, b:`` instead.
BTW, I think the explanation of the problem isn't as clear as it could be. The core problem, if I understand it right, is that contextlib.nested() is not equivalent to a real nested with statement because it calls the nested __enter__() methods too early. A real nested with statement translates into something like::
mgr1.__enter__() try: mgr2.__enter__() try: BLOCK except: ... except: if not mgr1.__exit__(*sys.exc_info()): raise
But contextlib.nested() calls all the __enter__() methods in its own __enter__() so it translates into something like::
mgr1.__enter__() mgr2.__enter__() try: BLOCK except: ...
The key problem here is that ``mgr2.__enter__()`` is outside of the try block, and the context manager has no way to put it inside. So the thing that contextlib.nested() really needs is a way to be able to insert statements into the BLOCK part of the code. (I'm not actually suggesting we go this route, but that seems to be what contextlib.nested() is really after.)
Yeah, it really seems pretty much limited to contextlib.nested(). I'd be happy to sacrifice the possibility to *exactly* emulate two nested with-statements. The cost of a new exception is huge -- everyone will have to explain its existence, and historically "you don't need to know about this little detail" isn't acceptable for Python docs. Little details you don't need to know about add up.
Guido van Rossum wrote:
Yeah, it really seems pretty much limited to contextlib.nested(). I'd be happy to sacrifice the possibility to *exactly* emulate two nested with-statements.
Then I really haven't explained the problem well at all. One of the premises of PEP 343 was "Got a frequently recurring block of code that only has one variant sequence of statements somewhere in the middle? Well, now you can factor that out by putting it in a generator, replacing the part that varies with a yield statement and decorating the generator with contextlib.contextmanager."
It turns out that there's a caveat that needs to go on the end of that though: "Be very, very sure that the yield statement can *never* be skipped or your context manager based version will raise a RuntimeError in cases where the original code would have just skipped over the variant section of code and resumed execution afterwards."
Nested context managers (whether through contextlib.nested or through syntactic support) just turns out to be a common case where you *don't necessarily know* just by looking at the code whether it can skip over the body of the code or not.
Suppose you have 3 context managers that are regularly used together (call them cmA(), cmB(), cmC() for now).
Writing that as:
with cmA(): with cmB(): with cmC(): do_something()
Or the tentatively proposed:
with cmA(), cmB(), cmC(): do_something()
is definitely OK, regardless of the details of the context managers.
However, whether or not you can bundle that up into a *new* context manager (regardless of nesting syntax) depends on whether or not an outer context manager can suppress an exception raised by an inner one.
@contextmanager def cmABC(): with cmA(): with cmB(): with cmC(): yield
with cmABC(): do_something()
The above is broken if cmB().__enter__() or cmC.__enter__() can raise an exception that cmA().__exit__() suppresses, or cmB.__enter__() raises an exception that cmB().__exit__() suppresses. So whereas the inline versions were clearly correct, the correctness of the second version currently depends on details of the context managers themselves. Changing the syntax to allow the three context managers to be written on one line does nothing to fix that semantic discrepancy between the original inline code and the factored out version.
PEP 377 is about changing the with statement semantics and the @contextmanager implementation so that the semantics of the factored out version actually matches that of the original inline code.
You can get yourself into similar trouble without nesting context managers - all it takes is some way of skipping the variant code in a context manager that wouldn't have raised an exception if the code was written out inline instead of being factored out into the context manager.
Suppose for instance you wanted to use a context manager as a different way of running tests:
@contextmanager def inline_test(self, *setup_args): try: self.setup(*setup_args) except: # Setup exception occurred, trap it and log it return try: yield except: # Test exception occurred, trap it and log it finally: try: self.teardown() except: # Teardown exception occurred, trap it and log it
with inline_test(setup1): test_one() with inline_test(setup2): test_two() with inline_test(setup3): test_three()
That approach isn't actually valid - a context manager is not permitted to decide in it's __enter__() method that executing the body of the with statement would be a bad idea.
The early return in the above makes it obvious that that CM is broken under the current semantics, but what about the following one:
@contextmanager def broken_cm(self): try: call_user_setup() try: yield finally: call_user_teardown() except UserCancel: show_message("Operation aborted by user")
That CM will raise RuntimeError if the user attempts to cancel an operation during the execution of the "call_user_setup()" method. Without SkipStatement or something like it, that can't be fixed.
Hell, I largely wrote PEP 377 to try to get out of having to document these semantic problems with the with statement - if I'm having trouble getting *python-dev* to grasp the problem, what hope do other users of Python have?
Cheers, Nick.
On Mon, Mar 16, 2009 at 2:37 PM, Nick Coghlan ncoghlan@gmail.com wrote:
Guido van Rossum wrote:
Yeah, it really seems pretty much limited to contextlib.nested(). I'd be happy to sacrifice the possibility to *exactly* emulate two nested with-statements.
Then I really haven't explained the problem well at all. One of the premises of PEP 343 was "Got a frequently recurring block of code that only has one variant sequence of statements somewhere in the middle? Well, now you can factor that out by putting it in a generator, replacing the part that varies with a yield statement and decorating the generator with contextlib.contextmanager."
It turns out that there's a caveat that needs to go on the end of that though: "Be very, very sure that the yield statement can *never* be skipped or your context manager based version will raise a RuntimeError in cases where the original code would have just skipped over the variant section of code and resumed execution afterwards."
Well, I don't think you can take that premise (promise? :-) literally anyways, since you cannot turn a loop into a with-statement either. So I would be fine with just adding the condition that the variant sequence should be executed exactly once.
Nested context managers (whether through contextlib.nested or through syntactic support) just turns out to be a common case where you *don't necessarily know* just by looking at the code whether it can skip over the body of the code or not.
Suppose you have 3 context managers that are regularly used together (call them cmA(), cmB(), cmC() for now).
Writing that as:
with cmA(): with cmB(): with cmC(): do_something()
Or the tentatively proposed:
with cmA(), cmB(), cmC(): do_something()
is definitely OK, regardless of the details of the context managers.
However, whether or not you can bundle that up into a *new* context manager (regardless of nesting syntax) depends on whether or not an outer context manager can suppress an exception raised by an inner one.
@contextmanager def cmABC(): with cmA(): with cmB(): with cmC(): yield
with cmABC(): do_something()
While all this may make sense to the original inventor of context managers (== you ;-), I personally find this example quite perverse. Do you have an example taken from real life?
The above is broken if cmB().__enter__() or cmC.__enter__() can raise an exception that cmA().__exit__() suppresses, or cmB.__enter__() raises an exception that cmB().__exit__() suppresses. So whereas the inline versions were clearly correct, the correctness of the second version currently depends on details of the context managers themselves. Changing the syntax to allow the three context managers to be written on one line does nothing to fix that semantic discrepancy between the original inline code and the factored out version.
I think I understand that, I just don't see a use case so important as to warrant introducing a brand new exception deriving from BaseException.
PEP 377 is about changing the with statement semantics and the @contextmanager implementation so that the semantics of the factored out version actually matches that of the original inline code.
You can get yourself into similar trouble without nesting context managers - all it takes is some way of skipping the variant code in a context manager that wouldn't have raised an exception if the code was written out inline instead of being factored out into the context manager.
Yeah, see above -- you can't write a context manager that implements a loop either.
Suppose for instance you wanted to use a context manager as a different way of running tests:
@contextmanager def inline_test(self, *setup_args): try: self.setup(*setup_args) except: # Setup exception occurred, trap it and log it return try: yield except: # Test exception occurred, trap it and log it finally: try: self.teardown() except: # Teardown exception occurred, trap it and log it
with inline_test(setup1): test_one() with inline_test(setup2): test_two() with inline_test(setup3): test_three()
That approach isn't actually valid
(but your proposal would make it valid right?)
- a context manager is not permitted
to decide in it's __enter__() method that executing the body of the with statement would be a bad idea.
A setup failure sounds like a catastrophic error to me.
The early return in the above makes it obvious that that CM is broken under the current semantics, but what about the following one:
@contextmanager def broken_cm(self): try: call_user_setup() try: yield finally: call_user_teardown() except UserCancel: show_message("Operation aborted by user")
That CM will raise RuntimeError if the user attempts to cancel an operation during the execution of the "call_user_setup()" method. Without SkipStatement or something like it, that can't be fixed.
Well pretty much anything that tries to catch asynchronous exceptions is doomed to be 100% correct. Again the example is too abstract to be convincing.
Hell, I largely wrote PEP 377 to try to get out of having to document these semantic problems with the with statement - if I'm having trouble getting *python-dev* to grasp the problem, what hope do other users of Python have?
Hell, if you can't come up with a real use case, why bother? :-)
Perhaps you could address my worry about introducing an obscure BaseException subclass that will forever add to the weight of the list of built-in exceptions in all documentation?
Guido van Rossum wrote:
Hell, I largely wrote PEP 377 to try to get out of having to document these semantic problems with the with statement - if I'm having trouble getting *python-dev* to grasp the problem, what hope do other users of Python have?
Hell, if you can't come up with a real use case, why bother? :-)
I figured I'd try for a solution that didn't offend my sense of aesthetics before caving in and working out how to better document the limitations of the status quo :)
Perhaps you could address my worry about introducing an obscure BaseException subclass that will forever add to the weight of the list of built-in exceptions in all documentation?
Since this is really just a matter of the aesthetics of the underlying design from my point of view rather than solving a real world problem, I don't have a good answer for you.
In the absence of reports of actual problems caused by this limitation, should I consider the PEP rejected?
Cheers, Nick.
On Mon, Mar 16, 2009 at 3:19 PM, Nick Coghlan ncoghlan@gmail.com wrote:
Guido van Rossum wrote:
Hell, I largely wrote PEP 377 to try to get out of having to document these semantic problems with the with statement - if I'm having trouble getting *python-dev* to grasp the problem, what hope do other users of Python have?
Hell, if you can't come up with a real use case, why bother? :-)
I figured I'd try for a solution that didn't offend my sense of aesthetics before caving in and working out how to better document the limitations of the status quo :)
Perhaps you could address my worry about introducing an obscure BaseException subclass that will forever add to the weight of the list of built-in exceptions in all documentation?
Since this is really just a matter of the aesthetics of the underlying design from my point of view rather than solving a real world problem, I don't have a good answer for you.
In the absence of reports of actual problems caused by this limitation, should I consider the PEP rejected?
Yes -- sorry for your wasted efforts.
Guido van Rossum wrote:
On Mon, Mar 16, 2009 at 3:19 PM, Nick Coghlan ncoghlan@gmail.com wrote:
Guido van Rossum wrote:
Hell, I largely wrote PEP 377 to try to get out of having to document these semantic problems with the with statement - if I'm having trouble getting *python-dev* to grasp the problem, what hope do other users of Python have?
Hell, if you can't come up with a real use case, why bother? :-)
I figured I'd try for a solution that didn't offend my sense of aesthetics before caving in and working out how to better document the limitations of the status quo :)
Perhaps you could address my worry about introducing an obscure BaseException subclass that will forever add to the weight of the list of built-in exceptions in all documentation?
Since this is really just a matter of the aesthetics of the underlying design from my point of view rather than solving a real world problem, I don't have a good answer for you.
In the absence of reports of actual problems caused by this limitation, should I consider the PEP rejected?
Yes -- sorry for your wasted efforts.
Not wasted - I prefer having this as a recognised limitation of the semantics rather than as an accident of the implementation.
Who knows, maybe somebody will come up with a real world use case some day and we can drag the PEP out and dust it off a bit :)
Cheers, Nick.
On Mar 16, 2009, at 3:40 PM, Nick Coghlan wrote:
Not wasted - I prefer having this as a recognised limitation of the semantics rather than as an accident of the implementation.
Well, I'm glad some good came from the issue report. =)
Who knows, maybe somebody will come up with a real world use case some day and we can drag the PEP out and dust it off a bit :)
I don't expect this to be at all compelling, but FWIW:
The identification of this issue came from an *experiment* attempting to create a *single* "daemonized()" CM that would execute the with- statement's block in a new child process and, of course, not execute it in the parent. At first, I ran into the RuntimeError in the parent process, and then after rewriting the CMs as classes, I realized the futility.
with daemonized(): run_some_subprocess()
Of course it was all possible if I used the component CMs directly:
with parent_trap(): with fork_but_raise_in_parent(): run_some_subprocess()
And thus:
def daemonized(): return contextlib.nested(parent_trap(), fork_but_raise_in_parent())
James Pye lists@jwp.name writes:
The identification of this issue came from an *experiment* attempting to create a *single* "daemonized()" CM that would execute the with- statement's block in a new child process and, of course, not execute it in the parent. At first, I ran into the RuntimeError in the parent process, and then after rewriting the CMs as classes, I realized the futility.
with daemonized(): run_some_subprocess()
This use case is addressed by my up-coming PEP, currently in draft form but submitted to the PEP editor. A reference implementation is at URL:http://pypi.python.org/pypi/python-daemon/.
I'd like to know whether this meets your needs; follow-ups privately to me or in ‘python-ideas’ as you think appropriate.
Ben Finney ben+python@benfinney.id.au writes:
James Pye lists@jwp.name writes:
with daemonized(): run_some_subprocess()
This use case is addressed by my up-coming PEP, currently in draft form but submitted to the PEP editor. A reference implementation is at URL:http://pypi.python.org/pypi/python-daemon/.
On second blush, it addresses a smaller use case, with the same syntax as your example. It should still be applicable at some level.
I'd like to know whether this meets your needs; follow-ups privately to me or in ‘python-ideas’ as you think appropriate.
Ditto.
James Pye wrote:
The identification of this issue came from an *experiment* attempting to create a *single* "daemonized()" CM that would execute the with-statement's block in a new child process and, of course, not execute it in the parent. At first, I ran into the RuntimeError in the parent process, and then after rewriting the CMs as classes, I realized the futility.
with daemonized(): run_some_subprocess()
Of course it was all possible if I used the component CMs directly:
with parent_trap(): with fork_but_raise_in_parent(): run_some_subprocess()
When updating the PEP with the rejection notice, it occurred to me that it is fairly easy to handle specific use cases like this reasonably cleanly by including a callable in the design that is always used inline in the body of the outermost with statement. For example:
@contextmanager def make_daemon() class SkipInParent(Exception): pass def startd(): # Fork process, then raise SkipInParent # in the parent process. The child process # continues running as a daemon. try: yield startd except SkipInParent: pass
with make_daemon() as startd: startd() # Daemon code goes here
With that approach, since it is startd() that raises the exception rather than __enter__() then __exit__() will always be given the chance to suppress it.
Cheers, Nick.
Nick Coghlan wrote:
One of the premises of PEP 343 was "Got a frequently recurring block of code that only has one variant sequence of statements somewhere in the middle? Well, now you can factor that out
Um, no -- it says explicitly right at the very top of PEP 343 that it's only about factoring out try/finally statements.
There's no way that
try: code_block finally: ...
can fail to enter the code block if you get as far as the "try". So it's not reasonable to expect the with statement to provide this ability.
Greg Ewing wrote:
Um, no -- it says explicitly right at the very top of PEP 343 that it's only about factoring out try/finally statements.
There's no way that
try: code_block finally: ...
can fail to enter the code block if you get as far as the "try". So it's not reasonable to expect the with statement to provide this ability.
We were working on the PEP for over a year though - expectations changed a bit when the ability to suppress exceptions was added (that wasn't in the original version of the PEP which was when most of the first few sections was written).
I agree that try/finally was by far the main use case - it just isn't the only use case or we would never have included the option to suppress exceptions at all.
Still, I'm happy to let this rest indefinitely - I doubt the daemonisation use case is going to be enough to change Guido's mind, and it really is rather difficult to come up with practical examples where the current behaviour is a genuine problem.
Cheers, Nick.
Aahz wrote:
On Sun, Mar 15, 2009, Michael Foord wrote:
It seems to me that we as a development community already made a decision when we switched to StopIteration as the primary mechanism for halting ``for`` loops.
If was previously IndexError that stopped for loops, so that was not new ;-).
On 12:56 pm, ncoghlan@gmail.com wrote:
PEP 377 is a proposal to allow context manager __enter__() methods to skip the body of the with statement by raising a specific (new) flow control exception.
Since there is a working reference implementation now, I thought it was time to open it up for broader discussion.
Why not allow a context manager to implement some other method, for the sake of argument let's say "__start__", which was invoked with a callable object and could choose to evaluate or not evaluate the statement body by simply not calling that object (or perhaps iterable, in the case of a generator)?
This PEP proposes that we have two ways to deal with the body of a 'with' statement: either the body is run or not. I have always wanted to have another option: run the body later.
Passing around an object representing the body of the with statement would allow for this use-case, as well as removing the ugly protrusion of yet another control-flow exception (which, as has already been noted, creates difficulties for some other python implementations).
For Jython, this proposal would not present any problems. Exceptions are in any event of lower cost than for CPython.
Given that we have now adopted Python bytecode for various scenarios where we cannot compile to Java bytecode, it would be nice to track any changes in the VM such as the proposed SETUP_WITH opcode. But I'm sure we'll continue to diff ceval.c, etc. (Consider this request perhaps fodder for the language summit?)
- Jim
On Sun, Mar 15, 2009 at 12:37 PM, glyph@divmod.com wrote:
On 12:56 pm, ncoghlan@gmail.com wrote:
PEP 377 is a proposal to allow context manager __enter__() methods to skip the body of the with statement by raising a specific (new) flow control exception.
Since there is a working reference implementation now, I thought it was time to open it up for broader discussion.
Why not allow a context manager to implement some other method, for the sake of argument let's say "__start__", which was invoked with a callable object and could choose to evaluate or not evaluate the statement body by simply not calling that object (or perhaps iterable, in the case of a generator)?
This PEP proposes that we have two ways to deal with the body of a 'with' statement: either the body is run or not. I have always wanted to have another option: run the body later.
Passing around an object representing the body of the with statement would allow for this use-case, as well as removing the ugly protrusion of yet another control-flow exception (which, as has already been noted, creates difficulties for some other python implementations).
Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/jbaker%40zyasoft.com
glyph@divmod.com wrote:
On 12:56 pm, ncoghlan@gmail.com wrote:
PEP 377 is a proposal to allow context manager __enter__() methods to skip the body of the with statement by raising a specific (new) flow control exception.
Since there is a working reference implementation now, I thought it was time to open it up for broader discussion.
Why not allow a context manager to implement some other method, for the sake of argument let's say "__start__", which was invoked with a callable object and could choose to evaluate or not evaluate the statement body by simply not calling that object (or perhaps iterable, in the case of a generator)?
So the with statement would in effect create a separate code object for the statement body that still shared the scope of the containing function, and then pass a zero-argument callable in to the new method to allow it to execute that code?
There are some practical hurdles to that idea (specifically, creating a callable which uses its parent's namespace rather than having its own), but the basic concept seems sound.
Rough spec for the concept:
Implementing __enter__/__exit__ on a CM would work as per PEP 343.
Implementing __with__ instead would give the CM complete control over whether or not to execute the block.
The implementation of contextlib.GeneratorContextManager would then change so that instead of providing __enter__/__exit__ as it does now it would instead provide __with__ as follows:
def __with__(self, exec_block): try: return self.gen.next() except StopIteration: pass else: try: exec_block() except: exc_type, value, traceback = sys.exc_info() try: self.gen.throw(type, value, traceback) raise RuntimeError("generator didn't stop after throw()") except StopIteration, exc: # Suppress the exception *unless* it's the same exception that # was passed to throw(). This prevents a StopIteration # raised inside the "with" statement from being suppressed return exc is not value except: # only re-raise if it's *not* the exception that was # passed to throw(), because __exit__() must not raise # an exception unless __exit__() itself failed. But throw() # has to raise the exception to signal propagation, so this # fixes the impedance mismatch between the throw() protocol # and the __exit__() protocol. if sys.exc_info()[1] is not value: raise else: try: self.gen.next() except StopIteration: return else: raise RuntimeError("generator didn't stop")
More radical in some ways that what I was suggesting, but also cleaner and more powerful.
Cheers, Nick.
Nick Coghlan wrote:
Rough spec for the concept:
Implementing __enter__/__exit__ on a CM would work as per PEP 343.
Implementing __with__ instead would give the CM complete control over whether or not to execute the block.
The implementation of contextlib.GeneratorContextManager would then change so that instead of providing __enter__/__exit__ as it does now it would instead provide __with__ as follows:
Expansion in the previous message wasn't quite right since it didn't give the executed block access to the result of __enter__().
Mark II:
def __with__(self, exec_block): try: enter_result = self.gen.next() except StopIteration: pass else: try: exec_block(enter_result) except: exc_type, value, traceback = sys.exc_info() try: self.gen.throw(type, value, traceback) raise RuntimeError("generator didn't stop after throw()") except StopIteration, exc: # Suppress the exception *unless* it's the same exception that # was passed to throw(). This prevents a StopIteration # raised inside the "with" statement from being suppressed return exc is not value except: # only re-raise if it's *not* the exception that was # passed to throw(), because __exit__() must not raise # an exception unless __exit__() itself failed. But throw() # has to raise the exception to signal propagation, so this # fixes the impedance mismatch between the throw() protocol # and the __exit__() protocol. if sys.exc_info()[1] is not value: raise else: try: self.gen.next() except StopIteration: return else: raise RuntimeError("generator didn't stop")
At 06:28 AM 3/16/2009 +1000, Nick Coghlan wrote:
There are some practical hurdles to that idea (specifically, creating a callable which uses its parent's namespace rather than having its own), but the basic concept seems sound.
Actually, that bit's pretty simple -- they're just "nonlocal" variables. But the cost of creating that function, and the cost of having cell variables in the surrounding function is potentially high. (In CPython at least, function calls are optimized in certain ways when the function only has "fast" locals, and no "cell" locals.)
The cost of creating the function (though not the code object) could be held till runtime, since the interpreter could wait until it's sure there's a __with__ method before doing a MAKE_CLOSURE on the code object.
Of course, at that point, what's the difference between:
with foo() as bar: baz
and...
@foo def bar(): baz
except for being slightly less verbose? (due to missing nonlocal statements, etc.)
P.J. Eby wrote:
Of course, at that point, what's the difference between:
with foo() as bar: baz
and...
@foo def bar(): baz
except for being slightly less verbose? (due to missing nonlocal statements, etc.)
That's not quite direct translation. Closer would be:
@foo() def _(bar): baz del _
since the "bar" of the with statement is injected into the namespace of the block. (Obviously, the foo would have be implemented differently in the two cases in order to have it operate the same way.)
I have thought about suggesting adding control flow to the with statement before, since I think if done properly, it might be able to negate the desire people have for multi-line lambdas/Ruby-style blocks, but I do wonder if it would be too confusing to have "with" sometimes mean "abstracting error handling code" and other times mean "abstracted control flow." But now it looks like there's no way around using the with-statement for control flow, since sometimes the block needs to be skipped anyway. So, I'm +.5 on the idea.
-- Carl
P.J. Eby wrote:
At 06:28 AM 3/16/2009 +1000, Nick Coghlan wrote:
There are some practical hurdles to that idea (specifically, creating a callable which uses its parent's namespace rather than having its own), but the basic concept seems sound.
Actually, that bit's pretty simple -- they're just "nonlocal" variables. But the cost of creating that function, and the cost of having cell variables in the surrounding function is potentially high. (In CPython at least, function calls are optimized in certain ways when the function only has "fast" locals, and no "cell" locals.)
Some additional complexities occurred to me today - they go by the names "return", "break" and "continue". With the current design those are perfectly well defined inside a with statement, but they pose a pretty serious hurdle for a practical implementation of glyph's idea.
So I'm going to stick with the more modest approach of a new control flow exception for PEP 377.
I still find the callable block idea somewhat intriguing, but if it ever happens it should be a new construct rather than overloading the meaning of the with statement.
Cheers, Nick.
Nick Coghlan a écrit :
Implementing __with__ instead would give the CM complete control over whether or not to execute the block.
please note, however, that this is an important change in the semantics of the with statement. As things are today, barring exceptional circunstances, the body of the with statement *will* be executed immediately. This allows to forget about the with statement when first reading code, as it doesn't change the intent of the programmer. What Glyph is proposing is more akin to Ruby code blocks.
cheers, Baptiste
Baptiste Carvello wrote:
Nick Coghlan a écrit :
Implementing __with__ instead would give the CM complete control over whether or not to execute the block.
please note, however, that this is an important change in the semantics of the with statement. As things are today, barring exceptional circunstances, the body of the with statement *will* be executed immediately. This allows to forget about the with statement when first reading code, as it doesn't change the intent of the programmer. What Glyph is proposing is more akin to Ruby code blocks.
Yep - the idea is intriguing, powerful, and far beyond the scope of what I'm trying to achieve with PEP 377. I suspect such blocks would have to be more along the lines of what PJE suggested in order to be practical anyway (i.e. nested functions where all "local" variables were implicitly declared as nonlocal), which then leads to the question of how much of a gain they would actually provide now that you *can* explicitly declare nonlocal variables.
Cheers, Nick.
participants (17)
-
"Martin v. Löwis" -
Aahz -
Baptiste Carvello -
Ben Finney -
Brett Cannon -
Carl Johnson -
glyph＠divmod.com -
Greg Ewing -
Guido van Rossum -
James Pye -
Jim Baker -
Michael Foord -
Nick Coghlan -
P.J. Eby -
Steven Bethard -
Terry Reedy -
Tristan Seligmann