with-except-finally blocks
Hello everyone,
I'd like to propose a little bit of syntactic sugar: allowing with-blocks to be followed by except- and finally-blocks just like try-blocks. For example:
with open('spam.txt') as file: print(file.read()) except IOError: print('No spam here...') finally: print('done.')
This proposed syntax is semantically equivalent to wrapping a with-block within a try-block like so:
try: with open('spam.txt') as file: print(file.read()) finally: print('done.') except IOError: print('No spam here...')
I see two advantages to the proposed syntax. First and most obviously, it saves an extra line and an extra indentation level. One line may not be a big deal, but one indentation level can really affect readability. Second and more conceptually, it makes sense to think about exception handling in the context of a with-block. More often than not, if you're using a with-block, you're expecting that something in that block could throw an exception. Usually with-blocks are used to make sure resources (e.g. files, database sessions, mutexes, etc.) are properly closed before the exception is propogated. But I very often want to do some custom clean-up as well (alert the user, etc.). Currently that requires wrapping the whole thing in a try-block, but allowing with-blocks to behave as try-blocks is a more direct way to express what is meant.
I was curious how often with-blocks are actually wrapped in try-blocks for no other purpose than catching exceptions raised in the with-block. So I searched through a number of open source projects looking (roughly) for that pattern:
Project with [1] try-with [2] ============== ======== ============ django 230 17 ipython 541 8 matplotlib 112 3 moinmoin 10 0 numpy 166 1 pillow/pil 1 0 pypy 254 4 scipy 163 2 sqlalchemy 36 0 twisted 72 1 ============== ======== ============ total 1585 36 (2.27%)
[1]: grep -Po '^\s*with .*:' **/*.py [2]: grep -Poz 'try:\s*with .*:' **/*.py
Assuming these projects are representative, about 2% of the with-blocks are directly wrapped by try-blocks. That's not a huge proportion, but it clearly shows that this pattern is being used "in the wild". Whether or not it's worth changing the language for the benefit of 2% of with-blocks is something to debate though.
What do people think of this idea?
-Kale Kundert
On Apr 15, 2015, at 23:39, Kale Kundert kale@thekunderts.net wrote:
Hello everyone,
I'd like to propose a little bit of syntactic sugar: allowing with-blocks to be followed by except- and finally-blocks just like try-blocks. For example:
with open('spam.txt') as file: print(file.read()) except IOError: print('No spam here...') finally: print('done.')
This proposed syntax is semantically equivalent to wrapping a with-block within a try-block like so:
try: with open('spam.txt') as file: print(file.read()) finally: print('done.') except IOError: print('No spam here...')
Why does the finally come before the except? Also, it's still indented with the with, which means you seem to be defining with-except-finally in terms of with-finally, and not defining with-finally at all. Did you mean to have the finally after the except and dedented?
Anyway, I like the idea, but does the grammar work? A with that has an optional except and finally at the same level seems potentially ambiguous. For example:
with spam: pass with eggs: pass finally: pass
If you think about it, it's pretty obvious that the finally goes with the second with, not the first. But can the parser tell that? Or a human who's scanning the code rather than thinking carefully about it? Or tools like auto-indenters?
(I think this problem doesn't arise for try, because it must have at least except or finally.)
It should be easy to edit the grammar and build the parser to at least the first part of the question; for the last part, trying to hack up a few different external tools like Emacs python-mode. But for the human question, I'm not sure how to answer it quite so easily...
I see two advantages to the proposed syntax. First and most obviously, it saves an extra line and an extra indentation level. One line may not be a big deal, but one indentation level can really affect readability. Second and more conceptually, it makes sense to think about exception handling in the context of a with-block. More often than not, if you're using a with-block, you're expecting that something in that block could throw an exception. Usually with-blocks are used to make sure resources (e.g. files, database sessions, mutexes, etc.) are properly closed before the exception is propogated. But I very often want to do some custom clean-up as well (alert the user, etc.). Currently that requires wrapping the whole thing in a try-block, but allowing with-blocks to behave as try-blocks is a more direct way to express what is meant.
I was curious how often with-blocks are actually wrapped in try-blocks for no other purpose than catching exceptions raised in the with-block. So I searched through a number of open source projects looking (roughly) for that pattern:
Project with [1] try-with [2] ============== ======== ============ django 230 17 ipython 541 8 matplotlib 112 3 moinmoin 10 0 numpy 166 1 pillow/pil 1 0 pypy 254 4 scipy 163 2 sqlalchemy 36 0 twisted 72 1 ============== ======== ============ total 1585 36 (2.27%)
[1]: grep -Po '^\s*with .*:' **/*.py [2]: grep -Poz 'try:\s*with .*:' **/*.py
Assuming these projects are representative, about 2% of the with-blocks are directly wrapped by try-blocks. That's not a huge proportion, but it clearly shows that this pattern is being used "in the wild". Whether or not it's worth changing the language for the benefit of 2% of with-blocks is something to debate though.
What do people think of this idea?
-Kale Kundert
Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On Thu, Apr 16, 2015 at 3:22 AM, Andrew Barnert abarnert@yahoo.com.dmarc.invalid wrote:
Anyway, I like the idea, but does the grammar work? A with that has an optional except and finally at the same level seems potentially ambiguous. For example:
with spam: pass with eggs: pass finally: pass
Is this any more (or less) ambiguous than:
if cond1: pass if cond2: pass else: pass
And we have no issue with that (at least not syntactically). With respect to readability however, ...
But as far as the example goes, I agree that it's probably a bit mixed up as was stated.
On Thursday, April 16, 2015 12:33 AM, James Edwards jheiv@jheiv.com wrote:
On Thu, Apr 16, 2015 at 3:22 AM, Andrew Barnert
abarnert@yahoo.com.dmarc.invalid wrote:
Anyway, I like the idea, but does the grammar work? A with that has an
optional except and finally at the same level seems potentially ambiguous. For example:
with spam: pass with eggs: pass finally: passIs this any more (or less) ambiguous than:
if cond1: pass if cond2: pass else: passAnd we have no issue with that (at least not syntactically).
Duh.
Just to make sure, I hacked up a quick patch. (Diff attached.) This gets you enough to test the syntax:
$ ./python.exe >>> import ast >>> print(ast.dump(ast.parse('with:\n pass\nwith:\n pass\nfinally:\n pass\n'))) Module(body=[With(items=[withitem(context_expr=Name(id='spam', ctx=Load()), optional_vars=None)], body=[Pass()], handlers=[], orelse=[], finalbody=[]), With(items=[withitem(context_expr=Name(id='eggs', ctx=Load()), optional_vars=None)], body=[Pass()], handlers=[], orelse=[], finalbody=[]), With(items=[], body=[Pass()], handlers=[], orelse=[], finalbody=[Pass()])])
That's one With statement with no except/else/finally, and one With statement with a finally, just as you'd expect.
(Of course the compiler just ignores the except/else/finally bits.)
On 04/16/2015 12:22 AM, Andrew Barnert wrote:
Why does the finally come before the except? Also, it's still indented with the with, which means you seem to be defining with-except-finally in terms of with-finally, and not defining with-finally at all. Did you mean to have the finally after the except and dedented?
Oops, my bad. I did mean to put the finally-block after the except-block:
try: with open('spam.txt') as file: print(file.read()) except IOError: print('No spam here...') finally: print('done.')
Is that any more ambiguous than for...else?
On Thu, Apr 16, 2015 at 2:22 AM, Andrew Barnert < abarnert@yahoo.com.dmarc.invalid> wrote:
On Apr 15, 2015, at 23:39, Kale Kundert kale@thekunderts.net wrote:
Hello everyone,
I'd like to propose a little bit of syntactic sugar: allowing
with-blocks to be
followed by except- and finally-blocks just like try-blocks. For
example:
with open('spam.txt') as file: print(file.read()) except IOError: print('No spam here...') finally: print('done.')
This proposed syntax is semantically equivalent to wrapping a with-block
within
a try-block like so:
try: with open('spam.txt') as file: print(file.read()) finally: print('done.') except IOError: print('No spam here...')
Why does the finally come before the except? Also, it's still indented with the with, which means you seem to be defining with-except-finally in terms of with-finally, and not defining with-finally at all. Did you mean to have the finally after the except and dedented?
Anyway, I like the idea, but does the grammar work? A with that has an optional except and finally at the same level seems potentially ambiguous. For example:
with spam: pass with eggs: pass finally: passIf you think about it, it's pretty obvious that the finally goes with the second with, not the first. But can the parser tell that? Or a human who's scanning the code rather than thinking carefully about it? Or tools like auto-indenters?
(I think this problem doesn't arise for try, because it must have at least except or finally.)
It should be easy to edit the grammar and build the parser to at least the first part of the question; for the last part, trying to hack up a few different external tools like Emacs python-mode. But for the human question, I'm not sure how to answer it quite so easily...
I see two advantages to the proposed syntax. First and most obviously,
it saves
an extra line and an extra indentation level. One line may not be a big
deal,
but one indentation level can really affect readability. Second and more conceptually, it makes sense to think about exception handling in the
context of
a with-block. More often than not, if you're using a with-block, you're expecting that something in that block could throw an exception. Usually with-blocks are used to make sure resources (e.g. files, database
sessions,
mutexes, etc.) are properly closed before the exception is propogated.
But I
very often want to do some custom clean-up as well (alert the user,
etc.).
Currently that requires wrapping the whole thing in a try-block, but
allowing
with-blocks to behave as try-blocks is a more direct way to express what
is meant.
I was curious how often with-blocks are actually wrapped in try-blocks
for no
other purpose than catching exceptions raised in the with-block. So I
searched
through a number of open source projects looking (roughly) for that
pattern:
Project with [1] try-with [2] ============== ======== ============ django 230 17 ipython 541 8 matplotlib 112 3 moinmoin 10 0 numpy 166 1 pillow/pil 1 0 pypy 254 4 scipy 163 2 sqlalchemy 36 0 twisted 72 1 ============== ======== ============ total 1585 36 (2.27%)
[1]: grep -Po '^\s*with .*:' **/*.py [2]: grep -Poz 'try:\s*with .*:' **/*.py
Assuming these projects are representative, about 2% of the with-blocks
are
directly wrapped by try-blocks. That's not a huge proportion, but it
clearly
shows that this pattern is being used "in the wild". Whether or not
it's worth
changing the language for the benefit of 2% of with-blocks is something
to
debate though.
What do people think of this idea?
-Kale Kundert
Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On Wed, Apr 15, 2015 at 11:39:37PM -0700, Kale Kundert wrote:
Hello everyone,
I'd like to propose a little bit of syntactic sugar: allowing with-blocks to be followed by except- and finally-blocks just like try-blocks. For example:
with open('spam.txt') as file: print(file.read()) except IOError: print('No spam here...') finally: print('done.')
This has two obvious, and slightly different, interpretations:
with open('spam.txt') as file: try: print(file.read()) except IOError: print('No spam here...') finally: print('done.')
try: with open('spam.txt') as file: print(file.read()) except IOError: print('No spam here...') finally: print('done.')
I'm not even going to ask you which of the two you intended, because whichever one you pick, half the time people will want the other. And of those times, half the time they will end up using this syntactic sugar anyway, and then be disturbed when it doesn't quite work the way they expected.
To put it another way, I expect this to be an oh-so-subtle bug magnet.
This proposed syntax is semantically equivalent to wrapping a with-block within a try-block like so:
try: with open('spam.txt') as file: print(file.read()) finally: print('done.') except IOError: print('No spam here...')
You can't describe "with...finally" in terms of "with...finally", it gives a RecursionError :-)
I assume this is an editing mistake, and you intended the finally clause to follow the except clause.
I see two advantages to the proposed syntax. First and most obviously, it saves an extra line and an extra indentation level. One line may not be a big deal, but one indentation level can really affect readability.
It certainly can affect readability, but not in the way you mean. The loss of that indentation level *hurts* readability, it doesn't help it, because it obscures the structure of the code.
Either the try block encloses the with, or the with encloses the try. They are not part of the same structure in the way that "if...elif... else" or "for...else" are.
[Aside: in the second case, the for-loop, "else" is not the best choice of keyword. The semantics are more like "then", but we are probably stuck with it now.]
Given either version:
with spam: try: X except: ...
try: with spam: X except: ...
the fact that X is indented two levels is a good thing, not a bad thing. It makes the structure of the code immediately visible, and there is absolutely no question of whether the with is enclosing the try or the try is enclosing the with.
Existing Python statements which introduce blocks strictly keep to the rule that things which are semantically at different levels are indented differently. (The exception is the single line form of some statements, e.g. "if condition: pass" on a single line.) We never write things like:
if condition: for x in sequence: do_this()
try: while flag: do_this() except Error: handle_error()
def func(arg): with func(arg) as spam: return spam.thing()
even in the cases where they wouldn't be syntactically ambiguous. Such things are strictly disallowed, and that is good.
Second and more conceptually, it makes sense to think about exception handling in the context of a with-block.
No more than any other block of code. There is nothing special about with blocks that go with exception handling, any more than (say) indexing into a list, processing a dict, or importing a module.
More often than not, if you're using a with-block, you're expecting that something in that block could throw an exception.
"More often than not"? Not according to your own investigation, where you found that only two percent of the time with blocks are associated with try blocks. That means that in the overwhelming majority of cases, nearly 98% of the time, we are NOT catching exceptions in the with-block.
That means that *at best* this syntactic sugar will be of little value, and at worst it will be actively harmful. I think it will be actively harmful, and give it a strong -1 vote.
On Apr 16, 2015, at 06:09, Steven D'Aprano steve@pearwood.info wrote:
Second and more conceptually, it makes sense to think about exception handling in the context of a with-block.
No more than any other block of code. There is nothing special about with blocks that go with exception handling, any more than (say) indexing into a list, processing a dict, or importing a module.
I think the key insight behind this is that a context manager is in some sense equivalent to a try/finally--and can in fact be implemented that way using @contextmanager. That isn't true for any of the other cases.
Which also implies that there is "one true answer" to your initial question; the implicit try-equivalent comes after the __enter__.
Nevertheless, I see your point, and I think you're right. This interpretation is clearly not obvious and universal (given that you didn't see it that way...). And anyone who _doesn't_ intuitively equate the context manager to the equivalent a try/finally is just as likely to expect either semantics as the other, and therefore it will be a bug magnet and/or a readability issue. (To use it safely, it's not sufficient for _you_ to understand the intended semantics, but for all of your possible readers, after all.)
On Thu, Apr 16, 2015 at 06:25:37AM -0700, Andrew Barnert wrote:
On Apr 16, 2015, at 06:09, Steven D'Aprano steve@pearwood.info wrote:
Second and more conceptually, it makes sense to think about exception handling in the context of a with-block.
No more than any other block of code. There is nothing special about with blocks that go with exception handling, any more than (say) indexing into a list, processing a dict, or importing a module.
I think the key insight behind this is that a context manager is in some sense equivalent to a try/finally--and can in fact be implemented that way using @contextmanager. That isn't true for any of the other cases.
You are correct that with... statements are, in a sense, equivalent to try...finally. That's hardly a secret or an insight -- I too have read the docs and the PEP :-) But I think it is irrelevant to this question. Anything we write might contain an internal try...finally. Consider:
spam()
It is, I hope, obvious that there is nothing special about spam that relates it to try...finally more than any other chunk of code. But what if I told you that *inside* spam there was a try...finally clause?
def spam(): try: eggs() finally: cheese()
I hope that you wouldn't change your mind and decide that spam was special and we should try to create syntactic sugar:
spam() finally: tomato()
because "spam is in some sense equivalent to a try/finally".
Now substitute with statements for spam. Yes, "with..." is fundamentally linked to a finally block. It's part of the definition of the with statement. But that finally block is controlled by the context manager, not us -- as users of the with statement, we don't get to control what happens in the finally clause unless we write our own try...finally block. Exactly the same as spam and its internal finally clause.
With the proposed syntax:
with something(): block finally: print("Done")
there are conceptually two independent finally blocks. There is the finally block hidden inside the context manager's __exit__ method, which is part of the with statement. As a consumer of the CM, we don't have any control over that. And there is our finally block, which prints "Done", which is independent of the context manager and under our control.
That raises another interesting question. Which finally clause will trigger first? With current syntax, the block structure is explicit and obvious:
with something(): try: block finally: print("Done")
"Done" is printed before the __exit__ method is called, because that finally clause is part of the with block. Whereas:
try: with something(): block finally: print("Done")
in this case, the __exit__ method is called first. The block structure makes that so obvious I don't even need to test it to be sure that it must be the case. (Famous last words!)
On Apr 16, 2015, at 09:25, Steven D'Aprano steve@pearwood.info wrote:
On Thu, Apr 16, 2015 at 06:25:37AM -0700, Andrew Barnert wrote: On Apr 16, 2015, at 06:09, Steven D'Aprano steve@pearwood.info wrote:
Second and more conceptually, it makes sense to think about exception handling in the context of a with-block.
No more than any other block of code. There is nothing special about with blocks that go with exception handling, any more than (say) indexing into a list, processing a dict, or importing a module.
I think the key insight behind this is that a context manager is in some sense equivalent to a try/finally--and can in fact be implemented that way using @contextmanager. That isn't true for any of the other cases.
You are correct that with... statements are, in a sense, equivalent to try...finally. That's hardly a secret or an insight -- I too have read the docs and the PEP :-) But I think it is irrelevant to this question. Anything we write might contain an internal try...finally. Consider:
spam()
It is, I hope, obvious that there is nothing special about spam that relates it to try...finally more than any other chunk of code. But what if I told you that *inside* spam there was a try...finally clause?
def spam(): try: eggs() finally: cheese()
That's clearly different. There is nothing inherent to function call syntax that relates to finally; there is something inherent to a with statement that does. You might just as well argue that there's nothing special about for blocks that go with iteration because any function definition may have a call to iter inside.
On top of that, inside a function is an entirely different scope; what happens inside a function is in many senses hidden or isolated from syntax. For example, compare this:
for i in range(10): if i == 3: break else: print('broke')
for i in range(10): if i == 3: eggs() else: print('broke')
If you told me there was a break inside eggs, it wouldn't make any difference. A call to eggs is still not equivalent to a break in this scope, just as a call to spam is still not equivalent to a finally in this scope.
I hope that you wouldn't change your mind and decide that spam was special and we should try to create syntactic sugar:
spam() finally: tomato()
because "spam is in some sense equivalent to a try/finally".
No more than I'd decide that eggs is special.
You could, of course, argue that spam is a different case from eggs because the one exception to scope locality is exceptions, and with statements, unlike break, are fundamentally about exceptions... But that requires assuming exactly the point you're trying to disprove.
Again, I agree with you that this proposal is probably a bad idea. But this argument against it doesn't work.
On 04/16, Steven D'Aprano wrote:
On Wed, Apr 15, 2015 at 11:39:37PM -0700, Kale Kundert wrote:
I'd like to propose a little bit of syntactic sugar: allowing with-blocks to be followed by except- and finally-blocks just like try-blocks. For example:
-1
'with' is highly specialized, and trying to make it more generic will be confusing.
Second and more conceptually, it makes sense to think about exception handling in the context of a with-block.
No more than any other block of code. There is nothing special about with blocks that go with exception handling [...]
I have to disagree: 'with' blocks are exactly a condensed try/finally, with the added capability of being able to suppress exceptions -- this makes them very special with respect to exception handling.
-- ~Ethan~
On Thu, Apr 16, 2015 at 08:35:14AM -0700, Ethan Furman wrote:
No more than any other block of code. There is nothing special about with blocks that go with exception handling [...]
I have to disagree: 'with' blocks are exactly a condensed try/finally, with the added capability of being able to suppress exceptions -- this makes them very special with respect to exception handling.
Yes, yes, that's a given that with statements encapsulate a finally clause. But that is missing the point. The proposal is not to add a finally clause to context managers, because they already have one. The proposal is to allow special syntax for the situation where you want a separate and independent finally.
with spam(): try: blockA except Error: blockB finally: blockC
The finally: blockC has nothing to do with the context manager's __exit__ method. We might as well be talking about this:
while spam(): try: blockA except Error: blockB finally: blockC
giving us syntactic sugar:
# not a serious proposal while spam(): blockA except Error: blockB finally: blockC
In fact, if I were a betting man, I'd venture a small wager that there are more while- or for-loops with try blocks inside them than with blocks. After all, the whole purpose of with statements is to avoid needing to write a try...finally!
On 2015-04-16 17:38, Steven D'Aprano wrote:
On Thu, Apr 16, 2015 at 08:35:14AM -0700, Ethan Furman wrote:
No more than any other block of code. There is nothing special about with blocks that go with exception handling [...]
I have to disagree: 'with' blocks are exactly a condensed try/finally, with the added capability of being able to suppress exceptions -- this makes them very special with respect to exception handling.
Yes, yes, that's a given that with statements encapsulate a finally clause. But that is missing the point. The proposal is not to add a finally clause to context managers, because they already have one. The proposal is to allow special syntax for the situation where you want a separate and independent finally.
with spam(): try: blockA except Error: blockB finally: blockC
The finally: blockC has nothing to do with the context manager's __exit__ method. We might as well be talking about this:
while spam(): try: blockA except Error: blockB finally: blockC
giving us syntactic sugar:
# not a serious proposal while spam(): blockA except Error: blockB finally: blockC
If I saw that I'd expect it to mean:
try: while spam(): blockA except Error: blockB finally: blockC
In fact, if I were a betting man, I'd venture a small wager that there are more while- or for-loops with try blocks inside them than with blocks. After all, the whole purpose of with statements is to avoid needing to write a try...finally!
On 04/17, Steven D'Aprano wrote:
On Thu, Apr 16, 2015 at 08:35:14AM -0700, Ethan Furman wrote:
Steven D'Aprano wrote:
No more than any other block of code. There is nothing special about with blocks that go with exception handling [...]
I have to disagree: 'with' blocks are exactly a condensed try/finally, with the added capability of being able to suppress exceptions -- this makes them very special with respect to exception handling.
Yes, yes, that's a given that with statements encapsulate a finally clause. But that is missing the point.
No, that's entirely the point. You claimed:
There is nothing special about with blocks that go with exception handling
when in fact exception handling (even if just ignoring) is half their reason for existence.
It was an unfortunate choice of words, not a big deal. Nobody who has called you on it has suggested the proposal is a win for Python syntax.
-- ~Ethan~
Steven D'Aprano steve@pearwood.info wrote:
This has two obvious, and slightly different, interpretations:
with open('spam.txt') as file: try: print(file.read()) except IOError: print('No spam here...') finally: print('done.')
try: with open('spam.txt') as file: print(file.read()) except IOError: print('No spam here...') finally: print('done.')
Sometimes I want the first, sometimes the second. And I expect a conscious mind-reading interpreter which gives me the one I want. If the interpreter in retrospect thinks it has given me the wrong one it should raise SorryError.
Sturla
participants (8)
-
Andrew Barnert -
Ethan Furman -
James Edwards -
Kale Kundert -
MRAB -
Ryan Gonzalez -
Steven D'Aprano -
Sturla Molden