Potential PEP: with/except
I've found that almost any time I'm writing a 'with' block, it's doing something that could throw an exception. As a result, each of those 'with' blocks needs to be nested within a 'try' block. Due to the nature of 'with', it is rarely (if ever) the case that the try block contains anything other than the with block itself. As a result, I would like to propose that the syntax for 'with' blocks be changed such that they can be accompanied by 'except', 'finally', and/or 'else' blocks as per a standard 'try' block. These would handle exceptions that occur in the 'with' block, including the execution of the applicable __enter__ and __exit__ methods. Example: try: with open(path) as myfile: ... # Do stuff with file except (OSError, IOError) as err: logger.error("Failed to read/open file {}: {}".format(path, err) The above would turn into simply: with open(path) as myfile: ... # Do stuff with file except (OSError, IOError) as err: logger.error(...) I think this is rather straightforward in meaning and easy to read, and simplifies some unnecessary nesting. I see this as the natural evolution of what 'with' is all about - replacing necessary try-finally blocks with something more elegant. We just didn't include the 'except' portion. I'm a bit hesitant to put this out there. I'm not worried about it getting shot down - that's kind of the point here. I'm just pretty strongly against to unnecessary syntactical additions to the language. This though, I think I can except. It introduces no new concepts and requires no special knowledge to use. There's no question about what is going on when you read it. -- Paul Ferrell pflarr@gmail.com
On Tue, Jan 22, 2019 at 3:11 PM Paul Ferrell <pflarr@gmail.com> wrote:
It definitely makes sense, both the problem and the proposed solution. The thing that concerns me is that any such problem and solution seems to apply equally to any other kind of block. Why not allow excepts on fo loops, for example?
-- CALVIN SPEALMAN SENIOR QUALITY ENGINEER cspealma@redhat.com M: +1.336.210.5107 <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
[Calvin Spealman]
Why not allow excepts on fo loops, for example?
Why not, indeed... I've heard there's a non-insignificant performance penalty for setting up a try statement, so it might be important to only set a for-loop up as a guarded for-loop upon reading the "except" statement (if the compiler can handle such behavior). On Tue, Jan 22, 2019 at 2:24 PM Calvin Spealman <cspealma@redhat.com> wrote:
On Wed, Jan 30, 2019 at 08:27:41PM -0600, Abe Dillon wrote:
I believe you have been misinformed. I admit I haven't tried it recently, but back in Python 2.5 days or so I ran some benchmarks which satisfied me that: - the cost of setting up a try block was effectively zero; - the cost of catching an exception is quite steep. I'd be surprised if Python 3 reverses that. -- Steven
On Tue, Jan 22, 2019, 12:11 PM Paul Ferrell <pflarr@gmail.com wrote:
The time machine strikes again. In fact, you can handle exceptions with a context manager object. Whatever you're with-ing must have a dunder exit method, which received any exceptions raised in the block as an argument. Return true and the exception is suppressed.
On the whole chain on context managers: I'm aware that 'exit' gets the exceptions raised, which is great in certain situations that are specific to the object. However, the common case is specific to the _usage_ of the object. Even if it were all just one type of object, like opening files, there are many different situations in which I need to handle the errors. Some may just need to be logged, some ignored, some may need to attempt a retry, some should cause a complete failure. Writing custom context managers for all of the different cases is the opposite of the desired result of slightly cleaner, less redundant code. Then consider I'm often using many different context managed objects in a code base, often simultaneously. This idea hit me because I keep running into the try/with/except pattern so often, and in so many different circumstances. On Tue, Jan 22, 2019 at 1:31 PM Michael Selik <mike@selik.org> wrote:
-- Paul Ferrell pflarr@gmail.com
On Wed, Jan 23, 2019 at 7:11 AM Paul Ferrell <pflarr@gmail.com> wrote:
Edge case: The "try/with/except" structure includes the entire 'with' header inside the try block, including the call to open(). But if the with block itself is handling the exceptions, the expression "open(path)" is actually evaluated before the exception handling gets going. So adding an except clause is NOT the same as just adding another context manager to the stack. I'm -0.25 on it, as I don't think overlaying in this way improves clarity. The current syntax makes it very obvious that the "open(path)" call is inside the try/except, but the proposed syntax isn't so clear (and as many people will expect it to be inside as outside). ChrisA
That is definitely an ambiguity worth considering (whether __enter__ is within the implied 'try'). Anecdotally, I showed the with/except example to my student (who's relatively new to python), to see how he interpreted it. He (correctly?) assumed the CM operations were within the 'try', and was pretty surprised when I told him that the except part of the with wasn't actually valid Python. On Tue, Jan 22, 2019 at 2:30 PM Chris Angelico <rosuav@gmail.com> wrote:
-- Paul Ferrell pflarr@gmail.com
On Wed, Jan 23, 2019 at 9:23 AM Paul Ferrell <pflarr@gmail.com> wrote:
That is definitely an ambiguity worth considering (whether __enter__ is within the implied 'try').
It's not even __enter__.
3 10 LOAD_GLOBAL 1 (print) 12 LOAD_CONST 1 ('Got f:') 14 LOAD_FAST 1 (f) 16 CALL_FUNCTION 2 18 POP_TOP 20 POP_BLOCK 22 BEGIN_FINALLY >> 24 WITH_CLEANUP_START 26 WITH_CLEANUP_FINISH 28 END_FINALLY 30 LOAD_CONST 0 (None) 32 RETURN_VALUE
At the time when "open(path)" is called, the 'with' block hasn't begun operating yet. The SETUP_WITH operation will call __enter__, but prior to that, we have to have an object to use as the context manager. Any exception thrown by opening the file will happen before __enter__ gets called. AIUI the __enter__ method of file objects doesn't actually do anything much (just validates that it's still open). ChrisA
On Tue, Jan 22, 2019 at 03:22:27PM -0700, Paul Ferrell wrote:
One of the more pernicious myths about language design is that if something surprises a beginner, it must be a bad idea. The reality is that beginners are the worst people to judge what is good or bad or consistent, because they don't have the knowledge or experience to recognise deep consistency or flaws in an feature. I'm just making a general observation here, not making a specific claim that this specific proposal is flawed (beyond the point I made earlier that it may be unnecessary and redundant). But anecdotes about "beginners were surprised by..." don't carry much weight with me. (Not "zero weight", it isn't as if I *want* to surprise beginners, and sometimes newcomers to a language can spot things which experts are so used to they don't notice any longer.) -- Steve
The first concern that comes to my mind is... When I see: with: ... except: ... Is that a shorthand for try: with: ... except: ... or for with: try: ... except: ... ? Both are plausible, and it makes a big difference, because 'with' already has an implicit 'except' block built in. -n On Tue, Jan 22, 2019, 12:12 Paul Ferrell <pflarr@gmail.com wrote:
On Tue, Jan 22, 2019 at 01:11:10PM -0700, Paul Ferrell wrote: [...]
What benefit does this give apart from saving one line and one indent? If either is in short supply, the code probably needs refactoring, not new syntax. The beauty of the current syntax is that try...except and with blocks are fully independent, composable blocks which can be learned and reasoned about seperately. You're proposing to add a new special-case syntax: while ... except ... that adds a new block structure that has to be implemented, documented, tested, maintained, taught and learned. It will inevitably lead to questions on mailing lists, IRC and Stackoverflow asking what is the difference between a separate try...with...except and a with...except, and when to choose one or the other. And of course then there will be the inevitable requests that we generalise it to other blocks: for ... except ... while ... except ... If this will allow us to write more expressive code, or do things we couldn't easily do before, then it might be worthwhile to add this additional complexity. But if all it does is save one line and one indent, then I believe it is redundant and I would be against it. -- Steve
I completely understand your perspective, and agree with most of it. It doesn't add new expressiveness, it adds a bit of polish (and I think completeness) to the relatively new concept of 'with' statements. Is this so intuitive that we don't actually have to teach it? Is it such a natural extension to 'with', that it would immediately be weird to find it missing in the future? If the answer to either of those questions is 'no', then I absolutely retract my idea. On Tue, Jan 22, 2019 at 3:22 PM Steven D'Aprano <steve@pearwood.info> wrote:
-- Paul Ferrell pflarr@gmail.com
I've been thinking more about this proposal, and realised why I've been feeling a slight sense of disquiet about it. I think it encourages an anti-pattern of catching too much. (Or at least a code smell.) Although we're all guilty of violating this principle from time to time, in general we ought to surround the minimum amount of code with a try...except that we need. Ideally (but rarely possible in practice) we want to surround a single operation which might raise at a time. Otherwise, we risk this sort of failure: try: n = len(sequence) result = process(n) except TypeError: # we implicitly assume that the ONLY source of # TypeError is calling len(sequence) handle_iterator(sequence) But what if process(n) itself raises TypeError? Perhaps because it takes two mandatory arguments, not one, and we've just hidden a bug in our code. Now obviously this specific example is just a toy, but the principle applies. When I see code like: try: with spam(arg) as x: block except SomeException: # implicitly assume that spam(arg) is the only # thing which can fail handle_failure_in_spam() what I see is a try block which may be too greedy, possibly hiding bugs in the code. So what we probably *actually* want is: try: tmp = spam(arg) except SomeException: handle_failure_in_spam() with tmp as x: block but who can be bothered writing that? At least until they've been bitten by the failure to do so. Given this: with spam(arg) as x: block except: ... what is the scope of this with...except clause? * just the call to spam(arg) - the call to spam(arg) and the call to x.__enter__ - spam(arg), x.__enter__ and the entire with block - just the call to x.__enter__ - just the block Visually, the beauty of the try...except syntax is that there is nothing else happening on the try line. The try statement is purely a delimiter, and it is *only* the indented block below it which is guarded. But the "with..." line in this proposal acts as both delimiter and code, and so it is ambiguous whether we want the delimiter to come before or after the code: try with block except with try block except try with except block Logically, I don't want this to guard the block. Doing so guards too much: it is bad enough when I'm lazy and surround the entire with statement in a single try...except, I don't want the language providing me a feature specifically to encourage me to do it. But visually, I would *never* guess that the block was not guarded by the with...except clause. Logically, I don't want it to cover the body of the with statement, but I hate to imagine having to explain to people why it doesn't. But the alternative is to enshrine in syntax something which *by design* guards too much and is a code smell. -- Steve
I have a neutral feeling about the proposal but I’d like to suggest something We can extend the try/with to other blocks, as suggested. What could be done to prevent any ambiguity is : try with blabla as blabla2 : ... except: ... Which is equivalent of : try: with blabla as blabl2: ... except: .... Any other combination should be explicit, except if there’s a nice syntax but I didn’t find it. Extending to other blocks would give : try for ... in ...: ... except: ... But if we use the « else » here, what’s going on ? It’s a one line saver (which is useless) but having many indented blocks doesn’t produce readable code in my opinion. Saving one indented block while keeping things clear is a good thing in my opinion. As long as it stays clear (which is not the case in the for block). But adding this feature to the with block without using it for other blocks is a bit strange, imho.
On Jan 22, 2019, at 8:28 PM, Steven D'Aprano <steve@pearwood.info> wrote:
I ended up at a similar conclusion this morning. I'm wary of enclosing too much code in the try block since it makes handling exceptions very difficult unless you have a rich hierarchy of exceptions -- something that I also warn against. I view the problem of enclosing too much in the try block as an extension of an overly broad except block. It makes it nearly impossible to know where the exception came from so you cannot handle it safely unless you have an exception hierarchy where each exception is raised from a single place in the code. A similar anti-pattern that I see regularly is to *catch too often* where there is a try-catch block every time that a method from another module or library is called and the catch portion translates the exception instead of handling it. It's interesting that the "exception translator" pattern leads to an overly rich exception hierarchy. It is interesting that you mention whether the try-catch would wrap __enter__ or not. The main reason that I am -1 on this proposal is that it introduces more ambiguity in what is happening. I was about to write pretty much what you have written with regards to the syntax muddying the waters about the scope of exception handling especially with regards to the meaning of the return value of `context_manager.__exit__`
But the alternative is to enshrine in syntax something which *by design* guards too much and is a code smell.
Very well put ;) - dave -- Safe wiring is not something to be learned after the fire trucks have left.
On Tue, Jan 22, 2019 at 3:11 PM Paul Ferrell <pflarr@gmail.com> wrote:
It definitely makes sense, both the problem and the proposed solution. The thing that concerns me is that any such problem and solution seems to apply equally to any other kind of block. Why not allow excepts on fo loops, for example?
-- CALVIN SPEALMAN SENIOR QUALITY ENGINEER cspealma@redhat.com M: +1.336.210.5107 <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
[Calvin Spealman]
Why not allow excepts on fo loops, for example?
Why not, indeed... I've heard there's a non-insignificant performance penalty for setting up a try statement, so it might be important to only set a for-loop up as a guarded for-loop upon reading the "except" statement (if the compiler can handle such behavior). On Tue, Jan 22, 2019 at 2:24 PM Calvin Spealman <cspealma@redhat.com> wrote:
On Wed, Jan 30, 2019 at 08:27:41PM -0600, Abe Dillon wrote:
I believe you have been misinformed. I admit I haven't tried it recently, but back in Python 2.5 days or so I ran some benchmarks which satisfied me that: - the cost of setting up a try block was effectively zero; - the cost of catching an exception is quite steep. I'd be surprised if Python 3 reverses that. -- Steven
On Tue, Jan 22, 2019, 12:11 PM Paul Ferrell <pflarr@gmail.com wrote:
The time machine strikes again. In fact, you can handle exceptions with a context manager object. Whatever you're with-ing must have a dunder exit method, which received any exceptions raised in the block as an argument. Return true and the exception is suppressed.
On the whole chain on context managers: I'm aware that 'exit' gets the exceptions raised, which is great in certain situations that are specific to the object. However, the common case is specific to the _usage_ of the object. Even if it were all just one type of object, like opening files, there are many different situations in which I need to handle the errors. Some may just need to be logged, some ignored, some may need to attempt a retry, some should cause a complete failure. Writing custom context managers for all of the different cases is the opposite of the desired result of slightly cleaner, less redundant code. Then consider I'm often using many different context managed objects in a code base, often simultaneously. This idea hit me because I keep running into the try/with/except pattern so often, and in so many different circumstances. On Tue, Jan 22, 2019 at 1:31 PM Michael Selik <mike@selik.org> wrote:
-- Paul Ferrell pflarr@gmail.com
On Wed, Jan 23, 2019 at 7:11 AM Paul Ferrell <pflarr@gmail.com> wrote:
Edge case: The "try/with/except" structure includes the entire 'with' header inside the try block, including the call to open(). But if the with block itself is handling the exceptions, the expression "open(path)" is actually evaluated before the exception handling gets going. So adding an except clause is NOT the same as just adding another context manager to the stack. I'm -0.25 on it, as I don't think overlaying in this way improves clarity. The current syntax makes it very obvious that the "open(path)" call is inside the try/except, but the proposed syntax isn't so clear (and as many people will expect it to be inside as outside). ChrisA
That is definitely an ambiguity worth considering (whether __enter__ is within the implied 'try'). Anecdotally, I showed the with/except example to my student (who's relatively new to python), to see how he interpreted it. He (correctly?) assumed the CM operations were within the 'try', and was pretty surprised when I told him that the except part of the with wasn't actually valid Python. On Tue, Jan 22, 2019 at 2:30 PM Chris Angelico <rosuav@gmail.com> wrote:
-- Paul Ferrell pflarr@gmail.com
On Wed, Jan 23, 2019 at 9:23 AM Paul Ferrell <pflarr@gmail.com> wrote:
That is definitely an ambiguity worth considering (whether __enter__ is within the implied 'try').
It's not even __enter__.
3 10 LOAD_GLOBAL 1 (print) 12 LOAD_CONST 1 ('Got f:') 14 LOAD_FAST 1 (f) 16 CALL_FUNCTION 2 18 POP_TOP 20 POP_BLOCK 22 BEGIN_FINALLY >> 24 WITH_CLEANUP_START 26 WITH_CLEANUP_FINISH 28 END_FINALLY 30 LOAD_CONST 0 (None) 32 RETURN_VALUE
At the time when "open(path)" is called, the 'with' block hasn't begun operating yet. The SETUP_WITH operation will call __enter__, but prior to that, we have to have an object to use as the context manager. Any exception thrown by opening the file will happen before __enter__ gets called. AIUI the __enter__ method of file objects doesn't actually do anything much (just validates that it's still open). ChrisA
On Tue, Jan 22, 2019 at 03:22:27PM -0700, Paul Ferrell wrote:
One of the more pernicious myths about language design is that if something surprises a beginner, it must be a bad idea. The reality is that beginners are the worst people to judge what is good or bad or consistent, because they don't have the knowledge or experience to recognise deep consistency or flaws in an feature. I'm just making a general observation here, not making a specific claim that this specific proposal is flawed (beyond the point I made earlier that it may be unnecessary and redundant). But anecdotes about "beginners were surprised by..." don't carry much weight with me. (Not "zero weight", it isn't as if I *want* to surprise beginners, and sometimes newcomers to a language can spot things which experts are so used to they don't notice any longer.) -- Steve
The first concern that comes to my mind is... When I see: with: ... except: ... Is that a shorthand for try: with: ... except: ... or for with: try: ... except: ... ? Both are plausible, and it makes a big difference, because 'with' already has an implicit 'except' block built in. -n On Tue, Jan 22, 2019, 12:12 Paul Ferrell <pflarr@gmail.com wrote:
On Tue, Jan 22, 2019 at 01:11:10PM -0700, Paul Ferrell wrote: [...]
What benefit does this give apart from saving one line and one indent? If either is in short supply, the code probably needs refactoring, not new syntax. The beauty of the current syntax is that try...except and with blocks are fully independent, composable blocks which can be learned and reasoned about seperately. You're proposing to add a new special-case syntax: while ... except ... that adds a new block structure that has to be implemented, documented, tested, maintained, taught and learned. It will inevitably lead to questions on mailing lists, IRC and Stackoverflow asking what is the difference between a separate try...with...except and a with...except, and when to choose one or the other. And of course then there will be the inevitable requests that we generalise it to other blocks: for ... except ... while ... except ... If this will allow us to write more expressive code, or do things we couldn't easily do before, then it might be worthwhile to add this additional complexity. But if all it does is save one line and one indent, then I believe it is redundant and I would be against it. -- Steve
I completely understand your perspective, and agree with most of it. It doesn't add new expressiveness, it adds a bit of polish (and I think completeness) to the relatively new concept of 'with' statements. Is this so intuitive that we don't actually have to teach it? Is it such a natural extension to 'with', that it would immediately be weird to find it missing in the future? If the answer to either of those questions is 'no', then I absolutely retract my idea. On Tue, Jan 22, 2019 at 3:22 PM Steven D'Aprano <steve@pearwood.info> wrote:
-- Paul Ferrell pflarr@gmail.com
I've been thinking more about this proposal, and realised why I've been feeling a slight sense of disquiet about it. I think it encourages an anti-pattern of catching too much. (Or at least a code smell.) Although we're all guilty of violating this principle from time to time, in general we ought to surround the minimum amount of code with a try...except that we need. Ideally (but rarely possible in practice) we want to surround a single operation which might raise at a time. Otherwise, we risk this sort of failure: try: n = len(sequence) result = process(n) except TypeError: # we implicitly assume that the ONLY source of # TypeError is calling len(sequence) handle_iterator(sequence) But what if process(n) itself raises TypeError? Perhaps because it takes two mandatory arguments, not one, and we've just hidden a bug in our code. Now obviously this specific example is just a toy, but the principle applies. When I see code like: try: with spam(arg) as x: block except SomeException: # implicitly assume that spam(arg) is the only # thing which can fail handle_failure_in_spam() what I see is a try block which may be too greedy, possibly hiding bugs in the code. So what we probably *actually* want is: try: tmp = spam(arg) except SomeException: handle_failure_in_spam() with tmp as x: block but who can be bothered writing that? At least until they've been bitten by the failure to do so. Given this: with spam(arg) as x: block except: ... what is the scope of this with...except clause? * just the call to spam(arg) - the call to spam(arg) and the call to x.__enter__ - spam(arg), x.__enter__ and the entire with block - just the call to x.__enter__ - just the block Visually, the beauty of the try...except syntax is that there is nothing else happening on the try line. The try statement is purely a delimiter, and it is *only* the indented block below it which is guarded. But the "with..." line in this proposal acts as both delimiter and code, and so it is ambiguous whether we want the delimiter to come before or after the code: try with block except with try block except try with except block Logically, I don't want this to guard the block. Doing so guards too much: it is bad enough when I'm lazy and surround the entire with statement in a single try...except, I don't want the language providing me a feature specifically to encourage me to do it. But visually, I would *never* guess that the block was not guarded by the with...except clause. Logically, I don't want it to cover the body of the with statement, but I hate to imagine having to explain to people why it doesn't. But the alternative is to enshrine in syntax something which *by design* guards too much and is a code smell. -- Steve
I have a neutral feeling about the proposal but I’d like to suggest something We can extend the try/with to other blocks, as suggested. What could be done to prevent any ambiguity is : try with blabla as blabla2 : ... except: ... Which is equivalent of : try: with blabla as blabl2: ... except: .... Any other combination should be explicit, except if there’s a nice syntax but I didn’t find it. Extending to other blocks would give : try for ... in ...: ... except: ... But if we use the « else » here, what’s going on ? It’s a one line saver (which is useless) but having many indented blocks doesn’t produce readable code in my opinion. Saving one indented block while keeping things clear is a good thing in my opinion. As long as it stays clear (which is not the case in the for block). But adding this feature to the with block without using it for other blocks is a bit strange, imho.
On Jan 22, 2019, at 8:28 PM, Steven D'Aprano <steve@pearwood.info> wrote:
I ended up at a similar conclusion this morning. I'm wary of enclosing too much code in the try block since it makes handling exceptions very difficult unless you have a rich hierarchy of exceptions -- something that I also warn against. I view the problem of enclosing too much in the try block as an extension of an overly broad except block. It makes it nearly impossible to know where the exception came from so you cannot handle it safely unless you have an exception hierarchy where each exception is raised from a single place in the code. A similar anti-pattern that I see regularly is to *catch too often* where there is a try-catch block every time that a method from another module or library is called and the catch portion translates the exception instead of handling it. It's interesting that the "exception translator" pattern leads to an overly rich exception hierarchy. It is interesting that you mention whether the try-catch would wrap __enter__ or not. The main reason that I am -1 on this proposal is that it introduces more ambiguity in what is happening. I was about to write pretty much what you have written with regards to the syntax muddying the waters about the scope of exception handling especially with regards to the meaning of the return value of `context_manager.__exit__`
But the alternative is to enshrine in syntax something which *by design* guards too much and is a code smell.
Very well put ;) - dave -- Safe wiring is not something to be learned after the fire trucks have left.
participants (11)
-
Abe Dillon -
Adrien Ricocotam -
Barry Scott -
Calvin Spealman -
Chris Angelico -
David Mertz -
David Shawley -
Michael Selik -
Nathaniel Smith -
Paul Ferrell -
Steven D'Aprano