Python Idea - extension of 'with'
We are all familiar with the `with` statement for context managers, so that the resources are correctly finalized/closed in case of an exception either in terms of the resources being allocated of the processing. It is very common though to wrap the `with block` in an outer `try` block, since although we know that the resource has been closed, at an 'application' level it is still neccessary to deal with the exception - an example might be : try: with open('config.cfg', 'r') as cfg: # Process the open file config = load_config(cfg) except FileNotFound: logging.info('Config file not found - using default configuration') except PermissionError: logging.warning('Cannot open config .cfg - using default configuration') config = default_config() else: logging.info('Using config from config.cfg') It struck me that this is probably quite a common idiom, and we have the main processing (of the opened resources) indented twice just to accommodate the outer try block. I was wondering whether a worthwhile extension might be to allow the `with` statement to have an `except` and `else` clauses which would have the same semantics as wrapping the `with` block with a try - for example the above would now look like: with open('config.cfg', 'r') as cfg: # Process the open file config = load_config(cfg) except FileNotFound: logging.info('Config file not found - using default configuration') except PermissionError: logging.warning('Cannot open config .cfg - using default configuration') config = default_config() else: logging.info('Using config from config.cfg') Treating the 'with' as an implied `try` would reduce the march to the right - now the key processing of the resource is now indented only one level - and the association of the exception from the `with` block is syntactically clear. I am not good enough with core development to put together a working prototype, but I can imagine that this simple extension would be worth while, but I would be more than happy to author a PEP for this if it gets some initial positive feedback. Open questions - that I have - should we allow `except`, `else` and `finally` clauses (or just `except` and `else` - do we need `finally` here). -- Anthony Flury *Moble*: +44 07743 282707 *Home*: +44 (0)1206 391294 *email*: anthony.flury@btinternet.com <mailto:anthony.flury@btinternet.com>
On Thu, Apr 8, 2021, 11:39 AM anthony.flury via Python-ideas < python-ideas@python.org> wrote:
I like the concept, but I don't like just having a plain with block implicitly acting as a try block because you have to read further to actually understand that yes, you're catching exceptions here. What about "try with ...:"? The combination of the two keywords fits the "try-with-resources" pattern in some other languages and makes it explicit up front that exceptions are about to be caught, while keeping just one level of indentation.
I think I like it. Q. Safe to assume this would catch exceptions from both the call to `open` as well as the call to `open.__enter__`? Q. What about something even more verbose (descriptive) like`try with open(...) as cfg:`? 🙂 Paul On Thu, 2021-04-08 at 15:59 +0100, anthony.flury via Python-ideas wrote:
On Fri, Apr 9, 2021 at 1:38 AM anthony.flury via Python-ideas <python-ideas@python.org> wrote:
Normally I'd be -1 on proposals whose main benefit is "one fewer indentation level". We don't need complexity that serves no other purpose than that. But the with/try combination is somewhat special - the 'with' block is inherently about exception handling. But there's this big question: On Fri, Apr 9, 2021 at 2:24 AM Paul Bryan <pbryan@anode.ca> wrote:
Q. Safe to assume this would catch exceptions from both the call to `open` as well as the call to `open.__enter__`?
No, not safe to assume. It's equally reasonable to define it as guarding only the body of the statement, or as guarding the header as well. The semantics have to be locked in one way or the other, and half the time that's going to be incorrect. IMO this would be a bug magnet on the difference between "exceptions that get handed to __exit__" and "exceptions that get caught by the associated except clause". For instance, what happens in this situation: with database_transaction(conn) as self.trn: self.trn.execute("insert blah blah blah") else: with database_transaction(conn) as self.trn: self.trn.execute("update set blah blah blah") (Yes, I know a good few database engines have a robust upsert/merge operation, but bear with me here.) The else clause implies that its body will not be executed if any exception was raised. But which of these exceptions would prevent that else from happening? * NameError on database_transaction or conn * ConnectionLostError inside database_transaction() * MemoryError in the transaction's __enter__() * AttributeError assigning to self.trn * ResourceError during the __del__ of the previous trn (freebie - that should be ignored regardless) * KeyboardInterrupt after assigning to self.trn but before beginning the block * QueryConflictError inside execute() (another freebie - obviously this one should) * CommitFailedError in __exit__, after no other exception * RollbackFailedError in __exit__, after some other exception (And yes, RollbackFailed is almost a freebie as well, since I have an "else" and no "except" here, but that'd be different with some actual except clauses, so it's worth clarifying.) If it's defined as simply nesting the with inside a try, then the answer is "all of the above". But that would be inconsistent with the existing meaning, which wouldn't cover nearly as much. Similar question: What would be the semantics of this? with contextlib.suppress(BaseException): a = b / c except BaseException as e: print(e) What types of exception could be caught and what types couldn't? ChrisA
On 4/8/21 11:25 AM, Chris Angelico wrote:
Well, if every exception is derived from BaseException (they are) and contextlib.suppress(BaseException) suppresses all BaseException-derived exceptions (it does) then the semantics of the above are: - "a" will be the result of "b / c" if no exception occurs - otherwise, "a" will be whatever it was before the with-block - no exception will ever be caught by the except-clause Generally speaking, no exception that a context manager handles (i.e. suppresses) will ever be available to be caught. -- ~Ethan~
On Fri, Apr 9, 2021 at 6:16 AM Ethan Furman <ethan@stoneleaf.us> wrote:
What about NameError looking up contextlib, or AttributeError looking up suppress? Will they be caught by that except clause, or not? Thank you for making my point: your assumption is completely the opposite of the OP's, given the same syntactic structure. ChrisA
On 4/8/21 1:21 PM, Chris Angelico wrote:
On Fri, Apr 9, 2021 at 6:16 AM Ethan Furman wrote:
On 4/8/21 11:25 AM, Chris Angelico wrote:
Ah, good point -- those two would get caught, as they happen before contextlib.suppress() gets control.
Thank you for making my point: your assumption is completely the opposite of the OP's, given the same syntactic structure.
The guidance should be: `try with` behaves exactly as `try/except with a with`, meaning that NameError and AttributeError for those two reasons would still be caught. Yes, it's easy to miss that at first glance, but it's just as easy to miss in the traditional try/except layout: ```python try: with contextlib.suppress(BaseException): # do stuff except BaseException as e: print(e) ``` How many people are going to look at that and think, "oh, NameError and AttributeError can still be caught" ? -- ~Ethan~
On Fri, Apr 9, 2021 at 6:41 AM Ethan Furman <ethan@stoneleaf.us> wrote:
At least in this form, it's clear that there's a sharp distinction between the stuff around the outside of the 'with' block and the stuff inside it. The semantics, as suggested, give 'with' blocks two distinct exception-management scopes, which mostly but do not entirely overlap. It's not a problem to rigorously define the semantics, but as evidenced here, people's intuitions will vary depending on which context manager is being used. It's absolutely obvious that the OP's example should let you catch errors from open(), and equally obvious that suppressing BaseException should mean that nothing gets caught. That's the problem here. As such, I'm -0.5 on this. It's a kinda nice feature, but all it really offers is one less indentation level, and the semantics are too confusing. ChrisA
On 4/8/21 2:40 PM, Chris Angelico wrote:
I, on the other hand, would love for something along those lines -- I find it massively annoying to have to wrap a with-block, which is supposed to deal with exceptions, inside a try/except block, because the exceptions that `with` deals with are too narrowly focused. Of course, I also prefer def positive(val): "silly example" if val > 0: return True else: return False over def positive(val): "silly example" if val > 0: return True return False because the first more clearly says "either this happens, or that happens". Likewise, having the excepts line up with the `with` visually ties the code together. Granted, this mostly boils down to personal preference. -- ~Ethan~
On Fri, Apr 9, 2021 at 8:22 AM Ethan Furman <ethan@stoneleaf.us> wrote:
And MY preference there would be "return val > 0" :) But I get your point. If it weren't for the potential confusion, I would absolutely agree with you. But look at the way for-else loops get treated. They have a VERY important place, yet they are constantly misunderstood. This would be worse, and it doesn't have as strong a place. ChrisA
On 2021-04-08 11:25, Chris Angelico wrote:
Hmm, I don't see the problem. I'd assume/want it to catch everything not already caught by the context manager. The scope would be everything inside the "try-with..." and not be mandatory of course. That's why the original example "try ..." comes first and is on the outside. If you wanted it to guard only *inside* the with statement, then you'd not be putting the try on the outside in the first place. You'd put it inside. I've used try-with many, many times, and can't currently remember any instances of with-try. -Mike
On 9/04/21 2:59 am, anthony.flury via Python-ideas wrote:
I was wondering whether a worthwhile extension might be to allow the `with` statement to have an `except` and `else` clauses
I don't think I would find this very useful. My application-level exception handling is rarely that close to the file operations, it's usually at a much higher level. -- Greg
My application-level
exception handling is rarely that close to the file operations, it's usually at a much higher level.
With is used for lots of other things, too. Does this same point apply to other use cases? For my part, I still find the extra indentation annoying required for with to work with files, even without a try block around it. :-( -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
08.04.21 17:59, anthony.flury via Python-ideas пише:
A year or two ago I proposed the same syntax with different semantic: to catch only exceptions in the context manager, not in the with block. Exceptions in the with block you can catch by adding try/except around the with block, exceptions in the with block and the context manager you can catch by adding try/except around the with statement, but there is no currently way to catch only exceptions in the context manager. It is quite a common problem, I encounter it several times per year since then. I still have a hope to add this feature, and it will conflict with your idea.
On 09/04/2021 19:02, Guido van Rossum wrote:
Surely it depends (if we adopt a proposal) how we document it. You could argue that very few syntax elements are entirely clear unless we explain it - which is what the point of the documentation. For example for someone who doesn't know what 'with' does, it isn't necessarily clear (just from the syntax) that 'with' ensures finalizing of resources when an exception occurs - the documentation has to explain that. IF we reject a syntax because it isn't self-explanatory that sounds like a bad precedence. -- Anthony Flury *Moble*: +44 07743 282707 *Home*: +44 (0)1206 391294 *email*: anthony.flury@btinternet.com <mailto:anthony.flury@btinternet.com>
09.04.21 21:02, Guido van Rossum пише:
Maybe. But there was also different idea for the "with" statement (with semantic similar to Pascal's). I understand the drawback of this syntax (error handlers are located too far from the guarded code), but I do not have other syntax which would look Pythonic.
Serhiy Storchaka writes:
FWIW, this is the semantics I would expect, for the reason you give:
Does this apply to the necessary setup in any other syntactic constructs? For example, in the abstract this is a "problem" in 'for':
it's not possible to catch the TypeError *only* in the outer loop. Of course this isn't a "real" problem in 'for', since this TypeError is surely not an exceptional situation you need to catch and handle at runtime, it's just a bug. I guess the biggest difference is that in 'with' the setup (corresponding to calling 'iter' on the iterable in 'for') is specified to be arbitrary code and can be arbitrarily complex. 'with' is the only construct where that's true, that I know of. (Of course the setups for other syntactic constructs are very simple.) Is that so? Steve
On Thu, Apr 8, 2021, 11:39 AM anthony.flury via Python-ideas < python-ideas@python.org> wrote:
I like the concept, but I don't like just having a plain with block implicitly acting as a try block because you have to read further to actually understand that yes, you're catching exceptions here. What about "try with ...:"? The combination of the two keywords fits the "try-with-resources" pattern in some other languages and makes it explicit up front that exceptions are about to be caught, while keeping just one level of indentation.
I think I like it. Q. Safe to assume this would catch exceptions from both the call to `open` as well as the call to `open.__enter__`? Q. What about something even more verbose (descriptive) like`try with open(...) as cfg:`? 🙂 Paul On Thu, 2021-04-08 at 15:59 +0100, anthony.flury via Python-ideas wrote:
On Fri, Apr 9, 2021 at 1:38 AM anthony.flury via Python-ideas <python-ideas@python.org> wrote:
Normally I'd be -1 on proposals whose main benefit is "one fewer indentation level". We don't need complexity that serves no other purpose than that. But the with/try combination is somewhat special - the 'with' block is inherently about exception handling. But there's this big question: On Fri, Apr 9, 2021 at 2:24 AM Paul Bryan <pbryan@anode.ca> wrote:
Q. Safe to assume this would catch exceptions from both the call to `open` as well as the call to `open.__enter__`?
No, not safe to assume. It's equally reasonable to define it as guarding only the body of the statement, or as guarding the header as well. The semantics have to be locked in one way or the other, and half the time that's going to be incorrect. IMO this would be a bug magnet on the difference between "exceptions that get handed to __exit__" and "exceptions that get caught by the associated except clause". For instance, what happens in this situation: with database_transaction(conn) as self.trn: self.trn.execute("insert blah blah blah") else: with database_transaction(conn) as self.trn: self.trn.execute("update set blah blah blah") (Yes, I know a good few database engines have a robust upsert/merge operation, but bear with me here.) The else clause implies that its body will not be executed if any exception was raised. But which of these exceptions would prevent that else from happening? * NameError on database_transaction or conn * ConnectionLostError inside database_transaction() * MemoryError in the transaction's __enter__() * AttributeError assigning to self.trn * ResourceError during the __del__ of the previous trn (freebie - that should be ignored regardless) * KeyboardInterrupt after assigning to self.trn but before beginning the block * QueryConflictError inside execute() (another freebie - obviously this one should) * CommitFailedError in __exit__, after no other exception * RollbackFailedError in __exit__, after some other exception (And yes, RollbackFailed is almost a freebie as well, since I have an "else" and no "except" here, but that'd be different with some actual except clauses, so it's worth clarifying.) If it's defined as simply nesting the with inside a try, then the answer is "all of the above". But that would be inconsistent with the existing meaning, which wouldn't cover nearly as much. Similar question: What would be the semantics of this? with contextlib.suppress(BaseException): a = b / c except BaseException as e: print(e) What types of exception could be caught and what types couldn't? ChrisA
On 4/8/21 11:25 AM, Chris Angelico wrote:
Well, if every exception is derived from BaseException (they are) and contextlib.suppress(BaseException) suppresses all BaseException-derived exceptions (it does) then the semantics of the above are: - "a" will be the result of "b / c" if no exception occurs - otherwise, "a" will be whatever it was before the with-block - no exception will ever be caught by the except-clause Generally speaking, no exception that a context manager handles (i.e. suppresses) will ever be available to be caught. -- ~Ethan~
On Fri, Apr 9, 2021 at 6:16 AM Ethan Furman <ethan@stoneleaf.us> wrote:
What about NameError looking up contextlib, or AttributeError looking up suppress? Will they be caught by that except clause, or not? Thank you for making my point: your assumption is completely the opposite of the OP's, given the same syntactic structure. ChrisA
On 4/8/21 1:21 PM, Chris Angelico wrote:
On Fri, Apr 9, 2021 at 6:16 AM Ethan Furman wrote:
On 4/8/21 11:25 AM, Chris Angelico wrote:
Ah, good point -- those two would get caught, as they happen before contextlib.suppress() gets control.
Thank you for making my point: your assumption is completely the opposite of the OP's, given the same syntactic structure.
The guidance should be: `try with` behaves exactly as `try/except with a with`, meaning that NameError and AttributeError for those two reasons would still be caught. Yes, it's easy to miss that at first glance, but it's just as easy to miss in the traditional try/except layout: ```python try: with contextlib.suppress(BaseException): # do stuff except BaseException as e: print(e) ``` How many people are going to look at that and think, "oh, NameError and AttributeError can still be caught" ? -- ~Ethan~
On Fri, Apr 9, 2021 at 6:41 AM Ethan Furman <ethan@stoneleaf.us> wrote:
At least in this form, it's clear that there's a sharp distinction between the stuff around the outside of the 'with' block and the stuff inside it. The semantics, as suggested, give 'with' blocks two distinct exception-management scopes, which mostly but do not entirely overlap. It's not a problem to rigorously define the semantics, but as evidenced here, people's intuitions will vary depending on which context manager is being used. It's absolutely obvious that the OP's example should let you catch errors from open(), and equally obvious that suppressing BaseException should mean that nothing gets caught. That's the problem here. As such, I'm -0.5 on this. It's a kinda nice feature, but all it really offers is one less indentation level, and the semantics are too confusing. ChrisA
On 4/8/21 2:40 PM, Chris Angelico wrote:
I, on the other hand, would love for something along those lines -- I find it massively annoying to have to wrap a with-block, which is supposed to deal with exceptions, inside a try/except block, because the exceptions that `with` deals with are too narrowly focused. Of course, I also prefer def positive(val): "silly example" if val > 0: return True else: return False over def positive(val): "silly example" if val > 0: return True return False because the first more clearly says "either this happens, or that happens". Likewise, having the excepts line up with the `with` visually ties the code together. Granted, this mostly boils down to personal preference. -- ~Ethan~
On Fri, Apr 9, 2021 at 8:22 AM Ethan Furman <ethan@stoneleaf.us> wrote:
And MY preference there would be "return val > 0" :) But I get your point. If it weren't for the potential confusion, I would absolutely agree with you. But look at the way for-else loops get treated. They have a VERY important place, yet they are constantly misunderstood. This would be worse, and it doesn't have as strong a place. ChrisA
On 2021-04-08 11:25, Chris Angelico wrote:
Hmm, I don't see the problem. I'd assume/want it to catch everything not already caught by the context manager. The scope would be everything inside the "try-with..." and not be mandatory of course. That's why the original example "try ..." comes first and is on the outside. If you wanted it to guard only *inside* the with statement, then you'd not be putting the try on the outside in the first place. You'd put it inside. I've used try-with many, many times, and can't currently remember any instances of with-try. -Mike
On 9/04/21 2:59 am, anthony.flury via Python-ideas wrote:
I was wondering whether a worthwhile extension might be to allow the `with` statement to have an `except` and `else` clauses
I don't think I would find this very useful. My application-level exception handling is rarely that close to the file operations, it's usually at a much higher level. -- Greg
My application-level
exception handling is rarely that close to the file operations, it's usually at a much higher level.
With is used for lots of other things, too. Does this same point apply to other use cases? For my part, I still find the extra indentation annoying required for with to work with files, even without a try block around it. :-( -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
08.04.21 17:59, anthony.flury via Python-ideas пише:
A year or two ago I proposed the same syntax with different semantic: to catch only exceptions in the context manager, not in the with block. Exceptions in the with block you can catch by adding try/except around the with block, exceptions in the with block and the context manager you can catch by adding try/except around the with statement, but there is no currently way to catch only exceptions in the context manager. It is quite a common problem, I encounter it several times per year since then. I still have a hope to add this feature, and it will conflict with your idea.
On 09/04/2021 19:02, Guido van Rossum wrote:
Surely it depends (if we adopt a proposal) how we document it. You could argue that very few syntax elements are entirely clear unless we explain it - which is what the point of the documentation. For example for someone who doesn't know what 'with' does, it isn't necessarily clear (just from the syntax) that 'with' ensures finalizing of resources when an exception occurs - the documentation has to explain that. IF we reject a syntax because it isn't self-explanatory that sounds like a bad precedence. -- Anthony Flury *Moble*: +44 07743 282707 *Home*: +44 (0)1206 391294 *email*: anthony.flury@btinternet.com <mailto:anthony.flury@btinternet.com>
09.04.21 21:02, Guido van Rossum пише:
Maybe. But there was also different idea for the "with" statement (with semantic similar to Pascal's). I understand the drawback of this syntax (error handlers are located too far from the guarded code), but I do not have other syntax which would look Pythonic.
Serhiy Storchaka writes:
FWIW, this is the semantics I would expect, for the reason you give:
Does this apply to the necessary setup in any other syntactic constructs? For example, in the abstract this is a "problem" in 'for':
it's not possible to catch the TypeError *only* in the outer loop. Of course this isn't a "real" problem in 'for', since this TypeError is surely not an exceptional situation you need to catch and handle at runtime, it's just a bug. I guess the biggest difference is that in 'with' the setup (corresponding to calling 'iter' on the iterable in 'for') is specified to be arbitrary code and can be arbitrarily complex. 'with' is the only construct where that's true, that I know of. (Of course the setups for other syntactic constructs are very simple.) Is that so? Steve
participants (11)
-
anthony.flury -
Chris Angelico -
Christopher Barker -
Ethan Furman -
Greg Ewing -
Guido van Rossum -
Jonathan Goble -
Mike Miller -
Paul Bryan -
Serhiy Storchaka -
Stephen J. Turnbull