with statement vs. try...except...finally

I'm wondering if the "with" statement should have exception clauses like the "try" statement, even though this seems to defeat part of the reason for the "with" statement. Currently I have a program segment that opens a file and reads a line, something like this (distilled to its elements for illustration): try: f = open('foo') line = f.readline() f.close() except IOError: line = 'default' So that I get a default value if anything goes awry whilst reading the file. If I write it using a "with" statement, I might have: line = 'default' with open('foo') as f: line = f.readline() Fine so far, but what if I want to be more granular? e.g. with "try...except": try: f = open('foo') except IOError: line = "can't open" try: line = f.readline() except IOError: line = "can't read" try: f.close() except IOError: line = "can't close" I can't see how to replace the above try-triplet with a "with" encapsulation. Or, do I have to wrap the "with" statement in try like this: try: with open('foo') as f: line = f.readline() except IOError: line = 'problem with read or close" -- Gerald Britton

Gerald Britton a écrit :
I'd say the triplet of "try...except" clauses above isn't OK, because if the file opening fails, teh code will try anyway to read it and to close it, leading to nameError and other uncaught exception. But the last clause, wrapped in try..except, seems fine to me. ++ Pascal

Good point, so can this be translated to a "with" statement: try: f = open('foo') try: line = f.readline() try: f.close() except IOError: line = "can't close" except IOError: line = "can't read" except IOError: line = "can't open" ?? I guess I'm wondering if we need a with...except construct so that we can get exceptions that happen after the with context is entered without wrapping the with statement in try...except. On Fri, May 29, 2009 at 3:49 PM, Pascal Chambon <chambon.pascal@wanadoo.fr> wrote:
-- Gerald Britton

This example is too simplified to understand if it's worth adding an except clause or not. If all you're doing is pulling the text out of the file, why can't you just make a function for this? lines = fancyread("myfile.txt") #Returns "couldn't open", "couldn't close", etc. in case of errors The point of the with statement is that your want to do more sophisticated stuff with the file contents, but still want the file closed at the end. But if you're doing more sophisticated stuff, it's not clear that you'd suddenly want to have "can't close" substituted in for whatever it was you whipped up by processing the file contents. So, I think we need a better example before we can judge the merits of an except clause. There's also the question of what the except clause would apply to. There are three places where a "with" could throw an exception: with open("myfile") as f: #Could barf on opening f.read() #Could barf in processing #Could barf during clean up Which of the three spots will the except clause deal with? All of them? As it is, barfing in the middle is already given to the context manager to clean up (though the context manager could always throw its own error during its own clean up), so it would be weird to have any other custom clean up that went in addition to what the context manager is already doing. -- Carl

On Sat, 30 May 2009 08:02:31 am Carl Johnson wrote:
Python has exceptions for exception handling. Unless you have a *really good reason*, you shouldn't go backwards to error codes, especially not such course-grained error codes that throw away useful information, and even more especially not error codes which can easily be mistaken for legitimate output: lines = fancyread(filename) if lines == "couldn't open": pass else: # oops, forgot to check for "couldn't read" for line in lines: process(line) will accidentally process each of 'c' 'o' 'u' ... 'r' 'e' 'a' 'd' if the file could be opened but there was an error reading from it. On the very rare case that I don't want to raise an exception on error, I return a tuple of (success_flag, result). If the function is successful, it returns (True, whatever) and if it fails, it returns (False, error-description). Sometimes I'll use the same idiom as string.find() and re.match(): return a sentinel (usually None) to stand in for "not found" or similar. I only do this if it represents something which is not an error condition, but an expected result. -- Steven D'Aprano

On Sat, 30 May 2009 06:58:55 am Gerald Britton wrote:
Would this hypothetical construct: with some_context_manager() as obj: do_something_with(obj) except Exception: error_handler() catch exceptions in the context manager, the do_something_with() block, or both? Give reasons for why you make that choice. Personally, I don't see the need for this. We can already do: # Catch errors in both the context manager and the with block try: with open(filename) as f: process(f) except Exception: pass or this: # Catch errors in just the with block with open(filename) as f: try: process(f) except Exception: pass or this: # Catch errors in the context manager and the with block separately: try: with open(filename) as f: try: process(f) except Exception: print "Error in the with block" except Exception: print "Error in the context manager" and if for some bizarre reason you want to catch errors in the context manager but not the body of with, you can manually emulate with: # untested try: try: f = open(filename).__enter__() except Exception: f = None process(f) finally: if f is not None: f.__exit__() That fourth case is ugly, but it's also rare. I can't imagine a case where you'd need it, but if you do, you can do it. The point is, you can make your error handlers as fine grained as you want, by wrapping just the bits you need. You can nest try blocks, or execute them sequentially. All the flexibility you want is there, at the cost of nothing but an extra line and an indent level. What does with...except gain you that you can't already do easily? -- Steven D'Aprano

On Sat, 30 May 2009 10:30:40 +1000 Steven D'Aprano <steve@pearwood.info> wrote:
That was the case that first drove me to think about extending with. The starting point was a loop: while retries < maxretries: try: with my_magic_lock: process() except ProcessException: handle() except: pass retries += 1 If entering my_magic_lock throws an exception, I don't care - I'm going to retry it anyway. We encountered a condition that caused my_magic_lock to throw ProcessException, so handle() would run without the lock and potentially corrupt our data. Extending with and wanting a way to throw an except *over* the first surrounding try:/except: both occurred to me. What I eventually did was push the loop into a wrapper context manager. <mike -- Mike Meyer <mwm@mired.org> http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org

On Fri, May 29, 2009 at 5:30 PM, Steven D'Aprano <steve@pearwood.info> wrote:
-1. As Steven hints, this will just cause more bugs because people won't guess the right semantics for corner cases, and corner cases are all what except clauses are about. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

On Fri, May 29, 2009 at 1:18 PM, Gerald Britton <gerald.britton@gmail.com>wrote:
FWIW, the same suggestion previously came up in the following thread: http://mail.python.org/pipermail/python-ideas/2009-February/003169.html -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC <http://stutzbachenterprises.com>

On Fri, May 29, 2009 at 3:30 PM, Daniel Stutzbach <daniel@stutzbachenterprises.com> wrote:
Sounds like it also got a thumbs-down from the BDFL: http://mail.python.org/pipermail/python-ideas/2009-February/003184.html Cheers, Chris -- http://blog.rebertia.com

Gerald Britton wrote:
The with statement is designed to simplify certain common use cases of the try statement. That simplification comes at the cost of reduced flexibility. That's OK though: when you need the extra fine-grained control then the original try statement is still around to help you out. Increasing the complexity of the with statement so that it can cover every conceivable try statement use case would defeat the whole point of adding the new statement in the first place. Cheers, Nick. P.S. You could always factor the above out into a simple function: def get_first_line(fname): try: f = open('foo') except IOError: line = "can't open" try: line = f.readline() except IOError: line = "can't read" try: f.close() except IOError: line = "can't close" return line line = get_first_line('foo') Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

Georg Brandl wrote:
Anyway, doesn't it make more sense to report not what *operation* failed, but what the *failure* was (the specifics of the IOError)?
It probably isn't a good idea to overanalyse the toy example ;) Hypocritically-yours, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

Gerald Britton a écrit :
I'd say the triplet of "try...except" clauses above isn't OK, because if the file opening fails, teh code will try anyway to read it and to close it, leading to nameError and other uncaught exception. But the last clause, wrapped in try..except, seems fine to me. ++ Pascal

Good point, so can this be translated to a "with" statement: try: f = open('foo') try: line = f.readline() try: f.close() except IOError: line = "can't close" except IOError: line = "can't read" except IOError: line = "can't open" ?? I guess I'm wondering if we need a with...except construct so that we can get exceptions that happen after the with context is entered without wrapping the with statement in try...except. On Fri, May 29, 2009 at 3:49 PM, Pascal Chambon <chambon.pascal@wanadoo.fr> wrote:
-- Gerald Britton

This example is too simplified to understand if it's worth adding an except clause or not. If all you're doing is pulling the text out of the file, why can't you just make a function for this? lines = fancyread("myfile.txt") #Returns "couldn't open", "couldn't close", etc. in case of errors The point of the with statement is that your want to do more sophisticated stuff with the file contents, but still want the file closed at the end. But if you're doing more sophisticated stuff, it's not clear that you'd suddenly want to have "can't close" substituted in for whatever it was you whipped up by processing the file contents. So, I think we need a better example before we can judge the merits of an except clause. There's also the question of what the except clause would apply to. There are three places where a "with" could throw an exception: with open("myfile") as f: #Could barf on opening f.read() #Could barf in processing #Could barf during clean up Which of the three spots will the except clause deal with? All of them? As it is, barfing in the middle is already given to the context manager to clean up (though the context manager could always throw its own error during its own clean up), so it would be weird to have any other custom clean up that went in addition to what the context manager is already doing. -- Carl

On Sat, 30 May 2009 08:02:31 am Carl Johnson wrote:
Python has exceptions for exception handling. Unless you have a *really good reason*, you shouldn't go backwards to error codes, especially not such course-grained error codes that throw away useful information, and even more especially not error codes which can easily be mistaken for legitimate output: lines = fancyread(filename) if lines == "couldn't open": pass else: # oops, forgot to check for "couldn't read" for line in lines: process(line) will accidentally process each of 'c' 'o' 'u' ... 'r' 'e' 'a' 'd' if the file could be opened but there was an error reading from it. On the very rare case that I don't want to raise an exception on error, I return a tuple of (success_flag, result). If the function is successful, it returns (True, whatever) and if it fails, it returns (False, error-description). Sometimes I'll use the same idiom as string.find() and re.match(): return a sentinel (usually None) to stand in for "not found" or similar. I only do this if it represents something which is not an error condition, but an expected result. -- Steven D'Aprano

On Sat, 30 May 2009 06:58:55 am Gerald Britton wrote:
Would this hypothetical construct: with some_context_manager() as obj: do_something_with(obj) except Exception: error_handler() catch exceptions in the context manager, the do_something_with() block, or both? Give reasons for why you make that choice. Personally, I don't see the need for this. We can already do: # Catch errors in both the context manager and the with block try: with open(filename) as f: process(f) except Exception: pass or this: # Catch errors in just the with block with open(filename) as f: try: process(f) except Exception: pass or this: # Catch errors in the context manager and the with block separately: try: with open(filename) as f: try: process(f) except Exception: print "Error in the with block" except Exception: print "Error in the context manager" and if for some bizarre reason you want to catch errors in the context manager but not the body of with, you can manually emulate with: # untested try: try: f = open(filename).__enter__() except Exception: f = None process(f) finally: if f is not None: f.__exit__() That fourth case is ugly, but it's also rare. I can't imagine a case where you'd need it, but if you do, you can do it. The point is, you can make your error handlers as fine grained as you want, by wrapping just the bits you need. You can nest try blocks, or execute them sequentially. All the flexibility you want is there, at the cost of nothing but an extra line and an indent level. What does with...except gain you that you can't already do easily? -- Steven D'Aprano

On Sat, 30 May 2009 10:30:40 +1000 Steven D'Aprano <steve@pearwood.info> wrote:
That was the case that first drove me to think about extending with. The starting point was a loop: while retries < maxretries: try: with my_magic_lock: process() except ProcessException: handle() except: pass retries += 1 If entering my_magic_lock throws an exception, I don't care - I'm going to retry it anyway. We encountered a condition that caused my_magic_lock to throw ProcessException, so handle() would run without the lock and potentially corrupt our data. Extending with and wanting a way to throw an except *over* the first surrounding try:/except: both occurred to me. What I eventually did was push the loop into a wrapper context manager. <mike -- Mike Meyer <mwm@mired.org> http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org

On Fri, May 29, 2009 at 5:30 PM, Steven D'Aprano <steve@pearwood.info> wrote:
-1. As Steven hints, this will just cause more bugs because people won't guess the right semantics for corner cases, and corner cases are all what except clauses are about. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

On Fri, May 29, 2009 at 1:18 PM, Gerald Britton <gerald.britton@gmail.com>wrote:
FWIW, the same suggestion previously came up in the following thread: http://mail.python.org/pipermail/python-ideas/2009-February/003169.html -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC <http://stutzbachenterprises.com>

On Fri, May 29, 2009 at 3:30 PM, Daniel Stutzbach <daniel@stutzbachenterprises.com> wrote:
Sounds like it also got a thumbs-down from the BDFL: http://mail.python.org/pipermail/python-ideas/2009-February/003184.html Cheers, Chris -- http://blog.rebertia.com

Gerald Britton wrote:
The with statement is designed to simplify certain common use cases of the try statement. That simplification comes at the cost of reduced flexibility. That's OK though: when you need the extra fine-grained control then the original try statement is still around to help you out. Increasing the complexity of the with statement so that it can cover every conceivable try statement use case would defeat the whole point of adding the new statement in the first place. Cheers, Nick. P.S. You could always factor the above out into a simple function: def get_first_line(fname): try: f = open('foo') except IOError: line = "can't open" try: line = f.readline() except IOError: line = "can't read" try: f.close() except IOError: line = "can't close" return line line = get_first_line('foo') Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

Georg Brandl wrote:
Anyway, doesn't it make more sense to report not what *operation* failed, but what the *failure* was (the specifics of the IOError)?
It probably isn't a good idea to overanalyse the toy example ;) Hypocritically-yours, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
participants (11)
-
Carl Johnson
-
Chris Rebert
-
Daniel Stutzbach
-
Georg Brandl
-
Gerald Britton
-
Guido van Rossum
-
Mike Meyer
-
Nick Coghlan
-
Pascal Chambon
-
Scott David Daniels
-
Steven D'Aprano