PEP: 463 Title: Exception-catching expressions Version: $Revision$ Last-Modified: $Date$ Author: Chris Angelico <rosuav@gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 15-Feb-2014 Python-Version: 3.5 Post-History: 16-Feb-2014, 21-Feb-2014 Abstract ======== Just as PEP 308 introduced a means of value-based conditions in an expression, this system allows exception-based conditions to be used as part of an expression. Motivation ========== A number of functions and methods have parameters which will cause them to return a specified value instead of raising an exception. The current system is ad-hoc and inconsistent, and requires that each function be individually written to have this functionality; not all support this. * dict.get(key, default) - second positional argument in place of KeyError * next(iter, default) - second positional argument in place of StopIteration * list.pop() - no way to return a default * seq[index] - no way to handle a bounds error * min(sequence, default=default) - keyword argument in place of ValueError * sum(sequence, start=default) - slightly different but can do the same job * statistics.mean(data) - no way to handle an empty iterator Rationale ========= The current system requires that a function author predict the need for a default, and implement support for it. If this is not done, a full try/except block is needed. Since try/except is a statement, it is impossible to catch exceptions in the middle of an expression. Just as if/else does for conditionals and lambda does for function definitions, so does this allow exception catching in an expression context. This provides a clean and consistent way for a function to provide a default: it simply raises an appropriate exception, and the caller catches it. With some situations, an LBYL technique can be used (checking if some sequence has enough length before indexing into it, for instance). This is not safe in all cases, but as it is often convenient, programmers will be tempted to sacrifice the safety of EAFP in favour of the notational brevity of LBYL. Additionally, some LBYL techniques (eg involving getattr with three arguments) warp the code into looking like literal strings rather than attribute lookup, which can impact readability. A convenient EAFP notation solves all of this. There's no convenient way to write a helper function to do this; the nearest is something ugly using either lambda:: def except_(expression, exception_list, default): try: return expression() except exception_list: return default() value = except_(lambda: 1/x, ZeroDivisionError, lambda: float("nan")) which is clunky, and unable to handle multiple exception clauses; or eval:: def except_(expression, exception_list, default): try: return eval(expression, globals_of_caller(), locals_of_caller()) except exception_list as exc: l = locals_of_caller().copy() l['exc'] = exc return eval(default, globals_of_caller(), l) def globals_of_caller(): return sys._getframe(2).f_globals def locals_of_caller(): return sys._getframe(2).f_locals value = except_("""1/x""",ZeroDivisionError,""" "Can't divide by zero" """) which is even clunkier, and relies on implementation-dependent hacks. (Writing globals_of_caller() and locals_of_caller() for interpreters other than CPython is left as an exercise for the reader.) Raymond Hettinger `expresses`__ a desire for such a consistent API. Something similar has been `requested`__ `multiple`__ `times`__ in the past. __ https://mail.python.org/pipermail/python-ideas/2014-February/025443.html __ https://mail.python.org/pipermail/python-ideas/2013-March/019760.html __ https://mail.python.org/pipermail/python-ideas/2009-August/005441.html __ https://mail.python.org/pipermail/python-ideas/2008-August/001801.html Proposal ======== Just as the 'or' operator and the three part 'if-else' expression give short circuiting methods of catching a falsy value and replacing it, this syntax gives a short-circuiting method of catching an exception and replacing it. This currently works:: lst = [1, 2, None, 3] value = lst[2] or "No value" The proposal adds this:: lst = [1, 2] value = lst[2] except IndexError: "No value" Specifically, the syntax proposed is:: expr except exception_list: default where expr, exception_list, and default are all expressions. First, expr is evaluated. If no exception is raised, its value is the value of the overall expression. If any exception is raised, exception_list is evaluated, and should result in either a type or a tuple, just as with the statement form of try/except. Any matching exception will result in the corresponding default expression being evaluated and becoming the value of the expression. As with the statement form of try/except, non-matching exceptions will propagate upward. Note that the current proposal does not allow the exception object to be captured. Where this is needed, the statement form must be used. (See below for discussion and elaboration on this.) This ternary operator would be between lambda and if/else in precedence. Consider this example of a two-level cache:: for key in sequence: x = (lvl1[key] except KeyError: (lvl2[key] except KeyError: f(key))) # do something with x This cannot be rewritten as:: x = lvl1.get(key, lvl2.get(key, f(key))) which, despite being shorter, defeats the purpose of the cache, as it must calculate a default value to pass to get(). The .get() version calculates backwards; the exception-testing version calculates forwards, as would be expected. The nearest useful equivalent would be:: x = lvl1.get(key) or lvl2.get(key) or f(key) which depends on the values being nonzero, as well as depending on the cache object supporting this functionality. Alternative Proposals ===================== Discussion on python-ideas brought up the following syntax suggestions:: value = expr except default if Exception [as e] value = expr except default for Exception [as e] value = expr except default from Exception [as e] value = expr except Exception [as e] return default value = expr except (Exception [as e]: default) value = expr except Exception [as e] try default value = expr except Exception [as e] continue with default value = default except Exception [as e] else expr value = try expr except Exception [as e]: default value = expr except default # Catches anything value = expr except(Exception) default # Catches only the named type(s) value = default if expr raise Exception value = expr or else default if Exception value = expr except Exception [as e] -> default value = expr except Exception [as e] pass default It has also been suggested that a new keyword be created, rather than reusing an existing one. Such proposals fall into the same structure as the last form, but with a different keyword in place of 'pass'. Suggestions include 'then', 'when', and 'use'. Also, in the context of the "default if expr raise Exception" proposal, it was suggested that a new keyword "raises" be used. All forms involving the 'as' capturing clause have been deferred from this proposal in the interests of simplicity, but are preserved in the table above as an accurate record of suggestions. Open Issues =========== Parentheses around the entire expression ---------------------------------------- Generator expressions require parentheses, unless they would be strictly redundant. Ambiguities with except expressions could be resolved in the same way, forcing nested except-in-except trees to be correctly parenthesized and requiring that the outer expression be clearly delineated. `Steven D'Aprano elaborates on the issue.`__ __ https://mail.python.org/pipermail/python-ideas/2014-February/025647.html Example usage ============= For each example, an approximately-equivalent statement form is given, to show how the expression will be parsed. These are not always strictly equivalent, but will accomplish the same purpose. It is NOT safe for the interpreter to translate one into the other. A number of these examples are taken directly from the Python standard library, with file names and line numbers correct as of early Feb 2014. Many of these patterns are extremely common. Retrieve an argument, defaulting to None:: cond = args[1] except IndexError: None # Lib/pdb.py:803: try: cond = args[1] except IndexError: cond = None Fetch information from the system if available:: pwd = os.getcwd() except OSError: None # Lib/tkinter/filedialog.py:210: try: pwd = os.getcwd() except OSError: pwd = None Attempt a translation, falling back on the original:: e.widget = self._nametowidget(W) except KeyError: W # Lib/tkinter/__init__.py:1222: try: e.widget = self._nametowidget(W) except KeyError: e.widget = W Read from an iterator, continuing with blank lines once it's exhausted:: line = readline() except StopIteration: '' # Lib/lib2to3/pgen2/tokenize.py:370: try: line = readline() except StopIteration: line = '' Retrieve platform-specific information (note the DRY improvement); this particular example could be taken further, turning a series of separate assignments into a single large dict initialization:: # sys.abiflags may not be defined on all platforms. _CONFIG_VARS['abiflags'] = sys.abiflags except AttributeError: '' # Lib/sysconfig.py:529: try: _CONFIG_VARS['abiflags'] = sys.abiflags except AttributeError: # sys.abiflags may not be defined on all platforms. _CONFIG_VARS['abiflags'] = '' Retrieve an indexed item, defaulting to None (similar to dict.get):: def getNamedItem(self, name): return self._attrs[name] except KeyError: None # Lib/xml/dom/minidom.py:573: def getNamedItem(self, name): try: return self._attrs[name] except KeyError: return None Translate numbers to names, falling back on the numbers:: g = grp.getgrnam(tarinfo.gname)[2] except KeyError: tarinfo.gid u = pwd.getpwnam(tarinfo.uname)[2] except KeyError: tarinfo.uid # Lib/tarfile.py:2198: try: g = grp.getgrnam(tarinfo.gname)[2] except KeyError: g = tarinfo.gid try: u = pwd.getpwnam(tarinfo.uname)[2] except KeyError: u = tarinfo.uid Perform some lengthy calculations in EAFP mode, handling division by zero as a sort of sticky NaN:: value = calculate(x) except ZeroDivisionError: float("nan") try: value = calculate(x) except ZeroDivisionError: value = float("nan") Calculate the mean of a series of numbers, falling back on zero:: value = statistics.mean(lst) except statistics.StatisticsError: 0 try: value = statistics.mean(lst) except statistics.StatisticsError: value = 0 Retrieving a message from either a cache or the internet, with auth check:: logging.info("Message shown to user: %s",((cache[k] except LookupError: (backend.read(k) except OSError: 'Resource not available') ) if check_permission(k) else 'Access denied' ) except BaseException: "This is like a bare except clause") try: if check_permission(k): try: _ = cache[k] except LookupError: try: _ = backend.read(k) except OSError: _ = 'Resource not available' else: _ = 'Access denied' except BaseException: _ = "This is like a bare except clause" logging.info("Message shown to user: %s", _) Looking up objects in a sparse list of overrides:: (overrides[x] or default except IndexError: default).ping() try: (overrides[x] or default).ping() except IndexError: default.ping() Narrowing of exception-catching scope ------------------------------------- The following examples, taken directly from Python's standard library, demonstrate how the scope of the try/except can be conveniently narrowed. To do this with the statement form of try/except would require a temporary variable, but it's far cleaner as an expression. Lib/ipaddress.py:343:: try: ips.append(ip.ip) except AttributeError: ips.append(ip.network_address) Becomes:: ips.append(ip.ip except AttributeError: ip.network_address) The expression form is nearly equivalent to this:: try: _ = ip.ip except AttributeError: _ = ip.network_address ips.append(_) Lib/tempfile.py:130:: try: dirlist.append(_os.getcwd()) except (AttributeError, OSError): dirlist.append(_os.curdir) Becomes:: dirlist.append(_os.getcwd() except (AttributeError, OSError): _os.curdir) Lib/asyncore.py:264:: try: status.append('%s:%d' % self.addr) except TypeError: status.append(repr(self.addr)) Becomes:: status.append('%s:%d' % self.addr except TypeError: repr(self.addr)) Comparisons with other languages ================================ (With thanks to Andrew Barnert for compiling this section.) `Ruby's`__ "begin…rescue…rescue…else…ensure…end" is an expression (potentially with statements inside it). It has the equivalent of an "as" clause, and the equivalent of bare except. And it uses no punctuation or keyword between the bare except/exception class/exception class with as clause and the value. (And yes, it's ambiguous unless you understand Ruby's statement/expression rules.) __ http://www.skorks.com/2009/09/ruby-exceptions-and-exception-handling/ :: x = begin computation() rescue MyException => e default(e) end; x = begin computation() rescue MyException default() end; x = begin computation() rescue default() end; x = begin computation() rescue MyException default() rescue OtherException other() end; In terms of this PEP:: x = computation() except MyException as e default(e) x = computation() except MyException default(e) x = computation() except default(e) x = computation() except MyException default() except OtherException other() `Erlang`__ has a try expression that looks like this:: __ http://erlang.org/doc/reference_manual/expressions.html#id79284 x = try computation() catch MyException:e -> default(e) end; x = try computation() catch MyException:e -> default(e); OtherException:e -> other(e) end; The class and "as" name are mandatory, but you can use "_" for either. There's also an optional "when" guard on each, and a "throw" clause that you can catch, which I won't get into. To handle multiple exceptions, you just separate the clauses with semicolons, which I guess would map to commas in Python. So:: x = try computation() except MyException as e -> default(e) x = try computation() except MyException as e -> default(e), OtherException as e->other_default(e) Erlang also has a "catch" expression, which, despite using the same keyword, is completely different, and you don't want to know about it. The ML family has two different ways of dealing with this, "handle" and "try"; the difference between the two is that "try" pattern-matches the exception, which gives you the effect of multiple except clauses and as clauses. In either form, the handler clause is punctuated by "=>" in some dialects, "->" in others. To avoid confusion, I'll write the function calls in Python style. Here's `SML's`__ "handle":: __ http://www.cs.cmu.edu/~rwh/introsml/core/exceptions.htm let x = computation() handle MyException => default();; Here's `OCaml's`__ "try":: __ http://www2.lib.uchicago.edu/keith/ocaml-class/exceptions.html let x = try computation() with MyException explanation -> default(explanation);; let x = try computation() with MyException(e) -> default(e) | MyOtherException() -> other_default() | (e) -> fallback(e);; In terms of this PEP, these would be something like:: x = computation() except MyException => default() x = try computation() except MyException e -> default() x = (try computation() except MyException as e -> default(e) except MyOtherException -> other_default() except BaseException as e -> fallback(e)) Many ML-inspired but not-directly-related languages from academia mix things up, usually using more keywords and fewer symbols. So, the `Oz`__ would map to Python as:: __ http://mozart.github.io/mozart-v1/doc-1.4.0/tutorial/node5.html x = try computation() catch MyException as e then default(e) Many Lisp-derived languages, like `Clojure,`__ implement try/catch as special forms (if you don't know what that means, think function-like macros), so you write, effectively:: __ http://clojure.org/special_forms#Special%20Forms--(try%20expr*%20catch-claus...) try(computation(), catch(MyException, explanation, default(explanation))) try(computation(), catch(MyException, explanation, default(explanation)), catch(MyOtherException, explanation, other_default(explanation))) In Common Lisp, this is done with a slightly clunkier `"handler-case" macro,`__ but the basic idea is the same. __ http://clhs.lisp.se/Body/m_hand_1.htm The Lisp style is, surprisingly, used by some languages that don't have macros, like Lua, where `xpcall`__ takes functions. Writing lambdas Python-style instead of Lua-style:: __ http://www.gammon.com.au/scripts/doc.php?lua=xpcall x = xpcall(lambda: expression(), lambda e: default(e)) This actually returns (true, expression()) or (false, default(e)), but I think we can ignore that part. Haskell is actually similar to Lua here (except that it's all done with monads, of course):: x = do catch(lambda: expression(), lambda e: default(e)) You can write a pattern matching expression within the function to decide what to do with it; catching and re-raising exceptions you don't want is cheap enough to be idiomatic. But Haskell infixing makes this nicer:: x = do expression() `catch` lambda: default() x = do expression() `catch` lambda e: default(e) And that makes the parallel between the lambda colon and the except colon in the proposal much more obvious:: x = expression() except Exception: default() x = expression() except Exception as e: default(e) `Tcl`__ has the other half of Lua's xpcall; catch is a function which returns true if an exception was caught, false otherwise, and you get the value out in other ways. And it's all built around the the implicit quote-and-exec that everything in Tcl is based on, making it even harder to describe in Python terms than Lisp macros, but something like:: __ http://wiki.tcl.tk/902 if {[ catch("computation()") "explanation"]} { default(explanation) } `Smalltalk`__ is also somewhat hard to map to Python. The basic version would be:: __ http://smalltalk.gnu.org/wiki/exceptions x := computation() on:MyException do:default() … but that's basically Smalltalk's passing-arguments-with-colons syntax, not its exception-handling syntax. Deferred sub-proposals ====================== Multiple except clauses ----------------------- An examination of use-cases shows that this is not needed as often as it would be with the statement form, and as its syntax is a point on which consensus has not been reached, the entire feature is deferred. In order to ensure compatibility with future versions, ensure that any consecutive except operators are parenthesized to guarantee the interpretation you expect. Multiple 'except' keywords can be used, and they will all catch exceptions raised in the original expression (only):: # Will catch any of the listed exceptions thrown by expr; # any exception thrown by a default expression will propagate. value = (expr except Exception1 [as e]: default1 except Exception2 [as e]: default2 # ... except ExceptionN [as e]: defaultN ) Using parentheses to force an alternative interpretation works as expected:: # Will catch an Exception2 thrown by either expr or default1 value = ( (expr except Exception1: default1) except Exception2: default2 ) # Will catch an Exception2 thrown by default1 only value = (expr except Exception1: (default1 except Exception2: default2) ) This last form is confusing and should be discouraged by PEP 8, but it is syntactically legal: you can put any sort of expression inside a ternary-except; ternary-except is an expression; therefore you can put a ternary-except inside a ternary-except. Open question: Where there are multiple except clauses, should they be separated by commas? It may be easier for the parser, that way:: value = (expr except Exception1 [as e]: default1, except Exception2 [as e]: default2, # ... except ExceptionN [as e]: defaultN, ) with an optional comma after the last, as per tuple rules. Downside: Omitting the comma would be syntactically valid, and would have almost identical semantics, but would nest the entire preceding expression in its exception catching rig - a matching exception raised in the default clause would be caught by the subsequent except clause. As this difference is so subtle, it runs the risk of being a major bug magnet. As a mitigation of this risk, this form:: value = expr except Exception1: default1 except Exception2: default2 could be syntactically forbidden, and parentheses required if the programmer actually wants that behaviour:: value = (expr except Exception1: default1) except Exception2: default2 This would prevent the accidental omission of a comma from changing the expression's meaning. Capturing the exception object ------------------------------ In a try/except block, the use of 'as' to capture the exception object creates a local name binding, and implicitly deletes that binding in a finally clause. As 'finally' is not a part of this proposal (see below), this makes it tricky to describe; also, this use of 'as' gives a way to create a name binding in an expression context. Should the default clause have an inner scope in which the name exists, shadowing anything of the same name elsewhere? Should it behave the same way the statement try/except does, and unbind the name? Should it bind the name and leave it bound? (Almost certainly not; this behaviour was changed in Python 3 for good reason.) Additionally, this syntax would allow a convenient way to capture exceptions in interactive Python; returned values are captured by "_", but exceptions currently are not. This could be spelled:
expr except Exception as e: e
(The inner scope idea is tempting, but currently CPython handles list comprehensions with a nested function call, as this is considered easier. It may be of value to simplify both comprehensions and except expressions, but that is a completely separate proposal to this PEP; alternatively, it may be better to stick with what's known to work. `Nick Coghlan elaborates.`__) __ https://mail.python.org/pipermail/python-ideas/2014-February/025702.html An examination of the Python standard library shows that, while the use of 'as' is fairly common (occurring in roughly one except clause in five), it is extremely *uncommon* in the cases which could logically be converted into the expression form. Its few uses can simply be left unchanged. Consequently, in the interests of simplicity, the 'as' clause is not included in this proposal. A subsequent Python version can add this without breaking any existing code, as 'as' is already a keyword. One example where this could possibly be useful is Lib/imaplib.py:568:: try: typ, dat = self._simple_command('LOGOUT') except: typ, dat = 'NO', ['%s: %s' % sys.exc_info()[:2]] This could become:: typ, dat = (self._simple_command('LOGOUT') except BaseException as e: ('NO', '%s: %s' % (type(e), e))) Or perhaps some other variation. This is hardly the most compelling use-case, but an intelligent look at this code could tidy it up significantly. In the absence of further examples showing any need of the exception object, I have opted to defer indefinitely the recommendation. Rejected sub-proposals ====================== finally clause -------------- The statement form try... finally or try... except... finally has no logical corresponding expression form. Therefore the finally keyword is not a part of this proposal, in any way. Bare except having different meaning ------------------------------------ With several of the proposed syntaxes, omitting the exception type name would be easy and concise, and would be tempting. For convenience's sake, it might be advantageous to have a bare 'except' clause mean something more useful than "except BaseException". Proposals included having it catch Exception, or some specific set of "common exceptions" (subclasses of a new type called ExpressionError), or have it look for a tuple named ExpressionError in the current scope, with a built-in default such as (ValueError, UnicodeError, AttributeError, EOFError, IOError, OSError, LookupError, NameError, ZeroDivisionError). All of these were rejected, for severa reasons. * First and foremost, consistency with the statement form of try/except would be broken. Just as a list comprehension or ternary if expression can be explained by "breaking it out" into its vertical statement form, an expression-except should be able to be explained by a relatively mechanical translation into a near-equivalent statement. Any form of syntax common to both should therefore have the same semantics in each, and above all should not have the subtle difference of catching more in one than the other, as it will tend to attract unnoticed bugs. * Secondly, the set of appropriate exceptions to catch would itself be a huge point of contention. It would be impossible to predict exactly which exceptions would "make sense" to be caught; why bless some of them with convenient syntax and not others? * And finally (this partly because the recommendation was that a bare except should be actively encouraged, once it was reduced to a "reasonable" set of exceptions), any situation where you catch an exception you don't expect to catch is an unnecessary bug magnet. Consequently, the use of a bare 'except' is down to two possibilities: either it is syntactically forbidden in the expression form, or it is permitted with the exact same semantics as in the statement form (namely, that it catch BaseException and be unable to capture it with 'as'). Bare except clauses ------------------- PEP 8 rightly advises against the use of a bare 'except'. While it is syntactically legal in a statement, and for backward compatibility must remain so, there is little value in encouraging its use. In an expression except clause, "except:" is a SyntaxError; use the equivalent long-hand form "except BaseException:" instead. A future version of Python MAY choose to reinstate this, which can be done without breaking compatibility. Parentheses around the except clauses ------------------------------------- Should it be legal to parenthesize the except clauses, separately from the expression that could raise? Example:: value = expr ( except Exception1 [as e]: default1 except Exception2 [as e]: default2 # ... except ExceptionN [as e]: defaultN ) This is more compelling when one or both of the deferred sub-proposals of multiple except clauses and/or exception capturing is included. In their absence, the parentheses would be thus:: value = expr except ExceptionType: default value = expr (except ExceptionType: default) The advantage is minimal, and the potential to confuse a reader into thinking the except clause is separate from the expression, or into thinking this is a function call, makes this non-compelling. The expression can, of course, be parenthesized if desired, as can the default:: value = (expr) except ExceptionType: (default) Short-hand for "except: pass" ----------------------------- The following was been suggested as a similar short-hand, though not technically an expression:: statement except Exception: pass try: statement except Exception: pass For instance, a common use-case is attempting the removal of a file:: os.unlink(some_file) except OSError: pass There is an equivalent already in Python 3.4, however, in contextlib:: from contextlib import suppress with suppress(OSError): os.unlink(some_file) As this is already a single line (or two with a break after the colon), there is little need of new syntax and a confusion of statement vs expression to achieve this. Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: