PEP 463, Exception-catching expressions, is stable and I believe ready for pronouncement. Drumroll please... http://www.python.org/dev/peps/pep-0463/ PEP: 463 Title: Exception-catching expressions Version: $Revision$ Last-Modified: $Date$ Author: Chris Angelico Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 15-Feb-2014 Python-Version: 3.5 Post-History: 20-Feb-2014, 16-Feb-2014 Abstract ======== Just as PEP 308 introduced a means of value-based conditions in an expression, this system allows exception-based conditions to be used as part of an expression. Motivation ========== A number of functions and methods have parameters which will cause them to return a specified value instead of raising an exception. The current system is ad-hoc and inconsistent, and requires that each function be individually written to have this functionality; not all support this. * dict.get(key, default) - second positional argument in place of KeyError * next(iter, default) - second positional argument in place of StopIteration * list.pop() - no way to return a default * seq[index] - no way to handle a bounds error * min(sequence, default=default) - keyword argument in place of ValueError * statistics.mean(data) - no way to handle an empty iterator Had this facility existed early in Python's history, there would have been no need to create dict.get() and related methods; the one obvious way to handle an absent key would be to respond to the exception. One method is written which signals the absence in one way, and one consistent technique is used to respond to the absence. Instead, we have dict.get(), and as of Python 3.4, we also have min(... default=default), and myriad others. We have a LBYL syntax for testing inside an expression, but there is currently no EAFP notation; compare the following:: # LBYL: if key in dic: process(dic[key]) else: process(None) # As an expression: process(dic[key] if key in dic else None) # EAFP: try: process(dic[key]) except KeyError: process(None) # As an expression: process(dic[key] except KeyError: None) Python generally recommends the EAFP policy, but must then proliferate utility functions like dic.get(key,None) to enable this. Rationale ========= The current system requires that a function author predict the need for a default, and implement support for it. If this is not done, a full try/except block is needed. Since try/except is a statement, it is impossible to catch exceptions in the middle of an expression. Just as if/else does for conditionals and lambda does for function definitions, so does this allow exception catching in an expression context. This provides a clean and consistent way for a function to provide a default: it simply raises an appropriate exception, and the caller catches it. With some situations, an LBYL technique can be used (checking if some sequence has enough length before indexing into it, for instance). This is not safe in all cases, but as it is often convenient, programmers will be tempted to sacrifice the safety of EAFP in favour of the notational brevity of LBYL. Additionally, some LBYL techniques (eg involving getattr with three arguments) warp the code into looking like literal strings rather than attribute lookup, which can impact readability. A convenient EAFP notation solves all of this. There's no convenient way to write a helper function to do this; the nearest is something ugly using either lambda:: def except_(expression, exception_list, default): try: return expression() except exception_list: return default() value = except_(lambda: 1/x, ZeroDivisionError, lambda: float("nan")) which is clunky, and unable to handle multiple exception clauses; or eval:: def except_(expression, exception_list, default): try: return eval(expression, globals_of_caller(), locals_of_caller()) except exception_list as exc: l = locals_of_caller().copy() l['exc'] = exc return eval(default, globals_of_caller(), l) def globals_of_caller(): return sys._getframe(2).f_globals def locals_of_caller(): return sys._getframe(2).f_locals value = except_("""1/x""",ZeroDivisionError,""" "Can't divide by zero" """) which is even clunkier, and relies on implementation-dependent hacks. (Writing globals_of_caller() and locals_of_caller() for interpreters other than CPython is left as an exercise for the reader.) Raymond Hettinger `expresses`__ a desire for such a consistent API. Something similar has been `requested`__ `multiple`__ `times`__ in the past. __ https://mail.python.org/pipermail/python-ideas/2014-February/025443.html __ https://mail.python.org/pipermail/python-ideas/2013-March/019760.html __ https://mail.python.org/pipermail/python-ideas/2009-August/005441.html __ https://mail.python.org/pipermail/python-ideas/2008-August/001801.html Proposal ======== Just as the 'or' operator and the three part 'if-else' expression give short circuiting methods of catching a falsy value and replacing it, this syntax gives a short-circuiting method of catching an exception and replacing it. This currently works:: lst = [1, 2, None, 3] value = lst[2] or "No value" The proposal adds this:: lst = [1, 2] value = (lst[2] except IndexError: "No value") Specifically, the syntax proposed is:: (expr except exception_list: default) where expr, exception_list, and default are all expressions. First, expr is evaluated. If no exception is raised, its value is the value of the overall expression. If any exception is raised, exception_list is evaluated, and should result in either a type or a tuple, just as with the statement form of try/except. Any matching exception will result in the corresponding default expression being evaluated and becoming the value of the expression. As with the statement form of try/except, non-matching exceptions will propagate upward. Parentheses are required around the entire expression, unless they would be completely redundant, according to the same rules as generator expressions follow. This guarantees correct interpretation of nested except-expressions, and allows for future expansion of the syntax - see below on multiple except clauses. Note that the current proposal does not allow the exception object to be captured. Where this is needed, the statement form must be used. (See below for discussion and elaboration on this.) This ternary operator would be between lambda and if/else in precedence. Consider this example of a two-level cache:: for key in sequence: x = (lvl1[key] except KeyError: (lvl2[key] except KeyError: f(key))) # do something with x This cannot be rewritten as:: x = lvl1.get(key, lvl2.get(key, f(key))) which, despite being shorter, defeats the purpose of the cache, as it must calculate a default value to pass to get(). The .get() version calculates backwards; the exception-testing version calculates forwards, as would be expected. The nearest useful equivalent would be:: x = lvl1.get(key) or lvl2.get(key) or f(key) which depends on the values being nonzero, as well as depending on the cache object supporting this functionality. Alternative Proposals ===================== Discussion on python-ideas brought up the following syntax suggestions:: value = expr except default if Exception [as e] value = expr except default for Exception [as e] value = expr except default from Exception [as e] value = expr except Exception [as e] return default value = expr except (Exception [as e]: default) value = expr except Exception [as e] try default value = expr except Exception [as e] continue with default value = default except Exception [as e] else expr value = try expr except Exception [as e]: default value = expr except default # Catches anything value = expr except(Exception) default # Catches only the named type(s) value = default if expr raise Exception value = expr or else default if Exception value = expr except Exception [as e] -> default value = expr except Exception [as e] pass default It has also been suggested that a new keyword be created, rather than reusing an existing one. Such proposals fall into the same structure as the last form, but with a different keyword in place of 'pass'. Suggestions include 'then', 'when', and 'use'. Also, in the context of the "default if expr raise Exception" proposal, it was suggested that a new keyword "raises" be used. All forms involving the 'as' capturing clause have been deferred from this proposal in the interests of simplicity, but are preserved in the table above as an accurate record of suggestions. The four forms most supported by this proposal are, in order:: value = (expr except Exception: default) value = (expr except Exception -> default) value = (expr except Exception pass default) value = (expr except Exception then default) All four maintain left-to-right evaluation order: first the base expression, then the exception list, and lastly the default. This is important, as the expressions are evaluated lazily. By comparison, several of the ad-hoc alternatives listed above must (by the nature of functions) evaluate their default values eagerly. The preferred form, using the colon, parallels try/except by using "except exception_list:", and parallels lambda by having "keyword name_list: subexpression"; it also can be read as mapping Exception to the default value, dict-style. Using the arrow introduces a token many programmers will not be familiar with, and which currently has no similar meaning, but is otherwise quite readable. The English word "pass" has a vaguely similar meaning (consider the common usage "pass by value/reference" for function arguments), and "pass" is already a keyword, but as its meaning is distinctly unrelated, this may cause confusion. Using "then" makes sense in English, but this introduces a new keyword to the language - albeit one not in common use, but a new keyword all the same. Left to right evaluation order is extremely important to readability, as it parallels the order most expressions are evaluated. Alternatives such as:: value = (expr except default if Exception) break this, by first evaluating the two ends, and then coming to the middle; while this may not seem terrible (as the exception list will usually be a constant), it does add to the confusion when multiple clauses meet, either with multiple except/if or with the existing if/else, or a combination. Using the preferred order, subexpressions will always be evaluated from left to right, no matter how the syntax is nested. Keeping the existing notation, but shifting the mandatory parentheses, we have the following suggestion:: value = expr except (Exception: default) value = expr except(Exception: default) This is reminiscent of a function call, or a dict initializer. The colon cannot be confused with introducing a suite, but on the other hand, the new syntax guarantees lazy evaluation, which a dict does not. The potential to reduce confusion is considered unjustified by the corresponding potential to increase it. Example usage ============= For each example, an approximately-equivalent statement form is given, to show how the expression will be parsed. These are not always strictly equivalent, but will accomplish the same purpose. It is NOT safe for the interpreter to translate one into the other. A number of these examples are taken directly from the Python standard library, with file names and line numbers correct as of early Feb 2014. Many of these patterns are extremely common. Retrieve an argument, defaulting to None:: cond = (args[1] except IndexError: None) # Lib/pdb.py:803: try: cond = args[1] except IndexError: cond = None Fetch information from the system if available:: pwd = (os.getcwd() except OSError: None) # Lib/tkinter/filedialog.py:210: try: pwd = os.getcwd() except OSError: pwd = None Attempt a translation, falling back on the original:: e.widget = (self._nametowidget(W) except KeyError: W) # Lib/tkinter/__init__.py:1222: try: e.widget = self._nametowidget(W) except KeyError: e.widget = W Read from an iterator, continuing with blank lines once it's exhausted:: line = (readline() except StopIteration: '') # Lib/lib2to3/pgen2/tokenize.py:370: try: line = readline() except StopIteration: line = '' Retrieve platform-specific information (note the DRY improvement); this particular example could be taken further, turning a series of separate assignments into a single large dict initialization:: # sys.abiflags may not be defined on all platforms. _CONFIG_VARS['abiflags'] = (sys.abiflags except AttributeError: '') # Lib/sysconfig.py:529: try: _CONFIG_VARS['abiflags'] = sys.abiflags except AttributeError: # sys.abiflags may not be defined on all platforms. _CONFIG_VARS['abiflags'] = '' Retrieve an indexed item, defaulting to None (similar to dict.get):: def getNamedItem(self, name): return (self._attrs[name] except KeyError: None) # Lib/xml/dom/minidom.py:573: def getNamedItem(self, name): try: return self._attrs[name] except KeyError: return None Translate numbers to names, falling back on the numbers:: g = (grp.getgrnam(tarinfo.gname)[2] except KeyError: tarinfo.gid) u = (pwd.getpwnam(tarinfo.uname)[2] except KeyError: tarinfo.uid) # Lib/tarfile.py:2198: try: g = grp.getgrnam(tarinfo.gname)[2] except KeyError: g = tarinfo.gid try: u = pwd.getpwnam(tarinfo.uname)[2] except KeyError: u = tarinfo.uid Look up an attribute, falling back on a default:: mode = (f.mode except AttributeError: 'rb') # Lib/aifc.py:882: if hasattr(f, 'mode'): mode = f.mode else: mode = 'rb' return (sys._getframe(1) except AttributeError: None) # Lib/inspect.py:1350: return sys._getframe(1) if hasattr(sys, "_getframe") else None Perform some lengthy calculations in EAFP mode, handling division by zero as a sort of sticky NaN:: value = (calculate(x) except ZeroDivisionError: float("nan")) try: value = calculate(x) except ZeroDivisionError: value = float("nan") Calculate the mean of a series of numbers, falling back on zero:: value = (statistics.mean(lst) except statistics.StatisticsError: 0) try: value = statistics.mean(lst) except statistics.StatisticsError: value = 0 Looking up objects in a sparse list of overrides:: (overrides[x] or default except IndexError: default).ping() try: (overrides[x] or default).ping() except IndexError: default.ping() Narrowing of exception-catching scope ------------------------------------- The following examples, taken directly from Python's standard library, demonstrate how the scope of the try/except can be conveniently narrowed. To do this with the statement form of try/except would require a temporary variable, but it's far cleaner as an expression. Lib/ipaddress.py:343:: try: ips.append(ip.ip) except AttributeError: ips.append(ip.network_address) Becomes:: ips.append(ip.ip except AttributeError: ip.network_address) The expression form is nearly equivalent to this:: try: _ = ip.ip except AttributeError: _ = ip.network_address ips.append(_) Lib/tempfile.py:130:: try: dirlist.append(_os.getcwd()) except (AttributeError, OSError): dirlist.append(_os.curdir) Becomes:: dirlist.append(_os.getcwd() except (AttributeError, OSError): _os.curdir) Lib/asyncore.py:264:: try: status.append('%s:%d' % self.addr) except TypeError: status.append(repr(self.addr)) Becomes:: status.append('%s:%d' % self.addr except TypeError: repr(self.addr)) In each case, the narrowed scope of the try/except ensures that an unexpected exception (for instance, AttributeError if "append" were misspelled) does not get caught by the same handler. This is sufficiently unlikely to be reason to break the call out into a separate line (as per the five line example above), but it is a small benefit gained as a side-effect of the conversion. Comparisons with other languages ================================ (With thanks to Andrew Barnert for compiling this section. Note that the examples given here do not reflect the current version of the proposal, and need to be edited.) `Ruby's`__ "begin…rescue…rescue…else…ensure…end" is an expression (potentially with statements inside it). It has the equivalent of an "as" clause, and the equivalent of bare except. And it uses no punctuation or keyword between the bare except/exception class/exception class with as clause and the value. (And yes, it's ambiguous unless you understand Ruby's statement/expression rules.) __ http://www.skorks.com/2009/09/ruby-exceptions-and-exception-handling/ :: x = begin computation() rescue MyException => e default(e) end; x = begin computation() rescue MyException default() end; x = begin computation() rescue default() end; x = begin computation() rescue MyException default() rescue OtherException other() end; In terms of this PEP:: x = computation() except MyException as e default(e) x = computation() except MyException default(e) x = computation() except default(e) x = computation() except MyException default() except OtherException other() `Erlang`__ has a try expression that looks like this __ http://erlang.org/doc/reference_manual/expressions.html#id79284 :: x = try computation() catch MyException:e -> default(e) end; x = try computation() catch MyException:e -> default(e); OtherException:e -> other(e) end; The class and "as" name are mandatory, but you can use "_" for either. There's also an optional "when" guard on each, and a "throw" clause that you can catch, which I won't get into. To handle multiple exceptions, you just separate the clauses with semicolons, which I guess would map to commas in Python. So:: x = try computation() except MyException as e -> default(e) x = try computation() except MyException as e -> default(e), OtherException as e->other_default(e) Erlang also has a "catch" expression, which, despite using the same keyword, is completely different, and you don't want to know about it. The ML family has two different ways of dealing with this, "handle" and "try"; the difference between the two is that "try" pattern-matches the exception, which gives you the effect of multiple except clauses and as clauses. In either form, the handler clause is punctuated by "=>" in some dialects, "->" in others. To avoid confusion, I'll write the function calls in Python style. Here's `SML's`__ "handle" __ http://www.cs.cmu.edu/~rwh/introsml/core/exceptions.htm :: let x = computation() handle MyException => default();; Here's `OCaml's`__ "try" __ http://www2.lib.uchicago.edu/keith/ocaml-class/exceptions.html :: let x = try computation() with MyException explanation -> default(explanation);; let x = try computation() with MyException(e) -> default(e) | MyOtherException() -> other_default() | (e) -> fallback(e);; In terms of this PEP, these would be something like:: x = computation() except MyException => default() x = try computation() except MyException e -> default() x = (try computation() except MyException as e -> default(e) except MyOtherException -> other_default() except BaseException as e -> fallback(e)) Many ML-inspired but not-directly-related languages from academia mix things up, usually using more keywords and fewer symbols. So, the `Oz`__ would map to Python as __ http://mozart.github.io/mozart-v1/doc-1.4.0/tutorial/node5.html :: x = try computation() catch MyException as e then default(e) Many Lisp-derived languages, like `Clojure,`__ implement try/catch as special forms (if you don't know what that means, think function-like macros), so you write, effectively __ http://clojure.org/special_forms#Special%20Forms--(try%20expr*%20catch-claus...) :: try(computation(), catch(MyException, explanation, default(explanation))) try(computation(), catch(MyException, explanation, default(explanation)), catch(MyOtherException, explanation, other_default(explanation))) In Common Lisp, this is done with a slightly clunkier `"handler-case" macro,`__ but the basic idea is the same. __ http://clhs.lisp.se/Body/m_hand_1.htm The Lisp style is, surprisingly, used by some languages that don't have macros, like Lua, where `xpcall`__ takes functions. Writing lambdas Python-style instead of Lua-style __ http://www.gammon.com.au/scripts/doc.php?lua=xpcall :: x = xpcall(lambda: expression(), lambda e: default(e)) This actually returns (true, expression()) or (false, default(e)), but I think we can ignore that part. Haskell is actually similar to Lua here (except that it's all done with monads, of course):: x = do catch(lambda: expression(), lambda e: default(e)) You can write a pattern matching expression within the function to decide what to do with it; catching and re-raising exceptions you don't want is cheap enough to be idiomatic. But Haskell infixing makes this nicer:: x = do expression() `catch` lambda: default() x = do expression() `catch` lambda e: default(e) And that makes the parallel between the lambda colon and the except colon in the proposal much more obvious:: x = expression() except Exception: default() x = expression() except Exception as e: default(e) `Tcl`__ has the other half of Lua's xpcall; catch is a function which returns true if an exception was caught, false otherwise, and you get the value out in other ways. And it's all built around the the implicit quote-and-exec that everything in Tcl is based on, making it even harder to describe in Python terms than Lisp macros, but something like __ http://wiki.tcl.tk/902 :: if {[ catch("computation()") "explanation"]} { default(explanation) } `Smalltalk`__ is also somewhat hard to map to Python. The basic version would be __ http://smalltalk.gnu.org/wiki/exceptions :: x := computation() on:MyException do:default() ... but that's basically Smalltalk's passing-arguments-with-colons syntax, not its exception-handling syntax. Deferred sub-proposals ====================== Multiple except clauses ----------------------- An examination of use-cases shows that this is not needed as often as it would be with the statement form, and as its syntax is a point on which consensus has not been reached, the entire feature is deferred. Multiple 'except' keywords could be used, and they will all catch exceptions raised in the original expression (only):: # Will catch any of the listed exceptions thrown by expr; # any exception thrown by a default expression will propagate. value = (expr except Exception1: default1 except Exception2: default2 # ... except ExceptionN: defaultN ) Currently, one of the following forms must be used:: # Will catch an Exception2 thrown by either expr or default1 value = ( (expr except Exception1: default1) except Exception2: default2 ) # Will catch an Exception2 thrown by default1 only value = (expr except Exception1: (default1 except Exception2: default2) ) Listing multiple exception clauses without parentheses is a syntax error (see above), and so a future version of Python is free to add this feature without breaking any existing code. Capturing the exception object ------------------------------ In a try/except block, the use of 'as' to capture the exception object creates a local name binding, and implicitly deletes that binding (to avoid creating a reference loop) in a finally clause. In an expression context, this makes little sense, and a proper sub-scope would be required to safely capture the exception object - something akin to the way a list comprehension is handled. However, CPython currently implements a comprehension's subscope with a nested function call, which has consequences in some contexts such as class definitions, and is therefore unsuitable for this proposal. Should there be, in future, a way to create a true subscope (which could simplify comprehensions, except expressions, with blocks, and possibly more), then this proposal could be revived; until then, its loss is not a great one, as the simple exception handling that is well suited to the expression notation used here is generally concerned only with the type of the exception, and not its value - further analysis below. This syntax would, admittedly, allow a convenient way to capture exceptions in interactive Python; returned values are captured by "_", but exceptions currently are not. This could be spelled:: >>> (expr except Exception as e: e) An examination of the Python standard library shows that, while the use of 'as' is fairly common (occurring in roughly one except clause in five), it is extremely *uncommon* in the cases which could logically be converted into the expression form. Its few uses can simply be left unchanged. Consequently, in the interests of simplicity, the 'as' clause is not included in this proposal. A subsequent Python version can add this without breaking any existing code, as 'as' is already a keyword. One example where this could possibly be useful is Lib/imaplib.py:568:: try: typ, dat = self._simple_command('LOGOUT') except: typ, dat = 'NO', ['%s: %s' % sys.exc_info()[:2]] This could become:: typ, dat = (self._simple_command('LOGOUT') except BaseException as e: ('NO', '%s: %s' % (type(e), e))) Or perhaps some other variation. This is hardly the most compelling use-case, but an intelligent look at this code could tidy it up significantly. In the absence of further examples showing any need of the exception object, I have opted to defer indefinitely the recommendation. Rejected sub-proposals ====================== finally clause -------------- The statement form try... finally or try... except... finally has no logical corresponding expression form. Therefore the finally keyword is not a part of this proposal, in any way. Bare except having different meaning ------------------------------------ With several of the proposed syntaxes, omitting the exception type name would be easy and concise, and would be tempting. For convenience's sake, it might be advantageous to have a bare 'except' clause mean something more useful than "except BaseException". Proposals included having it catch Exception, or some specific set of "common exceptions" (subclasses of a new type called ExpressionError), or have it look for a tuple named ExpressionError in the current scope, with a built-in default such as (ValueError, UnicodeError, AttributeError, EOFError, IOError, OSError, LookupError, NameError, ZeroDivisionError). All of these were rejected, for several reasons. * First and foremost, consistency with the statement form of try/except would be broken. Just as a list comprehension or ternary if expression can be explained by "breaking it out" into its vertical statement form, an expression-except should be able to be explained by a relatively mechanical translation into a near-equivalent statement. Any form of syntax common to both should therefore have the same semantics in each, and above all should not have the subtle difference of catching more in one than the other, as it will tend to attract unnoticed bugs. * Secondly, the set of appropriate exceptions to catch would itself be a huge point of contention. It would be impossible to predict exactly which exceptions would "make sense" to be caught; why bless some of them with convenient syntax and not others? * And finally (this partly because the recommendation was that a bare except should be actively encouraged, once it was reduced to a "reasonable" set of exceptions), any situation where you catch an exception you don't expect to catch is an unnecessary bug magnet. Consequently, the use of a bare 'except' is down to two possibilities: either it is syntactically forbidden in the expression form, or it is permitted with the exact same semantics as in the statement form (namely, that it catch BaseException and be unable to capture it with 'as'). Bare except clauses ------------------- PEP 8 rightly advises against the use of a bare 'except'. While it is syntactically legal in a statement, and for backward compatibility must remain so, there is little value in encouraging its use. In an expression except clause, "except:" is a SyntaxError; use the equivalent long-hand form "except BaseException:" instead. A future version of Python MAY choose to reinstate this, which can be done without breaking compatibility. Parentheses around the except clauses ------------------------------------- Should it be legal to parenthesize the except clauses, separately from the expression that could raise? Example:: value = expr ( except Exception1 [as e]: default1 except Exception2 [as e]: default2 # ... except ExceptionN [as e]: defaultN ) This is more compelling when one or both of the deferred sub-proposals of multiple except clauses and/or exception capturing is included. In their absence, the parentheses would be thus:: value = expr except ExceptionType: default value = expr (except ExceptionType: default) The advantage is minimal, and the potential to confuse a reader into thinking the except clause is separate from the expression, or into thinking this is a function call, makes this non-compelling. The expression can, of course, be parenthesized if desired, as can the default:: value = (expr) except ExceptionType: (default) As the entire expression is now required to be in parentheses (which had not been decided at the time when this was debated), there is less need to delineate this section, and in many cases it would be redundant. Short-hand for "except: pass" ----------------------------- The following was been suggested as a similar short-hand, though not technically an expression:: statement except Exception: pass try: statement except Exception: pass For instance, a common use-case is attempting the removal of a file:: os.unlink(some_file) except OSError: pass There is an equivalent already in Python 3.4, however, in contextlib:: from contextlib import suppress with suppress(OSError): os.unlink(some_file) As this is already a single line (or two with a break after the colon), there is little need of new syntax and a confusion of statement vs expression to achieve this. Common objections ================= Colons always introduce suites ------------------------------ While it is true that many of Python's syntactic elements use the colon to introduce a statement suite (if, while, with, for, etcetera), this is not by any means the sole use of the colon. Currently, Python syntax includes four cases where a colon introduces a subexpression: * dict display - { ... key:value ... } * slice notation - [start:stop:step] * function definition - parameter : annotation * lambda - arg list: return value This proposal simply adds a fifth: * except-expression - exception list: result Style guides and PEP 8 should recommend not having the colon at the end of a wrapped line, which could potentially look like the introduction of a suite, but instead advocate wrapping before the exception list, keeping the colon clearly between two expressions. Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: