Mailman 3 PEP 463: Exception-catching expressions - Python-Dev

Feb. 20, 2014

      PEP: 463
Title: Exception-catching expressions
Version: $Revision$
Last-Modified: $Date$
Author: Chris Angelico <rosuav@gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 15-Feb-2014
Python-Version: 3.5
Post-History: 16-Feb-2014, 21-Feb-2014

Abstract
========

Just as PEP 308 introduced a means of value-based conditions in an
expression, this system allows exception-based conditions to be used
as part of an expression.

Motivation
==========

A number of functions and methods have parameters which will cause
them to return a specified value instead of raising an exception.  The
current system is ad-hoc and inconsistent, and requires that each
function be individually written to have this functionality; not all
support this.

* dict.get(key, default) - second positional argument in place of
  KeyError

* next(iter, default) - second positional argument in place of
  StopIteration

* list.pop() - no way to return a default

* seq[index] - no way to handle a bounds error

* min(sequence, default=default) - keyword argument in place of
  ValueError

* sum(sequence, start=default) - slightly different but can do the
  same job

* statistics.mean(data) - no way to handle an empty iterator

Rationale
=========

The current system requires that a function author predict the need
for a default, and implement support for it.  If this is not done, a
full try/except block is needed.

Since try/except is a statement, it is impossible to catch exceptions
in the middle of an expression.  Just as if/else does for conditionals
and lambda does for function definitions, so does this allow exception
catching in an expression context.

This provides a clean and consistent way for a function to provide a
default: it simply raises an appropriate exception, and the caller
catches it.

With some situations, an LBYL technique can be used (checking if some
sequence has enough length before indexing into it, for instance). This is
not safe in all cases, but as it is often convenient, programmers will be
tempted to sacrifice the safety of EAFP in favour of the notational brevity
of LBYL. Additionally, some LBYL techniques (eg involving getattr with
three arguments) warp the code into looking like literal strings rather
than attribute lookup, which can impact readability. A convenient EAFP
notation solves all of this.

There's no convenient way to write a helper function to do this; the
nearest is something ugly using either lambda::

    def except_(expression, exception_list, default):
        try:
            return expression()
        except exception_list:
            return default()
    value = except_(lambda: 1/x, ZeroDivisionError, lambda: float("nan"))

which is clunky, and unable to handle multiple exception clauses; or
eval::

    def except_(expression, exception_list, default):
        try:
            return eval(expression, globals_of_caller(), locals_of_caller())
        except exception_list as exc:
            l = locals_of_caller().copy()
            l['exc'] = exc
            return eval(default, globals_of_caller(), l)

    def globals_of_caller():
        return sys._getframe(2).f_globals

    def locals_of_caller():
        return sys._getframe(2).f_locals

    value = except_("""1/x""",ZeroDivisionError,""" "Can't divide by zero" """)

which is even clunkier, and relies on implementation-dependent hacks.
(Writing globals_of_caller() and locals_of_caller() for interpreters
other than CPython is left as an exercise for the reader.)

Raymond Hettinger `expresses`__ a desire for such a consistent
API. Something similar has been `requested`__ `multiple`__ `times`__
in the past.

__ https://mail.python.org/pipermail/python-ideas/2014-February/025443.html
__ https://mail.python.org/pipermail/python-ideas/2013-March/019760.html
__ https://mail.python.org/pipermail/python-ideas/2009-August/005441.html
__ https://mail.python.org/pipermail/python-ideas/2008-August/001801.html

Proposal
========

Just as the 'or' operator and the three part 'if-else' expression give
short circuiting methods of catching a falsy value and replacing it,
this syntax gives a short-circuiting method of catching an exception
and replacing it.

This currently works::

    lst = [1, 2, None, 3]
    value = lst[2] or "No value"

The proposal adds this::

    lst = [1, 2]
    value = lst[2] except IndexError: "No value"

Specifically, the syntax proposed is::

    expr except exception_list: default

where expr, exception_list, and default are all expressions.  First,
expr is evaluated.  If no exception is raised, its value is the value
of the overall expression.  If any exception is raised, exception_list
is evaluated, and should result in either a type or a tuple, just as
with the statement form of try/except.  Any matching exception will
result in the corresponding default expression being evaluated and
becoming the value of the expression.  As with the statement form of
try/except, non-matching exceptions will propagate upward.

Note that the current proposal does not allow the exception object to
be captured. Where this is needed, the statement form must be used.
(See below for discussion and elaboration on this.)

This ternary operator would be between lambda and if/else in
precedence.

Consider this example of a two-level cache::
    for key in sequence:
        x = (lvl1[key] except KeyError: (lvl2[key] except KeyError: f(key)))
        # do something with x

This cannot be rewritten as::
        x = lvl1.get(key, lvl2.get(key, f(key)))

which, despite being shorter, defeats the purpose of the cache, as it must
calculate a default value to pass to get(). The .get() version calculates
backwards; the exception-testing version calculates forwards, as would be
expected. The nearest useful equivalent would be::
        x = lvl1.get(key) or lvl2.get(key) or f(key)
which depends on the values being nonzero, as well as depending on the cache
object supporting this functionality.

Alternative Proposals
=====================

Discussion on python-ideas brought up the following syntax suggestions::

    value = expr except default if Exception [as e]
    value = expr except default for Exception [as e]
    value = expr except default from Exception [as e]
    value = expr except Exception [as e] return default
    value = expr except (Exception [as e]: default)
    value = expr except Exception [as e] try default
    value = expr except Exception [as e] continue with default
    value = default except Exception [as e] else expr
    value = try expr except Exception [as e]: default
    value = expr except default # Catches anything
    value = expr except(Exception) default # Catches only the named type(s)
    value = default if expr raise Exception
    value = expr or else default if Exception
    value = expr except Exception [as e] -> default
    value = expr except Exception [as e] pass default

It has also been suggested that a new keyword be created, rather than
reusing an existing one.  Such proposals fall into the same structure
as the last form, but with a different keyword in place of 'pass'.
Suggestions include 'then', 'when', and 'use'. Also, in the context of
the "default if expr raise Exception" proposal, it was suggested that a
new keyword "raises" be used.

All forms involving the 'as' capturing clause have been deferred from
this proposal in the interests of simplicity, but are preserved in the
table above as an accurate record of suggestions.

Open Issues
===========

Parentheses around the entire expression
----------------------------------------

Generator expressions require parentheses, unless they would be
strictly redundant.  Ambiguities with except expressions could be
resolved in the same way, forcing nested except-in-except trees to be
correctly parenthesized and requiring that the outer expression be
clearly delineated.  `Steven D'Aprano elaborates on the issue.`__

__ https://mail.python.org/pipermail/python-ideas/2014-February/025647.html

Example usage
=============

For each example, an approximately-equivalent statement form is given,
to show how the expression will be parsed.  These are not always
strictly equivalent, but will accomplish the same purpose.  It is NOT
safe for the interpreter to translate one into the other.

A number of these examples are taken directly from the Python standard
library, with file names and line numbers correct as of early Feb 2014.
Many of these patterns are extremely common.

Retrieve an argument, defaulting to None::
        cond = args[1] except IndexError: None

        # Lib/pdb.py:803:
        try:
            cond = args[1]
        except IndexError:
            cond = None

Fetch information from the system if available::
            pwd = os.getcwd() except OSError: None

            # Lib/tkinter/filedialog.py:210:
            try:
                pwd = os.getcwd()
            except OSError:
                pwd = None

Attempt a translation, falling back on the original::
        e.widget = self._nametowidget(W) except KeyError: W

        # Lib/tkinter/__init__.py:1222:
        try:
            e.widget = self._nametowidget(W)
        except KeyError:
            e.widget = W

Read from an iterator, continuing with blank lines once it's
exhausted::
        line = readline() except StopIteration: ''

        # Lib/lib2to3/pgen2/tokenize.py:370:
        try:
            line = readline()
        except StopIteration:
            line = ''

Retrieve platform-specific information (note the DRY improvement);
this particular example could be taken further, turning a series of
separate assignments into a single large dict initialization::
        # sys.abiflags may not be defined on all platforms.
        _CONFIG_VARS['abiflags'] = sys.abiflags except AttributeError: ''

        # Lib/sysconfig.py:529:
        try:
            _CONFIG_VARS['abiflags'] = sys.abiflags
        except AttributeError:
            # sys.abiflags may not be defined on all platforms.
            _CONFIG_VARS['abiflags'] = ''

Retrieve an indexed item, defaulting to None (similar to dict.get)::
    def getNamedItem(self, name):
        return self._attrs[name] except KeyError: None

    # Lib/xml/dom/minidom.py:573:
    def getNamedItem(self, name):
        try:
            return self._attrs[name]
        except KeyError:
            return None

Translate numbers to names, falling back on the numbers::
            g = grp.getgrnam(tarinfo.gname)[2] except KeyError: tarinfo.gid
            u = pwd.getpwnam(tarinfo.uname)[2] except KeyError: tarinfo.uid

            # Lib/tarfile.py:2198:
            try:
                g = grp.getgrnam(tarinfo.gname)[2]
            except KeyError:
                g = tarinfo.gid
            try:
                u = pwd.getpwnam(tarinfo.uname)[2]
            except KeyError:
                u = tarinfo.uid

Perform some lengthy calculations in EAFP mode, handling division by
zero as a sort of sticky NaN::

    value = calculate(x) except ZeroDivisionError: float("nan")

    try:
        value = calculate(x)
    except ZeroDivisionError:
        value = float("nan")

Calculate the mean of a series of numbers, falling back on zero::

    value = statistics.mean(lst) except statistics.StatisticsError: 0

    try:
        value = statistics.mean(lst)
    except statistics.StatisticsError:
        value = 0

Retrieving a message from either a cache or the internet, with auth
check::

    logging.info("Message shown to user: %s",((cache[k]
        except LookupError:
            (backend.read(k) except OSError: 'Resource not available')
        )
        if check_permission(k) else 'Access denied'
    ) except BaseException: "This is like a bare except clause")

    try:
        if check_permission(k):
            try:
                _ = cache[k]
            except LookupError:
                try:
                    _ = backend.read(k)
                except OSError:
                    _ = 'Resource not available'
        else:
            _ = 'Access denied'
    except BaseException:
        _ = "This is like a bare except clause"
    logging.info("Message shown to user: %s", _)

Looking up objects in a sparse list of overrides::

    (overrides[x] or default except IndexError: default).ping()

    try:
        (overrides[x] or default).ping()
    except IndexError:
        default.ping()

Narrowing of exception-catching scope
-------------------------------------

The following examples, taken directly from Python's standard library,
demonstrate how the scope of the try/except can be conveniently narrowed.
To do this with the statement form of try/except would require a temporary
variable, but it's far cleaner as an expression.

Lib/ipaddress.py:343::
            try:
                ips.append(ip.ip)
            except AttributeError:
                ips.append(ip.network_address)
Becomes::
            ips.append(ip.ip except AttributeError: ip.network_address)
The expression form is nearly equivalent to this::
            try:
                _ = ip.ip
            except AttributeError:
                _ = ip.network_address
            ips.append(_)

Lib/tempfile.py:130::
    try:
        dirlist.append(_os.getcwd())
    except (AttributeError, OSError):
        dirlist.append(_os.curdir)
Becomes::
    dirlist.append(_os.getcwd() except (AttributeError, OSError): _os.curdir)

Lib/asyncore.py:264::
            try:
                status.append('%s:%d' % self.addr)
            except TypeError:
                status.append(repr(self.addr))
Becomes::
            status.append('%s:%d' % self.addr except TypeError: repr(self.addr))

Comparisons with other languages
================================

(With thanks to Andrew Barnert for compiling this section.)

`Ruby's`__ "begin…rescue…rescue…else…ensure…end" is an expression
(potentially with statements inside it).  It has the equivalent of an "as"
clause, and the equivalent of bare except.  And it uses no punctuation or
keyword between the bare except/exception class/exception class with as
clause and the value.  (And yes, it's ambiguous unless you understand
Ruby's statement/expression rules.)

__ http://www.skorks.com/2009/09/ruby-exceptions-and-exception-handling/

::

    x = begin computation() rescue MyException => e default(e) end;
    x = begin computation() rescue MyException default() end;
    x = begin computation() rescue default() end;
    x = begin computation() rescue MyException default() rescue
OtherException other() end;

In terms of this PEP::

    x = computation() except MyException as e default(e)
    x = computation() except MyException default(e)
    x = computation() except default(e)
    x = computation() except MyException default() except OtherException other()

`Erlang`__ has a try expression that looks like this::

__ http://erlang.org/doc/reference_manual/expressions.html#id79284

    x = try computation() catch MyException:e -> default(e) end;
    x = try computation() catch MyException:e -> default(e);
OtherException:e -> other(e) end;

The class and "as" name are mandatory, but you can use "_" for either.
There's also an optional "when" guard on each, and a "throw" clause that
you can catch, which I won't get into.  To handle multiple exceptions,
you just separate the clauses with semicolons, which I guess would map
to commas in Python.  So::

    x = try computation() except MyException as e -> default(e)
    x = try computation() except MyException as e -> default(e),
OtherException as e->other_default(e)

Erlang also has a "catch" expression, which, despite using the same keyword,
is completely different, and you don't want to know about it.

The ML family has two different ways of dealing with this, "handle" and
"try"; the difference between the two is that "try" pattern-matches the
exception, which gives you the effect of multiple except clauses and as
clauses.  In either form, the handler clause is punctuated by "=>" in
some dialects, "->" in others.

To avoid confusion, I'll write the function calls in Python style.

Here's `SML's`__ "handle"::
__ http://www.cs.cmu.edu/~rwh/introsml/core/exceptions.htm

    let x = computation() handle MyException => default();;

Here's `OCaml's`__ "try"::
__ http://www2.lib.uchicago.edu/keith/ocaml-class/exceptions.html

    let x = try computation() with MyException explanation ->
default(explanation);;

    let x = try computation() with

        MyException(e) -> default(e)
      | MyOtherException() -> other_default()
      | (e) -> fallback(e);;

In terms of this PEP, these would be something like::

    x = computation() except MyException => default()
    x = try computation() except MyException e -> default()
    x = (try computation()
         except MyException as e -> default(e)
         except MyOtherException -> other_default()
         except BaseException as e -> fallback(e))

Many ML-inspired but not-directly-related languages from academia mix things
up, usually using more keywords and fewer symbols. So, the `Oz`__ would map
to Python as::
__ http://mozart.github.io/mozart-v1/doc-1.4.0/tutorial/node5.html

    x = try computation() catch MyException as e then default(e)

Many Lisp-derived languages, like `Clojure,`__ implement try/catch as special
forms (if you don't know what that means, think function-like macros), so you
write, effectively::
__ http://clojure.org/special_forms#Special%20Forms--(try%20expr*%20catch-claus...)

    try(computation(), catch(MyException, explanation, default(explanation)))

    try(computation(),
        catch(MyException, explanation, default(explanation)),
        catch(MyOtherException, explanation, other_default(explanation)))

In Common Lisp, this is done with a slightly clunkier `"handler-case" macro,`__
but the basic idea is the same.

__ http://clhs.lisp.se/Body/m_hand_1.htm

The Lisp style is, surprisingly, used by some languages that don't have
macros, like Lua, where `xpcall`__ takes functions. Writing lambdas
Python-style instead of Lua-style::
__ http://www.gammon.com.au/scripts/doc.php?lua=xpcall

    x = xpcall(lambda: expression(), lambda e: default(e))

This actually returns (true, expression()) or (false, default(e)), but
I think we can ignore that part.

Haskell is actually similar to Lua here (except that it's all done
with monads, of course)::

    x = do catch(lambda: expression(), lambda e: default(e))

You can write a pattern matching expression within the function to decide
what to do with it; catching and re-raising exceptions you don't want is
cheap enough to be idiomatic.

But Haskell infixing makes this nicer::

    x = do expression() `catch` lambda: default()
    x = do expression() `catch` lambda e: default(e)

And that makes the parallel between the lambda colon and the except
colon in the proposal much more obvious::

    x = expression() except Exception: default()
    x = expression() except Exception as e: default(e)

`Tcl`__ has the other half of Lua's xpcall; catch is a function which returns
true if an exception was caught, false otherwise, and you get the value out
in other ways.  And it's all built around the the implicit quote-and-exec
that everything in Tcl is based on, making it even harder to describe in
Python terms than Lisp macros, but something like::
__ http://wiki.tcl.tk/902

    if {[ catch("computation()") "explanation"]} { default(explanation) }

`Smalltalk`__ is also somewhat hard to map to Python. The basic version
would be::
__ http://smalltalk.gnu.org/wiki/exceptions

    x := computation() on:MyException do:default()

… but that's basically Smalltalk's passing-arguments-with-colons
syntax, not its exception-handling syntax.

Deferred sub-proposals
======================

Multiple except clauses
-----------------------

An examination of use-cases shows that this is not needed as often as
it would be with the statement form, and as its syntax is a point on
which consensus has not been reached, the entire feature is deferred.

In order to ensure compatibility with future versions, ensure that any
consecutive except operators are parenthesized to guarantee the
interpretation you expect.

Multiple 'except' keywords can be used, and they will all catch
exceptions raised in the original expression (only)::

    # Will catch any of the listed exceptions thrown by expr;
    # any exception thrown by a default expression will propagate.
    value = (expr
        except Exception1 [as e]: default1
        except Exception2 [as e]: default2
        # ... except ExceptionN [as e]: defaultN
    )

Using parentheses to force an alternative interpretation works as
expected::

    # Will catch an Exception2 thrown by either expr or default1
    value = (
        (expr except Exception1: default1)
        except Exception2: default2
    )
    # Will catch an Exception2 thrown by default1 only
    value = (expr except Exception1:
        (default1 except Exception2: default2)
    )

This last form is confusing and should be discouraged by PEP 8, but it
is syntactically legal: you can put any sort of expression inside a
ternary-except; ternary-except is an expression; therefore you can put
a ternary-except inside a ternary-except.

Open question: Where there are multiple except clauses, should they be
separated by commas?  It may be easier for the parser, that way::

    value = (expr
        except Exception1 [as e]: default1,
        except Exception2 [as e]: default2,
        # ... except ExceptionN [as e]: defaultN,
    )

with an optional comma after the last, as per tuple rules.  Downside:
Omitting the comma would be syntactically valid, and would have almost
identical semantics, but would nest the entire preceding expression in
its exception catching rig - a matching exception raised in the
default clause would be caught by the subsequent except clause.  As
this difference is so subtle, it runs the risk of being a major bug
magnet.

As a mitigation of this risk, this form::

    value = expr except Exception1: default1 except Exception2: default2

could be syntactically forbidden, and parentheses required if the
programmer actually wants that behaviour::

    value = (expr except Exception1: default1) except Exception2: default2

This would prevent the accidental omission of a comma from changing
the expression's meaning.

Capturing the exception object
------------------------------

In a try/except block, the use of 'as' to capture the exception object
creates a local name binding, and implicitly deletes that binding in a
finally clause.  As 'finally' is not a part of this proposal (see
below), this makes it tricky to describe; also, this use of 'as' gives
a way to create a name binding in an expression context.  Should the
default clause have an inner scope in which the name exists, shadowing
anything of the same name elsewhere?  Should it behave the same way the
statement try/except does, and unbind the name?  Should it bind the
name and leave it bound? (Almost certainly not; this behaviour was
changed in Python 3 for good reason.)

Additionally, this syntax would allow a convenient way to capture
exceptions in interactive Python; returned values are captured by "_",
but exceptions currently are not. This could be spelled:
...
...
...
expr except Exception as e: e
(The inner scope idea is tempting, but currently CPython handles list
comprehensions with a nested function call, as this is considered
easier.  It may be of value to simplify both comprehensions and except
expressions, but that is a completely separate proposal to this PEP;
alternatively, it may be better to stick with what's known to
work. `Nick Coghlan elaborates.`__)

__ https://mail.python.org/pipermail/python-ideas/2014-February/025702.html

An examination of the Python standard library shows that, while the use
of 'as' is fairly common (occurring in roughly one except clause in five),
it is extremely *uncommon* in the cases which could logically be converted
into the expression form.  Its few uses can simply be left unchanged.
Consequently, in the interests of simplicity, the 'as' clause is not
included in this proposal.  A subsequent Python version can add this without
breaking any existing code, as 'as' is already a keyword.

One example where this could possibly be useful is Lib/imaplib.py:568::
        try: typ, dat = self._simple_command('LOGOUT')
        except: typ, dat = 'NO', ['%s: %s' % sys.exc_info()[:2]]
This could become::
        typ, dat = (self._simple_command('LOGOUT')
            except BaseException as e: ('NO', '%s: %s' % (type(e), e)))
Or perhaps some other variation. This is hardly the most compelling use-case,
but an intelligent look at this code could tidy it up significantly.  In the
absence of further examples showing any need of the exception object, I have
opted to defer indefinitely the recommendation.

Rejected sub-proposals
======================

finally clause
--------------
The statement form try... finally or try... except... finally has no
logical corresponding expression form.  Therefore the finally keyword
is not a part of this proposal, in any way.

Bare except having different meaning
------------------------------------

With several of the proposed syntaxes, omitting the exception type name
would be easy and concise, and would be tempting. For convenience's sake,
it might be advantageous to have a bare 'except' clause mean something
more useful than "except BaseException". Proposals included having it
catch Exception, or some specific set of "common exceptions" (subclasses
of a new type called ExpressionError), or have it look for a tuple named
ExpressionError in the current scope, with a built-in default such as
(ValueError, UnicodeError, AttributeError, EOFError, IOError, OSError,
LookupError, NameError, ZeroDivisionError). All of these were rejected,
for severa reasons.

* First and foremost, consistency with the statement form of try/except
would be broken. Just as a list comprehension or ternary if expression
can be explained by "breaking it out" into its vertical statement form,
an expression-except should be able to be explained by a relatively
mechanical translation into a near-equivalent statement. Any form of
syntax common to both should therefore have the same semantics in each,
and above all should not have the subtle difference of catching more in
one than the other, as it will tend to attract unnoticed bugs.

* Secondly, the set of appropriate exceptions to catch would itself be
a huge point of contention. It would be impossible to predict exactly
which exceptions would "make sense" to be caught; why bless some of them
with convenient syntax and not others?

* And finally (this partly because the recommendation was that a bare
except should be actively encouraged, once it was reduced to a "reasonable"
set of exceptions), any situation where you catch an exception you don't
expect to catch is an unnecessary bug magnet.

Consequently, the use of a bare 'except' is down to two possibilities:
either it is syntactically forbidden in the expression form, or it is
permitted with the exact same semantics as in the statement form (namely,
that it catch BaseException and be unable to capture it with 'as').

Bare except clauses
-------------------

PEP 8 rightly advises against the use of a bare 'except'. While it is
syntactically legal in a statement, and for backward compatibility must
remain so, there is little value in encouraging its use. In an expression
except clause, "except:" is a SyntaxError; use the equivalent long-hand
form "except BaseException:" instead. A future version of Python MAY choose
to reinstate this, which can be done without breaking compatibility.

Parentheses around the except clauses
-------------------------------------

Should it be legal to parenthesize the except clauses, separately from
the expression that could raise? Example::

    value = expr (
        except Exception1 [as e]: default1
        except Exception2 [as e]: default2
        # ... except ExceptionN [as e]: defaultN
    )

This is more compelling when one or both of the deferred sub-proposals
of multiple except clauses and/or exception capturing is included.  In
their absence, the parentheses would be thus::
    value = expr except ExceptionType: default
    value = expr (except ExceptionType: default)

The advantage is minimal, and the potential to confuse a reader into
thinking the except clause is separate from the expression, or into thinking
this is a function call, makes this non-compelling.  The expression can, of
course, be parenthesized if desired, as can the default::
    value = (expr) except ExceptionType: (default)

Short-hand for "except: pass"
-----------------------------

The following was been suggested as a similar
short-hand, though not technically an expression::

    statement except Exception: pass

    try:
        statement
    except Exception:
        pass

For instance, a common use-case is attempting the removal of a file::
    os.unlink(some_file) except OSError: pass

There is an equivalent already in Python 3.4, however, in contextlib::
    from contextlib import suppress
    with suppress(OSError): os.unlink(some_file)

As this is already a single line (or two with a break after the colon),
there is little need of new syntax and a confusion of statement vs
expression to achieve this.

Copyright
=========

This document has been placed in the public domain.

..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End:

PEP 463: Exception-catching expressions

Jan Kaliszewski

Isaac Morland

Jan Kaliszewski