[Python-Dev] cpython: Rename contextlib.ignored() to contextlib.ignore().

[Nick Coghlan]

On 17 October 2013 01:24, Barry Warsaw <barry at python.org> wrote:
> On Oct 16, 2013, at 08:31 AM, Eric Snow wrote:
>
>>When a module's maintainer makes a decision on a relatively insignificant
>>addition to the module, I'd expect little resistance or even comment (the
>>original commit was months ago).  That's why I'm surprised by the reaction
>>to this change.  It just seems like the whole thing is being blown way out
>>of proportion to the detriment of other interesting problems. Sure, things
>>could have been done differently.  But in this case it's not that big a
>>deal.
>
> Any project as big and diverse as Python needs a hierarchical structure of
> trust and responsibility.  I see it *roughly* as core dev <  module maintainer
> < release manager < bdfl delegate < bdfl.
>
> However, it's imperative to remain vigilantly transparent so that everyone
> understands the rationale and motivation behind a change, even if they
> disagree with it.  Trust is extended upwards when this transparency is
> extended downloads.  "'Cause I said so" only works at the top of the chain. ;)

Right, and that's why I never opted out of the thread entirely,
despite being seriously tempted at times :)

It's also caused me to reflect on a few things over the past few days,
including why the bar for contextlib is (at this point in time)
comparatively lower than the one for functools, collections and
itertools. I was also asked a very good question off-list as to why
TransformDict escalated to needing a PEP, while I still believe a
couple of issues is adequate for keeping this minor addition to
contextlib (for the record: the patch Zero posted to
http://bugs.python.org/issue19266 has been committed, so the pattern
is now called "suppress").

An interesting lightning talk at BrisPy last night also included a key
point: the speaker spent a lot of time creating custom internal tools
for process engineers to use as an alternative to getting expensive
Matlab licenses, and his experience was that object-oriented
programming was *just* beyond what most process engineers could cope
with in terms of writing their own code.

If you look at the history of various constructs in Python, it takes
years for idioms and patterns to build up around them, even after the
core syntax additions is made to the language. Once those idioms are
in place, though, you get to a point where you can first teach people
how to use them effectively and only later teach them how to build
their own. Because the patterns have been enshrined in things with
*names*, you also don't need to be able to recognise them on sight -
you have something to look up the first you set, rather than having to
interpret a particular shape of code as indicating a particular idiom.

This pattern recurs again and again:

- you learn how to call functions before you learn how to write them
- you learn how to instantiate and use classes before you learn how to
write new ones
- you learn how to import modules before you learn how to write your own
- you learn how to iterate before you learn how to write iterators
- you learn how to apply decorators before you learn how to build them
- you learn how to invoke context managers before you learn how to
write your own
- you often don't even have to learn how to use descriptors per se, as
it's often hidden in learning to use decorators. Learning to create
your own then lets you realise how much of what you thought you knew
about classes is actually just based on the way a few standard
descriptors work.
- you also often don't even have to learn how to use metaclasses, as
it's often hidden behind class inheritance (e.g. from abc.ABC or
enum.Enum, or from a base class in an ORM or web framework). Learning
to create your own metaclasses, on the other hand, can be completely
mindbending.

Over time, this progressively lowers the barrier to entry for Python
programming, as the intent is to enable users to *do* more without
necessarily needing to learn (much) more, as well as allowing code
authors to more clearly communicate their *intent* to code readers by
using *named* patterns rather than leaving them implicit in the code
structure.

Exception handling, however, is a notable exception(!) to that general
pattern. Appropriately *handling* exceptions is actually harder than
raising them:

    raise Exception

vs:

    try:
        <code that may raise an exception>
    except Exception:
        <do something about it>

The introduction of context managers several years ago basically
provides an opportunity to change that over time: by standardising
various common try/except idioms, it becomes feasible to postpone the
explanations of the underlying details.

Compared to the extensive toolkits we already have for functional
abstractions, iterator based abstractions and data containers, the
general purpose context manipulation idioms are still quite weak.
contextlib.closing was part of the initial toolkit in Python 2.5,
while contextlib.nested was later removed as fundamentally broken in
the presence of resources that acquire their resource in __init__
(like files).

contextlib.contextmanager is a tool for *writing* context managers not
using them, as is contextlib.ContextDecorator.

contextlib.ExitStack is similarly on the "power tool" side of the
fence, being the context management equivalent of the iter() and
next() builtins (letting you cleanly manipulate the individual steps
of a with statement, just as those two builtins let you manipulate the
individual steps of a for loop).

By contrast, suppress() and redirect_stdout() are the *first* general
purpose context managers added to contextlib since its incarnation in
Python 2.5 (although there have been many various domain specific
context manager additions elsewhere in the standard library).

It's worth unpacking the "simple" six line definition of
contextlib.suppress, to see how many Python concepts it actually
requires you to understand:

    @contextmanager
    def suppress(*exceptions):
        try:
            yield
        except exceptions:
            pass

By my count, it's at least:

1. defining functions
2. applying decorators
3. accepting arbitrary positional arguments
4. generators
5. implementing context managers with generators
6. exception handling
7. using dynamic tuples for exception handling
8. the interaction between generators, exception handling and context management

This function is *short* because of the information density provided
by the various constructs it uses, but it's *far* from being simple.

We then turn to the motivating example for the addition, taking an
eight (or five) line construct and turning it into a four (or even
two!) line construct:

First, the eight line version:

    try:
        os.remove("somefile.tmp")
    except FileNotFoundError:
        pass

    try:
        os.remove("someotherfile.tmp")
    except FileNotFoundError:
        pass

And the five line version (which only applies in this particular
example because the two commands are almost identical, and carries the
cost of conveying less information in the traceback purely through
line position):

    for name in ("somefile.tmp", "someotherfile.tmp"):
        try:
            os.remove(name)
        except FileNotFoundError:
            pass

Now, the four line version:

    with suppress(FileNotFoundError):
       os.remove("somefile.tmp")

    with suppress(FileNotFoundError):
       os.remove("someotherfile.tmp")

And even a two line version:

    with suppress(FileNotFoundError): os.remove("somefile.tmp")
    with suppress(FileNotFoundError): os.remove("someotherfile.tmp")

With the try/except version, it's much harder to tell *at a glance*
what the code is doing, because significant parts of the intent (the
specific exception being caught and the "pass" keyword that indicates
it is being suppressed) are hidden down and to the right.

Now, the other point raised in the thread is that there's a more
general idiom that's potentially of interest, which is to keep track
of whether or not an exception was thrown. I actually agree with that,
and it's something I now plan to explore in contextlib2 and then (if I
decide I actually like the construct) a PEP for Python 3.5.
Specifically, it would be a replacement for the existing search loop
idiom that was based on a context manager in order to generalise to
more than one level of loop nesting. However, the potential
ramifications of offering such a feature are vastly greater than those
of the far more limited contextlib.suppress context manager (which
only covers one very specific exception handling pattern), so it would
need to go through the PEP process.

it would be something along the lines of the following (note: I'm
*not* proposing this for 3.4. There's no way I'd propose it or
anything like it for the standard library without trialling it in
contextlib2 first, and letting it bake there for at least few months.
It may even make sense to tie the failed/missing idiom to suppress()
as was suggested in this thread rather than to the new construct):

    # Arbitrarily nested search loop
    with exit_label() as found:
        for i in range(x):
            for j in range(y):
                if matches(i, j):
                    found.exit((i, j))
    if found:
        print(found.value)

    # Delayed exception handling
    with exit_label(OSError) as failed:
        os.remove("somefile.tmp")
    # Do other things...
    if failed:
        print(failed.exc)

    with exit_label(ValueError) as missing:
        location = data.find(substr)
   if missing:


    # Implementation sketch
    def exit_label(exception):
        return _ExitLabel(exception)

    class _ExitLabel:
        def __init__(exception):
            if not exception:
                class ExitToLabel(Exception):
                    pass
                exception = ExitToLabel
            self._exception = exception
            self._sentinel = sentinel = object()
            self._value = sentinel
            self._exc = sentinel
            self._entered = False

        def __enter__(self):
            if self._entered:
                raise RuntimeError("Cannot reuse exit label")
            self._entered = True
            return self

        def __exit__(self, exc_type, exc_value, exc_tb):
           self._exc = exc_value
           if isinstance(exc_value, self._exceptions):
                traceback.clear_frames(exc_value.__traceback__)
                return True # Suppress the exception
            return False # Propagate the exception

        def __bool__(self):
           return self._exc and self._exc is not self._sentinel

         def exit(self, value=None):
             self._value = value
             raise self._exception

        @property
        def value(self):
            if self._value is self._sentinel:
                raise RuntimeError("Label value not yet set")
            return self._value

        @property
        def exc(self):
            if self._exc is self._exc:
                raise RuntimeError("Label exception result not yet set")
            return self._exc


Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia