[Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body

[Nick Coghlan]

Guido van Rossum wrote:
> Yeah, it really seems pretty much limited to contextlib.nested(). I'd
> be happy to sacrifice the possibility to *exactly* emulate two nested
> with-statements.

Then I really haven't explained the problem well at all. One of the
premises of PEP 343 was "Got a frequently recurring block of code that
only has one variant sequence of statements somewhere in the middle?
Well, now you can factor that out by putting it in a generator,
replacing the part that varies with a yield statement and decorating the
generator with contextlib.contextmanager."

It turns out that there's a caveat that needs to go on the end of that
though: "Be very, very sure that the yield statement can *never* be
skipped or your context manager based version will raise a RuntimeError
in cases where the original code would have just skipped over the
variant section of code and resumed execution afterwards."

Nested context managers (whether through contextlib.nested or through
syntactic support) just turns out to be a common case where you *don't
necessarily know* just by looking at the code whether it can skip over
the body of the code or not.

Suppose you have 3 context managers that are regularly used together
(call them cmA(), cmB(), cmC() for now).

Writing that as:

  with cmA():
    with cmB():
      with cmC():
        do_something()

Or the tentatively proposed:

  with cmA(), cmB(), cmC():
        do_something()

is definitely OK, regardless of the details of the context managers.

However, whether or not you can bundle that up into a *new* context
manager (regardless of nesting syntax) depends on whether or not an
outer context manager can suppress an exception raised by an inner one.

  @contextmanager
  def cmABC():
    with cmA():
      with cmB():
        with cmC():
          yield

  with cmABC():
    do_something()

The above is broken if cmB().__enter__() or cmC.__enter__() can raise an
exception that cmA().__exit__() suppresses, or cmB.__enter__() raises an
exception that cmB().__exit__() suppresses. So whereas the inline
versions were clearly correct, the correctness of the second version
currently depends on details of the context managers themselves.
Changing the syntax to allow the three context managers to be written on
one line does nothing to fix that semantic discrepancy between the
original inline code and the factored out version.

PEP 377 is about changing the with statement semantics and the
@contextmanager implementation so that the semantics of the factored out
version actually matches that of the original inline code.

You can get yourself into similar trouble without nesting context
managers - all it takes is some way of skipping the variant code in a
context manager that wouldn't have raised an exception if the code was
written out inline instead of being factored out into the context manager.

Suppose for instance you wanted to use a context manager as a different
way of running tests:

  @contextmanager
  def inline_test(self, *setup_args):
    try:
      self.setup(*setup_args)
    except:
      # Setup exception occurred, trap it and log it
      return
    try:
      yield
    except:
      # Test exception occurred, trap it and log it
    finally:
      try:
        self.teardown()
      except:
        # Teardown exception occurred, trap it and log it

  with inline_test(setup1):
    test_one()
  with inline_test(setup2):
    test_two()
  with inline_test(setup3):
    test_three()

That approach isn't actually valid - a context manager is not permitted
to decide in it's __enter__() method that executing the body of the with
statement would be a bad idea.

The early return in the above makes it obvious that that CM is broken
under the current semantics, but what about the following one:

  @contextmanager
  def broken_cm(self):
    try:
      call_user_setup()
      try:
        yield
      finally:
        call_user_teardown()
    except UserCancel:
      show_message("Operation aborted by user")

That CM will raise RuntimeError if the user attempts to cancel an
operation during the execution of the "call_user_setup()" method.
Without SkipStatement or something like it, that can't be fixed.

Hell, I largely wrote PEP 377 to try to get out of having to document
these semantic problems with the with statement - if I'm having trouble
getting *python-dev* to grasp the problem, what hope do other users of
Python have?

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------