[Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body

[Guido van Rossum]

On Mon, Mar 16, 2009 at 2:37 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Guido van Rossum wrote:
>> Yeah, it really seems pretty much limited to contextlib.nested(). I'd
>> be happy to sacrifice the possibility to *exactly* emulate two nested
>> with-statements.
>
> Then I really haven't explained the problem well at all. One of the
> premises of PEP 343 was "Got a frequently recurring block of code that
> only has one variant sequence of statements somewhere in the middle?
> Well, now you can factor that out by putting it in a generator,
> replacing the part that varies with a yield statement and decorating the
> generator with contextlib.contextmanager."
>
> It turns out that there's a caveat that needs to go on the end of that
> though: "Be very, very sure that the yield statement can *never* be
> skipped or your context manager based version will raise a RuntimeError
> in cases where the original code would have just skipped over the
> variant section of code and resumed execution afterwards."

Well, I don't think you can take that premise (promise? :-) literally
anyways, since you cannot turn a loop into a with-statement either. So
I would be fine with just adding the condition that the variant
sequence should be executed exactly once.

> Nested context managers (whether through contextlib.nested or through
> syntactic support) just turns out to be a common case where you *don't
> necessarily know* just by looking at the code whether it can skip over
> the body of the code or not.
>
> Suppose you have 3 context managers that are regularly used together
> (call them cmA(), cmB(), cmC() for now).
>
> Writing that as:
>
>  with cmA():
>    with cmB():
>      with cmC():
>        do_something()
>
> Or the tentatively proposed:
>
>  with cmA(), cmB(), cmC():
>        do_something()
>
> is definitely OK, regardless of the details of the context managers.
>
> However, whether or not you can bundle that up into a *new* context
> manager (regardless of nesting syntax) depends on whether or not an
> outer context manager can suppress an exception raised by an inner one.
>
>  @contextmanager
>  def cmABC():
>    with cmA():
>      with cmB():
>        with cmC():
>          yield
>
>  with cmABC():
>    do_something()

While all this may make sense to the original inventor of context
managers (== you ;-), I personally find this example quite perverse.
Do you have an example taken from real life?

> The above is broken if cmB().__enter__() or cmC.__enter__() can raise an
> exception that cmA().__exit__() suppresses, or cmB.__enter__() raises an
> exception that cmB().__exit__() suppresses. So whereas the inline
> versions were clearly correct, the correctness of the second version
> currently depends on details of the context managers themselves.
> Changing the syntax to allow the three context managers to be written on
> one line does nothing to fix that semantic discrepancy between the
> original inline code and the factored out version.

I think I understand that, I just don't see a use case so important as
to warrant introducing a brand new exception deriving from
BaseException.

> PEP 377 is about changing the with statement semantics and the
> @contextmanager implementation so that the semantics of the factored out
> version actually matches that of the original inline code.
>
> You can get yourself into similar trouble without nesting context
> managers - all it takes is some way of skipping the variant code in a
> context manager that wouldn't have raised an exception if the code was
> written out inline instead of being factored out into the context manager.

Yeah, see above -- you can't write a context manager that implements a
loop either.

> Suppose for instance you wanted to use a context manager as a different
> way of running tests:
>
>  @contextmanager
>  def inline_test(self, *setup_args):
>    try:
>      self.setup(*setup_args)
>    except:
>      # Setup exception occurred, trap it and log it
>      return
>    try:
>      yield
>    except:
>      # Test exception occurred, trap it and log it
>    finally:
>      try:
>        self.teardown()
>      except:
>        # Teardown exception occurred, trap it and log it
>
>  with inline_test(setup1):
>    test_one()
>  with inline_test(setup2):
>    test_two()
>  with inline_test(setup3):
>    test_three()
>
> That approach isn't actually valid

(but your proposal would make it valid right?)

> - a context manager is not permitted
> to decide in it's __enter__() method that executing the body of the with
> statement would be a bad idea.

A setup failure sounds like a catastrophic error to me.

> The early return in the above makes it obvious that that CM is broken
> under the current semantics, but what about the following one:
>
>  @contextmanager
>  def broken_cm(self):
>    try:
>      call_user_setup()
>      try:
>        yield
>      finally:
>        call_user_teardown()
>    except UserCancel:
>      show_message("Operation aborted by user")
>
> That CM will raise RuntimeError if the user attempts to cancel an
> operation during the execution of the "call_user_setup()" method.
> Without SkipStatement or something like it, that can't be fixed.

Well pretty much anything that tries to catch asynchronous exceptions
is doomed to be 100% correct. Again the example is too abstract to be
convincing.

> Hell, I largely wrote PEP 377 to try to get out of having to document
> these semantic problems with the with statement - if I'm having trouble
> getting *python-dev* to grasp the problem, what hope do other users of
> Python have?

Hell, if you can't come up with a real use case, why bother? :-)

Perhaps you could address my worry about introducing an obscure
BaseException subclass that will forever add to the weight of the list
of built-in exceptions in all documentation?

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)