[Python-ideas] Enhanced context managers with ContextManagerExit and None

[Kristján Valur Jónsson]

Phew, it is a bit awkward to discuss this in two separate places.  Those interested are invited to take a peek at the issue as well:
http://bugs.python.org/issue18677

________________________________________
Frá: Nick Coghlan [ncoghlan at gmail.com]
Sent: 13. ágúst 2013 15:52
To: Kristján Valur Jónsson
Cc: python-ideas at python.org; Ronald Oussoren
Efni: Re: [Python-ideas] Enhanced context managers with ContextManagerExit and None

> nested() was deprecated and removed because it didn't handle files (or
> any other CM that does resource acquisition in __init__) correctly.
> The fact you can't factor out arbitrary context managers had nothing
> to do with it.
Ok, I appreciate that, although I assumed otherwise.  Imho, it is not nested that is broken but "CM that do resource aquisition in __init__()".  I like to think of them as "hybrid".  __enter__ is for resource aquisition.  The only reason to do it in __init__ is if the object is its own context manager.  But that is not a very good pattern.
By deprecating and removing nested(), effectively we are giving up and saying: context managers should only be instantiated with a "with" statement, inline, and should only be used with "with" statements.
There is nothing magic about "nested()" causing it to be a "bug magnet".  The bug magnet is that some context managers don't take well to being instantiated early.  We should have fixed that, rather than effectively prohibiting any programming involving context managers.

> At the moment, a CM cannot prevent execution of the body - it must be
> paired with an if statement or an inner call that may raise an
> exception, keeping the flow control at least somewhat visible at the
> point of execution.

so:
with deal_with_error, acquire_resource as r:
   foo(r)
has more visible flow control than:
with acquire_resource_or_deal_with_error as r:
   foo(r)

With the combined with statement, you visually have a single condition manager, if not actually.

> The following is also an illegal context manager:
>
>   @contextmanager
>  def bad_cm(broken=False):
>     if not broken:
>            yield
Not with my patch :).  In fact this now works for all cm_a and cm_b:
@contextmanager
def pair(cm_a, cm_b):
    with cm_a as a, cm_b as b:
        yield a, b

contextlib._GeneratorContextManager takes care of raising ContextManagerExit if nothing is yielded.

> But that's not what you're asking about. You're asking for the ability
> to collapse two independent try statements into one.
No, not any more than I'm asking to collapse two "if" statements into one.
However, a CM is not a try statement.  It is a first class object, just like a function is.  If we can do abstract programming with functions, pass callables around, lambdas, do currying, and so on and so forth, why should we not be able to do so with context managers?



There are already things you can't factor out as functions - that's
why we have generators and context managers. It's also a fact that
there are things you can't factor out as single context managers. This
is
why we have nested context managers and also still have explicit
try/except/else/finally statements.


> Expand it out to the underlying constructs and you will see this code
> is outright buggy, because the exception handler is too broad:

>    try:
>        if not condition:
>            raise ContextManagerExit
>         execute_code() # ContextManagerExit will be eaten here
>    except ContextManagerExit:
>         pass
Now you are just being pedantic :)
ContextManagerExit is a private exception that can only be raised by if_a, so there are no stray exceptions.
Anyway, this was just to demonstrate how flow control _can_ be done with condition managers if one wanted to do so,
intentionally. 

> In current python, it is impossible to create a combined context manager that does this:
>
> if_c = nested(if_a(), if_b(condition))
>
> with if_c:
>     execute_code()  #A single context manager cannot both raise an exception from __enter__() _and_ have it supressed.
>
>This is a feature, not a bug: the with statement body will *always*
>execute, unless __enter__ raises an exception. Don't be misled by the
>ability to avoid repeating the with keyword when specifying multiple
>context managers in the same statement: semantically, that's
>equivalent to multiple nested with statements, so the outer one always
>executes, and the inner ones can only skip the body by raising an
>exception from __enter__.

I realize this, and this is the whole point of my my proposal.  That __enter__ can raise an exception and have that exception silenced by the context manager machinery, not having to build that silencing around the machinery yourself.

The point is: With two arbitrary context managers, cm_a and cm_b, cm_a _can_ silence the exception that was raised by cm_b's __enter__() method.  This may be by design, or by accident, but it is possible.  And this means that an equivalent cm_c = nested(cm_a, cm_b) is not possible.  I want to be able to deal with this edge case so that can programmatically, in addition to syntactically, nest two context managers.

> I'd be open to adding the following context manager to contextlib:
>
>    @contextmanager
>   def skip(keep=False):
Great, but that's not what my proposal is about.  It's not about flow control, but composability.

> An empty contextlib.ExitStack() instance is already a perfectly
> serviceable "do nothing" context manager, we don't need (and won't
> get) another one.
Did you miss the performance argument?  Of course I can write a do-nothing context manager.  I'm not suggesting "None" because I'm too lasy to write my own.  I'm suggesting it because context managers are very useful for other things than manageing resources, namely optional monitoring of he program:

do myfunc():
  ...
  with app_timer:
    stuff()

if app_is_being_monitored:
    app_timer = real_app_timing_contextmanager
else:
    app_timer = None.

A context manager, even a do-nothing one, is currently expensive, consisting of two dynamic function calls.  Having a "special" do-nothing context manager singleton would be beneficial in such cases when performance is important.

Cheers,

Kristján

Cheers,
Nick.