Hi everyone,

I'd like to bounce this proposal off everyone and see if it's worth formulating as a PEP. I haven't found any prior discussion of it, but as we all know, searches can easily miss things, so if this is old hat please LMK.

Summary: The construction

with expr1 as var1, expr2 as var2, ...:
    body

fails (with an AttributeError) unless each expression returns a value satisfying the context manager protocol. Instead, we should permit any expression to be used. If a value does not expose an __enter__ method, it should behave as though its __enter__ method is return self; if it does not have an __exit__ method, it should behave as though that method is return False.

Rationale: The with statement has proven to be a valued extension to Python. In addition to providing improved readability for block scoping, it has strongly encouraged the use of scoped cleanups for objects which require them, such as files and mutices, in the process eliminating a lot of annoying bugs. I would argue that at present, whenever dealing with an object which requires such cleanup at a known time, with should be the default way of doing it, and not doing so is the sort of thing one should be explaining in a code comment. However, the current syntax makes a few common patterns harder to implement than they should be.

For example, this is a good pattern:

with functionReturningFile(...) as input:
   doSomething(input)

There are many cases where an Optional[file] makes sense as a parameter, as well; for example, an optional debug output stream, or an input source which may either be a file (if provided) or some non-file source (by default). Likewise, there are many cases where a function may naturally return an Optional[file], e.g. "open the file if the user has provided the filename." However, the following is not valid Python:

with functionReturningOptionalFile(...) as input:
   doSomething(input)

To handle this case, one has a few options. One may only use the 'with' in the known safe cases:

inputFile = functionReturningOptionalFile(...)
if inputFile:
    with inputFile as input:
        doSomething(input)
else:
    doSomething(None)

(NB that this requires factoring the with statement body into its own function, which may separately reduce readability and/or introduce overhead); one may dispense with the 'with' clause and do it in the pre-PEP343 way:

try:
    input = functionReturningOptionalFile(...)
    doSomething(input)
finally:
    if input:
        input.close()

(This sacrifices all the benefits of the with statement, and requires the caller to explicitly call the cleanup methods, increasing error-proneness); or one may construct an explicit 'dev-null' class and return it instead of the file:

class DevNullFile(object):
    .... implement the entire File API, including a context manager ...

(This can only be described as god-awful, especially for complex API's like files)

One obvious option would be to allow None to act as a context manager as well. We might contrast this with PEP 336, "Make None Callable." This was rejected (rightly, I think) because "it is considered a feature that None raises an error if called." For example, it means that if a function variable has been nulled, attempting to call it later raises an error, as this usually indicates a code mistake. In the case where that is not correct, it is easy to assign a noop lambda to the function variable instead of None, thus allowing the error-checking and the function-deactivating behaviors to both persist, and in a clear and easily understandable way.

In this case, OTOH, the AttributeError raised if None is passed to a with statement has significantly lower value. As the example above illustrates, there are many cases where None is an entirely legitimate value to want to pass, and unlike in the other situation, there is no equally easy way to pass it. Furthermore, if the passing of None is an error in some case, it is more useful to see that error at the site where the variable is actually used in the with statement body -- the thing for which it does not make sense to use None -- rather than at a structural declaration which essentially defines a variable scope. 

This is also the reason why such a change would impact relatively little existing code: code already has to be structured to prevent this from happening. If the assigned expression in the with statement could only return None as a result of a code bug, and a piece of existing code is relying on the with statement to catch it, it would instead fall through and be caught by their own body code, presumably giving a more coherent error anyway. This is a nonzero change in behavior, but it's well within the scope of behavior changes which normally occur from version to version.

One alternative to this proposal would be to have only None allowed to act as a context manager. However, None is not particularly special in this regard; the logic above applies to any function which might return a Union type. Furthermore, allowing it for any type would permit the following construction as well:

with var1 as expr1, var2 as expr2, ...
    .... body ...

where the common factor between the variables is no longer their need for a guaranteed cleanup operation, but simply that they are semantically all tied to a single scope of the code. This improves code clarity, as it allows the syntax to follow the intent more closely, and also eliminates one other ugliness. In present Python, the required syntax for the above would be

var1 = expr1
var3 = expr3
with var2 as expr2, var4 as expr4:
    ... body ...

where the variables in the 'with' statement are those which satisfy the context manager protocol, and the ones above it are those which do not satisfy the protocol. The split between the two is entirely tied to a nonlocal fact about the code, namely the implementation of the return values of each of the expressions, making it nonobvious which is which. Worse, if the expressions depend on each other in sequence, this may have to be broken up into

var1 = expr1
with var2 as expr2:
    var3 = expr3(var1, var2)
    with var4 as expr4(var3, ...):
        .... body ...

This seems to lose on every measure of clarity and maintainability relative to the single compound 'with' statement above.

Finally, one may ask if an (effective) default implementation of a protocol is ever a good idea. "Hidden defaults" are a great way to trigger surprising behavior, after all. However, in this case I would argue that the proposed default behavior is sufficiently obvious that there is no risk. Someone seeing a compound 'with' statement of the above form would naturally assume that its consequence is (a) to set each varN to the corresponding exprN, and (b) to execute any scope-initializers tied to exprN. Likewise, someone would naturally assume that nothing at all happens at scope exit, which is exactly the behavior of __exit__ being 'return False'. In fact, this increases local code clarity, since the counter-case -- where the implementation of each defaults (effectively) to raising an AttributeError -- is nonobvious and so requires that "nonlocal knowledge" of the code to assemble with statements.

Specific implementation proposal: Actually defining __enter__ and __exit__ methods for each object would be a lot of overhead for no good value. Instead, we can easily implement this as a change to the specified behavior of the 'with' statement, simply by changing the error-handling behavior in the SETUP_WITH and WITH_CLEANUP_START cases in ceval.c. If this does proceed to the PEP stage, I'll put together a changelist, but it's very straightforward. Null values for enter and exit are no longer errors; if enter is null, then instead of decrementing the refcount of mgr and calling enter, we leave the mgr refcount alone and push it onto the stack in place of the result. If exit is null, we simply push it onto the stack just like we would normally, and ignore it in WITH_CLEANUP_START.