Proposal: Allowing any variable to be used in a 'with... as...' expression
Hi everyone, I'd like to bounce this proposal off everyone and see if it's worth formulating as a PEP. I haven't found any prior discussion of it, but as we all know, searches can easily miss things, so if this is old hat please LMK. *Summary: *The construction with expr1 as var1, expr2 as var2, ...: body fails (with an AttributeError) unless each expression returns a value satisfying the context manager protocol. Instead, we should permit any expression to be used. If a value does not expose an __enter__ method, it should behave as though its __enter__ method is return self; if it does not have an __exit__ method, it should behave as though that method is return False. *Rationale: *The with statement has proven to be a valued extension to Python. In addition to providing improved readability for block scoping, it has strongly encouraged the use of scoped cleanups for objects which require them, such as files and mutices, in the process eliminating a lot of annoying bugs. I would argue that at present, whenever dealing with an object which requires such cleanup at a known time, with should be the default way of doing it, and *not* doing so is the sort of thing one should be explaining in a code comment. However, the current syntax makes a few common patterns harder to implement than they should be. For example, this is a good pattern: with functionReturningFile(...) as input: doSomething(input) There are many cases where an Optional[file] makes sense as a parameter, as well; for example, an optional debug output stream, or an input source which may either be a file (if provided) or some non-file source (by default). Likewise, there are many cases where a function may naturally return an Optional[file], e.g. "open the file if the user has provided the filename." However, the following is *not* valid Python: with functionReturningOptionalFile(...) as input: doSomething(input) To handle this case, one has a few options. One may only use the 'with' in the known safe cases: inputFile = functionReturningOptionalFile(...) if inputFile: with inputFile as input: doSomething(input) else: doSomething(None) (NB that this requires factoring the with statement body into its own function, which may separately reduce readability and/or introduce overhead); one may dispense with the 'with' clause and do it in the pre-PEP343 way: try: input = functionReturningOptionalFile(...) doSomething(input) finally: if input: input.close() (This sacrifices all the benefits of the with statement, and requires the caller to explicitly call the cleanup methods, increasing error-proneness); or one may construct an explicit 'dev-null' class and return it instead of the file: class DevNullFile(object): .... implement the entire File API, including a context manager ... (This can only be described as god-awful, especially for complex API's like files) One obvious option would be to allow None to act as a context manager as well. We might contrast this with PEP 336 <https://www.python.org/dev/peps/pep-0336/>, "Make None Callable." This was rejected (rightly, I think) because "it is considered a feature that None raises an error if called." For example, it means that if a function variable has been nulled, attempting to call it later raises an error, as this usually indicates a code mistake. In the case where that is not correct, it is easy to assign a noop lambda to the function variable instead of None, thus allowing the error-checking and the function-deactivating behaviors to both persist, and in a clear and easily understandable way. In this case, OTOH, the AttributeError raised if None is passed to a with statement has significantly lower value. As the example above illustrates, there are many cases where None is an entirely legitimate value to want to pass, and unlike in the other situation, there is no equally easy way to pass it. Furthermore, if the passing of None *is* an error in some case, it is more useful to see that error at the site where the variable is actually used in the with statement body -- the thing for which it does not make sense to use None -- rather than at a structural declaration which essentially defines a variable scope. This is also the reason why such a change would impact relatively little existing code: code already has to be structured to prevent this from happening. If the assigned expression in the with statement could only return None as a result of a code bug, and a piece of existing code is relying on the with statement to catch it, it would instead fall through and be caught by their own body code, presumably giving a more coherent error anyway. This is a nonzero change in behavior, but it's well within the scope of behavior changes which normally occur from version to version. One alternative to this proposal would be to have only None allowed to act as a context manager. However, None is not particularly special in this regard; the logic above applies to any function which might return a Union type. Furthermore, allowing it for any type would permit the following construction as well: with var1 as expr1, var2 as expr2, ... .... body ... where the common factor between the variables is no longer their need for a guaranteed cleanup operation, but simply that they are semantically all tied to a single scope of the code. This improves code clarity, as it allows the syntax to follow the intent more closely, and also eliminates one other ugliness. In present Python, the required syntax for the above would be var1 = expr1 var3 = expr3 with var2 as expr2, var4 as expr4: ... body ... where the variables in the 'with' statement are those which satisfy the context manager protocol, and the ones above it are those which do not satisfy the protocol. The split between the two is entirely tied to a nonlocal fact about the code, namely the implementation of the return values of each of the expressions, making it nonobvious which is which. Worse, if the expressions depend on each other in sequence, this may have to be broken up into var1 = expr1 with var2 as expr2: var3 = expr3(var1, var2) with var4 as expr4(var3, ...): .... body ... This seems to lose on every measure of clarity and maintainability relative to the single compound 'with' statement above. Finally, one may ask if an (effective) default implementation of a protocol is ever a good idea. "Hidden defaults" are a great way to trigger surprising behavior, after all. However, in this case I would argue that the proposed default behavior is sufficiently obvious that there is no risk. Someone seeing a compound 'with' statement of the above form would naturally assume that its consequence is (a) to set each varN to the corresponding exprN, and (b) to execute any scope-initializers tied to exprN. Likewise, someone would naturally assume that nothing at all happens at scope exit, which is exactly the behavior of __exit__ being 'return False'. In fact, this *increases* local code clarity, since the counter-case -- where the implementation of each defaults (effectively) to raising an AttributeError -- is nonobvious and so requires that "nonlocal knowledge" of the code to assemble with statements. *Specific implementation proposal: *Actually defining __enter__ and __exit__ methods for each object would be a lot of overhead for no good value. Instead, we can easily implement this as a change to the specified behavior of the 'with' statement, simply by changing the error-handling behavior in the SETUP_WITH <https://github.com/python/cpython/blob/d5d9e81ce9a7efc5bc14a5c21398d1ef6f626884/Python/ceval.c#L3119> and WITH_CLEANUP_START <https://github.com/python/cpython/blob/d5d9e81ce9a7efc5bc14a5c21398d1ef6f626884/Python/ceval.c#L3119> cases in ceval.c. If this does proceed to the PEP stage, I'll put together a changelist, but it's very straightforward. Null values for enter and exit are no longer errors; if enter is null, then instead of decrementing the refcount of mgr and calling enter, we leave the mgr refcount alone and push it onto the stack in place of the result. If exit is null, we simply push it onto the stack just like we would normally, and ignore it in WITH_CLEANUP_START.
Good idea, +1 from me. On Sun, May 19, 2019, 3:17 AM Yonatan Zunger <zunger@humu.com> wrote:
Hi everyone,
I'd like to bounce this proposal off everyone and see if it's worth formulating as a PEP. I haven't found any prior discussion of it, but as we all know, searches can easily miss things, so if this is old hat please LMK.
*Summary: *The construction
with expr1 as var1, expr2 as var2, ...: body
fails (with an AttributeError) unless each expression returns a value satisfying the context manager protocol. Instead, we should permit any expression to be used. If a value does not expose an __enter__ method, it should behave as though its __enter__ method is return self; if it does not have an __exit__ method, it should behave as though that method is return False.
*Rationale: *The with statement has proven to be a valued extension to Python. In addition to providing improved readability for block scoping, it has strongly encouraged the use of scoped cleanups for objects which require them, such as files and mutices, in the process eliminating a lot of annoying bugs. I would argue that at present, whenever dealing with an object which requires such cleanup at a known time, with should be the default way of doing it, and *not* doing so is the sort of thing one should be explaining in a code comment. However, the current syntax makes a few common patterns harder to implement than they should be.
For example, this is a good pattern:
with functionReturningFile(...) as input: doSomething(input)
There are many cases where an Optional[file] makes sense as a parameter, as well; for example, an optional debug output stream, or an input source which may either be a file (if provided) or some non-file source (by default). Likewise, there are many cases where a function may naturally return an Optional[file], e.g. "open the file if the user has provided the filename." However, the following is *not* valid Python:
with functionReturningOptionalFile(...) as input: doSomething(input)
To handle this case, one has a few options. One may only use the 'with' in the known safe cases:
inputFile = functionReturningOptionalFile(...) if inputFile: with inputFile as input: doSomething(input) else: doSomething(None)
(NB that this requires factoring the with statement body into its own function, which may separately reduce readability and/or introduce overhead); one may dispense with the 'with' clause and do it in the pre-PEP343 way:
try: input = functionReturningOptionalFile(...) doSomething(input) finally: if input: input.close()
(This sacrifices all the benefits of the with statement, and requires the caller to explicitly call the cleanup methods, increasing error-proneness); or one may construct an explicit 'dev-null' class and return it instead of the file:
class DevNullFile(object): .... implement the entire File API, including a context manager ...
(This can only be described as god-awful, especially for complex API's like files)
One obvious option would be to allow None to act as a context manager as well. We might contrast this with PEP 336 <https://www.python.org/dev/peps/pep-0336/>, "Make None Callable." This was rejected (rightly, I think) because "it is considered a feature that None raises an error if called." For example, it means that if a function variable has been nulled, attempting to call it later raises an error, as this usually indicates a code mistake. In the case where that is not correct, it is easy to assign a noop lambda to the function variable instead of None, thus allowing the error-checking and the function-deactivating behaviors to both persist, and in a clear and easily understandable way.
In this case, OTOH, the AttributeError raised if None is passed to a with statement has significantly lower value. As the example above illustrates, there are many cases where None is an entirely legitimate value to want to pass, and unlike in the other situation, there is no equally easy way to pass it. Furthermore, if the passing of None *is* an error in some case, it is more useful to see that error at the site where the variable is actually used in the with statement body -- the thing for which it does not make sense to use None -- rather than at a structural declaration which essentially defines a variable scope.
This is also the reason why such a change would impact relatively little existing code: code already has to be structured to prevent this from happening. If the assigned expression in the with statement could only return None as a result of a code bug, and a piece of existing code is relying on the with statement to catch it, it would instead fall through and be caught by their own body code, presumably giving a more coherent error anyway. This is a nonzero change in behavior, but it's well within the scope of behavior changes which normally occur from version to version.
One alternative to this proposal would be to have only None allowed to act as a context manager. However, None is not particularly special in this regard; the logic above applies to any function which might return a Union type. Furthermore, allowing it for any type would permit the following construction as well:
with var1 as expr1, var2 as expr2, ... .... body ...
where the common factor between the variables is no longer their need for a guaranteed cleanup operation, but simply that they are semantically all tied to a single scope of the code. This improves code clarity, as it allows the syntax to follow the intent more closely, and also eliminates one other ugliness. In present Python, the required syntax for the above would be
var1 = expr1 var3 = expr3 with var2 as expr2, var4 as expr4: ... body ...
where the variables in the 'with' statement are those which satisfy the context manager protocol, and the ones above it are those which do not satisfy the protocol. The split between the two is entirely tied to a nonlocal fact about the code, namely the implementation of the return values of each of the expressions, making it nonobvious which is which. Worse, if the expressions depend on each other in sequence, this may have to be broken up into
var1 = expr1 with var2 as expr2: var3 = expr3(var1, var2) with var4 as expr4(var3, ...): .... body ...
This seems to lose on every measure of clarity and maintainability relative to the single compound 'with' statement above.
Finally, one may ask if an (effective) default implementation of a protocol is ever a good idea. "Hidden defaults" are a great way to trigger surprising behavior, after all. However, in this case I would argue that the proposed default behavior is sufficiently obvious that there is no risk. Someone seeing a compound 'with' statement of the above form would naturally assume that its consequence is (a) to set each varN to the corresponding exprN, and (b) to execute any scope-initializers tied to exprN. Likewise, someone would naturally assume that nothing at all happens at scope exit, which is exactly the behavior of __exit__ being 'return False'. In fact, this *increases* local code clarity, since the counter-case -- where the implementation of each defaults (effectively) to raising an AttributeError -- is nonobvious and so requires that "nonlocal knowledge" of the code to assemble with statements.
*Specific implementation proposal: *Actually defining __enter__ and __exit__ methods for each object would be a lot of overhead for no good value. Instead, we can easily implement this as a change to the specified behavior of the 'with' statement, simply by changing the error-handling behavior in the SETUP_WITH <https://github.com/python/cpython/blob/d5d9e81ce9a7efc5bc14a5c21398d1ef6f626884/Python/ceval.c#L3119> and WITH_CLEANUP_START <https://github.com/python/cpython/blob/d5d9e81ce9a7efc5bc14a5c21398d1ef6f626884/Python/ceval.c#L3119> cases in ceval.c. If this does proceed to the PEP stage, I'll put together a changelist, but it's very straightforward. Null values for enter and exit are no longer errors; if enter is null, then instead of decrementing the refcount of mgr and calling enter, we leave the mgr refcount alone and push it onto the stack in place of the result. If exit is null, we simply push it onto the stack just like we would normally, and ignore it in WITH_CLEANUP_START. _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On 5/18/2019 8:13 PM, Yonatan Zunger wrote:
Hi everyone,
I'd like to bounce this proposal off everyone and see if it's worth formulating as a PEP. I haven't found any prior discussion of it, but as we all know, searches can easily miss things, so if this is old hat please LMK.
*Summary: *The construction
with expr1 as var1, expr2 as var2, ...: body
fails (with an AttributeError) unless each expression returns a value satisfying the context manager protocol. Instead, we should permit any expression to be used. If a value does not expose an __enter__ method, it should behave as though its __enter__ method is return self; if it does not have an __exit__ method, it should behave as though that method is return False.
So far, -1. I think trying to use a non-context manager, in particular None, as a context manager, *should* raise. Since 'with' was introduced, we have gradually added the CM methods to most everything that *should* be a CM. If you think something else specific needs conversion, please open a new thread.
*Rationale: *The with statement has proven to be a valued extension to Python. In addition to providing improved readability for block scoping, it has strongly encouraged the use of scoped cleanups for objects which require them, such as files and mutices, in the process eliminating a lot of annoying bugs. I would argue that at present, whenever dealing with an object which requires such cleanup at a known time, with should be the default way of doing it, and /not/ doing so is the sort of thing one should be explaining in a code comment. However, the current syntax makes a few common patterns harder to implement than they should be.
For example, this is a good pattern:
with functionReturningFile(...) as input: doSomething(input)
There are many cases where an Optional[file] makes sense as a parameter, as well; for example, an optional debug output stream, or an input source which may either be a file (if provided) or some non-file source (by default). Likewise, there are many cases where a function may naturally return an Optional[file], e.g. "open the file if the user has provided the filename." However, the following is /not/ valid Python:
with functionReturningOptionalFile(...) as input: doSomething(input)
To handle this case, one has a few options. One may only use the 'with' in the known safe cases:
inputFile = functionReturningOptionalFile(...) if inputFile:
Or more expicitly, 'if inputFile is not None:'
with inputFile as input: doSomething(input) else: doSomething(None)
When a function returns something useful or None, I think an immediate test is generally good practice. It will usually make the code clearer to readers. -- Terry Jan Reedy
Terry, let me make sure I'm understanding your responses. (1) Only certain things should be CM's, and those things should be explicitly denoted as such. (2)
When a function returns something useful or None, I think an immediate test is generally good practice. It will usually make the code clearer to readers.
Thinking about the first one, the purpose of the context manager protocol is to allow cleanups at scope exit, and I agree that only specified items should have these, and that this should be explicitly decorated. In this context, what I'm proposing is rather that the 'with' statement should be allowed to accept expressions which aren't context managers as well -- even though the 'with' statement provides no direct value in that case, it simplifies larger code structure to do so, because (as in the examples above) there are plenty of cases where the presence of a nontrivial context manager is conditional. Essentially, a "noop with" improves code health, while not extending the context manager protocol itself. For the second one, I completely agree that an immediate test is good practice, *if the return of None is actually an error. *The problem is that there are many cases in which this is not an error, but the 'with' statement forces it to be treated as one anyway. If the desired programmer behavior is that None would be an error in a certain context, it's much clearer for the programmer to state so explicitly at the point of assignment, and/or for the error to be raised at the point where a None value is invalid. The current rule uses the 'with' statement as an implicit error checker, but it goes off at inappropriate times because that was never its real purpose. WDYT about this? Did I capture what you were asking correctly, and do these responses make sense? On Sat, May 18, 2019 at 6:39 PM Terry Reedy <tjreedy@udel.edu> wrote:
On 5/18/2019 8:13 PM, Yonatan Zunger wrote:
Hi everyone,
I'd like to bounce this proposal off everyone and see if it's worth formulating as a PEP. I haven't found any prior discussion of it, but as we all know, searches can easily miss things, so if this is old hat please LMK.
*Summary: *The construction
with expr1 as var1, expr2 as var2, ...: body
fails (with an AttributeError) unless each expression returns a value satisfying the context manager protocol. Instead, we should permit any expression to be used. If a value does not expose an __enter__ method, it should behave as though its __enter__ method is return self; if it does not have an __exit__ method, it should behave as though that method is return False.
So far, -1. I think trying to use a non-context manager, in particular None, as a context manager, *should* raise. Since 'with' was introduced, we have gradually added the CM methods to most everything that *should* be a CM. If you think something else specific needs conversion, please open a new thread.
*Rationale: *The with statement has proven to be a valued extension to Python. In addition to providing improved readability for block scoping, it has strongly encouraged the use of scoped cleanups for objects which require them, such as files and mutices, in the process eliminating a lot of annoying bugs. I would argue that at present, whenever dealing with an object which requires such cleanup at a known time, with should be the default way of doing it, and /not/ doing so is the sort of thing one should be explaining in a code comment. However, the current syntax makes a few common patterns harder to implement than they should be.
For example, this is a good pattern:
with functionReturningFile(...) as input: doSomething(input)
There are many cases where an Optional[file] makes sense as a parameter, as well; for example, an optional debug output stream, or an input source which may either be a file (if provided) or some non-file source (by default). Likewise, there are many cases where a function may naturally return an Optional[file], e.g. "open the file if the user has provided the filename." However, the following is /not/ valid Python:
with functionReturningOptionalFile(...) as input: doSomething(input)
To handle this case, one has a few options. One may only use the 'with' in the known safe cases:
inputFile = functionReturningOptionalFile(...) if inputFile:
Or more expicitly, 'if inputFile is not None:'
with inputFile as input: doSomething(input) else: doSomething(None)
When a function returns something useful or None, I think an immediate test is generally good practice. It will usually make the code clearer to readers.
-- Terry Jan Reedy
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On Sat, May 18, 2019 at 07:44:26PM -0700, Yonatan Zunger wrote:
Essentially, a "noop with" improves code health,
I don't think I can accept that as a mere assertion. I'd like to see some concrete evidence that it does.
For example, this is a good pattern:
with functionReturningFile(...) as input: doSomething(input)
Right.
There are many cases where an Optional[file] makes sense as a parameter, as well; for example, an optional debug output stream, or an input source which may either be a file (if provided) or some non-file source (by default). Likewise, there are many cases where a function may naturally return an Optional[file], e.g. "open the file if the user has provided the filename."
If you need to use the word "if" to describe what a piece of code does, then it makes sense for the code to include the "if" keyword.
However, the following is /not/ valid Python:
with functionReturningOptionalFile(...) as input: doSomething(input)
Nor would any of these be safe: functionReturningFileOrNone().write("hello world") (expression_returning_list_or_None).append(1) functionReturningNumberOrNone() + 1 especially since it is so easy to accidentally return None when you didn't intend to: def functionReturningNumber(arg): if arg > 0: return something # oops forgot the case arg <= 0 So I think that it is a mistake to treat None where it would normally not be accepted as a no-op. It is better practice to explicitly test for None and branch, or to use a some sort of None-aware operator or wrapper.
To handle this case, one has a few options. One may only use the 'with' in the known safe cases:
inputFile = functionReturningOptionalFile(...) if inputFile:
Or more expicitly, 'if inputFile is not None:'
with inputFile as input: doSomething(input) else: doSomething(None)
Can you give an example of real-world code where the body of the with and the body of the else are identical? I would expect the more common case will be: if file := MaybeFileOrNone() is None: handle(None) else: with file as f: process(f) or for those using pre-walrus operator Pythons: file = MaybeFileOrNone() if file is None: handle(None) else: with file as f: process(f) -- Steven
On 5/18/2019 10:44 PM, Yonatan Zunger wrote:
Terry, let me make sure I'm understanding your responses.
(1) Only certain things should be CM's, and those things should be explicitly denoted as such.
Thinking about the first one, the purpose of the context manager protocol is to allow cleanups at scope exit, and I agree that only specified items should have these, and that this should be explicitly decorated.
Agreed
In this context, what I'm proposing is rather that the 'with' statement should be allowed to accept expressions which aren't context managers as well --
Disagree. This will masks bugs. The Python design philosophy is that bugs should be visible and caught as soon as possible. It is intentionally *not* stripped of all redundancy and compressed to the minimum text posible. There are languages which do not have or strongly avoid exceptions. One way is to give objects artificial default values in expressions, such as 0 for + and - and 1 for + and /. Another way is to give expressions a default backup value NUL or Bottom that propagates when used, much as NaN does is floating point arithmetic. Python is not such a language.
even though the 'with' statement provides no direct value in that case,
I see it as negative value.
it simplifies larger code structure to do so,
It does not simplify the logic.
because (as in the examples above) there are plenty of cases where the presence of a nontrivial context manager is conditional.
You have not even begun to show 'plenty'. You only showed a hypothetical made-up function name. The real open() either returns a file or raises an exception. I believe this is normal for functions that return a context manager object.
Essentially, a "noop with" improves code health, while not extending the context manager protocol itself.
Masking bugs decreases code health.
(2)
When a function returns something useful or None, I think an
immediate
test is generally good practice. It will usually make the code
clearer
to readers.
For the second one, I completely agree that an immediate test is good practice,
Ok
/if the return of None is actually an error. /
We may disagree on when returning None is an error for Python. Functions that *always* return None because their only effect is a side-effect are not an issue here. Functions that sometimes return a usable object and sometimes raise or return None are always at least a bit of a nuisance. There is no universal agreement on when to signal 'I cannot do that' 'in band' (return None) or 'out of band' (raise exception). A general discussion is out of scope here, but I suggested above than returning None instead of a context manager object is a design error.
The current rule uses the 'with' statement as an implicit error checker, but it goes off at inappropriate times because that was never its real purpose.
Why are you picking on 'with' statements? Every expression that is not universally valid for all objects is an implicit error checker. In Python, 'a + b' raises when appropriate, but that is not its main (real) purpose. -- Terry Jan Reedy
If a function that tends to return a context manager returns None, that should not mean an error occurred. If an error or unexpected condition occurred, an exception should be thrown. Errors and exceptions should result in the code within the with statement not executing. We could add a new constant to Python, “Dont.” It’s a global falsey Singleton and a context manager that causes the code inside “with” not to execute.
On May 18, 2019, at 10:44 PM, Yonatan Zunger <zunger@humu.com> wrote:
an
On Mon, Jun 24, 2019 at 4:03 AM James Lu <jamtlu@gmail.com> wrote:
If a function that tends to return a context manager returns None, that should not mean an error occurred. If an error or unexpected condition occurred, an exception should be thrown. Errors and exceptions should result in the code within the with statement not executing.
We could add a new constant to Python, “Dont.” It’s a global falsey Singleton and a context manager that causes the code inside “with” not to execute.
A cleaner way to handle this would be an exception. Unfortunately, the simple trick of suppressing the exception in __exit__ doesn't work, as __exit__ isn't called if __enter__ itself fails. Maybe you can use ExitStack to manage this? ChrisA
On Sat, Jun 22, 2019 at 12:14:02PM -0400, James Lu wrote:
If a function that tends to return a context manager returns None, that should not mean an error occurred.
I suppose that *technically* this is true. It might be designed to return a context manager or None. But: 1. Because of the way Python returns None from functions when you fall out the end without an explicit return, "returns None is a programming error (a bug)" is the safe way to bet. 2. If it is intentional, and documented, the usual practice is to explicitly check for the exceptional value: mo = re.match(a, b) if mo: print(mo.group()) We don't give None a .group() method that returns None, so that lazy coders can write this and suppress the exception: assert hasattr(None, "group") print(re.match(a, b).group()) # doesn't raise any more, yay! because there are too many ways that None *is* an error and we don't want to cover them up. Getting an exception if you have an unexpected None is a *good thing*. So I don't think that with blocks should just automatically skip running None. For every context manager that intentionally returns None, there are probably a hundred that do it accidentally, where it is a bug. I can see at least three ways to deal with the very few "context manager or None" cases, that require no changes to the language: 1. Have the context manager explicitly return None and require the caller to explicitly check for it: x = Context(arg) if x is not None: with x: ... This adds two lines and one extra level of indentation, which is not too bad for something self-documenting and explicit. In Python 3.8 we can reduce it to one line and one extra level of indentation: if (x := Context(arg)) is not None: with x: ... which might be the best solution of all. 2. Have the context manager raise an exception (not an error!) and require the caller to explicitly catch it: try: with Context(arg) as x: ... except ExceptionalCase: pass This adds three lines and one extra level of indentation. On the other hand, if your context manager can raise on actual errors as well, this is the most natural way to solve the problem: try: with Context(arg) as x: ... except SomethingBadError as err: handle(err) except ExceptionalCase: pass 3. Write the context manager to return a do-nothing mock-up instead of None. You might be able to use the Mock object in the standard library for this (I have never used Mock, so I don't know if it is suitable) but worst case it adds a one-off cost to the writer of the context manager, but it gives the caller do-nothing handling for free: with ContextOrMock(arg) as x: ... If x happens to be a do-nothing mock, the code in the block will do nothing.
If an error or unexpected condition occurred, an exception should be thrown. Errors and exceptions should result in the code within the with statement not executing.
Isn't that how the with statement already works?
We could add a new constant to Python, “Dont.” It’s a global falsey Singleton and a context manager that causes the code inside “with” not to execute.
Don't what? I might be more sympathetic to that if the name was more descriptive. Say, "SkipWithBlock", and we make it an exception. To skip the with-block, have the context manager raise SkipWithBlock from inside its __enter__ method. This is less disruptive because: - it won't hide the exception from buggy context managers that accidentally return None; - it requires an explicit raise inside the context manager; - although it adds a new built-in, it is an exception, and psychologically people tend to think of exceptions as seperate from the builtins (at least *I* do, and people I've spoken to). All this supposes that there is a moderately common need for a context manager to skip the with block, and that the existing solutions aren't sufficient. -- Steven
On Sat, May 18, 2019 at 8:17 PM Yonatan Zunger <zunger@humu.com> wrote:
Instead, we should permit any expression to be used. If a value does not expose an __enter__ method, it should behave as though its __enter__ method is return self; if it does not have an __exit__ method, it should behave as though that method is return False.
I'm not sure why you would want this. But it is really easy to implement with a named function already. I think it would be better to experiment with a wrapper function first, maybe put it on PyPI, before changing the language for something whose purpose is unclear (to me). I think you probably mean something other than what you actually write. It doesn't really make sense for "any expression" as far as I can tell. What would it possibly mean to write: with (2+2) as foo: print(foo) But anyway, my toy simple implementation which satisfies the stated requirement:
from contextlib import contextmanager @contextmanager ... def zungerfy(obj): ... if hasattr(obj, '__enter__'): ... yield obj.__enter__() ... else: ... yield obj ... if hasattr(obj, '__exit__'): ... obj.__exit__() ... with zungerfy(Foo(42)) as foo: ... print("Hello", foo.val) ... ... Hello 42 with zungerfy(open('README.md')) as foo: ... print("Hello", foo.readline())
... Hello ## About the course -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
I think you probably mean something other than what you actually write. It doesn't really make sense for "any expression" as far as I can tell. What would it possibly mean to write:
with (2+2) as foo: print(foo)
I have occasionally thought it would be nice to do something like this (and I could, but I haven't, so I guess I don't think it that strongly):
@contextmanager ... def bind(val): ... yield val ... with bind(2+2) as four: ... print(four) ... 4
-- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On Sun, May 19, 2019 at 11:46 AM David Mertz <mertz@gnosis.cx> wrote:
I think you probably mean something other than what you actually write. It doesn't really make sense for "any expression" as far as I can tell. What would it possibly mean to write:
with (2+2) as foo: print(foo)
I have occasionally thought it would be nice to do something like this (and I could, but I haven't, so I guess I don't think it that strongly):
@contextmanager ... def bind(val): ... yield val ... with bind(2+2) as four: ... print(four) ... 4
Thing is, this is slightly deceptive. People will assume/expect that the name binding 'four' ends at the end of the 'with' block. In actual fact, something like this has an entrance but no exit - there is absolutely no meaning to the unindent. Functionality would be identical to "with bind(2+2) as four: pass" followed by the same code. Python's 'with' block guards a section of code. At the end of that code, something has to get undone - a file gets closed, a database transaction gets committed/rolled back, a suppressed exception returns to normal state, etc. You can't have the __exit__ method do "del four", so with current code, there's no code-significant meaning to this block. So there are two broad options: 1) Keep it having no functional meaning, just a semantic declaration "hey, this is the only place I'm using this". Not hugely valuable, but maybe people would like it. Probably best to have the three-line bind() function. 2) Redefine the 'with' block or create a new syntactic form such that the variable actually creates a subscope. That way, at the end of the block, the name would revert to its former meaning. x = 1 with local 2 as x: print(x) # 2 print(x) # 1 This would have definite value, but would be a much larger change to the language. And variants of it have been proposed and rejected before. ChrisA
I would definitely love that kind of subscoping syntax, but as you say, that would be a much larger change. :) The use of this for things like '2+2' would be, as you say, syntactic sugar; the compiler could even be clever and strip it out of the bytecode entirely. Its only purpose in that context would be to clarify place of use, and to allow simplification of multi-element 'with' assignments like in the last example I gave in the original email. The principal purpose of this change, however, would be to enable things like optional value returns, in which in at least one fork of a conditional the 'with' statement would remain nontrivial. On Sat, May 18, 2019 at 7:02 PM Chris Angelico <rosuav@gmail.com> wrote:
On Sun, May 19, 2019 at 11:46 AM David Mertz <mertz@gnosis.cx> wrote:
I think you probably mean something other than what you actually
write. It doesn't really make sense for "any expression" as far as I can tell. What would it possibly mean to write:
with (2+2) as foo: print(foo)
I have occasionally thought it would be nice to do something like this (and I could, but I haven't, so I guess I don't think it that strongly):
@contextmanager ... def bind(val): ... yield val ... with bind(2+2) as four: ... print(four) ... 4
Thing is, this is slightly deceptive. People will assume/expect that the name binding 'four' ends at the end of the 'with' block. In actual fact, something like this has an entrance but no exit - there is absolutely no meaning to the unindent. Functionality would be identical to "with bind(2+2) as four: pass" followed by the same code.
Python's 'with' block guards a section of code. At the end of that code, something has to get undone - a file gets closed, a database transaction gets committed/rolled back, a suppressed exception returns to normal state, etc. You can't have the __exit__ method do "del four", so with current code, there's no code-significant meaning to this block.
So there are two broad options:
1) Keep it having no functional meaning, just a semantic declaration "hey, this is the only place I'm using this". Not hugely valuable, but maybe people would like it. Probably best to have the three-line bind() function.
2) Redefine the 'with' block or create a new syntactic form such that the variable actually creates a subscope. That way, at the end of the block, the name would revert to its former meaning.
x = 1 with local 2 as x: print(x) # 2 print(x) # 1
This would have definite value, but would be a much larger change to the language. And variants of it have been proposed and rejected before.
ChrisA _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On 5/18/2019 10:01 PM, Chris Angelico wrote:
2) Redefine the 'with' block or create a new syntactic form such that the variable actually creates a subscope. That way, at the end of the block, the name would revert to its former meaning.
x = 1 with local 2 as x: print(x) # 2 print(x) # 1
Since Python allows both multiple character names and unicode names, there is no shortage of names, and hence hardly any need for such temporary rebinding. Function and class bodies must be sub-scoped each for their own reasons.
This would have definite value,
Not much. If one wants to explicitly localize the use of (new) names, one can explicitly delete them. x1 = 1 ... x2, x3 = 2, 3 h = hypot(x2, x3) del x2, x3 # Okay reader, you can forget about x2, x3. ... print(1) -- Terry Jan Reedy
On Mon, May 20, 2019 at 4:46 AM Terry Reedy <tjreedy@udel.edu> wrote:
On 5/18/2019 10:01 PM, Chris Angelico wrote:
2) Redefine the 'with' block or create a new syntactic form such that the variable actually creates a subscope. That way, at the end of the block, the name would revert to its former meaning.
x = 1 with local 2 as x: print(x) # 2 print(x) # 1
Since Python allows both multiple character names and unicode names, there is no shortage of names, and hence hardly any need for such temporary rebinding. Function and class bodies must be sub-scoped each for their own reasons.
This would have definite value,
Not much. If one wants to explicitly localize the use of (new) names, one can explicitly delete them.
x1 = 1 ... x2, x3 = 2, 3 h = hypot(x2, x3) del x2, x3 # Okay reader, you can forget about x2, x3. ... print(1)
Every other Python construct that represents a "grouping" of code is delimited with indentation. Why should variables be delimited with "del"? Yes, there's the argument that using "del" works on current versions of Python and any proposed syntactic feature wouldn't, but that doesn't mean that every proposed feature has no value. Maybe it's not sufficient value to add to the language, but IMO it's more useful than redefining the 'with' block to silently return the value it was given. ChrisA
On 5/19/2019 3:00 PM, Chris Angelico wrote:
On Mon, May 20, 2019 at 4:46 AM Terry Reedy <tjreedy@udel.edu> wrote:
On 5/18/2019 10:01 PM, Chris Angelico wrote:
2) Redefine the 'with' block or create a new syntactic form such that the variable actually creates a subscope. That way, at the end of the block, the name would revert to its former meaning.
x = 1 with local 2 as x: print(x) # 2 print(x) # 1
Since Python allows both multiple character names and unicode names, there is no shortage of names, and hence hardly any need for such temporary rebinding. Function and class bodies must be sub-scoped each for their own reasons.
This would have definite value,
Not much. If one wants to explicitly localize the use of (new) names, one can explicitly delete them.
x1 = 1 ... x2, x3 = 2, 3 h = hypot(x2, x3) del x2, x3 # Okay reader, you can forget about x2, x3. ... print(1)
Every other Python construct that represents a "grouping" of code is delimited with indentation. Why should variables be delimited with "del"?
Indents are a precious resource. To me, it is a plus that del does not use one -- Terry Jan Reedy
Chris Angelico writes:
On Mon, May 20, 2019 at 4:46 AM Terry Reedy <tjreedy@udel.edu> wrote:
On 5/18/2019 10:01 PM, Chris Angelico wrote:
2) Redefine the 'with' block or create a new syntactic form such that the variable actually creates a subscope. That way, at the end of the block, the name would revert to its former meaning.
This has been discussed in the past, in the context of constructs like comprehensions. The sense was that given Python's declaration-less variables, it's usually better to keep the scope hierarchy relatively flat. (The use of the word "sense" rather than "conclusion" is deliberate.) Comprehensions were judged to be an exception, and the dummy variables in them are now confined to the scope of the comprehension. I suspect that many Pythonistas would argue that while in the general context sometimes limiting the scope of a name is useful, equally often it's a symptom of a function that has grown too big, and does too many things.
Oh, I neglected to include the definition of my boring Foo class:
class Foo(object): ... def __init__(self, val): ... self.val = val ...
The only important thing about it is that it DOES NOT have __enter__() or __exit__() methods. On Sat, May 18, 2019 at 9:40 PM David Mertz <mertz@gnosis.cx> wrote:
On Sat, May 18, 2019 at 8:17 PM Yonatan Zunger <zunger@humu.com> wrote:
Instead, we should permit any expression to be used. If a value does not expose an __enter__ method, it should behave as though its __enter__ method is return self; if it does not have an __exit__ method, it should behave as though that method is return False.
I'm not sure why you would want this. But it is really easy to implement with a named function already. I think it would be better to experiment with a wrapper function first, maybe put it on PyPI, before changing the language for something whose purpose is unclear (to me).
I think you probably mean something other than what you actually write. It doesn't really make sense for "any expression" as far as I can tell. What would it possibly mean to write:
with (2+2) as foo: print(foo)
But anyway, my toy simple implementation which satisfies the stated requirement:
from contextlib import contextmanager @contextmanager ... def zungerfy(obj): ... if hasattr(obj, '__enter__'): ... yield obj.__enter__() ... else: ... yield obj ... if hasattr(obj, '__exit__'): ... obj.__exit__() ... with zungerfy(Foo(42)) as foo: ... print("Hello", foo.val) ... ... Hello 42 with zungerfy(open('README.md')) as foo: ... print("Hello", foo.readline())
... Hello ## About the course
-- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
-- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On Sun, 19 May 2019 at 12:17, Yonatan Zunger <zunger@humu.com> wrote:
Hi everyone,
I'd like to bounce this proposal off everyone and see if it's worth formulating as a PEP. I haven't found any prior discussion of it, but as we all know, searches can easily miss things, so if this is old hat please LMK.
Summary: The construction
with expr1 as var1, expr2 as var2, ...: body
fails (with an AttributeError) unless each expression returns a value satisfying the context manager protocol. Instead, we should permit any expression to be used. If a value does not expose an __enter__ method, it should behave as though its __enter__ method is return self; if it does not have an __exit__ method, it should behave as though that method is return False.
What you describe sounds like the Maybe monad or the Option enum in Rust to me. with Maybe(expr1) as var1, expr2 as var2, ...: body doesn't seem ugly to me, and would make clear whether expr1 was something that had to have a value or could, maybe, be None. Whether the code within body would benefit most from Maybe unwrapping expr1, or from Maybe offering one of the common monad interfaces like map or iteration, is a related, interesting question. -Rob
Robert Collins writes:
What you describe sounds like the Maybe monad or the Option enum in Rust to me.
with Maybe(expr1) as var1, expr2 as var2, ...: body
doesn't seem ugly to me, and would make clear whether expr1 was something that had to have a value or could, maybe, be None.
AIUI `None` is a red herring. Yonatan is thinking more generally about any kind of polymorphism, where not all members of the union are context managers. Then he makes the obvious generalization to unions where none of the members are context managers. I was also thinking basically along your line, where rather than having the with construct do checking, having a ContextManagerProxy (probably to be spelled "CMProxy") factory which would return its argument unchanged if it was a context manager, and otherwise return a proxy that contains the argument as a member, provides __enter__ and __exit__ methods, and proxies all of the other attributes of the original via __getattribute__ or __getattr__. Sure, this is less efficient but I prefer having such behavior explicitly marked.
On 5/18/2019 8:13 PM, Yonatan Zunger wrote: ...
For example, this is a good pattern:
with functionReturningFile(...) as input: doSomething(input)
There are many cases where an Optional[file] makes sense as a parameter, as well; for example, an optional debug output stream, or an input source which may either be a file (if provided) or some non-file source (by default). Likewise, there are many cases where a function may naturally return an Optional[file], e.g. "open the file if the user has provided the filename." However, the following is /not/ valid Python:
with functionReturningOptionalFile(...) as input: doSomething(input)
To handle this case, one has a few options. One may only use the 'with' in the known safe cases:
inputFile = functionReturningOptionalFile(...) if inputFile: with inputFile as input: doSomething(input) else: doSomething(None)
Either all possible arguments to doSomething() are going to have to support the same duck-typed API, or doSomething() is going to have to have special logic to handle some types (specifically None in this case). Rather than propagate this "might be None" behavior in to doSomething(), you'd be much better off with this code (as proposed by Steven D'Aprano, but more similar to your example above): if inputFile := functionReturningOptionalFile(...) is None: doSomethingWithNone() else: with inputFile as input: doSomething(input) You're now making the decision about what to do with None as early as possible, instead of passing it off to any function you're calling. Now doSomething() only need concern itself with file-like objects. So I think adding "make 'with' work with anything" would just enable bad APIs, or duck-typing values just to make them look like other types, without actually doing anything (your DevNullFile example further down). Neither of these is desirable. Eric
participants (10)
-
Batuhan Taskaya -
Chris Angelico -
David Mertz -
Eric V. Smith -
James Lu -
Robert Collins -
Stephen J. Turnbull -
Steven D'Aprano -
Terry Reedy -
Yonatan Zunger