A couple of with statement ideas

I've been considering which broad concerns I want to focus on for 3.3, and have come to the conclusion that I'm going to have plenty to keep me busy between helping to smooth over some of the rough edges left over from the PEP 3118 implementation, as well as championing the module aliasing concept I first brought up a few weeks ago. That means there are a couple of with statement ideas that I still like, but almost certainly won't have the time to champion myself. So, I'm lobbing them over the fence in a half-baked form. If anyone feels strongly enough about them to get them into a PEP-worthy state, go right ahead (that's the whole point of this message). If nobody else cares enough to pick them up... well, that will simply justify my decision that they weren't important enough to worry about. Idea 1: Implicit context managers PEP 343 included the idea of a "__context__" method that permitted the creation of objects with an implicit associated context manager. It was eventually dropped because we couldn't find a good way to explain it to users at the time. Instead, you see things like "decimal.localcontext()" and various other objects that need to be called every time you want to use them as a context manager. If you want an object to have a "native" context manager, you have to either write __enter__ and __exit__ manually, or use a mixin or class decorator that adapts the __enter__/__exit__ interface to a single method (e.g. the ContextManager class and manage_context method from GarlicSim that Ram Rachum posted about here some time ago). Now that everyone has had time to get used to the way context managers work, I believe the concept may usefully make a return using the "implicit context manager" terminology. While the "__context__" name has since been claimed by PEP 3134, the alternative name used in PEP 346 ("__with__") is still a possibility. The idea of bringing back this concept would be to allow an object to implement *either* a __with__ method that returns a context manager, or else implement __enter__/__exit__ directly. The behaviour would be similar to the relationship between iterators and iterables, except that a missing "__with__" implementation would imply the "return self" semantics that iterators must implement explicitly. Anyone picking up this idea should be prepared for a lot of pushback, as the last few years have shown us that the with statement is perfectly usable without this feature, and the GarlicSim example shows that this is already feasible using existing mechanisms (even I am at best lukewarm on the concept). Idea 2: Son of PEP 377 (letting context managers skip the body of the with statement) The niggles that lead me to write PEP 377 still bug me. There are some nested with statements that are perfectly valid as inline code but will throw RuntimeError if you make them into a single context manager via contextlib.contextmanager. That PEP was rightly rejected as having too great an impact on "normal" code for something that is a comparatively exotic corner case. As I noted in issue 5251, I've since thought of an alternative, lower impact solution that may prove more acceptable to Guido and others: an optional "__entered__" method for context managers that, if present, would be executed *inside* the scope of the try/finally block before the body of the with statement itself started executing. Any exceptions raised in that method would be passed to the __exit__ method, thus increasing the flexibility of the context management protocol, while having minimal impact Most significantly, contextlib.GeneratorContextManager could be adjusted to make use of this feature to correctly skip the body of the with statement when the internal generator doesn't yield (attempt to invoke __enter__ a second time would still trigger RuntimeError, though). The translation of the problematic nested with statements into a single generator based context manager would then work correctly, functioning in the same way as the equivalent inline code. I really like this idea, since it provides a genuinely new capability to the context management protocol in a relatively low impact manner. However, as noted in the intro, there are other, more immediately practical, questions I want to deal with first, so who knows if or when I'll be able to devote a significant amount of time to this one. So, if anyone is feeling particularly keen to champion a PEP and dig into the internals of contextlib and CPython's with statement implementation, there's a couple of ideas for you to mull over :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Nick Coghlan wrote:
Now that everyone has had time to get used to the way context managers work,
"Everyone"? On comp.lang.python and the python tutor mailing list, I don't believe I've seen any questions about the use of or writing of context managers, which implies either that the feature is so intuitive and simple that there's no questions to ask, or that they aren't (yet) widely used. I'd like to comment on the rest of your post, but I don't have a clue what you're talking about *wink* -- Steven

On Fri, Feb 25, 2011 at 7:59 AM, Steven D'Aprano <steve@pearwood.info> wrote:
It's a true statement, given a suitably small definition of "everyone" :) It's at least a much larger set than it was back when AMK noticed the deep terminology confusion in the first version of the with statement and context management documentation (which was when Guido applied the Zen and dropped the __context__ method from the protocol). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

From: Nick Coghlan
I'm in favour of the idea, but the terminology problem still needs to be solved. I think it's important that the name of the object implementing this protocol not have the word "context" in it *anywhere*. I like __with__ as the special method name, as it very obviously suggests a tight connection with the with-statement. The only term I can think of right now for the object is "withable object". It's a severe abuse of the English language, I know, but unfortunately there doesn't seem to be a concise verb meaning "enter a temporary execution context". -- Greg This email may be confidential and subject to legal privilege, it may not reflect the views of the University of Canterbury, and it is not guaranteed to be virus free. If you are not an intended recipient, please notify the sender immediately and erase all copies of the message and any attachments. Please refer to http://www.canterbury.ac.nz/emaildisclaimer for more information.

On Fri, Feb 25, 2011 at 12:55 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
If the field returns a context manager, then the natural name to my mind would be __context_manager__. What I don't like about __with__ is that it's not a noun and doesn't tell me what value the attribute has or what I would do with it. Why do you think "it's important that the name ... not have the word "context" in it *anywhere*"? --- Bruce New Puzzazz newsletter: http://j.mp/puzzazz-news-2011-02 Make your web app more secure: http://j.mp/gruyere-security

On 25Feb2011 13:14, Bruce Leban <bruce@leapyear.org> wrote: | On Fri, Feb 25, 2011 at 12:55 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> | wrote: | > From: Nick Coghlan | > > It's at least a much larger set than it was back when AMK noticed the | > > deep terminology confusion in the first version of the with statement | > > and context management documentation (which was when Guido applied the | > > Zen and dropped the __context__ method from the protocol). | > | > I'm in favour of the idea, but the terminology problem still | > needs to be solved. I think it's important that the name of the | > object implementing this protocol not have the word "context" in | > it *anywhere*. | > | > I like __with__ as the special method name, as it very obviously | > suggests a tight connection with the with-statement. | | If the field returns a context manager, then the natural name to my mind | would be __context_manager__. It's very long... but accurate. | What I don't like about __with__ is that it's not a noun and doesn't tell me | what value the attribute has or what I would do with it. "enter" and "exit" aren't nouns either. I guess they are events though, whereas __with__ is supposed to return something. Grammar aside I like __with__, personally, since __context__ seems to be out. Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ BROCCOLI!! THE ONLY VEGETABLE THAT SOUNDS LIKE AN ADVERB!! - ken@aiai.ed.ac.uk (Ken Johnson)

From: Bruce Leban [mailto:bruce@leapyear.org]
Why do you think "it's important that the name ... not have the word "context" in it *anywhere*"?
Maybe it's not that important. I'm just trying to avoid any chance of lingering confusion. -- Greg This email may be confidential and subject to legal privilege, it may not reflect the views of the University of Canterbury, and it is not guaranteed to be virus free. If you are not an intended recipient, please notify the sender immediately and erase all copies of the message and any attachments. Please refer to http://www.canterbury.ac.nz/emaildisclaimer for more information.

On Sat, Feb 26, 2011 at 8:46 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
The best I have at the moment is "objects with an implicit context manager", but that's still a huge improvement over where we were when PEP 343 was implemented. Back then, we weren't sure whether "context manager" referred to the objects with __context__ or the objects with __enter__/__exit__. Since Guido decided to drop the first option completely, the question is now solidly resolved in favour of the latter, so it opens up the possibility of adding back a __with__, __cm__, __context_manager__ or __manager__ method using a new term or phrase for the objects that implement it. The iterator/iterable precedent suggests manager->manageable as a possibility, but "manageable objects" isn't easy to write *or* to say. "Managed objects" could work, though (despite being slightly less technically correct). It sounds like there's enough interest in that idea that it's worth pursuing - it still needs someone to write/champion the PEP though. No comments on the PEP 377 variant though, which is a little disappointing. I see the fact that, depending on the details of cmA() and cmB() this code: @contextmanager def cmAB(): with cmA(), cmB(): yield with cmAB(): # Do stuff may throw RuntimeError, while the inline equivalent works just fine: with cmA(), cmB(): # Do stuff as a bit of a design flaw in the underlying structure of the context management protocol. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Nick Coghlan wrote:
Urk. Maybe eliminating the word "context" is the wrong thing to do, because "managed object" sounds far too vague -- it's far from clear *how* it's being managed. Also highly likely to be confused somehow with "managed code" in the .NET world (which is a confusingly vague term in itself). My current thought is "context manager provider", long-winded though it is. -- Greg

Greg Ewing wrote:
"Inquisitionize"? It's even Pythonic! ;) Unless I misunderstood, this (__with__ or whatever it ends up being called) would be an alternate method of implementing a context manager, so why not just call these "context managers" just like objects with __enter__ and __exit__ are? - Tal Einat

On Mon, Feb 28, 2011 at 8:36 AM, Tal Einat <taleinat@gmail.com> wrote:
Because that's precisely the terminology confusion that got __context__ dropped from PEP 343 in the first place. To make the standard comparison, even iterators and iterables are not the same thing, even though the former are a subset of the latter. In this case, where objects would be expected to implement __with__ or __enter__/__exit__, but never both, the distinction should be kept even more clear. Basically, __with__ should be a context manager factory function, while context managers themselves are still required to implement __enter__/__exit__. Personally, my preference still goes to "objects with an implicit context manager". That said, until someone steps forward to write the PEP and make the case for bringing this idea back *at all*, the detailed terminology discussion is fairly moot. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Nick Coghlan wrote:
Now that everyone has had time to get used to the way context managers work,
"Everyone"? On comp.lang.python and the python tutor mailing list, I don't believe I've seen any questions about the use of or writing of context managers, which implies either that the feature is so intuitive and simple that there's no questions to ask, or that they aren't (yet) widely used. I'd like to comment on the rest of your post, but I don't have a clue what you're talking about *wink* -- Steven

On Fri, Feb 25, 2011 at 7:59 AM, Steven D'Aprano <steve@pearwood.info> wrote:
It's a true statement, given a suitably small definition of "everyone" :) It's at least a much larger set than it was back when AMK noticed the deep terminology confusion in the first version of the with statement and context management documentation (which was when Guido applied the Zen and dropped the __context__ method from the protocol). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

From: Nick Coghlan
I'm in favour of the idea, but the terminology problem still needs to be solved. I think it's important that the name of the object implementing this protocol not have the word "context" in it *anywhere*. I like __with__ as the special method name, as it very obviously suggests a tight connection with the with-statement. The only term I can think of right now for the object is "withable object". It's a severe abuse of the English language, I know, but unfortunately there doesn't seem to be a concise verb meaning "enter a temporary execution context". -- Greg This email may be confidential and subject to legal privilege, it may not reflect the views of the University of Canterbury, and it is not guaranteed to be virus free. If you are not an intended recipient, please notify the sender immediately and erase all copies of the message and any attachments. Please refer to http://www.canterbury.ac.nz/emaildisclaimer for more information.

On Fri, Feb 25, 2011 at 12:55 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
If the field returns a context manager, then the natural name to my mind would be __context_manager__. What I don't like about __with__ is that it's not a noun and doesn't tell me what value the attribute has or what I would do with it. Why do you think "it's important that the name ... not have the word "context" in it *anywhere*"? --- Bruce New Puzzazz newsletter: http://j.mp/puzzazz-news-2011-02 Make your web app more secure: http://j.mp/gruyere-security

On 25Feb2011 13:14, Bruce Leban <bruce@leapyear.org> wrote: | On Fri, Feb 25, 2011 at 12:55 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> | wrote: | > From: Nick Coghlan | > > It's at least a much larger set than it was back when AMK noticed the | > > deep terminology confusion in the first version of the with statement | > > and context management documentation (which was when Guido applied the | > > Zen and dropped the __context__ method from the protocol). | > | > I'm in favour of the idea, but the terminology problem still | > needs to be solved. I think it's important that the name of the | > object implementing this protocol not have the word "context" in | > it *anywhere*. | > | > I like __with__ as the special method name, as it very obviously | > suggests a tight connection with the with-statement. | | If the field returns a context manager, then the natural name to my mind | would be __context_manager__. It's very long... but accurate. | What I don't like about __with__ is that it's not a noun and doesn't tell me | what value the attribute has or what I would do with it. "enter" and "exit" aren't nouns either. I guess they are events though, whereas __with__ is supposed to return something. Grammar aside I like __with__, personally, since __context__ seems to be out. Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ BROCCOLI!! THE ONLY VEGETABLE THAT SOUNDS LIKE AN ADVERB!! - ken@aiai.ed.ac.uk (Ken Johnson)

From: Bruce Leban [mailto:bruce@leapyear.org]
Why do you think "it's important that the name ... not have the word "context" in it *anywhere*"?
Maybe it's not that important. I'm just trying to avoid any chance of lingering confusion. -- Greg This email may be confidential and subject to legal privilege, it may not reflect the views of the University of Canterbury, and it is not guaranteed to be virus free. If you are not an intended recipient, please notify the sender immediately and erase all copies of the message and any attachments. Please refer to http://www.canterbury.ac.nz/emaildisclaimer for more information.

On Sat, Feb 26, 2011 at 8:46 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
The best I have at the moment is "objects with an implicit context manager", but that's still a huge improvement over where we were when PEP 343 was implemented. Back then, we weren't sure whether "context manager" referred to the objects with __context__ or the objects with __enter__/__exit__. Since Guido decided to drop the first option completely, the question is now solidly resolved in favour of the latter, so it opens up the possibility of adding back a __with__, __cm__, __context_manager__ or __manager__ method using a new term or phrase for the objects that implement it. The iterator/iterable precedent suggests manager->manageable as a possibility, but "manageable objects" isn't easy to write *or* to say. "Managed objects" could work, though (despite being slightly less technically correct). It sounds like there's enough interest in that idea that it's worth pursuing - it still needs someone to write/champion the PEP though. No comments on the PEP 377 variant though, which is a little disappointing. I see the fact that, depending on the details of cmA() and cmB() this code: @contextmanager def cmAB(): with cmA(), cmB(): yield with cmAB(): # Do stuff may throw RuntimeError, while the inline equivalent works just fine: with cmA(), cmB(): # Do stuff as a bit of a design flaw in the underlying structure of the context management protocol. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Nick Coghlan wrote:
Urk. Maybe eliminating the word "context" is the wrong thing to do, because "managed object" sounds far too vague -- it's far from clear *how* it's being managed. Also highly likely to be confused somehow with "managed code" in the .NET world (which is a confusingly vague term in itself). My current thought is "context manager provider", long-winded though it is. -- Greg

Greg Ewing wrote:
"Inquisitionize"? It's even Pythonic! ;) Unless I misunderstood, this (__with__ or whatever it ends up being called) would be an alternate method of implementing a context manager, so why not just call these "context managers" just like objects with __enter__ and __exit__ are? - Tal Einat

On Mon, Feb 28, 2011 at 8:36 AM, Tal Einat <taleinat@gmail.com> wrote:
Because that's precisely the terminology confusion that got __context__ dropped from PEP 343 in the first place. To make the standard comparison, even iterators and iterables are not the same thing, even though the former are a subset of the latter. In this case, where objects would be expected to implement __with__ or __enter__/__exit__, but never both, the distinction should be kept even more clear. Basically, __with__ should be a context manager factory function, while context managers themselves are still required to implement __enter__/__exit__. Personally, my preference still goes to "objects with an implicit context manager". That said, until someone steps forward to write the PEP and make the case for bringing this idea back *at all*, the detailed terminology discussion is fairly moot. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (8)
-
Bruce Leban
-
Cameron Simpson
-
Greg Ewing
-
MRAB
-
Nick Coghlan
-
Raymond Hettinger
-
Steven D'Aprano
-
Tal Einat