On 24 August 2017 at 15:38, Barry Warsaw wrote:

Yes, but in conversations about Python, the term “context” (in the context of context managers) comes up way more often than the term “scope”. I actually think Python’s scoping rules are fairly easy to grasp, as there aren’t that many levels or ways to access them, and the natural, common interactions are basically implicit when thinking about the code you’re writing.

So while “context”, “environment”, and “scope” are certainly overloaded terms in Python, the first two have very specific existing, commonplace constructs within Python, and “scope” is both the least overloaded of the three and most closely matches what is actually going on.

A different tack would more closely align with PEP 550’s heritage in thread-local storage, calling these things “execution storage”. I think I read Guido suggest elsewhere using a namespace here so that in common code you’d only have to change the “threading.local()” call to migrate to PEP 550. It might be neat if you could do something like:

I strongly agree with Barry's reservations about using the term "context" here. I've not been following the discussion (I was away when it started and there's too many emails to go through to catch up) but I've found the use of the term "context" to be a particular problem in trying to understand what's going on just skimming the messages. I don't have a strong opinion on what name should be used, but I am definitely against using the term "context". Paul

Barry Warsaw wrote:

I actually think Python’s scoping rules are fairly easy to grasp,

The problem is that the word "scope", as generally used in relation to programming languages, has to do with visibility of names. A variable is "in scope" at a particular point in the code if you can acccess it just by writing its name there. The things we're talking about are never "in scope" in that sense. What we have is something similar to a dynamic scope, but a special action is required to access bindings in it. I can't think of any established term for things like that. The closest I'ves seen is one dialect of Scheme that called it a "fluid environment". In that dialect, fluid-let didn't create bindings in the normal scope, and you had to use specific functions to access them. -- Greg

On Thu, 24 Aug 2017 17:06:09 -0400 Barry Warsaw wrote:

On Aug 24, 2017, at 16:01, francismb wrote:

...

On 08/24/2017 03:52 PM, Barry Warsaw wrote:

...

Guido van Rossum wrote:

...

On Tue, Aug 22, 2017 at 7:12 PM, Nathaniel Smith wrote:

I worry that that's going to lead more people astray thinking this has something to do with contextlib, which it really doesn't (it's much more closely related to threading.local()).

If it's hard to find a name due collision with related meanings or already taken buckets then I would suggest just inventing a new word. An artificial one.

What do you thing?

I propose RaymondLuxuryYach_t, but we’ll have to pronounce it ThroatwobblerMangrove.

Sorry, but I think we should prononce it ThroatwobblerMangrove. Regards Antoine.

On Aug 26, 2017, at 12:43, francismb wrote:

...

...

I propose RaymondLuxuryYach_t, but we’ll have to pronounce it ThroatwobblerMangrove.

Sorry, but I think we should prononce it ThroatwobblerMangrove.

too hard to pronounce but at least is unique, I would prefer thredarena but I see naming is hard ... :-)

Oh mollusks, I thought you said bacon. -Barry

On 25 August 2017 at 20:28, Antoine Pitrou wrote:

On Fri, 25 Aug 2017 15:36:55 +1000 Nick Coghlan wrote:

...

On 24 August 2017 at 23:52, Barry Warsaw wrote:

...

Guido van Rossum wrote:

...

On Tue, Aug 22, 2017 at 7:12 PM, Nathaniel Smith wrote:

I worry that that's going to lead more people astray thinking this has something to do with contextlib, which it really doesn't (it's much more closely related to threading.local()).

This is my problem with using "Context" for this PEP. Although I can't keep up with all names being thrown around, it seems to me that in Python we already have a well-established meaning for "contexts" -- context managers, and the protocols they implement as they participate in `with` statements. We have contextlib which reinforces this. What's being proposed in PEP 550 is so far removed from this concept that I think it's just going to cause confusion (well, it does in me anyway!).

While I understand the concern, I think context locals and contextlib are more closely related than folks realise, as one of the main problems that the PEP is aiming to solve is that with statements (and hence context managers) *do not work as expected* when their body includes "yield", "yield from" or "await" .

If I write:

def read_chunks(fn, chunk_size=8192): with open(fn, "rb") as f: while True: data = f.read(chunk_size) if not data: break yield data

The "with" statement here works fine even though its body includes a "yield" (and if there had been an added "await" things would probably not be different).

The class of context managers you're talking about is in my experience a small minority (I've hardly ever used them myself, and I don't think I have ever written one).

I actually agree with this, as the vast majority of context managers are about managing the state of frame locals, rather than messing about with implicitly shared state. It's similar to the way that thread locals are vastly outnumbered by ordinary frame locals. That's part of what motivated my suggested distinction between explicit context (such as your example here) and implicit context (the trickier cases that PEP 550 aims to help handle).

So I don't think the two concepts are as closely related as you seem to think.

Of the 12 examples in https://www.python.org/dev/peps/pep-0343/#examples, two of them related to manipulating the decimal thread local context, and a third relates to manipulating a hidden "queue signals for later or process them immediately?" flag, so the use cases that PEP 550 covers have always been an intended subset of the use cases the PEP 343 covers. It's just that the explicit use cases either already work sensibly in the face of frame suspension (e.g. keeping a file open, since you're still using it), or have other reasons why you wouldn't want to suspend the frame after entering that particular context (e.g. if you suspend a frame with a lock held, there's no real way for the interpreter to guess whether that's intentional or not, so it has to assume keeping it locked is intentional, and expect you to release it explicitly if that's what you want) And while PEP 550 doesn't handle the stream redirection case natively (since it doesn't allow for suspend/resume callbacks the way PEP 525 does), it at least allows for the development of a context-aware output stream wrapper API where: * you replace the target stream globally with a context-aware wrapper that delegates attribute access to a particular context local if that's set and to the original stream otherwise * you have a context manager that sets & reverts the context local variable rather than manipulating the process global state directly

That said, I also think "context" is the best term (barring "environment" perhaps) to describe what PEP 550 is talking about. Qualifying it ("logical", etc.) helps disambiguate

+1 Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 25 August 2017 at 23:36, Yury Selivanov wrote:

On Fri, Aug 25, 2017 at 9:23 AM, Nick Coghlan wrote: [..]

...

And while PEP 550 doesn't handle the stream redirection case natively (since it doesn't allow for suspend/resume callbacks the way PEP 525 does), it at least allows for the development of a context-aware output stream wrapper API where:

PEP 525 can't handle streams redirection -- it can do it only for single-threaded programs.

Good point, whereas the hypothetical context-aware wrapper I proposed would be both thread-safe (since the process global would be changed once before the program went multi-threaded and then left alone) *and* potentially lower overhead (one context local lookup per stream attribute access, rather than a global state change every time a frame was suspended or resumed with a stream redirected)

sys.stdout/stderr/stdin are global variables, that's how the API is specified. API users assume that the change is process-wide.

Yeah, I wasn't suggesting any implicit changes to the way those work. However, it does occur to me that if we did add a new "contextlocals" API, then: 1. It could offer context-aware wrappers for stdin/stdout/stderr 2. If could offer context-aware alternatives to the stream redirection context managers in contextlib That approach would work even better than replacing sys.stdin/out/err themselves with wrappers since the wrappers wouldn't be vulnerable to being overwritten by other code that mutated the sys module. Anyway, that's not a serious proposal right now, but I do think it's decent validation of the power and flexibility of the proposed implicit state management model. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Aug 25, 2017, at 10:18, Yury Selivanov wrote:

I has a similar idea when I discovered that PEP 550 can't be used directly to fix sys.std* streams redirection. Another idea:

1. We alter PyModule to make it possible to add properties (descriptor protocol, or we implement custom __getattr__). I think we can make it so that only sys module would be able to actually use it, so it's not going to be a new feature -- just a hack for CPython.

2. We make sys.std* attributes properties, with getters and setters.

3. sys.std* setters will: issue a DeprecationWarning; set whatever the user wants to set in a global variable + set a flag (let's call it "sys.__stdout_global_modified") that sys.std* were modified.

4. sys.std* getters will use PEP 550 to lookup when __stdout_global_modified is false. If it's true -- we fallback to globals.

5. We deprecate the current API and add new APIs for the redirection system that uses PEP 550 explicitly.

6. In Python 4 we remove the old sys.std* API.

Thit is still *very* fragile: any code that writes to sys.stdout breaks all assumptions. But it offers a way to raise a warning when old-API is being used - something that we'll probably need if we add new APIs to fix this problem.

It’s ideas like this that do make me think of scopes when talking about global state and execution contexts. I understand that the current PEP 550 invokes an explicit separate namespace, but thinking bigger, if the usual patterns of just writing to sys.std{out,err} still worked and in the presence of single “threaded” execution it just did the normal thing, but in the presence of threads, async, etc. it *also* did the right thing, then code wouldn’t need to change just because you started to adopt async. That implies some scoping rules to make “sys.stdout” refer to the local execution’s sys.stdout if it were set, and the global sys.stdout if it were not. This would of course be a much deeper change to Python, with lots of tricky semantics and corner cases to get right. But it might be worth it to provide an execution model and an API that would be harder to get wrong because Python just Does the Right Thing. It’s difficult because you also have to be able to reason about what’s going on, and it’ll be imperative to be able to debug and examine the state of your execution when things go unexpected. That’s always harder when mixing dynamic scope with lexical scope, which I think is what PEP 550 is ultimately getting at. Cheers, -Barry

I think the issue with sys.std* is a distraction for this discussion. The issue also seems overstated, and I wouldn't want to change it. The ability to set these is mostly used in small programs that are also single-threaded. Libraries should never mess with them -- it's easy to explicitly pass an output file around, and for errors you should use logging or in a pinch you can write to sys.stderr, but you shouldn't set it. -- --Guido van Rossum (python.org/~guido)

On Fri, Aug 25, 2017 at 8:18 AM, Yury Selivanov wrote:

Another idea:

1. We alter PyModule to make it possible to add properties (descriptor protocol, or we implement custom __getattr__). I think we can make it so that only sys module would be able to actually use it, so it's not going to be a new feature -- just a hack for CPython.

FWIW, I've been toying with a similar problem and solution for a while. I'd like to clean up the sys module, including grouping some of the attributes (e.g. the import state), turn the get/set pairs into properties, and deprecate direct usage of some of the attributes. Though supporting descriptors on module objects would work [1] and be useful (particularly deprecating module attrs), it's sufficiently worthy of a PEP that I haven't taken the time. Instead, the approach I settled on was to rename sys to _sys and add a sys written in Python that proxies _sys. Here's a rough first pass: https://github.com/ericsnowcurrently/cpython/tree/sys-module It's effectively the same thing as ModuleType supporting descriptors, but specific only to sys. One problem with both approaches is that we'd be changing the type of the sys module. There's a relatively common idiom in the stdlib (and elsewhere) of using "type(sys)" to get ModuleType. Changing the type of the sys module breaks that. -eric [1] This is doable with a custom __getattribute__ on ModuleType, though it will impact attribute lookup on all modules. I suppose there could be a subclass that does the right thing... Anyway, this is more python-ideas territory.