Hi,
This is a third iteration of the PEP.
There was some really good feedback on python-ideas and
the discussion thread became hard to follow again, so I decided to
update the PEP only three days after I published the previous
version.
Summary of the changes can be found in the "Version History"
section: https://www.python.org/dev/peps/pep-0550/#version-history
There are a few open questions left, namely the terminology
and design of ContextKey API. On the former topic, I'm quite
happy with the latest version: Execution Context, Logical
Context, and Context Key.
Thank you,
Yury
PEP: 550
Title: Execution Context
Version: $Revision$
Last-Modified: $Date$
Author: Yury Selivanov
On 19 August 2017 at 06:33, Yury Selivanov
Hi,
This is a third iteration of the PEP.
There was some really good feedback on python-ideas and the discussion thread became hard to follow again, so I decided to update the PEP only three days after I published the previous version.
Summary of the changes can be found in the "Version History" section: https://www.python.org/dev/peps/pep-0550/#version-history
There are a few open questions left, namely the terminology and design of ContextKey API. On the former topic, I'm quite happy with the latest version: Execution Context, Logical Context, and Context Key.
Nice, I quite like this version of the naming scheme and the core design in general. While Guido has a point using the same noun for two different things being somewhat confusing, I think the parallel here is the one between the local scope and the lexical (nonlocal) scope for variable names - just as your lexical scope is a nested stack of local scopes in outer functions, your execution context is your current logical context plus a nested stack of outer logical contexts.
Generator Object Modifications ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
To achieve this, we make a small set of modifications to the generator object:
* New ``__logical_context__`` attribute. This attribute is readable and writable for Python code.
* When a generator object is instantiated its ``__logical_context__`` is initialized with an empty ``LogicalContext``.
* Generator's ``.send()`` and ``.throw()`` methods are modified as follows (in pseudo-C)::
if gen.__logical_context__ is not NULL: tstate = PyThreadState_Get()
tstate.execution_context.push(gen.__logical_context__)
try: # Perform the actual `Generator.send()` or # `Generator.throw()` call. return gen.send(...) finally: gen.__logical_context__ = tstate.execution_context.pop() else: # Perform the actual `Generator.send()` or # `Generator.throw()` call. return gen.send(...)
I think this pseudo-code expansion includes a few holdovers from the original visibly-immutable API design. Given the changes since then, I think this would be clearer if the first branch used sys.run_with_logical_context(), since the logical context references at the Python layer now behave like shared mutable objects, and the apparent immutability of sys.run_with_execution_context() comes from injecting a fresh logical context every time. Also +1 to the new design considerations questions that explicitly postpones consideration of any of my "What about..."" questions from python-ideas to future PEPs. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sat, Aug 19, 2017 at 4:17 AM, Nick Coghlan
* Generator's ``.send()`` and ``.throw()`` methods are modified as follows (in pseudo-C)::
if gen.__logical_context__ is not NULL: tstate = PyThreadState_Get()
tstate.execution_context.push(gen.__logical_context__)
try: # Perform the actual `Generator.send()` or # `Generator.throw()` call. return gen.send(...) finally: gen.__logical_context__ = tstate.execution_context.pop() else: # Perform the actual `Generator.send()` or # `Generator.throw()` call. return gen.send(...)
I think this pseudo-code expansion includes a few holdovers from the original visibly-immutable API design.
Given the changes since then, I think this would be clearer if the first branch used sys.run_with_logical_context(), since the logical context references at the Python layer now behave like shared mutable objects, and the apparent immutability of sys.run_with_execution_context() comes from injecting a fresh logical context every time.
This is a good idea, I like it! It will indeed simplify the explanation. Yury
On Fri, 18 Aug 2017 16:33:27 -0400
Yury Selivanov
There are a few open questions left, namely the terminology and design of ContextKey API. On the former topic, I'm quite happy with the latest version: Execution Context, Logical Context, and Context Key.
I don't really like it. "Logical Context" is vague (there are lots of things called "context" in other libraries, so a bit of specificity would help avoid confusion), and it's not clear what is "logical" about it anyway. "Local Context" actually seemed better to me (as it reminded of threading.local() or the general notion of thread-local storage). Regards Antoine.
On Sat, Aug 19, 2017, 01:43 Antoine Pitrou
On Fri, 18 Aug 2017 16:33:27 -0400 Yury Selivanov
wrote: There are a few open questions left, namely the terminology and design of ContextKey API. On the former topic, I'm quite happy with the latest version: Execution Context, Logical Context, and Context Key.
I don't really like it. "Logical Context" is vague (there are lots of things called "context" in other libraries, so a bit of specificity would help avoid confusion), and it's not clear what is "logical" about it anyway. "Local Context" actually seemed better to me (as it reminded of threading.local() or the general notion of thread-local storage).
I have to admit that I didn't even pick up on that name change. I could go either way. I do appreciate dropping ContextItem, though, because "CI" makes me think of continuous integration. And the overall shape of the API for public consumption LGTM. -brett
Regards
Antoine.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
On 08/19/2017 01:40 AM, Antoine Pitrou wrote:
On Fri, 18 Aug 2017 16:33:27 -0400 Yury Selivanov wrote:
There are a few open questions left, namely the terminology and design of ContextKey API. On the former topic, I'm quite happy with the latest version: Execution Context, Logical Context, and Context Key.
I don't really like it. "Logical Context" is vague (there are lots of things called "context" in other libraries, so a bit of specificity would help avoid confusion), and it's not clear what is "logical" about it anyway. "Local Context" actually seemed better to me (as it reminded of threading.local() or the general notion of thread-local storage).
I am also not seeing the link between "logical" and "this local layer of environmental changes that won't be seen by those who called me". Maybe ContextLayer? Or marry the two and call it LocalContextLayer. -- ~Ethan~
The way we came to "logical context" was via "logical thread (of control)",
which is distinct from OS thread. But I think we might need to search for
another term...
On Aug 19, 2017 11:56 AM, "Ethan Furman"
On 08/19/2017 01:40 AM, Antoine Pitrou wrote:
On Fri, 18 Aug 2017 16:33:27 -0400 Yury Selivanov wrote:
There are a few open questions left, namely the terminology
and design of ContextKey API. On the former topic, I'm quite happy with the latest version: Execution Context, Logical Context, and Context Key.
I don't really like it. "Logical Context" is vague (there are lots of things called "context" in other libraries, so a bit of specificity would help avoid confusion), and it's not clear what is "logical" about it anyway. "Local Context" actually seemed better to me (as it reminded of threading.local() or the general notion of thread-local storage).
I am also not seeing the link between "logical" and "this local layer of environmental changes that won't be seen by those who called me".
Maybe ContextLayer? Or marry the two and call it LocalContextLayer.
-- ~Ethan~
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido% 40python.org
On 20 August 2017 at 10:21, Guido van Rossum
The way we came to "logical context" was via "logical thread (of control)", which is distinct from OS thread. But I think we might need to search for another term...
Right. Framing it in pragmatic terms, the two entities that we're attempting to name are: 1. The mutable storage that ContextKey.set() writes to 2. The dynamic context that ContextKey.get() queries Right now, we're using ExecutionContext for the latter, and LogicalContext for the former, and I can definitely see Antoine's point that those names don't inherently convey any information about which is which. Personally, I still like the idea of moving ExecutionContext into the "mutable storage" role, and then finding some other name for the stack of execution contexts that ck.get() queries. For example, if we named the latter after what it's *for*, we could call it the DynamicQueryContext, and end up with the following invocation functions: # Replacing ExecutionContext in the current PEP DynamicQueryContext sys.get_dynamic_query_context() sys.new_dynamic_query_context() sys.run_with_dynamic_query_context() # Suggests immutability -> good! # Suggests connection to ck.get() -> good! # Replacing LogicalContext in the current PEP ExecutionContext sys.new_execution_context() sys.run_with_execution_context() __execution_context__ attribute on generators (et al) # Neutral on mutability/immutability # Neutral on ck.set()/ck.get() An alternative would be to dispense with the ExecutionContext name entirely, and instead use DynamicWriteContext and DynamicQueryContext. If we did that, I'd suggest omitting "dynamic" from the function and attribute names (while keeping it on the types), and end up with: # Replacing ExecutionContext in the current PEP DynamicQueryContext sys.get_query_context() sys.new_query_context() sys.run_with_query_context() # Suggests immutability -> good! # Suggests connection to ck.get() -> good! # Replacing LogicalContext in the current PEP DynamicWriteContext sys.new_write_context() sys.run_with_write_context() __write_context__ attribute on generators (et al) # Suggests mutability -> good! # Suggests connection to ck.set() -> good! In this variant, the phrase "execution context" could become a general term that covered *all* of the active state that a running piece of code has access to (the dynamic context, thread locals, closure variables, module globals, process globals, etc), rather than referring to any particular runtime entity. Cheers, Nick. P.S. Since we have LookupError (rather than QueryError) as the shared base exception type for KeyError and IndexError, it would also be entirely reasonable to replace "Query" in the above suggestions with "Lookup" (DynamicLookupContext, sys.get_lookup_context(), etc). That would also have the benefit of being less jargony, and more like conversational English. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 08/19/2017 10:41 PM, Nick Coghlan wrote:
On 20 August 2017 at 10:21, Guido van Rossum wrote:
The way we came to "logical context" was via "logical thread (of control)", which is distinct from OS thread. But I think we might need to search for another term...
Right. Framing it in pragmatic terms, the two entities that we're attempting to name are:
1. The mutable storage that ContextKey.set() writes to 2. The dynamic context that ContextKey.get() queries
Right now, we're using ExecutionContext for the latter, and LogicalContext for the former, and I can definitely see Antoine's point that those names don't inherently convey any information about which is which.
[snip]
# Replacing ExecutionContext in the current PEP DynamicQueryContext sys.get_dynamic_query_context() sys.new_dynamic_query_context() sys.run_with_dynamic_query_context() # Suggests immutability -> good! # Suggests connection to ck.get() -> good!
# Replacing LogicalContext in the current PEP ExecutionContext sys.new_execution_context() sys.run_with_execution_context() __execution_context__ attribute on generators (et al) # Neutral on mutability/immutability # Neutral on ck.set()/ck.get()
[snippety snip]
# Replacing ExecutionContext in the current PEP DynamicQueryContext sys.get_query_context() sys.new_query_context() sys.run_with_query_context() # Suggests immutability -> good! # Suggests connection to ck.get() -> good!
# Replacing LogicalContext in the current PEP DynamicWriteContext sys.new_write_context() sys.run_with_write_context() __write_context__ attribute on generators (et al) # Suggests mutability -> good! # Suggests connection to ck.set() -> good!
This is just getting more confusing for me. Going back to Yury's original names for now... Relating this naming problem back to globals() and locals(), the correlation works okay for locals/LocalContext, but breaks down at the globals() level because globals() is a specific set of variables -- namely, module-level assignments, while ExecutionContext would be the equivalent of globals, nonlocals, and locals all together. A more accurate name for ExecutionContext might be ParentContext, but that would imply that the LocalContext is not included, and it is (if I finally understand what's going on, of course). So I like ExecutionContext for the stack of WhateverWeCallTheOtherContext contexts. But what do we call it? Again, if I understand what's going on, a normal, threadless, non-async, generator-bereft, plain vanilla Python program is going to have only one LocalContext no matter how many nor how deep the function call chain goes -- so in that sense Local isn't really the best word, but Context all by itself is /really/ unhelpful, and Local does imply "the most current Context Layer". Of all the names proposed so far, I think LocalContext is the best reminder of the thing that CK.key writes to. For the piled layers of LocalContexts that CK.key.get searches through, either ExecutionContext or perhaps ContextEnvironment or even ContextStack works for me (the stack portion not being an implementation detail, but a promise of how it effectively works). -- ~Ethan~
On 22 August 2017 at 09:39, Greg Ewing
Ethan Furman wrote:
So I like ExecutionContext for the stack of WhateverWeCallTheOtherContext contexts. But what do we call it?
How about ExecutionContextFrame, by analogy with stack/stack frame.
My latest suggestion to Yury was to see how the PEP reads with it called ImplicitContext, such that: * the active execution context is a stack of implicit contexts * ContextKey.set() updates the innermost implicit context * Contextkey.get() reads the whole stack of active implicit contexts * by default, generators (both sync and async) would have their own implicit context, but you could make them use the context of method callers by doing "gen.__implicit_context__ = None" * by default, coroutines would use their method caller's context, but async frameworks would make sure to give top-level tasks their own independent contexts That proposal came from an initial attempt at redrafting the Abstract and Rationale sections, where it turns out that one of the things the current version of the PEP is somewhat taking for granted is that the reader already has a particular understanding of the difference between explicit state management (i.e. passing things around as function arguments and instance attributes) and implicit state management (i.e. relying on process globals and thread locals). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, Aug 21, 2017 at 10:09 PM, Nick Coghlan
My latest suggestion to Yury was to see how the PEP reads with it called ImplicitContext, such that:
* the active execution context is a stack of implicit contexts * ContextKey.set() updates the innermost implicit context * Contextkey.get() reads the whole stack of active implicit contexts * by default, generators (both sync and async) would have their own implicit context, but you could make them use the context of method callers by doing "gen.__implicit_context__ = None" * by default, coroutines would use their method caller's context, but async frameworks would make sure to give top-level tasks their own independent contexts
That proposal came from an initial attempt at redrafting the Abstract and Rationale sections, where it turns out that one of the things the current version of the PEP is somewhat taking for granted is that the reader already has a particular understanding of the difference between explicit state management (i.e. passing things around as function arguments and instance attributes) and implicit state management (i.e. relying on process globals and thread locals).
I think I like ImplicitContext. Maybe we can go with this as a working title at least. I think we should also rethink the form the key framework-facing APIs will take, and how they are presented in the PEP -- I am now leaning towards explaining this from the start as an immutable linked list of immutable mappings, where the OS-thread state gets updated to a new linked list when it is changed (either by ContextKey.set or by the various stack manipulations). I think this falls under several Zen-of-Python points: EIBTI, and "If the implementation is easy to explain, it may be a good idea." -- --Guido van Rossum (python.org/~guido http://python.org/%7Eguido)
On Sat, 19 Aug 2017 17:21:03 -0700
Guido van Rossum
The way we came to "logical context" was via "logical thread (of control)", which is distinct from OS thread. But I think we might need to search for another term...
Perhaps "task context"? A "task" might be a logical thread, OS thread, or anything else that deserves a distinct set of implicit parameters. Regards Antoine.
On Sun, Aug 20, 2017, 03:08 Antoine Pitrou
On Sat, 19 Aug 2017 17:21:03 -0700 Guido van Rossum
wrote: The way we came to "logical context" was via "logical thread (of control)", which is distinct from OS thread. But I think we might need to search for another term...
Perhaps "task context"? A "task" might be a logical thread, OS thread, or anything else that deserves a distinct set of implicit parameters.
Maybe this is skirting too loose to the dynamic scoping, but maybe ContextFrame? This does start to line up with frames of execution which I know is a bit low-level, but then again most people will never need to know about this corner of Python. -brett
Regards
Antoine.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
On Sun, Aug 20, 2017, 03:08 Antoine Pitrou
wrote: On Sat, 19 Aug 2017 17:21:03 -0700 Guido van Rossum
wrote: The way we came to "logical context" was via "logical thread (of control)", which is distinct from OS thread. But I think we might need to search for another term...
Perhaps "task context"? A "task" might be a logical thread, OS thread, or anything else that deserves a distinct set of implicit parameters.
I think you're on to something here, though I hesitate to use "task"
because asyncio.Task is a specific implementation of it.
On Sun, Aug 20, 2017 at 7:04 PM, Brett Cannon
Maybe this is skirting too loose to the dynamic scoping, but maybe ContextFrame? This does start to line up with frames of execution which I know is a bit low-level, but then again most people will never need to know about this corner of Python.
I've been thinking that the missing link here may be the execution stack. A logical thread (LT) has a "logical stack" (LS). While a thread of control is a fairly fuzzy concept (once you include things that aren't OS threads), an execution stack is a concrete object, even if not all logical threads represent their execution stack in the same way. For example, a suspended asyncio Task has a stack that is represented by a series of stack frames linked together by await (or yield from), and greenlet apparently uses a different representation again (their term is micro-thread -- maybe we could also do something with that?). Here's the result of some more thinking about this PEP that I've been doing while writing and rewriting this message (with a surprising ending). Let's say that the details of scheduling an LT and managing its mapping onto an LS is defined by a "framework". In this terminology, OS threads are a framework, as are generators, asyncio, and greenlet. There are potentially many different such frameworks. (Some others include Twisted, Tornado and concurrent.futures.ThreadPoolExecutor.) The PEP's big idea is to recognize that there are also many different, non-framework, libraries (e.g. Decimal or Flask) that need to associate some data with an LT. The PEP therefore proposes APIs that allow libraries to do this without caring about what framework is managing the LT, and vice versa (the framework doesn't have to care about how libraries use the per-LT data). The proposed APIs uses two sets of concepts: one set for the framework and one for the library. The library-facing API is simple: create a ContextKey (CK) instance as a global variable in the library, and use its get() and set() methods to access and manipulate the data for that library associated with the current logical thread (LT). Its role is similar to threading.local(), although the API and implementation are completely different, and threading.local() is tied to a specific framework (OS threads). For frameworks the API is more complicated. There are two new classes, LogicalContext (LC) and ExecutionContext (EC). The PEP gives pseudo code suggesting that LC is/contains a dict (whose items are (CK, value) pairs) and an EC is/contains a list of LCs. But in actuality that's only one possible implementation (and not the one proposed for CPython). The key idea is rather that a framework needs to be able to take the data associated with one LT and clone it as the starting point for the data associated for a new LT. This cloning is done by sys.get_execution_context(), and the PEP proposes to use a Hash Array Mapped Trie (HAMT) as the basis for the implementation of LC and EC, to make this cloning fast. IIUC it needs to be fast to match the speed with which many frameworks create and destroy their LTs. The PEP proposes a bunch of new functions in sys for frameworks to manipulate LCs and ECs and their association with the current OS-level thread. Note that OS threads are important here because in the end all frameworks build on top of them. Honestly I'm not sure we need the distinction between LC and EC. If you read carefully some of the given example code seems to confuse them. If we could get away with only a single framework-facing concept, I would be happy calling it ExecutionContext. (Another critique of the proposal I have is that it adds too many similarly-named functions to sys. But this email is already too long and I need to go to bed.) -- --Guido van Rossum (python.org/~guido http://python.org/%7Eguido)
On 21 August 2017 at 15:03, Guido van Rossum
Honestly I'm not sure we need the distinction between LC and EC. If you read carefully some of the given example code seems to confuse them. If we could get away with only a single framework-facing concept, I would be happy calling it ExecutionContext.
Unfortunately, I don't think we can, and that's why I tried to reframe the discussion in terms of "Where ContextKey.set() writes to" and "Where ContextKey.get() looks things up". Consider the following toy generator: def tracking_gen(): start_tracking_iterations() while True: tally_iteration() yield task_id = ContextKey("task_id") iter_counter = ContextKey("iter_counter") def start_tracking_iterations(): iter_counter.set(collection.Counter()) def tally_iteration(): current_task = task_id.get() # Set elsewhere iter_counter.get()[current_task] += 1 Now, this isn't a very *sensible* generator (since it could just use a regular object instance for tracking instead of a context variable), but nevertheless, it's one that we would expect to work, and it's one that we would expect to exhibit the following properties: 1. When tally_iteration() calls task_id.get(), we expect that to be resolved in the context calling next() on the instance, *not* the context where the generator was first created 2. When tally_iteration() calls iter_counter.get(), we expect that to be resolved in the same context where start_tracking_iterations() called iter_counter.set() This has consequences for the design in the PEP: * what we want to capture at generator creation time is the context where writes will happen, and we also want that to be the innermost context used for lookups * other than that innermost context, we want everything else to be dynamic * this means that "mutable context saved on the generator" and "entire dynamic context visible when the generator runs" aren't the same thing And hence the introduction of the LocalContext/LogicalContext terminology for the former, and the ExecutionContext terminology for the latter. It's also where the analogy with ChainMap came from (although I don't think this has made it into the PEP itself): * LogicalContext is the equivalent of the individual mappings * ExecutionContext is the equivalent of ChainMap * ContextKey.get() replaces ChainMap.__getitem__ * ContextKey.set(value) replaces ChainMap.__setitem__ * ContextKey.set(None) replaces ChainMap.__delitem__ While the context is defined conceptually as a nested chain of key:value mappings, we avoid using the mapping syntax because of the way the values can shift dynamically out from under you based on who called you - while the ChainMap analogy is hopefully helpful to understanding, we don't want people taking it too literally or things will become more confusing rather than less. Despite that risk, taking the analogy further is where the DynamicWriteContext + DynamicLookupContext terminology idea came from: * like ChainMap.new_child(), adjusting the DynamicWriteContext changes what ck.set() affects, and also sets the innermost context for ck.get() * like using a different ChainMap, adjusting the DynamicLookupContext changes what ck.get() can see (unlike ChainMap, it also isolates ck.set() by default) I'll also note that the first iteration of the PEP didn't really make this distinction, and it caused a problem that Nathaniel pointed out: generators would "snapshot" their entire dynamic context when first created, and then never adjust it for external changes between iterations. This meant that if you adjusted something like the decimal context outside the generator after creating it, it would ignore those changes - instead of having the problem of changes inside the generator leaking out, we instead had the problem of changes outside the generator *not* making their way in, even if you wanted them to. Due to that heritage, fixing some of the examples could easily have been missed in the v2 rewrite that introduced the distinction between the two kinds of context.
(Another critique of the proposal I have is that it adds too many similarly-named functions to sys. But this email is already too long and I need to go to bed.)
If it helps any, one of the ideas that has come up is to put all of the proposed context manipulation APIs in contextlib rather than in sys, and I think that's a reasonable idea (I don't think any of us actually like the notion of adding that many new subsystem specific APIs directly to sys). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, Aug 21, 2017 at 7:12 AM, Nick Coghlan
On 21 August 2017 at 15:03, Guido van Rossum
wrote: Honestly I'm not sure we need the distinction between LC and EC. If you read carefully some of the given example code seems to confuse them. If we could get away with only a single framework-facing concept, I would be happy calling it ExecutionContext.
Unfortunately, I don't think we can, and that's why I tried to reframe the discussion in terms of "Where ContextKey.set() writes to" and "Where ContextKey.get() looks things up".
Consider the following toy generator:
def tracking_gen(): start_tracking_iterations() while True: tally_iteration() yield
task_id = ContextKey("task_id") iter_counter = ContextKey("iter_counter")
def start_tracking_iterations(): iter_counter.set(collection.Counter())
def tally_iteration(): current_task = task_id.get() # Set elsewhere iter_counter.get()[current_task] += 1
Now, this isn't a very *sensible* generator (since it could just use a regular object instance for tracking instead of a context variable), but nevertheless, it's one that we would expect to work, and it's one that we would expect to exhibit the following properties:
1. When tally_iteration() calls task_id.get(), we expect that to be resolved in the context calling next() on the instance, *not* the context where the generator was first created 2. When tally_iteration() calls iter_counter.get(), we expect that to be resolved in the same context where start_tracking_iterations() called iter_counter.set()
This has consequences for the design in the PEP:
* what we want to capture at generator creation time is the context where writes will happen, and we also want that to be the innermost context used for lookups * other than that innermost context, we want everything else to be dynamic * this means that "mutable context saved on the generator" and "entire dynamic context visible when the generator runs" aren't the same thing
And hence the introduction of the LocalContext/LogicalContext terminology for the former, and the ExecutionContext terminology for the latter.
OK, this is a sensible explanation. I think the PEP would benefit from including some version of it early on (though perhaps shortened a bit).
It's also where the analogy with ChainMap came from (although I don't think this has made it into the PEP itself):
* LogicalContext is the equivalent of the individual mappings * ExecutionContext is the equivalent of ChainMap * ContextKey.get() replaces ChainMap.__getitem__ * ContextKey.set(value) replaces ChainMap.__setitem__ * ContextKey.set(None) replaces ChainMap.__delitem__
While the context is defined conceptually as a nested chain of key:value mappings, we avoid using the mapping syntax because of the way the values can shift dynamically out from under you based on who called you - while the ChainMap analogy is hopefully helpful to understanding, we don't want people taking it too literally or things will become more confusing rather than less.
Agreed. However now I am confused as to how the HAMT fits in. Yury says somewhere that the HAMT will be used for the EC and then cloning the EC is just returning a pointer to the same EC. But even if I interpret that as making a new EC containing a pointer to the same underlying HAMT, I don't see how that will preserve the semantics that different logical threads, running interleaved (like different generators being pumped alternatingly), will see updates to LCs that are lower on the stack of LCs in the EC. (I see this with the stack-of-dicts version, but not with the immutable HAMT inplementation.)
Despite that risk, taking the analogy further is where the DynamicWriteContext + DynamicLookupContext terminology idea came from:
* like ChainMap.new_child(), adjusting the DynamicWriteContext changes what ck.set() affects, and also sets the innermost context for ck.get() * like using a different ChainMap, adjusting the DynamicLookupContext changes what ck.get() can see (unlike ChainMap, it also isolates ck.set() by default)
Here I'm lost again. In the PEP's pseudo code, your first bullet seems to be the operation "push a new LC on the stack of the current EC". Does the second bullet just mean "switch to a different EC"?
I'll also note that the first iteration of the PEP didn't really make this distinction, and it caused a problem that Nathaniel pointed out: generators would "snapshot" their entire dynamic context when first created, and then never adjust it for external changes between iterations. This meant that if you adjusted something like the decimal context outside the generator after creating it, it would ignore those changes - instead of having the problem of changes inside the generator leaking out, we instead had the problem of changes outside the generator *not* making their way in, even if you wanted them to.
OK, this really needs to be made very clear early in the PEP. Maybe this final sentence provides the key requirement: changes outside the generator should make it into the generator when next() is invoked, unless the generator itself has made an override; but changes inside the generator should not leak out through next().
Due to that heritage, fixing some of the examples could easily have been missed in the v2 rewrite that introduced the distinction between the two kinds of context.
At this point I would like to suggest that maybe you and/or Nathaniel could volunteer as co-authors for the PEP. You could then also help Yury clean up his grammar (e.g. adding "the" in various places) and improve the general structure of the PEP.
(Another critique of the proposal I have is that it adds too many similarly-named functions to sys. But this email is already too long and I need to go to bed.)
If it helps any, one of the ideas that has come up is to put all of the proposed context manipulation APIs in contextlib rather than in sys, and I think that's a reasonable idea (I don't think any of us actually like the notion of adding that many new subsystem specific APIs directly to sys).
I don't think it belongs in contextlib. That module is about contexts for use in with-statements; here we are not particularly concerned with those but with manipulating state that is associated with a logical thread. I think it makes more sense to add a new module for this purpose. I also think that some of the framework-facing APIs should probably be methods rather than functions. -- --Guido van Rossum (python.org/~guido)
On Mon, Aug 21, 2017 at 3:10 PM, Guido van Rossum
Agreed. However now I am confused as to how the HAMT fits in. Yury says somewhere that the HAMT will be used for the EC and then cloning the EC is just returning a pointer to the same EC. But even if I interpret that as making a new EC containing a pointer to the same underlying HAMT, I don't see how that will preserve the semantics that different logical threads, running interleaved (like different generators being pumped alternatingly), will see updates to LCs that are lower on the stack of LCs in the EC. (I see this with the stack-of-dicts version, but not with the immutable HAMT inplementation.)
Few important things (using the current PEP 550 terminology): * ExecutionContext is a *dynamic* stack of LogicalContexts. * LCs do not reference other LCs. * ContextKey.set() can only modify the *top* LC in the stack. If LC is a mutable mapping: # EC = [LC1, LC2, LC3, LC4({a: b, foo: bar})] a.set(c) # LC4 = EC.top() # LC4[a] = c # EC = [LC1, LC2, LC3, LC4({a: c, foo: bar})] If LC are implemented with immutable mappings: # EC = [LC1, LC2, LC3, LC4({a: b, foo: bar})] a.set(c) # LC4 = EC.pop() # LC4_1 = LC4.copy() # LC4_1[a] = c # EC.push(LC4_1) # EC = [LC1, LC2, LC3, LC4_1({a: c, foo: bar})] Any code that uses EC will not see any difference, because it can only work with the top LC. Back to generators. Generators have their own empty LCs when created to store their *local* EC modifications. When a generator is *being* iterated, it pushes its LC to the EC. When the iteration step is finished, it pops its LC from the EC. If you have nested generators, they will dynamically build a stack of their LCs while they are iterated. Therefore, generators *naturally* control the stack of EC. We can't execute two generators simultaneously in one thread (we can only iterate them one by one), so the top LC always belongs to the current generator that is being iterated: def nested_gen(): # EC = [outer_LC, gen1_LC, nested_gen_LC] yield # EC = [outer_LC, gen1_LC, nested_gen_LC] yield def gen1(): # EC = [outer_LC, gen1_LC] n = nested_gen() yield # EC = [outer_LC, gen1_LC] next(n) # EC = [outer_LC, gen1_LC] yield next(n) # EC = [outer_LC, gen1_LC] def gen2(): # EC = [outer_LC, gen2_LC] yield # EC = [outer_LC, gen2_LC] yield g1 = gen1() g2 = gen2() next(g1) next(g2) next(g1) next(g2) HAMT is a way to efficiently implement immutable mappings with O(log32 N) set operation, that's it. If we implement immutable mappings using regular dicts and copy, set() would be O(log N). [..]
I'll also note that the first iteration of the PEP didn't really make this distinction, and it caused a problem that Nathaniel pointed out: generators would "snapshot" their entire dynamic context when first created, and then never adjust it for external changes between iterations. This meant that if you adjusted something like the decimal context outside the generator after creating it, it would ignore those changes - instead of having the problem of changes inside the generator leaking out, we instead had the problem of changes outside the generator *not* making their way in, even if you wanted them to.
OK, this really needs to be made very clear early in the PEP. Maybe this final sentence provides the key requirement: changes outside the generator should make it into the generator when next() is invoked, unless the generator itself has made an override; but changes inside the generator should not leak out through next().
It's covered here with two examples: https://www.python.org/dev/peps/pep-0550/#ec-semantics-for-generators Yury
On Mon, Aug 21, 2017 at 12:50 PM, Yury Selivanov
Few important things (using the current PEP 550 terminology):
* ExecutionContext is a *dynamic* stack of LogicalContexts. * LCs do not reference other LCs. * ContextKey.set() can only modify the *top* LC in the stack.
If LC is a mutable mapping:
# EC = [LC1, LC2, LC3, LC4({a: b, foo: bar})]
a.set(c) # LC4 = EC.top() # LC4[a] = c
# EC = [LC1, LC2, LC3, LC4({a: c, foo: bar})]
If LC are implemented with immutable mappings:
# EC = [LC1, LC2, LC3, LC4({a: b, foo: bar})]
a.set(c) # LC4 = EC.pop() # LC4_1 = LC4.copy() # LC4_1[a] = c # EC.push(LC4_1)
# EC = [LC1, LC2, LC3, LC4_1({a: c, foo: bar})]
Any code that uses EC will not see any difference, because it can only work with the top LC.
OK, good. This makes more sense, especially if I read "the EC" as shorthand for the EC stored in the current thread's per-thread state. The immutable mapping (if used) is used for the LC, not for the EC, and in this case cloning an EC would simply make a shallow copy of its underlying list -- whereas without the immutable mapping, cloning the EC would also require making shallow copies of each LC. And I guess the linked-list implementation (Approach #3 in the PEP) makes EC cloning an O(1) operation. Note that there is a lot of hand-waving and shorthand in this explanation, but I think I finally follow the design. It is going to be a big task to write this up in a didactic way -- the current PEP needs a fair amount of help in that sense. (If you want to become a better writer, I've recently enjoyed reading Steven Pinker's *The Sense of Style*: The Thinking Person's Guide to Writing in the 21st Century. Amongst other fascinating topics, it explains why so often what we think is clearly written can cause so much confusion.)
Back to generators. Generators have their own empty LCs when created to store their *local* EC modifications.
When a generator is *being* iterated, it pushes its LC to the EC. When the iteration step is finished, it pops its LC from the EC. If you have nested generators, they will dynamically build a stack of their LCs while they are iterated.
Therefore, generators *naturally* control the stack of EC. We can't execute two generators simultaneously in one thread (we can only iterate them one by one), so the top LC always belongs to the current generator that is being iterated:
def nested_gen(): # EC = [outer_LC, gen1_LC, nested_gen_LC] yield # EC = [outer_LC, gen1_LC, nested_gen_LC] yield
def gen1(): # EC = [outer_LC, gen1_LC] n = nested_gen() yield # EC = [outer_LC, gen1_LC] next(n) # EC = [outer_LC, gen1_LC] yield next(n) # EC = [outer_LC, gen1_LC]
def gen2(): # EC = [outer_LC, gen2_LC] yield # EC = [outer_LC, gen2_LC] yield
g1 = gen1() g2 = gen2()
next(g1) next(g2) next(g1) next(g2)
This, combined with your later clarification:
In the current version of the PEP, generators are initialized with an empty LogicalContext. When they are being iterated (started or resumed), their LogicalContext is pushed to the EC. When the iteration is stopped (or paused), they pop their LC from the EC.
makes it clear how the proposal works for generators. There's an important piece that I hadn't figured out from Nick's generator example, because I had mistakenly assumed that something *would* be captured at generator create time. It's a reasonable mistake to make, I think -- the design space here is just huge and there are many variations that don't affect typical code but do differ in edge cases. Your clear statement "nothing needs to be captured" is helpful to avoid this misunderstanding.
HAMT is a way to efficiently implement immutable mappings with O(log32 N) set operation, that's it. If we implement immutable mappings using regular dicts and copy, set() would be O(log N).
This sounds like abuse of the O() notation. Mathematically O(log N) and O(log32 N) surely must be equivalent, since log32 N is just K*(log N) for some constant K (about 0.288539), and the constant disappears in the O(), as O(K*f(N)) and O(f(N)) are equivalent. Now, I'm happy to hear that a HAMT-based implementation is faster than a dict+copy-based implementation, but I don't think your use of O() makes sense here.
[..]
I'll also note that the first iteration of the PEP didn't really make this distinction, and it caused a problem that Nathaniel pointed out: generators would "snapshot" their entire dynamic context when first created, and then never adjust it for external changes between iterations. This meant that if you adjusted something like the decimal context outside the generator after creating it, it would ignore those changes - instead of having the problem of changes inside the generator leaking out, we instead had the problem of changes outside the generator *not* making their way in, even if you wanted them to.
OK, this really needs to be made very clear early in the PEP. Maybe this final sentence provides the key requirement: changes outside the generator should make it into the generator when next() is invoked, unless the generator itself has made an override; but changes inside the generator should not leak out through next().
It's covered here with two examples: https://www.python.org/dev/peps/pep-0550/#ec-semantics-for-generators
I think what's missing is the fact that this is one of the key motivating reasons for the design (starting with v2 of the PEP). When I encountered that section I just skimmed it, assuming it was mostly just showing how to apply the given semantics to generators. I also note some issues with the use of tense here -- it's a bit confusing to follow which parts of the text refer to defects of the current (pre-PEP) situation and which parts refer to how the proposal would solve these defects. -- --Guido van Rossum (python.org/~guido http://python.org/%7Eguido)
On Mon, Aug 21, 2017 at 8:06 PM, Guido van Rossum
On Mon, Aug 21, 2017 at 12:50 PM, Yury Selivanov
wrote: Few important things (using the current PEP 550 terminology):
* ExecutionContext is a *dynamic* stack of LogicalContexts. * LCs do not reference other LCs. * ContextKey.set() can only modify the *top* LC in the stack.
If LC is a mutable mapping:
# EC = [LC1, LC2, LC3, LC4({a: b, foo: bar})]
a.set(c) # LC4 = EC.top() # LC4[a] = c
# EC = [LC1, LC2, LC3, LC4({a: c, foo: bar})]
If LC are implemented with immutable mappings:
# EC = [LC1, LC2, LC3, LC4({a: b, foo: bar})]
a.set(c) # LC4 = EC.pop() # LC4_1 = LC4.copy() # LC4_1[a] = c # EC.push(LC4_1)
# EC = [LC1, LC2, LC3, LC4_1({a: c, foo: bar})]
Any code that uses EC will not see any difference, because it can only work with the top LC.
OK, good. This makes more sense, especially if I read "the EC" as shorthand for the EC stored in the current thread's per-thread state.
That's exactly what I meant by "the EC".
The immutable mapping (if used) is used for the LC, not for the EC, and in this case cloning an EC would simply make a shallow copy of its underlying list -- whereas without the immutable mapping, cloning the EC would also require making shallow copies of each LC. And I guess the linked-list implementation (Approach #3 in the PEP) makes EC cloning an O(1) operation.
All correct.
Note that there is a lot of hand-waving and shorthand in this explanation, but I think I finally follow the design. It is going to be a big task to write this up in a didactic way -- the current PEP needs a fair amount of help in that sense.
Elvis Pranskevichus (our current What's New editor and my colleague) offered me to help with the PEP. He's now working on a partial rewrite. I've been working on this PEP for about a month now and at this point it makes it difficult for me to dump this knowledge in a nice and readable way (in any language that I know, FWIW).
(If you want to become a better writer, I've recently enjoyed reading Steven Pinker's The Sense of Style: The Thinking Person's Guide to Writing in the 21st Century. Amongst other fascinating topics, it explains why so often what we think is clearly written can cause so much confusion.)
Will definitely check it out, thank you!
Back to generators. Generators have their own empty LCs when created to store their *local* EC modifications.
When a generator is *being* iterated, it pushes its LC to the EC. When the iteration step is finished, it pops its LC from the EC. If you have nested generators, they will dynamically build a stack of their LCs while they are iterated.
Therefore, generators *naturally* control the stack of EC. We can't execute two generators simultaneously in one thread (we can only iterate them one by one), so the top LC always belongs to the current generator that is being iterated:
def nested_gen(): # EC = [outer_LC, gen1_LC, nested_gen_LC] yield # EC = [outer_LC, gen1_LC, nested_gen_LC] yield
def gen1(): # EC = [outer_LC, gen1_LC] n = nested_gen() yield # EC = [outer_LC, gen1_LC] next(n) # EC = [outer_LC, gen1_LC] yield next(n) # EC = [outer_LC, gen1_LC]
def gen2(): # EC = [outer_LC, gen2_LC] yield # EC = [outer_LC, gen2_LC] yield
g1 = gen1() g2 = gen2()
next(g1) next(g2) next(g1) next(g2)
This, combined with your later clarification:
In the current version of the PEP, generators are initialized with an empty LogicalContext. When they are being iterated (started or resumed), their LogicalContext is pushed to the EC. When the iteration is stopped (or paused), they pop their LC from the EC.
makes it clear how the proposal works for generators. There's an important piece that I hadn't figured out from Nick's generator example, because I had mistakenly assumed that something *would* be captured at generator create time. It's a reasonable mistake to make,
Yeah, it is very subtle.
HAMT is a way to efficiently implement immutable mappings with O(log32 N) set operation, that's it. If we implement immutable mappings using regular dicts and copy, set() would be O(log N).
This sounds like abuse of the O() notation. Mathematically O(log N) and O(log32 N) surely must be equivalent, since log32 N is just K*(log N) for some constant K (about 0.288539), and the constant disappears in the O(), as O(K*f(N)) and O(f(N)) are equivalent. Now, I'm happy to hear that a HAMT-based implementation is faster than a dict+copy-based implementation, but I don't think your use of O() makes sense here.
I made a typo there: when implementing an immutable mapping with Python dicts, setting a key is an O(N) operation (not O(log N)): we need to make a shallow copy of a dict and then add an item to it. (the PEP doesn't have this typo) With HAMT, set() is O(log32 N): https://github.com/python/peps/blob/master/pep-0550-hamt_vs_dict.png Yury
On Mon, Aug 21, 2017 at 8:06 PM, Guido van Rossum
OK, this really needs to be made very clear early in the PEP. Maybe this final sentence provides the key requirement: changes outside the generator should make it into the generator when next() is invoked, unless the generator itself has made an override; but changes inside the generator should not leak out through next().
It's covered here with two examples: https://www.python.org/dev/peps/pep-0550/#ec-semantics-for-generators
I think what's missing is the fact that this is one of the key motivating reasons for the design (starting with v2 of the PEP). When I encountered that section I just skimmed it, assuming it was mostly just showing how to apply the given semantics to generators. I also note some issues with the use of tense here -- it's a bit confusing to follow which parts of the text refer to defects of the current (pre-PEP) situation and which parts refer to how the proposal would solve these defects.
I see. The proposal always uses present tense to describe things it adds, and I now see that this is indeed very confusing. This needs to be fixed. Yury
On Mon, Aug 21, 2017 at 5:12 PM, Nick Coghlan
On 21 August 2017 at 15:03, Guido van Rossum
wrote: Honestly I'm not sure we need the distinction between LC and EC. If you read carefully some of the given example code seems to confuse them. If we could get away with only a single framework-facing concept, I would be happy calling it ExecutionContext.
Unfortunately, I don't think we can, and that's why I tried to reframe the discussion in terms of "Where ContextKey.set() writes to" and "Where ContextKey.get() looks things up".
Consider the following toy generator:
def tracking_gen(): start_tracking_iterations() while True: tally_iteration() yield
task_id = ContextKey("task_id") iter_counter = ContextKey("iter_counter")
def start_tracking_iterations(): iter_counter.set(collection.Counter())
def tally_iteration(): current_task = task_id.get() # Set elsewhere iter_counter.get()[current_task] += 1
Now, this isn't a very *sensible* generator (since it could just use a regular object instance for tracking instead of a context variable), but nevertheless, it's one that we would expect to work, and it's one that we would expect to exhibit the following properties:
1. When tally_iteration() calls task_id.get(), we expect that to be resolved in the context calling next() on the instance, *not* the context where the generator was first created 2. When tally_iteration() calls iter_counter.get(), we expect that to be resolved in the same context where start_tracking_iterations() called iter_counter.set()
This has consequences for the design in the PEP:
* what we want to capture at generator creation time is the context where writes will happen, and we also want that to be the innermost context used for lookups
I don't get it. How is this a consequence of the above two points? And why do we need to capture something (a "context") at generator creation time? -- Koos
* other than that innermost context, we want everything else to be dynamic * this means that "mutable context saved on the generator" and "entire dynamic context visible when the generator runs" aren't the same thing
And hence the introduction of the LocalContext/LogicalContext terminology for the former, and the ExecutionContext terminology for the latter.
[...]
-- + Koos Zevenhoven + http://twitter.com/k7hoven +
On Mon, Aug 21, 2017 at 5:14 PM, Koos Zevenhoven
This has consequences for the design in the PEP:
* what we want to capture at generator creation time is the context where writes will happen, and we also want that to be the innermost context used for lookups
I don't get it. How is this a consequence of the above two points? And why do we need to capture something (a "context") at generator creation time?
We don't need to "capture" anything when a generator is created (it was something that PEP 550 version 1 was doing). In the current version of the PEP, generators are initialized with an empty LogicalContext. When they are being iterated (started or resumed), their LogicalContext is pushed to the EC. When the iteration is stopped (or paused), they pop their LC from the EC. Yury
On Tue, Aug 22, 2017 at 12:25 AM, Yury Selivanov
On Mon, Aug 21, 2017 at 5:14 PM, Koos Zevenhoven
wrote: [..] This has consequences for the design in the PEP:
* what we want to capture at generator creation time is the context where writes will happen, and we also want that to be the innermost context used for lookups
I don't get it. How is this a consequence of the above two points? And why do we need to capture something (a "context") at generator creation time?
We don't need to "capture" anything when a generator is created (it was something that PEP 550 version 1 was doing).
Ok, good.
In the current version of the PEP, generators are initialized with an empty LogicalContext. When they are being iterated (started or resumed), their LogicalContext is pushed to the EC. When the iteration is stopped (or paused), they pop their LC from the EC.
Another quick one before I go: Do we really need to push and pop a LC on each next() call, even if it most likely will never be touched? -- Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven +
On Mon, Aug 21, 2017 at 5:39 PM, Koos Zevenhoven
In the current version of the PEP, generators are initialized with an empty LogicalContext. When they are being iterated (started or resumed), their LogicalContext is pushed to the EC. When the iteration is stopped (or paused), they pop their LC from the EC.
Another quick one before I go: Do we really need to push and pop a LC on each next() call, even if it most likely will never be touched?
Yes, otherwise it will be hard to maintain the consistency of the stack. There will be an optimization: if the LC is empty, we will push NULL to the stack, thus avoiding the cost of allocating an object. I measured the overhead -- generators will become 0.5-1% slower in microbenchmarks, but only when they do pretty much nothing. If a generator contains more Python code than a bare "yield" expression, the overhead will be harder to detect. Yury
On Tue, Aug 22, 2017 at 12:44 AM, Yury Selivanov
On Mon, Aug 21, 2017 at 5:39 PM, Koos Zevenhoven
wrote: [..] In the current version of the PEP, generators are initialized with an empty LogicalContext. When they are being iterated (started or resumed), their LogicalContext is pushed to the EC. When the iteration is stopped (or paused), they pop their LC from the EC.
Another quick one before I go: Do we really need to push and pop a LC on each next() call, even if it most likely will never be touched?
Yes, otherwise it will be hard to maintain the consistency of the stack.
There will be an optimization: if the LC is empty, we will push NULL to the stack, thus avoiding the cost of allocating an object.
But if LCs are immutable, there needs to be only one empty-LC instance. That would avoid special-casing NULL in code.
-- Koos
I measured the overhead -- generators will become 0.5-1% slower in microbenchmarks, but only when they do pretty much nothing. If a generator contains more Python code than a bare "yield" expression, the overhead will be harder to detect.
-- + Koos Zevenhoven + http://twitter.com/k7hoven +
On Tue, Aug 22, 2017 at 2:06 AM, Koos Zevenhoven
There will be an optimization: if the LC is empty, we will push NULL to the stack, thus avoiding the cost of allocating an object.
But if LCs are immutable, there needs to be only one empty-LC instance. That would avoid special-casing NULL in code.
Yes, this is true. Yury
Hi Yury, On 08/18/2017 10:33 PM, Yury Selivanov wrote:
* ``.get()`` method: return the current EC value for the context key. Context keys return ``None`` when the key is missing, so the method never fails. Is the difference between `Key not found` and `value is None` important here?
Thanks, --francis
participants (10)
-
Antoine Pitrou
-
Brett Cannon
-
Ethan Furman
-
francismb
-
Greg Ewing
-
Guido van Rossum
-
Guido van Rossum
-
Koos Zevenhoven
-
Nick Coghlan
-
Yury Selivanov