[Python-Dev] PEP 550 v4

[Yury Selivanov]

On Mon, Aug 28, 2017 at 9:50 PM, Guido van Rossum <guido at python.org> wrote:
> On Mon, Aug 28, 2017 at 6:07 PM, Nathaniel Smith <njs at pobox.com> wrote:
>>
>> The important difference between generators/coroutines and normal
>> function calls is that with normal function calls, the link between
>> the caller and callee is fixed for the entire lifetime of the inner
>> frame, so there's no way for the context to shift under your feet. If
>> all we had were normal function calls, then (green-) thread locals
>> using the save/restore trick would be enough to handle all the use
>> cases above -- it's only for generators/coroutines where the
>> save/restore trick breaks down. This means that pushing/popping LCs
>> when crossing into/out of a generator frame is the minimum needed to
>> get the desired semantics, and it keeps the LC stack small (important
>> since lookups can be O(n) in the worst case), and it minimizes the
>> backcompat breakage for operations like decimal.setcontext() where
>> people *do* expect to call it in a subroutine and have the effects be
>> visible in the caller.
>
>
> I like this way of looking at things. Does this have any bearing on
> asyncio.Task? To me those look more like threads than like generators. Or
> possibly they should inherit the lookup chain from the point when the Task
> was created, [..]

We explain why tasks have to inherit the lookup chain from the point
where they are created in the PEP (in the new High-level Specification
section):
https://www.python.org/dev/peps/pep-0550/#coroutines-and-asynchronous-tasks

In short, without inheriting the chain we can't wrap coroutines into
tasks (like wrapping an await in wait_for() would break the code, if
we don't inherit the chain).

In the latest version (v4) we made all coroutines to have their own
Logical Context, which, as we discovered today, makes us unable to set
context variables in __aenter__ coroutines.  This will be fixed in the
next version.

> FWIW we *could* have a policy that OS threads also
> inherit the lookup chain from their creator, but I doubt that's going to fly
> with backwards compatibility.

Backwards compatibility is indeed an issue.  Inheriting the chain for
threads would mean another difference between PEP 550 and
'threading.local()', that could cause backwards incompatible behaviour
for decimal/numpy when they are updated to new APIs.

For decimal, for example, we could use the following pattern to
fallback to use the default decimal context for ECs (threads) that
don't have it set:

    ctx = decimal_var.get(default=default_decimal_ctx)

We can also add an 'initializer' keyword-argument to 'new_context_var'
to specify a callable that will be used to give a default value to the
var.

Another issue, is that with the current C API, we can only inherit EC
for threads started with 'threading.Thread'. There's no reliable way
to inherit the chain if a thread was initialized by a C extension.

IMO, inheriting the lookup chain in threads makes sense when we use
them for pools, like concurrent.futures.ThreadPoolExecutor.  When
threads are used as long-running subprograms, inheriting the chain
should be an opt-in.

Yury