[Python-Dev] PEP 550 v4: coroutine policy

[Yury Selivanov]

Long story short, I think we need to rollback our last decision to
prohibit context propagation up the call stack in coroutines.  In PEP
550 v3 and earlier, the following snippet would work just fine:

   var = new_context_var()

   async def bar():
       var.set(42)

   async def foo():
       await bar()
       assert var.get() == 42   # with previous PEP 550 semantics

   run_until_complete(foo())

But it would break if a user wrapped "await bar()" with "wait_for()":

   var = new_context_var()

   async def bar():
       var.set(42)

   async def foo():
       await wait_for(bar(), 1)
       assert var.get() == 42  # AssertionError !!!

   run_until_complete(foo())

Therefore, in the current (v4) version of the PEP, we made all
coroutines to have their own LC (just like generators), which makes
both examples always raise an AssertionError.  This makes it easier
for async/await users to refactor their code: they simply cannot
propagate EC changes up the call stack, hence any coroutine can be
safely wrapped into a task.

Nathaniel and Stefan Krah argued on the mailing list that this change
in semantics makes the PEP harder to understand.  Essentially, context
changes propagate up the call stack for regular code, but not for
asynchronous.  For regular code the PEP behaves like TLS, but for
asynchronous it behaves like dynamic scoping.

IMO, on its own, this argument is not strong enough to rollback to the
older PEP 550 semantics, but I think I've discovered a stronger one:
asynchronous context managers.

With the current version (v4) of the PEP, it's not possible to set
context variables in __aenter__ and in
@contextlib.asynccontextmanager:

    class Foo:

        async def __aenter__(self):
             context_var.set('aaa')    # won't be visible outside of __aenter__

So I guess we have no other choice other than reverting this spec
change for coroutines.  The very first example in this email should
start working again.

This means that PEP 550 will have a caveat for async code: don't rely
on context propagation up the call stack, unless you are writing
__aenter__ and __aexit__ that are guaranteed to be called without
being wrapped into a Task.

BTW, on the topic of dynamic scoping.  Context manager protocols (both
sync and async) is the fundamental reason why we couldn't implement
dynamic scoping in Python even if we wanted to.  With a classical
dynamic scoping in a functional language, __enter__ would have its own
scope, which the code in the 'with' block would never be able to
access.  Thus I propose to stop associating PEP 550 concepts with
(dynamic) scoping.

Thanks,
Yury