Re: [Python-Dev] PEP 550 v4: coroutine policy

On Tue, Aug 29, 2017 at 12:32 PM, Antoine Pitrou wrote:

Le 29/08/2017 à 21:18, Yury Selivanov a écrit :

...

On Tue, Aug 29, 2017 at 2:40 PM, Antoine Pitrou wrote:

...

On Mon, 28 Aug 2017 17:24:29 -0400 Yury Selivanov wrote:

...

Long story short, I think we need to rollback our last decision to prohibit context propagation up the call stack in coroutines. In PEP 550 v3 and earlier, the following snippet would work just fine:

var = new_context_var()

async def bar(): var.set(42)

async def foo(): await bar() assert var.get() == 42 # with previous PEP 550 semantics

run_until_complete(foo())

But it would break if a user wrapped "await bar()" with "wait_for()":

var = new_context_var()

async def bar(): var.set(42)

async def foo(): await wait_for(bar(), 1) assert var.get() == 42 # AssertionError !!!

run_until_complete(foo())

[...]

...

Why wouldn't the bar() coroutine inherit the LC at the point it's instantiated (i.e. where the synchronous bar() call is done)?

We want tasks to have their own isolated contexts. When a task is started, it runs its code in parallel with its "parent" task.

I'm sorry, but I don't understand what it all means.

To pose the question differently: why is example #1 supposed to be different, philosophically, than example #2? Both spawn a coroutine, both wait for its execution to end. There is no reason that adding a wait_for() intermediary (presumably because the user wants to add a timeout) would significantly change the execution semantics of bar().

...

wait_for() in the above example creates an asyncio.Task implicitly, and that's why we don't see 'var' changed to '42' in foo().

I don't understand why a non-obvious behaviour detail (the fact that wait_for() creates an asyncio.Task implicitly) should translate into a fundamental difference in observable behaviour. I find it counter-intuitive and error-prone.

For better or worse, asyncio users generally need to be aware of the distinction between coroutines/Tasks/Futures and which functions create or return which -- it's essentially the same as the distinction between running some code in the current thread versus spawning a new thread to run it (and then possibly waiting for the result). Mostly the docs tell you when a function converts a coroutine into a Task, e.g. if you look at the docs for 'ensure_future' or 'wait_for' or 'wait' they all say this explicitly. Or in some cases like 'gather' and 'shield', it's implicit because they take arbitrary futures, and creating a task is how you convert a coroutine into a future. As a rule of thumb, I think it's accurate to say that any function that takes a coroutine object as an argument always converts it into a Task.

...

This is a slightly complicated case, but it's addressable with a good documentation and recommended best practices.

It would be better addressed with consistent behaviour that doesn't rely on specialist knowledge, though :-/

This is the core of the Curio/Trio critique of asyncio: in asyncio, operations that implicitly initiate concurrent execution are all over the API. This is the root cause of asyncio's problems with buffering and backpressure, it makes it hard to shut down properly (it's hard to know when everything has finished running), it's related to the "spooky cancellation at a distance" issue where cancelling one task can cause another Task to get a cancelled exception, etc. If you use the recommended "high level" API for streams, then AFAIK it's still impossible to close your streams properly at shutdown (you can request that a close happen "sometime soon", but you can't tell when it's finished). Obviously asyncio isn't going anywhere, so we should try to solve/mitigate these issues where we can, but asyncio's API fundamentally assumes that users will be very aware and careful about which operations create which kinds of concurrency. So I sort of feel like, if you can use asyncio at all, then you can handle wait_for creating a new LC. -n [1] https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-a... -- Nathaniel J. Smith -- https://vorpus.org