<div dir="ltr"><div class="gmail_default" style="font-family:monospace,monospace"><span style="font-family:arial,sans-serif">Let me respond to my own email. I'm sorry I wrote a long email, but I figured I'll have to take the time to write this down carefully (and even in a new thread with a clear title) so that people would know what the discussion was about. Probably I could have done better structuring that email, but I seriously ran out of time.</span></div><div class="gmail_default" style="font-family:monospace,monospace"><span style="font-family:arial,sans-serif"><br></span></div><div class="gmail_default" style="font-family:monospace,monospace"><span style="font-family:arial,sans-serif">This is directly related to how "a normal user" writing async code would be affected by the semantics of this (context arguments/variables). It's also related to the semantics of contexts combined with normal generator functions, partly because the situation is somewhat similar, and partly because we might want the same basic rules to apply in both situations. </span></div><div class="gmail_default" style="font-family:monospace,monospace"><span style="font-family:arial,sans-serif"><br></span></div><div class="gmail_default" style="font-family:monospace,monospace"><span style="font-family:arial,sans-serif">(Side note: This also has to do with more obscure cases like multiple different async frameworks in the same process (or in the same program, or perhaps the same server, or even larger </span><div class="gmail_default" style="color:rgb(136,136,136);display:inline"></div><span style="color:rgb(136,136,136)">–</span><span style="font-family:arial,sans-serif"> whatever the constraints are).</span><br></div><div class="gmail_default" style="font-family:monospace,monospace"><span style="font-family:arial,sans-serif"><br></span></div><div class="gmail_default" style="font-family:monospace,monospace"><span style="font-family:arial,sans-serif">Any of the context propagation and isolation/leakage semantics I have described (or that I recall anyone else describing) could be implemented in the PEP 555 approach without any problems. The difference is just an if statement branch or two in C code.</span><span style="font-family:arial,sans-serif"><br></span></div><div class="gmail_default" style="font-family:monospace,monospace"><span style="font-family:arial,sans-serif"><br></span></div><div class="gmail_default">So, see below for some more discussion between (it would be useful if some people could reply to this email and say if and why they agree or disagree with something below -- also non-experts that roughly understand what I'm talking about):</div><div class="gmail_default" style="font-family:monospace,monospace"><span style="font-family:arial,sans-serif"><br></span></div><div class="gmail_default" style="font-family:monospace,monospace"><span style="font-family:arial,sans-serif">On Fri, Oct 13, 2017 at 6:49 PM, Koos Zevenhoven </span><span dir="ltr" style="font-family:arial,sans-serif"><<a href="mailto:k7hoven@gmail.com" target="_blank">k7hoven@gmail.com</a>></span><span style="font-family:arial,sans-serif"> wrote:</span><br></div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><font face="monospace, monospace">This is a continuation of the PEP 555 discussion in</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"><a href="https://mail.python.org/pipermail/python-ideas/2017-September/046916.html" target="_blank">https://mail.python.org/<wbr>pipermail/python-ideas/2017-<wbr>September/046916.html</a><br></font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">And this month in</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"><a href="https://mail.python.org/pipermail/python-ideas/2017-October/047279.html" target="_blank">https://mail.python.org/<wbr>pipermail/python-ideas/2017-<wbr>October/047279.html</a></font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">If you are new to the discussion, the best point to start reading this might be at my second full paragraph below ("The status quo...").</font></div><div style="font-family:monospace,monospace"><span style="font-family:arial,sans-serif"><br></span></div><div style="font-family:monospace,monospace"><span style="font-family:arial,sans-serif">On Fri, Oct 13, 2017 at 10:25 AM, Nick Coghlan </span><span dir="ltr" style="font-family:arial,sans-serif"><<a href="mailto:ncoghlan@gmail.com" target="_blank">ncoghlan@gmail.com</a>></span><span style="font-family:arial,sans-serif"> <wbr>wrote:</span><br></div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="font-family:monospace,monospace;margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="gmail-m_9145725328918873316gmail-">On 13 October 2017 at 10:56, Guido van Rossum <span dir="ltr"><<a href="mailto:guido@python.org" target="_blank">guido@python.org</a>></span> <wbr>wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I'm out of energy to debate every point (Steve said it well -- that decimal/generator example is too contrived), but I found one nit in Nick's email that I wish to correct.<br><div class="gmail_extra"><br><div class="gmail_quote"><span class="gmail-m_9145725328918873316gmail-m_-7756605063962862266gmail-">On Wed, Oct 11, 2017 at 1:28 AM, Nick Coghlan <span dir="ltr"><<a href="mailto:ncoghlan@gmail.com" target="_blank">ncoghlan@gmail.com</a>></span> <wbr>wrote:<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">As a less-contrived example, consider context managers implemented as generators.</div><div class="gmail_quote"><br></div><div class="gmail_quote">We want those to run with the execution context that's active when they're used in a with statement, not the one that's active when they're created (the fact that generator-based context managers can only be used once mitigates the risk of creation time context capture causing problems, but the implications would still be weird enough to be worth avoiding).</div></div></div></blockquote><div><br></div></span><div>Here I think we're in agreement about the desired semantics, but IMO all this requires is some special casing for @contextlib.contextmanager. To me this is the exception, not the rule -- in most *other* places I would want the yield to switch away from the caller's context.</div><span class="gmail-m_9145725328918873316gmail-m_-7756605063962862266gmail-"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra">For native coroutines, we want them to run with the execution context that's active when they're awaited or when they're prepared for submission to an event loop, not the one that's active when they're created.</div></div></blockquote><div><br></div></span><div>This caught my eye as wrong. Considering that asyncio's tasks (as well as curio's and trio's) *are* native coroutines, we want complete isolation between the context active when `await` is called and the context active inside the `async def` function.</div></div></div></div></blockquote><div><br></div></span><div>The rationale for this behaviour *does* arise from a refactoring argument:</div><div><br></div><div> async def original_async_function():</div><div> with some_context():</div><div> do_some_setup()<br></div><div> raw_data = await some_operation()</div><div> data = do_some_postprocessing(raw_dat<wbr>a)<br></div><div><br></div><div>Refactored:</div><div><br></div><div><div> async def async_helper_function():</div> do_some_setup()<br><div> raw_data = await some_operation()</div><div> return do_some_postprocessing(raw_dat<wbr>a)<br></div><div><br></div></div><div><div> async def refactored_async_function():</div><div> with some_context():</div> data = await async_helper_function()<div><br></div></div></div></div></div></blockquote><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace"><div>*This* type of refactoring argument I *do* subscribe to.</div></div><div style="font-family:monospace,monospace"> </div><blockquote class="gmail_quote" style="font-family:monospace,monospace;margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div></div>However, considering that coroutines are almost always instantiated at the point where they're awaited, I do concede that creation time context capture would likely also work out OK for the coroutine case, which would leave contextlib.contextmanager as the only special case (and it would turn off both creation-time context capture *and* context isolation).</div></div></div></blockquote><div style="font-family:monospace,monospace"><div><br></div></div><div><div style="font-family:monospace,monospace">The difference between context propagation through coroutine function calls and awaits comes up when you need help from "the" event loop, which means things like creating new tasks from coroutines. However, we cannot even assume that the loop is the only one. So far, it makes no difference where you call the coroutine function. It is only when you await it or schedule it for execution in a loop when something can actually happen.</div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace">The status quo is that there's nothing that prevents you from calling a coroutine function from within one event loop and then awaiting it in another. So if we want an event loop to be able to pass information down the call chain in such a way that the information is available *throughout the whole task that it is driving*, then the contexts needs to a least propagate through `await`s.</div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace">This was my starting point 2.5 years ago, when Yury was drafting this status quo (PEP 492). It looked a lot of PEP 492 was inevitable, but that there will be a problem, where each API that uses "blocking IO" somewhere under the hood would need a duplicate version for asyncio (and one for each third-party async framework!). I felt it was necessary to think about a solution before PEP 492 is accepted, and this became a fairly short-lived thread here on python-ideas:</div><div style="font-family:monospace,monospace"><br></div><div><font face="monospace, monospace"><a href="https://mail.python.org/pipermail/python-ideas/2015-May/033267.html" target="_blank">https://mail.python.org/<wbr>pipermail/python-ideas/2015-<wbr>May/033267.html</a></font><br></div></div><div style="font-family:monospace,monospace"><div style="display:inline"><br></div></div><div style="font-family:monospace,monospace"><div style="font-family:monospace,monospace">This year, the discussion on Yury's PEP 550 somehow ended up with a very similar need before I got involved, apparently for independent reasons.</div><br></div><div style="font-family:monospace,monospace"><div style="font-family:monospace,monospace;display:inline">A design for solving this need (and others) is also found in my first draft of PEP 555, found at </div></div><div style="font-family:monospace,monospace"><div style="font-family:monospace,monospace;display:inline"><br></div></div><div><div style="display:inline"><font face="monospace, monospace"><a href="https://mail.python.org/pipermail/python-ideas/2017-September/046916.html" target="_blank">https://mail.python.org/<wbr>pipermail/python-ideas/2017-<wbr>September/046916.html</a></font><br></div></div><div style="font-family:monospace,monospace"><div style="font-family:monospace,monospace;display:inline"><br></div></div><div style="font-family:monospace,monospace"><div style="font-family:monospace,monospace;display:inline">Essentially, it's a way of *passing information down the call chain* when it's inconvenient or impossible to pass the information as normal function arguments. I now call the concept "context arguments".</div></div><div style="font-family:monospace,monospace"><div style="font-family:monospace,monospace;display:inline"><br></div></div><div><div style="font-family:monospace,monospace">More recently, I put some focus on the direct needs of normal users (as opposed direct needs of async framework authors). </div><div style="font-family:monospace,monospace"><br></div><div><span style="font-family:monospace,monospace">Those thoughts are most "easily" discussed in terms of generator functions, which are very similar to coroutine functions: A generator function is often thought of as a function that returns an iterable of lazily evaluated values. In this type of usage, the relevant "function call" happens when calling the generator function. The subsequent calls to next() (or a yield from) are thought of as merely getting the items in the iterable, even if they do actually run code in </span><font face="monospace, monospace">the generator's frame. The discussion on this is found starting from this email:</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"><a href="https://mail.python.org/pipermail/python-ideas/2017-October/047279.html" target="_blank">https://mail.python.org/<wbr>pipermail/python-ideas/2017-<wbr>October/047279.html</a><br></font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">However, also coroutines are evaluated lazily. The question is, when should we consider the "subroutine call" to happen: when the coroutine funct</font><span style="font-family:monospace,monospace">ion is called, or when the resulting object is awaited. Often these two are indeed on the same line of code, so it does not matter. But as I discuss above, there are definitely cases where it matters. This has mostly to do with the interactions of different tasks within one event loop, or code where multiple event loops interact.</span></div><div><span style="font-family:monospace,monospace"><br></span></div><div><font face="monospace, monospace">As mentioned above, there are cases where propagating the context through next() and await is desirable. However, there are also cases where the coroutine call is important. This comes up in the case of multiple interacting tasks or multiple event loops. </font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">To start with, probably a more example-friendly case, however, is running an event loop and a coroutine from synchronous code:</font></div><div style="font-family:monospace,monospace"><br></div><div><div><span style="font-family:monospace,monospace">import asyncio</span></div><div><span style="font-family:monospace,monospace"><br></span></div><div><span style="font-family:monospace,monospace">async def do_something_context_aware():</span><br></div><div><font face="monospace, monospace"> do_something_that_depends_on(<wbr>current_context())</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">loop = asyncio.get_event_loop()</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">with some_context():</font></div><div><font face="monospace, monospace"> coro = do_something_context_aware()</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">loop.run_until_complete(coro)<br></font></div><div><font face="monospace, monospace"><br></font></div></div></div><div><font face="monospace, monospace"><div style="display:inline"></div><div style="display:inline"></div><div style="display:inline"></div><div style="display:inline"></div></font></div><div><font face="monospace, monospace"><div style="display:inline">Now, if the coroutine function call `do_something_context_aware()` does not save the current context on `coro`, then there is no way some_context() can affect the code that will run inside the coroutine, even if that is what we are explicitly trying to do here.</div></font></div><div><font face="monospace, monospace"><div style="display:inline"><br></div></font></div><div><div style="font-family:monospace,monospace">The easy solution is to delegate the context transfer to the scheduling function (run_until_complete), and require that the context is passed to that function:</div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace">with some_context():</div><div style="font-family:monospace,monospace"> coro = do_something_context_aware()</div><div style="font-family:monospace,monospace"> loop.run_until_complete(coro)</div><br></div><div><div style="font-family:monospace,monospace">This gives the async framework (here asyncio) a chance to make sure the context propagates as expected. In general, I'm in favor of giving async frameworks some freedom in how this is implemented. However, to give the framework even more freedom, the coroutine call, do_something_context_aware(), could save the current context branch on `coro`, which run_until_complete can attach to the Task that gets created.</div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace">The bigger question is, what should happen when a coroutine awaits on another coroutine directly, without giving the framework a change to interfere:</div></div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace">async def inner():</div><div style="font-family:monospace,monospace"> do_context_aware_stuff()</div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace">async def outer():</div><div style="font-family:monospace,monospace"> with first_context():</div><div style="font-family:monospace,monospace"> coro = inner()</div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace"> with second_context():</div><div style="font-family:monospace,monospace"> await coro</div><div style="font-family:monospace,monospace"><br></div><div><div style="font-family:monospace,monospace">The big question is: In the above, which context should the coroutine be run in?</div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace">"The" event loop does not have a chance to interfere, so we cannot delegate the decision.</div><br></div></div></div></div></div></blockquote><div><div class="gmail_default" style="font-family:monospace,monospace"></div><div class="gmail_default" style="font-family:monospace,monospace">Note that I did not write the above as what real code is expected to look like. It's just to underline the semantic difference that the context can change between the call and the await. Indeed, one might say that people don't write code like that. And maybe async/await is still sufficiently young that one can sort of say "this is how we showed people how to do it, so that's how they'll do it" [*].</div></div><div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">But let's make the above code just a little bit more complicated, so that it becomes easier to believe that the semantic difference here really matters, and cannot be hand-waved away:</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">async def outer():</div><div class="gmail_default" style="font-family:monospace,monospace"> with some_context():</div><div class="gmail_default" style="font-family:monospace,monospace"> a = inner()</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace"> with other_context():</div><div class="gmail_default" style="font-family:monospace,monospace"> b = inner()</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace"> await gather(a, b) # execute coroutines a and b concurrently</div><br></div><div><div class="gmail_default" style="font-family:monospace,monospace"></div><div class="gmail_default" style="font-family:monospace,monospace">If the coroutine function call, inner(), does not save a pointer to the current context at that point, then the code would just ignore the with statements completely and run both coroutines in the outer context, which is clearly not what an author of such code would want the code to do.</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">It is certainly possible to fix the problem with requiring wrapping the coroutines with stuff, but that would lead to nobody ever knowing what the semantics will be without checking if the the coroutine has been wrapped or not.</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">On the other hand, we could make the code *just work*, and that would be completely in line with what I've been promoting also as the semantics for generator functions in this thread:</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace"><a href="https://mail.python.org/pipermail/python-ideas/2017-October/047279.html">https://mail.python.org/pipermail/python-ideas/2017-October/047279.html</a><br></div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">I am definitely *not* talking about this kind of semantics because of something *I personally* need: In fact, I arrived at these thoughts because my designs for solving "my" original problem had turned into a more general-purpose mechanism (PEP 555) that would eventually also affect how code written by completely normal users of with statements and generator functions would behave. And certainly the situation is *very* similar to the case of coroutine functions, as (only?) Guido seems to acknowledge.</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">But then how to address "my" original problem where the context would propagate through awaits, and next/send? From what others have written, it seems there are also other situations where that is desired. There are several ways to solve the problem as an extension to PEP 555, but below is one:</div><br></div><div><div class="gmail_default" style="font-family:monospace,monospace"><span style="font-family:arial,sans-serif"> </span><br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div class="gmail_extra"><div class="gmail_quote"><div></div><div><div style="font-family:monospace,monospace">We need both versions: the one that propagates first_context() into the coroutine, and the one that propagates second_context() into it. Or, using my metaphor from the other thread, we need "both the forest and the trees". </div><br></div><div><div style="font-family:monospace,monospace">A solution to this would be to have two types of context arguments:</div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace">1. (calling) context arguments </div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace">and </div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace">2. execution context arguments</div><br></div><div><div style="font-family:monospace,monospace"><br></div></div></div></div></div></div></blockquote><div><br></div><div><div class="gmail_default" style="font-family:monospace,monospace">So yes, I'm actually serious about this possibility. Now it would be up to library and framework authors to pick the right variant of the two. And this is definitely something that could be documented very clearly.</div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div class="gmail_extra"><div class="gmail_quote"><div><div style="font-family:monospace,monospace"></div><div style="font-family:monospace,monospace">Both of these would have their own stack of (argument, value) assignment pairs, explained in the implementation part of the first PEP 555 draft. While this is a complication, the performance overhead of these is so small, that doubling the overhead should not be a performance concern. The question is, do we want these two types of stacks, or do we want to work around it somehow, for instance using context-local storage, implemented on top of the first kind, to implement something like the second kind. However, that again raises some issues of how to propagate the context-local storage down the ambiguous call chain.</div></div><span class="gmail-HOEnZb"><font color="#888888"><div><div style="display:inline"><font face="monospace, monospace"><br></font></div></div></font></span></div></div></div></div></blockquote><div><br></div><div><div class="gmail_default" style="font-family:monospace,monospace">This would also reduce the need to decorate and wrap generators and decorator functions, although in some cases that would still be needed.</div></div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">If something was not clear, but seems relevant to what I'm trying to discuss here, please ask :)</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace"><div class="gmail_default" style="color:rgb(136,136,136);display:inline"></div><span style="color:rgb(136,136,136)">–</span><div class="gmail_default" style="color:rgb(136,136,136);display:inline"></div><span style="color:rgb(136,136,136)">– Koos</span><br></div><div> </div></div><br><div class="gmail_default" style="font-family:monospace,monospace">[*] Maybe it would not even be too late to make minor changes in the PEP 492 semantics of coroutine functions at this point if there was a good enough reason. But in fact, I think the current semantics might be perfectly fine, and I'm definitely not suggesting any changes to existing semantics here. Only extensions to the existing semantics.</div><br clear="all"><div><br></div>-- <br><div class="gmail_signature">+ Koos Zevenhoven + <a href="http://twitter.com/k7hoven" target="_blank">http://twitter.com/k7hoven</a> +</div>
</div></div>