[Python-ideas] (PEP 555 subtopic) Propagation of context in async code

[Koos Zevenhoven]

On Fri, Oct 13, 2017 at 7:38 PM, Yury Selivanov <yselivanov.ml at gmail.com>
wrote:

> On Fri, Oct 13, 2017 at 11:49 AM, Koos Zevenhoven <k7hoven at gmail.com>
> wrote:
> [..]
> > This was my starting point 2.5 years ago, when Yury was drafting this
> status
> > quo (PEP 492). It looked a lot of PEP 492 was inevitable, but that there
> > will be a problem, where each API that uses "blocking IO" somewhere under
> > the hood would need a duplicate version for asyncio (and one for each
> > third-party async framework!). I felt it was necessary to think about a
> > solution before PEP 492 is accepted, and this became a fairly short-lived
> > thread here on python-ideas:
>
> Well, it's obvious why the thread was "short-lived".  Don't mix
> non-blocking and blocking code and don't nest asyncio loops.  But I
> believe this new subtopic is a distraction.


​Nesting is not the only way to have interaction between two event loops.​
​ But whenever anyone *does* want to nest two loops, they are perhaps more
likely to be loops of different frameworks.​

​You believe that the semantics in async code is a distraction?


> You should start a new
> thread on Python-ideas if you want to discuss the acceptance of PEP
> 492 2.5 years ago.
>

I
​'m definitely not interested in discussing the acceptance of PEP 492.
​


>
> [..]
> > The bigger question is, what should happen when a coroutine awaits on
> > another coroutine directly, without giving the framework a change to
> > interfere:
> >
> >
> > async def inner():
> >     do_context_aware_stuff()
> >
> > async def outer():
> >     with first_context():
> >         coro = inner()
> >
> >     with second_context():
> >         await coro
> >
> > The big question is: In the above, which context should the coroutine be
> run
> > in?
>
> The real big question is how people usually write code.  And the
> answer is that they *don't write it like that* at all.  Many context
> managers in many frameworks (aiohttp, tornado, and even asyncio)
> require you to wrap your await expressions in them.  Not coroutine
> instantiation.
>

​You know very well that I've been talking about how people usually write
code etc. But we still need to handle the corner cases too.​


>
> A more important point is that existing context solutions for async
> frameworks can only support a with statement around an await
> expression. And people that use such solutions know that 'with ...:
> coro = inner()' isn't going to work at all.
>
> Therefore wrapping coroutine instantiation in a 'with' statement is
> not a pattern.  It can only become a pattern, if whatever execution
> context PEP accepted in Python 3.7 encouraged people to use it.
>
>
​The code is to illustrate semantics, not an example of real code. The
point is to highlight that the context has changed between when the
coroutine function was called and when it is awaited. That's certainly a
thing that can happen in real code, even if it is not the most typical
case. I do mention this in my previous email.


> [..]
> > Both of these would have their own stack of (argument, value) assignment
> > pairs, explained in the implementation part of the first PEP 555 draft.
> > While this is a complication, the performance overhead of these is so
> small,
> > that doubling the overhead should not be a performance concern.
>
> Please stop handwaving performance.  Using big O notation:
>
>
​There is discussion on perfomance elsewhere, now also in this other
subthread:

https://mail.python.org/pipermail/python-ideas/2017-October/047327.html

PEP 555, worst complexity for uncached lookup: O(N), where 'N' is the
> total number of all context values for all context keys for the
> current frame stack.


​Not true. See the above link. Lookups are fast (*and* O(1), if we want
them to be).

​PEP 555 stacks are independent of frames, BTW.​



> For a recursive function you can easily have a
> situation where cache is invalidated often, and code starts to run
> slower and slower.
>

​Not true either. The lookups are O(1) in a recursive function, with and
without nested contexts.​

​I started this thread for discussion about semantics in an async context.
Stefan asked about performance in the other thread, so I posted there.

––Koos
​


> PEP 550 v1, worst complexity for uncached lookup: O(1), see [1].
>
> PEP 550 v2+, worst complexity for uncached lookup: O(k), where 'k' is
> the number of nested generators for the current frame. Usually k=1.
>
> While caching will mitigate PEP 555' bad performance characteristics
> in *tight loops*, the performance of uncached path must not be
> ignored.
>
> Yury
>
> [1] https://www.python.org/dev/peps/pep-0550/#appendix-hamt-
> performance-analysis
>



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20171013/51ce9488/attachment.html>