On Tue, Oct 10, 2017 at 8:34 AM, Koos Zevenhoven <k7hoven@gmail.com> wrote:
On Tue, Oct 10, 2017 at 4:22 AM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
On Mon, Oct 9, 2017 at 8:37 PM, Koos Zevenhoven <k7hoven@gmail.com> wrote:
You can cause unbound growth in PEP 550 too. All you have to do is nest an unbounded number of generators.
You can only nest up to 'sys.get_recursion_limit()' number of generators.
With PEP 555 you can do:
while True: context_var.assign(42).__enter__()
Well, in PEP 550, you can explicitly stack an unbounded number of LogicalContexts in a while True loop.
No, you can't. PEP 550 doesn't have APIs to "stack ... LogicalContexts".
Or you can run out of memory using plain lists even faster:
l = [42]
while True: l *= 2 # ensure exponential blow-up
I don't see why your example with context_var.assign(42).__enter__() would be any more likely.
Of course you can write broken code. The point is that contexts work like scopes/mappings, and it's counter-intuitive that setting a variable with 'cv.assign(..).__enter__()' will break the world. If a naive user tries to convert their existing decimal-like API to use your PEP, everything would work initially, but then blow up in production. [..]
Really, it was my mistake to ever make you think that context_var.assign(42).__enter__() can be compared to .set(42) in PEP 550. I'll say it once more: PEP 555 context arguments have no equivalent of the PEP-550 .set(..).
Any API exposing a context manager should have an alternative try..finally API. In your case it's 'context_var.assign(42).__enter__()'. 'With' statements are sugar in Python. It's unprecedented to design API solely around them.
In PEP 555, nesting generators doesn't do anything really, unless you actually assign to context arguments in the generators. Only those who use it will pay.
Same for 550. If a generator doesn't set context variables, its LC will be an empty mapping (or NULL if you want to micro-optimize things). Nodes for the chain will come from a freelist. The effective overhead for generators is a couple operations on pointers, and thus visible only in microbenchmarks.
Sure, you can implement push and pop and maintain a freelist by just doing operations on pointers. But that would be a handful of operations. Maybe you'd even manage to avoid INCREFs and DECREFs by not exposing things as Python objects.
But I guarantee you, PEP 555 is simpler in this regard.
[..] I wrote several implementations of PEP 550 so far. No matter what you put in genobject.send(): one pointer op or two, the results are the same: in microbenchmarks generators become 1-2% slower. In macrobenchmarks of generators you can't observe any slowdown. And if we want the fastest possible context implementation, we can chose PEP 550 v1, which is the simplest solution. In any case, the performance argument is invalid, please stop using it.
But seriously, you will always end up in a weird situation if you call an unbounded number of contextmanager.__enter__() methods without calling __exit__(). Nothing new about that. But entering a handful of assignment contexts and leaving them open until a script ends is not the end of the world. I don't think anyone should do that though.
You'll say that it's not how the API is supposed to be used, and we say that we want to convert things like decimal and numpy to use the new mechanism. That question was also hand-waved by you: numpy and decimal will have to come up with new/better APIs to use PEP 555. Well, that's just not good enough.
What part of my explanation of this are you unhappy with? For instance, the 12th (I think) email in this thread, which is my response to Nathaniel. Could you reply to that and tell us your concern?
I'm sorry, I'm not going to find some 12th email in some thread. I stated in this thread the following: not being able to use PEP 555 to fix *existing* decimal & numpy APIs is not good enough. And decimal & numpy is only one example, there's tons of code out there that can benefit from its APIs to be fixed to support for async code in Python 3.7.
Well, anyone interested can read that 12th email in this thread. In short, my recommendation for libraries would be as follows:
* If the library does not provide a context manager yet, they should add one, using PEP 555. That will then work nicely in coroutines and generators.
* If the library does have a context manager, implement it using PEP 555. Or to be safe, add a new API function, so behavior in existing async code won't change.
* If the library needs to support some kind of set_state(..) operation, implement it by getting the state using a PEP 555 context argument and mutating its contents.
* Fall back to thread-local storage if no context argument is present or if the Python version does not support context arguments.
The last bullet point is the problem. Everybody is saying to you that it's not acceptable. It's your choice to ignore that. [..]
What do you mean by "just sweep it under the carpet"? Capturing the context at the moment of generators creation is a design choice with some consequences (that I illustrated in my previous email). There are cons and pros of doing that.
"Capturing the context at generator creation" and "isolating generators completely" are two different things.
I've described pros of the former. The latter has no pros that I'm aware of, except if sweeping things under the carpet is considered as one.
Yes, the latter works in some use cases, but in others it does not. For instance, if an async framework wants to make some information available throughout the async task. If you isolate generators, then async programmers will have to avoid generators, because they don't have access to the information the framework is trying to provide.
This is plain incorrect. Please read PEP 550v1 before continuing the discussion about it.
Also, if you refactor your generator into subgenerators using `yield from`, the subgenerators will not see the context set by the outer generator.
Subgenerators see the context changes in the outer generator in all versions of PEP 550. The point you didn't like is that in all versions of PEP 550 subgenerators could not leak any context to the outer generator. Please don't confuse these two. Yury