[Python-ideas] yield from multiple iterables (was Re: The async API of the future: yield-from)
tismer at stackless.com
Sat Oct 20 19:48:33 CEST 2012
On 20.10.12 18:30, Guido van Rossum wrote:
> On Fri, Oct 19, 2012 at 10:37 PM, Greg Ewing
> <greg.ewing at canterbury.ac.nz> wrote:
>> Nick Coghlan wrote:
>>> Please don't lose sight of the fact that yield-based suspension points
>>> looking like something other than an ordinary function call is a
>>> *feature*, not a bug.
> (Ironically, Christian just revived an old thread where Nick was of a
> different opinion.)
>> People keep asserting that, but I don't think we have enough
>> experience with the explicit-switching-point-markers-all-the-
>> way-up style of coding to tell whether it's a good idea or not.
> Hm. I would say I have a lot of real world experience with this style:
> App Engine's NDB. It's in use by 1000s of people. I've written a lot
> of complicated database code in it (integrating several layers of
> caching). This style really does hold up well.
> Now, I think that if it could yield from, NDB would be even better,
> but for most code it would be a very minimal change, and the issue
> here (the requirement to mark suspension points) is the same.
> In C# they also have a lot of experience with this style -- functions
> declared as async must be waited for using await, and the type checker
> enforces that it's await all the way up (I think a function using
> await must be declared as async, too).
>> My gut feeling is that the explicit markers will help at the
>> lowest levels, where you're trying to protect a critical section,
>> but at the upper levels they will just be noise that causes
>> unnecessary worry.
> Actually, an earlier experience (like 25 years earier :-) suggests
> that it's at the higher levels where you get in trouble without the
> markers -- because you still have critical sections in end user code,
> but it's impossible to remember which functions you call may cause a
> task switch.
>> In one of Guido's earlier posts (which I can't find now,
>> unfortunately), he said something that made it sound like
>> he was coming around to that point of view too, but he
>> seems to have gone back on that recently.
> I was probably more waxing philosophically on the reasons why people
> like greenlets/gevent (if they like it). I feel I am pretty
> consistently in favor of marking switch points, at least in the
> context we are currently discussing (where high-speed async event
> handling is the thing to do).
> For less-performant situations I'm fine with writing classic
> synchronous-looking code, and running it in multiple OS threads for
> concurrency reasons. But the purpose of designing a new async API is
> to break the barrier of one thread per connection.
It is of course a bit confusing to find out who thought what and when ;-)
And yes, I see your point, but I have difficulties to see how it is done
best. If I take Stackless as an example, well, there would everything
potentially be marked as some codef, because it is simply everywhere
But just for the fact that something _supports_ suspension or switching
is IMHO not enough reason to clutter the code everywhere.
What I think is need is a way to distinguish critical code paths.
Not sure how this should be.
Maybe it's my limited understanding.
The generator-based functions do not get switched from alone.
If they want to do that, they call some special function, and I would
mark them for doing that.
But all the tree up? I can't see the benefit so much.
Maybe it would be less verbose to have decorators that assert something
does _not_ switch, like guards? Or maybe add properties?
I agree that one needs certain information about the program that
can easily be extracted. Perhaps this could be done with an analysing
tool. This tool would only need additional hints if things are very dynamic,
like variables holding certain constructs which are known at runtime
Christian Tismer :^) <mailto:tismer at stackless.com>
Software Consulting : Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/
14482 Potsdam : PGP key -> http://pgp.uni-mainz.de
phone +49 173 24 18 776 fax +49 (30) 700143-0023
PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04
whom do you want to sponsor today? http://www.stackless.com/
More information about the Python-ideas