[Python-ideas] yield from multiple iterables (was Re: The async API of the future: yield-from)

[Guido van Rossum]

On Fri, Oct 19, 2012 at 10:37 PM, Greg Ewing
<greg.ewing at canterbury.ac.nz> wrote:
> Nick Coghlan wrote:
>
>> Please don't lose sight of the fact that yield-based suspension points
>> looking like something other than an ordinary function call is a
>> *feature*, not a bug.

(Ironically, Christian just revived an old thread where Nick was of a
different opinion.)

> People keep asserting that, but I don't think we have enough
> experience with the explicit-switching-point-markers-all-the-
> way-up style of coding to tell whether it's a good idea or not.

Hm. I would say I have a lot of real world experience with this style:
App Engine's NDB. It's in use by 1000s of people. I've written a lot
of complicated database code in it (integrating several layers of
caching). This style really does hold up well.

Now, I think that if it could yield from, NDB would be even better,
but for most code it would be a very minimal change, and the issue
here (the requirement to mark suspension points) is the same.

In C# they also have a lot of experience with this style -- functions
declared as async must be waited for using await, and the type checker
enforces that it's await all the way up (I think a function using
await must be declared as async, too).

> My gut feeling is that the explicit markers will help at the
> lowest levels, where you're trying to protect a critical section,
> but at the upper levels they will just be noise that causes
> unnecessary worry.

Actually, an earlier experience (like 25 years earier :-) suggests
that it's at the higher levels where you get in trouble without the
markers -- because you still have critical sections in end user code,
but it's impossible to remember which functions you call may cause a
task switch.

> In one of Guido's earlier posts (which I can't find now,
> unfortunately), he said something that made it sound like
> he was coming around to that point of view too, but he
> seems to have gone back on that recently.

I was probably more waxing philosophically on the reasons why people
like greenlets/gevent (if they like it). I feel I am pretty
consistently in favor of marking switch points, at least in the
context we are currently discussing (where high-speed async event
handling is the thing to do).

For less-performant situations I'm fine with writing classic
synchronous-looking code, and running it in multiple OS threads for
concurrency reasons. But the purpose of designing a new async API is
to break the barrier of one thread per connection.

-- 
--Guido van Rossum (python.org/~guido)