On Fri, Oct 12, 2012 at 10:05 PM, Greg Ewing wrote: [Long sections snipped, all very clear]

Guido van Rossum wrote:

...

(6) Spawning off multiple async subtasks

Futures: f1 = subtask1(args1) # Note: no yield!!! f2 = subtask2(args2) res1, res2 = yield f1, f2

Yield-from: ??????????

*** Greg, can you come up with a good idiom to spell concurrency at this level? Your example only has concurrency in the philosophers example, but it appears to interact directly with the scheduler, and the philosophers don't return values. ***

I don't regard the need to interact directly with the scheduler as a problem. That's because in the world I envisage, there would only be *one* scheduler, for much the same reason that there can really only be one async event handling loop in any given program. It would be part of the standard library and have a well-known API that everyone uses.

If you don't want things to be that way, then maybe this is a good use for yielding things to the scheduler. Yielding a generator could mean "spawn this as a concurrent task".

You could go further and say that yielding a tuple of generators means to spawn them all concurrently, wait for them all to complete and send back a tuple of the results. The yield-from code would then look pretty much the same as the futures code.

Sadly it looks that r = yield from (f1(), f2()) ends up interpreting the tuple as the iterator, and you end up with r = (f1(), f2()) (i.e., a tuple of generators) rather than the desired r = ((yield from f1()), (yield from f2()))

However, I'm inclined to think that this is too much functionality to build directly into the scheduler, and that it would be better provided by a class or function that builds on more primitive facilities.

Possibly. In NDB it is actually a very common operation which looks quite elegant. But your solution below is fine (and helps by giving people a specific entry in the documentation they can look up!)

So it would look something like

Yield-from: task1 = subtask1(args1) task2 = subtask2(args2) res1, res2 = yield from par(task1, task2)

where the implementation of par() is left as an exercise for the reader.

So, can par() be as simple as def par(*args): results = [] for task in args: result = yield from task results.append(result) return results ??? Or does it need to interact with the scheduler to ensure fairness? (Not having built one of these, my intuition for how the primitives fit together is still lacking, so excuse me for asking naive questions.) Of course there's the question of what to do when one of the tasks raises an error -- I haven't quite figured that out in NDB either, it runs all the tasks to completion but the caller only sees the first exception. I briefly considered having an "multi-exception" but it felt too weird -- though I'm not married to that decision.

...

(7) Checking whether an operation is already complete

Futures: if f.done(): ...

I'm inclined to think that this is not something the scheduler needs to be directly concerned with. If it's important for one task to know when another task is completed, it's up to those tasks to agree on a way of communicating that information between them.

Although... is there a way to non-destructively test whether a generator is exhausted? If so, this could easily be provided as a scheduler primitive.

Nick answered this affirmatively.

...

(8) Getting the result of an operation multiple times

Futures:

f = async_op(args) # squirrel away a reference to f somewhere else r = yield f # ... later, elsewhere r = f.result()

Is this really a big deal? What's wrong with having to store the return value away somewhere if you want to use it multiple times?

I suppose that's okay.

...

(9) Canceling an operation

Futures: f.cancel()

This would be another scheduler primitive.

Yield-from: cancel(task)

This would remove the task from the ready list or whatever queue it's blocked on, and probably throw an exception into it to give it a chance to clean up.

Ah, of course. (I said I was asking newbie questions. Consider me your first newbie!)

...

(10) Registering additional callbacks

Futures: f.add_done_callback(callback)

Another candidate for a higher-level facility, I think. The API might look something like

Yield-from: cbt = task_with_callbacks(task) cbt.add_callback(callback) yield from cbt.run()

I may have a go at coming up with implementations for some of these things and send them in later posts.

Or better, add them to the tutorial. (Or an advanced tutorial, "common async patterns". That would actually be a useful collection of use cases for whatever we end up building.) Here's another pattern that I can't quite figure out. It started when Ben Darnell posted a link to Tornado's chat demo (https://github.com/facebook/tornado/blob/master/demos/chat/chatdemo.py). I didn't understand it and asked him offline what it meant. Essentially, it's a barrier pattern where multiple tasks (each representing a different HTTP request, and thus not all starting at the same time) render a partial web page and then block until a new HTTP request comes in that provides the missing info. (For technical reasons they only do this once, and then the browsers re-fetch the URL.) When the missing info is available, it must wake up all blocked task and give then the new info. I wrote a Futures-based version of this -- not the whole thing, but the block-until-more-info-and-wakeup part. Here it is (read 'info' for 'messages'): Each waiter executes this code when it is ready to block: f = Future() # Explicitly create a future! waiters.add(f) messages = yield f <process messages and quit> I'd write a helper for the first two lines: def register(): f = Future() waiters.add(f) return f Then the waiter's code becomes: messages = yield register() <process messages and quit> When new messages become available, the code just sends the same results to all those Futures: def wakeup(messages): for waiter in waiters: waiter.set_result(messages) waiters.clear() (OO sauce left to the reader. :-) If you wonder where the code is that hooks up the waiter.set_result() call with the yield, that's done by the scheduler: when a task yields a Future, it adds a callback to the Future that reschedules the task when the Future's result is set. Edge cases: - Were the waiter to lose interest, it could remove its Future from the list of waiters, but no harm is done leaving it around either. (NDB doesn't have this feature, but if you have a way to remove callbacks, setting the result of a Future that nobody cares about has no ill effect. You could even use a weak set...) - It's possible to broadcast an exception to all waiters by using waiter.set_exception(). -- --Guido van Rossum (python.org/~guido)

On Sun, Oct 14, 2012 at 10:27 AM, Terry Reedy wrote:

On 10/14/2012 10:36 AM, Guido van Rossum wrote:

...

So, can par() be as simple as

def par(*args): results = [] for task in args: result = yield from task results.append(result) return results

???

Or does it need to interact with the scheduler to ensure fairness? (Not having built one of these, my intuition for how the primitives fit together is still lacking, so excuse me for asking naive questions.)

Of course there's the question of what to do when one of the tasks raises an error -- I haven't quite figured that out in NDB either, it runs all the tasks to completion but the caller only sees the first exception. I briefly considered having an "multi-exception" but it felt too weird -- though I'm not married to that decision.

One answer is to append the exception object to results and let the requesting code sort out what to do.

def par(*args): results = [] for task in args: try:

result = yield from task results.append(result) except Exception as exc: results.append(exc) return results

But then the caller would have to sort through the results and check for exceptions. I want the caller to be able to use try/except as well. So far the best I've come up with is to recommend that if you care about distinguishing multiple exceptions, use separate yields surrounded by separate try/except blocks. Note that the tasks can still run concurrently, just create all the futures before doing the first yield. -- --Guido van Rossum (python.org/~guido)

On Sun, Oct 14, 2012 at 12:38 PM, Terry Reedy wrote:

On 10/14/2012 1:42 PM, Guido van Rossum wrote:

...

On Sun, Oct 14, 2012 at 10:27 AM, Terry Reedy wrote:

...

On 10/14/2012 10:36 AM, Guido van Rossum wrote:

...

Of course there's the question of what to do when one of the tasks raises an error -- I haven't quite figured that out in NDB either, it runs all the tasks to completion but the caller only sees the first exception. I briefly considered having an "multi-exception" but it felt too weird -- though I'm not married to that decision.

...

...

One answer is to append the exception object to results and let the requesting code sort out what to do.

def par(*args): results = [] for task in args: try:

result = yield from task results.append(result) except Exception as exc: results.append(exc) return results

But then the caller would have to sort through the results and check for exceptions. I want the caller to be able to use try/except as well.

OK. Then ...

def par(*args): results = [] exceptions = False for task in args: try: result = yield from task results.append(result) except Exception as exc: results.append(exc) exceptions = True if not exceptions: return results else: exc = MultiXException() exc.results = results raise exc

Is this is what you meant by 'multi-exception'?

Yes.

caller:

try: results = <whatever> except MultiXException as exc: errprocess(exc.results)

In NDB I have yet to encounter a situation where I care. -- --Guido van Rossum (python.org/~guido)

On Sun, Oct 14, 2012 at 7:36 AM, Guido van Rossum wrote:

...

So it would look something like

Yield-from: task1 = subtask1(args1) task2 = subtask2(args2) res1, res2 = yield from par(task1, task2)

where the implementation of par() is left as an exercise for the reader.

So, can par() be as simple as

def par(*args): results = [] for task in args: result = yield from task results.append(result) return results

???

Or does it need to interact with the scheduler to ensure fairness? (Not having built one of these, my intuition for how the primitives fit together is still lacking, so excuse me for asking naive questions.)

It's not just fairness, it needs to interact with the scheduler to get any parallelism at all if the sub-generators have more than one step. Consider: def task1(): print "1A" yield print "1B" yield print "1C" # and so on... def task2(): print "2A" yield print "2B" yield print "2C" def outer(): yield from par(task1(), task2()) Both tasks are started immediately, but can't progress further until they are yielded from to advance the iterator. So with this version of par() you get 1A, 2A, 1B, 1C..., 2B, 2C. To get parallelism I think you have to schedule each sub-generator separately instead of just yielding from them (which negates some of the benefits of yield from like easy error handling). Even if there is a clever version of par() that works more like yield from, you'd need to go back to explicit scheduling if you wanted parallel execution without forcing everything to finish at the same time (which is simple with Futures).

Of course there's the question of what to do when one of the tasks raises an error -- I haven't quite figured that out in NDB either, it runs all the tasks to completion but the caller only sees the first exception. I briefly considered having an "multi-exception" but it felt too weird -- though I'm not married to that decision.

In general for this kind of parallel operation I think it's fine to say that one (unspecified) exception is raised in the outer function and the rest are hidden. With futures, "(r1, r2) = yield (f1, f2)" is just shorthand for "r1 = yield f1; r2 = yield f2", so separating the yields to have separate try/except blocks is no problem. WIth yield from it's not as good because the second operation can't proceed while the outer function is waiting for the first. -Ben

On Sun, Oct 14, 2012 at 3:27 PM, Guido van Rossum wrote:

On Sun, Oct 14, 2012 at 3:09 PM, Ben Darnell wrote:

...

On Sun, Oct 14, 2012 at 7:36 AM, Guido van Rossum wrote:

...

...

So it would look something like

Yield-from: task1 = subtask1(args1) task2 = subtask2(args2) res1, res2 = yield from par(task1, task2)

where the implementation of par() is left as an exercise for the reader.

So, can par() be as simple as

def par(*args): results = [] for task in args: result = yield from task results.append(result) return results

???

Or does it need to interact with the scheduler to ensure fairness? (Not having built one of these, my intuition for how the primitives fit together is still lacking, so excuse me for asking naive questions.)

It's not just fairness, it needs to interact with the scheduler to get any parallelism at all if the sub-generators have more than one step. Consider:

def task1(): print "1A" yield print "1B" yield print "1C" # and so on...

def task2(): print "2A" yield print "2B" yield print "2C"

def outer(): yield from par(task1(), task2())

Hm, that's a little unrealistic -- in practice you'll rarely see code that yields unless it is also blocking for I/O. I presume that if both tasks immediately block for I/O, the one whose I/O completes first gets the run next; and if it then blocks again, it'll again depend on whose I/O finishes first.

(Admittedly this has little to do with fairness now.)

...

Both tasks are started immediately, but can't progress further until they are yielded from to advance the iterator. So with this version of par() you get 1A, 2A, 1B, 1C..., 2B, 2C.

Really? When you call a generator, it doesn't run until the first yield; it gets suspended before the first bytecode of the body. So if anything, you might get 1A, 1B, 1C, 2A, 2B, 2C. (Which would prove your point just as much of course.)

Ah, OK. I was mistaken about the "first yield" part, but the rest stands. The problem is that as soon as task1 blocks on IO, the entire current task (which includes outer(), par(), and both children) gets unscheduled. no part of task2 gets scheduled until it gets yielded from, because the scheduler can't see it until then.

Sadly I don't have a framework lying around where I can test this easily -- I'm pretty sure that the equivalent code in NDB interacts with the scheduler in a way that ensures round-robin scheduling.

...

To get parallelism I think you have to schedule each sub-generator separately instead of just yielding from them (which negates some of the benefits of yield from like easy error handling).

Honestly I don't mind of the scheduler has to be messy, as long the mess is hidden from the caller.

Agreed.

...

Even if there is a clever version of par() that works more like yield from, you'd need to go back to explicit scheduling if you wanted parallel execution without forcing everything to finish at the same time (which is simple with Futures).

Why wouldn't all generators that aren't blocked for I/O just run until their next yield, in a round-robin fashion? That's fair enough for me.

But as I said, my intuition for how things work in Greg's world is not very good.

The good and bad parts of this proposal both stem from the fact that yield from is very similar to just inlining everything together. This gives you the exception handling semantics that you expect from synchronous code, but it means that the scheduler can't distinguish between subtasks; you have to explicitly schedule them as top-level tasks.

...

...

Of course there's the question of what to do when one of the tasks raises an error -- I haven't quite figured that out in NDB either, it runs all the tasks to completion but the caller only sees the first exception. I briefly considered having an "multi-exception" but it felt too weird -- though I'm not married to that decision.

In general for this kind of parallel operation I think it's fine to say that one (unspecified) exception is raised in the outer function and the rest are hidden. With futures, "(r1, r2) = yield (f1, f2)" is just shorthand for "r1 = yield f1; r2 = yield f2", so separating the yields to have separate try/except blocks is no problem. With yield from it's not as good because the second operation can't proceed while the outer function is waiting for the first.

Hmmm, I think I see your point. This seems to follow if (as Greg insists) you don't have any decorators on the generators.

OTOH I am okay with only getting one of the exceptions. But I think all of the remaining tasks should still be run to completion -- maybe the caller just cared about their side effects. Or maybe this should be an option to par().

That's probably a good idea. -Ben

-- --Guido van Rossum (python.org/~guido)

Ben Darnell wrote: The problem is that as soon as task1 blocks on IO, the entire

current task (which includes outer(), par(), and both children) gets unscheduled. no part of task2 gets scheduled until it gets yielded from, because the scheduler can't see it until then.

The suggested implementation of par() that I posted does explicitly schedule the subtasks. Then it repeatedly yields, giving them a chance to run, until they all complete. -- Greg

On Sun, Oct 14, 2012 at 10:58 PM, Greg Ewing wrote:

Guido van Rossum wrote:

...

Why wouldn't all generators that aren't blocked for I/O just run until their next yield, in a round-robin fashion? That's fair enough for me.

But as I said, my intuition for how things work in Greg's world is not very good.

That's exactly how my scheduler behaves.

...

OTOH I am okay with only getting one of the exceptions. But I think all of the remaining tasks should still be run to completion -- maybe the caller just cared about their side effects. Or maybe this should be an option to par().

This is hard to answer without considering real use cases, but my feeling is that if I care enough about the results of the subtasks to wait until they've all completed before continuing, then if anything goes wrong in any of them, I might as well abandon the whole computation.

If that's not the case, I'd be happy to wrap each one in a try-except that doesn't propagate the exception to the main task, but just records the information that the subtask failed somewhere, for the main task to check afterwards.

Another direction to approach this is to consider that par() ought to be just an optimisation -- the result should be the same as if you'd written sequential code to perform the subtasks one after another. And in that case, an exception in one would prevent any of the following ones from executing, so it's fine if par() behaves like that, too.

I'd think of such a par() more as something that saves me typing than as an optimization. Anyway, the key functionality I cannot live without here is to start multiple tasks concurrently. It seems that without par() or some other scheduling primitive, you cannot do that: if I write a = foo_task() # Search google b = bar_task() # Search bing ra = yield from a rb = yield from b # now compare search results the tasks run sequentially. A good par() should run then concurrently. But there needs to be another way to get a task running immediately and concurrently; I believe that would be a = spawn(foo_task()) right? One could then at any later point use ra = yield from a One could also combine these and do e.g. a = spawn(foo_task()) b = spawn(bar_task()) <do more work locally> ra, rb = yield from par(a, b) Have I got the spelling for spawn() right? In many other systems (e.g. threads, greenlets) this kind of operation takes a callable, not the result of calling a function (albeit a generator). If it takes a generator, would it return the same generator or a different one to wait for? -- --Guido van Rossum (python.org/~guido)

Calvin Spealman wrote:

If we allow spawn(task()) then we're not getting nice tracebacks anyway, so I think we should allow

future1 = yield task1() # spawn task future2 = yield task2() # spawn other task

I don't think it's necessary to allow 'yield task' as a method of spawning in order to get nice tracebacks for spawned tasks. In the Task-object-based system I'm thinking about, if an exception reaches the top level of a Task, it gets stored in the Task object until another task wait()s for it, and then it continues to propagate. This makes sense, because the wait() establishes a task-subtask relationship, so the traceback should proceed from the subtask to the waiting task.

Both are primitives we need to support as first-class operation. That is, without some wrapper like spawn().

In my system, spawn() isn't a wrapper -- it *is* the primitive way to create an independent task. And I think it's the only one we need. -- Greg

Calvin Spealman wrote:

What if two tasks call wait() on the same subtask which raises an error?

That would be disallowed. A given Task would only be allowed to have its wait() method called once. The reason for this restriction is because of the way tracebacks are attached to exception objects in Python 3, which means that exceptions are effectively single-use now. If it weren't for that, the exception could simply be raised in *both* waiters.

I think we should let errors propagate through yield-from, primarily. That's what it exists for.

Yes, and that's exactly what my wait() mechanism does. You call the wait() method using yield-from. The important idea is that just because you spawn a task, it doesn't necessarily follow that you want to be regarded as the *parent* of that task and receive its exceptions. That only becomes clear when you wait() for it.

...

In my system, spawn() isn't a wrapper -- it *is* the primitive way to create an independent task. And I think it's the only one we need.

It has to know what scheduler to talk to, right?

Yes, but in my world, there is only *one* scheduler. I understand that not everyone thinks that's a good idea, and I'm thinking about ways to remove that restriction. But I'm not yet sure that it *should* be removed even if it can. It seems to me that having multiple schedulers is inviting many of the same problems as having multiple event loops, and this whole disussion is centred on the idea that there should only be one of those. Just to be clear, I'm not saying there should only be one scheduler *implementation* in existence -- only that there should only be one *instance* of some scheduler implementation in any given program (or thread, if you're using those). And there should be a standard interface for it and an agreed way of finding the instance. What you're saying is that the standard interface should consist of yielded instructions and the instance should be found implicitly using dynamic scoping. This is *very* different from the kind of interface used for everything else in Python, and I'm not yet convinced that such a large amount of weirdness is justified. -- Greg

Guido van Rossum wrote:

On Tue, Oct 16, 2012 at 2:14 PM, Greg Ewing wrote:

...

The important idea is that just because you spawn a task, it doesn't necessarily follow that you want to be regarded as the *parent* of that task and receive its exceptions.

Maybe. But the opposite doesn't follow either. It's a toss-up between the spawner and the waiter.

So maybe spawn() should have an option indicating that the spawning task is to receive exceptions occuring in the spawned task. -- Greg

On Wed, Oct 17, 2012 at 5:27 AM, Christian Tismer wrote:

On 17.10.12 12:38, Greg Ewing wrote:

...

Guido van Rossum wrote:

...

On Tue, Oct 16, 2012 at 2:14 PM, Greg Ewing wrote:

...

The important idea is that just because you spawn a task, it doesn't necessarily follow that you want to be regarded as the *parent* of that task and receive its exceptions.

Maybe. But the opposite doesn't follow either. It's a toss-up between the spawner and the waiter.

So maybe spawn() should have an option indicating that the spawning task is to receive exceptions occuring in the spawned task.

No idea if that helps here, but the same problem occurred for us as well. It is not always clear if an exception should be handled in a certain context, or if it should be passed on and get raised later in the context that is concerned.

For that, Stackless has introduced a _bomb_ object that encapsulates an exception, in order to let it pass through the normal call/yield/return interface. It is used to send an exception over a channel, which will explode (raise that exception) when the receiver picks it later up.

Hmm... That sounds a little like your iriginal design for the channel only supported transferring values. At least for NDB, all channels support exceptions and tracebacks as an explicit alternative to the value.

I could think of something similar as a way to collect very many results in a join construct that collects everything without the need to handle each exception in the very moment it was raised.

That would make it possible to collect results efficiently using 'yield from' and inspect the results later.

Probably nothing new, just mentioned an idea...

I do think we're hashing out important ideas... -- --Guido van Rossum (python.org/~guido)

On Tue, Oct 16, 2012 at 5:14 PM, Greg Ewing wrote:

Calvin Spealman wrote:

...

What if two tasks call wait() on the same subtask which raises an error?

That would be disallowed. A given Task would only be allowed to have its wait() method called once.

The reason for this restriction is because of the way tracebacks are attached to exception objects in Python 3, which means that exceptions are effectively single-use now. If it weren't for that, the exception could simply be raised in *both* waiters.

...

I think we should let errors propagate through yield-from, primarily. That's what it exists for.

Yes, and that's exactly what my wait() mechanism does. You call the wait() method using yield-from.

The important idea is that just because you spawn a task, it doesn't necessarily follow that you want to be regarded as the *parent* of that task and receive its exceptions. That only becomes clear when you wait() for it.

...

...

In my system, spawn() isn't a wrapper -- it *is* the primitive way to create an independent task. And I think it's the only one we need.

It has to know what scheduler to talk to, right?

Yes, but in my world, there is only *one* scheduler.

Practically speaking, that is nice. But, are there use cases for multiple schedulers we should support? I also like the idea of the scheduler being an iterable, and thus itself being something you can schedule. Turtles all the way down.

I understand that not everyone thinks that's a good idea, and I'm thinking about ways to remove that restriction. But I'm not yet sure that it *should* be removed even if it can. It seems to me that having multiple schedulers is inviting many of the same problems as having multiple event loops, and this whole disussion is centred on the idea that there should only be one of those.

Just to be clear, I'm not saying there should only be one scheduler *implementation* in existence -- only that there should only be one *instance* of some scheduler implementation in any given program (or thread, if you're using those). And there should be a standard interface for it and an agreed way of finding the instance.

What you're saying is that the standard interface should consist of yielded instructions and the instance should be found implicitly using dynamic scoping. This is *very* different from the kind of interface used for everything else in Python, and I'm not yet convinced that such a large amount of weirdness is justified.

I don't follow the part about "found implicitly using dynamic scoping". What do you mean? In my model, the tasks never find the scheduler at all. They don't directly access it at all.

-- Greg _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

-- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy

But the scheduler we're talking about is all pure Python. If the interface is well known and universally used, and there's a good implementation of it in the standard library, why would anyone want another one?

Probably because they already have another one and can't get rid of it. Whether or not we are trying to include GUI development in this, I can guarantee that people will try and use it with a GUI message loop (to avoid blocking on IO, largely). In this case we'd almost certainly need a different implementation for Wx/Tcl/whatever. "Universally used" is a nice idea, but it will take a long time to get there. A well known interface, especially one that doesn't require the loop itself (i.e. it doesn't have a blocking run() function), lets users write thin wrappers, like the one we did for Tcl: http://pastebin.com/FuZwc1Ur (CallableContext (the 'scheduler') base class is in http://pastebin.com/ndS53Cd8). There needs to be an way to change which one is used at runtime, but there only needs to be one per thread. Cheers, Steve

Piet Delport wrote:

In particular, this suspend() could be used to integrate fairly directly with callback-based APIs: for example, if you have a Twisted Deferred, you could do:

result = yield suspend(d.addCallback)

I've been thinking about how to express this using the primitives provided by the scheduler in my tutorial. I don't actually have a primitive that simply suspends a task; instead, I have one that moves the current task from the ready list to a specified list: scheduler.block(queue) Similarly, I don't have a primitive that explicitly resumes a particular task[1] -- only one that takes the first task off a specified list and resumes it: scheduler.unblock(queue) I think this is a good idea if we want to be able to cancel tasks, because a cancelled task ought to cleanly disappear from the system, without any risk that something will try to schedule it again. This is achievable if we maintain the invariant that a task always belongs to some queue, and the scheduler knows about that queue. Given these primitives, we can define def wakeup_callback(queue): lambda: scheduler.unblock(queue) and def wait_for_callback(add_callback): q = [] add_callback(wakeup_callback(q)) scheduler.block(q) yield This is starting to look rather like a semaphore. If we assume semaphores as a facility provided by the library, then it becomes very straightforward: def wait_for_callback(add_callback): s = Semaphore() add_callback(s.notify) yield from s.wait() That assumes the callback is single-use. But a semaphore can also handle multi-use callbacks: you just keep the semaphore around and repeatedly wait on it. You will get woken up once for each time the callback is called. s = Semaphore() something.add_callback(s.notify) while we_are_still_interested(): yield from s.wait() ... --- [1] Actually I do, but I'm thinking it shouldn't be exposed as part of the public API for reasons given here. -- Greg

2012/10/18 Greg Ewing

I've converted my tutorial on generator-based tasks for Python 3.3, tidied it up a bit and posted it here:

http://www.cosc.canterbury.ac.nz/greg.ewing/python/tasks/

I liked it. I was kind of confused about use of yield/yield from in this style of async. Now things seems to be pretty clear. -- Carlo Pires

Yuval Greenfield wrote:

block() removes the current task from the ready_list, but is the current task guaranteed to be my task?

Yes, block() always operates on the currently running task.

If so, then I'd never run again after the yield in acquire(), that is unless a gracious other player unblocks me.

Yes, the unblocking needs to be done by another task, or by something outside the task system such as an I/O callback.

I'm not sure it makes sense for scheduler functions to store waiting tasks in a queue owned by the app and invisible from the scheduler. This can cause *invisible deadlocks* such as:

schedule(philosopher("Socrates", 8, 3, 1, forks[0], forks[2]), "Socrates") schedule(philosopher("Euclid", 5, 1, 4, forks[2], forks[0]), "Euclid")

Deadlocks are a potential problem in any system involving concurrency, and have to be dealt with on a case-by-case basis. Simply having the scheduler know where all the tasks are will not prevent deadlocks. It might make it possible for the scheduler to *detect* deadlocks, but you still have to do something about them. Having said that, I'm thinking about writing a more elaborate version of my scheduler that does keep track of which queue a task is waiting on, mainly so that tasks can be cancelled cleanly.

Is there a coroutine strategy for tackling these challenges? Or will I just get better at overcoming them with practice?

If you've been using threads all your life as you say, then you're probably already pretty good at dealing with them. All of the same techniques apply. -- Greg

Guido van Rossum wrote:

But there needs to be another way to get a task running immediately and concurrently; I believe that would be

a = spawn(foo_task())

right? One could then at any later point use

ra = yield from a

Hmmm. I suppose it *could* be made to work that way, but I'm not sure it's a good idea, because it blurs the distinction between invoking a subtask synchronously and waiting for the result of a previously spawned independent task. Recently I've been thinking about an implementation where it would look like this. First you do t = spawn(foo_task()) but what you get back is *not* a generator; rather it's a Task object which wraps a generator and provides various operations. One of them would be r = yield from t.wait() which waits for the task to complete and then returns its value (or if it raised an exception, propagates the exception). Other operations that a Task object might support include t.unblock() # wake up a blocked task t.cancel() # unschedule and clean up the task t.throw(exception) # raise an exception in the task (I haven't included t.block(), because I think that should be a stand-alone function that operates on the current task. Telling some other task to block feels like a dodgy thing to do.)

One could also combine these and do e.g.

a = spawn(foo_task()) b = spawn(bar_task()) <do more work locally> ra, rb = yield from par(a, b)

If you're happy to bail out at the first exception, you wouldn't strictly need a par() function for this, you could just do a = spawn(foo_task()) b = spawn(bar_task()) ra = yield from a.wait() rb = yield from b.wait()

Have I got the spelling for spawn() right? In many other systems (e.g. threads, greenlets) this kind of operation takes a callable, not the result of calling a function (albeit a generator).

That's a result of the fact that a generator doesn't start running as soon as you call it. If you don't like that, the spawn() operation could be defined to take an uncalled generator and make the call for you. But I think it's useful to make the call yourself, because it gives you an opportunity to pass parameters to the task.

If it takes a generator, would it return the same generator or a different one to wait for?

In your version above where you wait for the task simply by calling it with yield-from, spawn() would have to return a generator (or something with the same interface). But it couldn't be the same generator -- it would have to be a wrapper that takes care of blocking until the subtask is finished. -- Greg

Guido van Rossum wrote:

On Tue, Oct 16, 2012 at 12:20 AM, Greg Ewing wrote:

...

it blurs the distinction between invoking a subtask synchronously and waiting for the result of a previously spawned independent task.

Are you sure you really want to distinguish between those though?

I think I do. Partly because I feel that not doing so would make code harder to reason about. Async stuff is difficult enough as it is without hiding the boundaries between one thread of control and another. There are technical reasons as well. If you use 'yield from' to wait for completion of an independent task, then it would seem like you should be able to do this: t1 = task1() t2 = task2() spawn(t1) spawn(t2) r1 = yield from t1 r2 = yield from t2 But that can't work -- the object that you wait on has to be different from the generator instance passed to spawn(). The reason is that if the task finishes before anyone waits on it, the return value needs to be stored somewhere. Having spawn() return an object that deliberately does *not* have the interface of a generator, and having to explicitly wait for it, makes it much less likely that anyone will make that kind of mistake. If you wrote t1 = task1() t2 = task2() spawn(t1) spawn(t2) r1 = yield from t1.wait() r2 = yield from t2.wait() you would quickly get an exception, because generators don't have a wait() method. -- Greg

Guido van Rossum wrote:

On Fri, Oct 12, 2012 at 10:05 PM, Greg Ewing wrote:

...

You could go further and say that yielding a tuple of generators means to spawn them all concurrently, wait for them all to complete and send back a tuple of the results. The yield-from code would then look pretty much the same as the futures code.

Sadly it looks that

r = yield from (f1(), f2())

ends up interpreting the tuple as the iterator,

That's not yielding a tuple of generators. This is: r = yield (f1(), f2()) Note the absence of 'from'.

So, can par() be as simple as

def par(*args): results = [] for task in args: result = yield from task results.append(result) return results

No, it can't be as simple as that, because that will just execute the tasks sequentially. It would have to be something like this: def par(*tasks): n = len(tasks) results = [None] * n for i, task in enumerate(tasks): def thunk(): nonlocal n results[i] = yield from task n -= 1 scheduler.schedule(thunk) while n > 0: yield return results Not exactly straightforward, but that's why we write it once and put it in the library. :-)

Of course there's the question of what to do when one of the tasks raises an error -- I haven't quite figured that out in NDB either, it runs all the tasks to completion but the caller only sees the first exception. I briefly considered having an "multi-exception" but it felt too weird -- though I'm not married to that decision.

Hmmm. Probably what should happen is that all the other tasks get cancelled and then the exception gets propagated to the caller of par(). If we assume another couple of primitives: scheduler.cancel(task) -- cancels the task scheduler.throw(task, exc) -- raises an exception in the task then we could implement it this way: def par(*tasks): n = len(tasks) results = [None] * n this = scheduler.current_task for i, task in enumerate(tasks): def thunk(): nonlocal n try: results[i] = yield from task except BaseException as e: for t in tasks: scheduler.cancel(t) scheduler.throw(this, e) n -= 1 scheduler.schedule(thunk) while n > 0: yield return results

...

...

(10) Registering additional callbacks

While we're at it: class task_with_callbacks(): def __init__(self, task): self.task = task self.callbacks = [] def add_callback(self, cb): self.callbacks.append(cb) def run(self): result = yield from self.task for cb in self.callbacks: cb() return result

Here's another pattern that I can't quite figure out. ... Essentially, it's a barrier pattern where multiple tasks (each representing a different HTTP request, and thus not all starting at the same time) render a partial web page and then block until a new HTTP request comes in that provides the missing info.

This should be fairly straightforward. waiters = [] # Tasks waiting for the event When a task wants to wait: scheduler.block(waiters) When the event occurs: for t in waiters: scheduler.schedule(t) del waiters[:] Incidentally, this is a commonly encountered pattern known as a "condition queue" in IPC parlance. I envisage that the async library would provide encapsulations of this and other standard IPC mechanisms such as mutexes, semaphores, channels, etc. -- Greg

Guido van Rossum wrote:

But, as Christian Tismer wrote, we need to have some kind of idea of what the primitives are that we want to support.

Well, I was just responding to your asking what the yield-from equivalent would be to the corresponding thing using Futures. I assumed from the fact that you asked that it was something Futures-using people like to do a lot, so it would be worth putting into a library. There may be other ways to approach it, though. Suppose we had a primitive that just waits for a single task to finish and returns its value. Then we could do this: def par(*tasks): for task in tasks: scheduler.schedule(task) return [yield from scheduler.wait_for(task) for task in tasks] That's straightforward enough that maybe it doesn't even need to be a library function, just a well-known pattern.

Maybe you meant condition variable? It looks like threading.Condition with notify_all().

Something like that -- the terminology probably varies a bit from one library to another. The basic concept is "set of tasks waiting for some condition to become true".

Anyway, I agree we need some primitives like these, but I'm not sure how to choose the set of essentials.

I think that most, maybe all, of the standard synchronisation mechanisms, like mutexes and semaphores, can be built out of the primitives I've already introduced -- essentially just block() and yield. So anything of this kind that we provide will be more in the way of convenience features than essential primitives. -- Greg

On Mon, Oct 15, 2012 at 8:25 PM, Calvin Spealman wrote:

The more I follow this thread the less I understand the point of introducing a new use for yield-from in this discussion.

+1. To me, "yield from" is just a tool that brings generators back to parity with functions when it comes to breaking up a larger algorithm into smaller pieces. Where you would break a function out into subfunctions and call them normally, with a generator you can break out subgenerators and invoke them with yield from. Any meaningful use of "yield from" in the coroutine context *has* to ultimate devolve to an operation that: 1. Asks the scheduler to schedule another operation 2. Waits for that operation to complete Guido's approach to that problem is that step 1 is handled by calling functions that in turn call methods on a thread-local scheduler. These methods return Future objects, which can subsequently be yielded to the scheduler to say "I'm waiting for this future to be set". I *thought* Greg's way combined step 1 and step 2 into a single operation: the objects you yield *not only* say what you want to wait for, but also what you want to do. However, his example par() implementation killed that idea, since it turned out to need to schedule tasks explicitly rather than their being a "execute this in parallel" option. So now I'm back to think that Greg and Guido are talking about different levels. *Any* scheduling option will be able to be collapsed into an async task invoked by "yield from" by writing: def simple_async_task(): return yield start_task() The part that still needs to be figured out is how you turn that suspend/resume communications channel between the lowest level of the task stack and the scheduling loop into something usable, as well as how you handle iteration in a sensible way (I described my preferred approach when writing about the API I'd like to see for an async version of as_completed). I haven't seen anything to suggest that "yield from"'s role should change from what it is in 3.3: a way to factor out generators into multiple pieces with out breaking send() and throw(). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Mon, Oct 15, 2012 at 10:31 PM, Calvin Spealman wrote:

Currently, "with yield expr:" is not valid syntax, surprisingly.

It's not that surprising, it's the general requirement that yield expressions must be enclosed in parentheses except when used standalone or in a simple assignment statement. "with (yield expr):" is valid syntax though, so I'm reluctant to endorse doing anything substantially different if the parentheses are omitted. I think the combination of "yield from" to delegate control (including exception handling) completely to a subgenerator and "context manager + for loop + explicit yield" when an operation needs to yield multiple times and the exception handling behaviour should be left to the caller (as in the "as_completed" case) should cover the necessary behaviours. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Tue, Oct 16, 2012 at 12:16 AM, Calvin Spealman wrote:

I'm still -1 on delegating control to subgenerators with yield-from, versus having the scheduler just deal with them directly. I think it is far less flexible.

Um, yield from is to generators as calls are to functions... delegating to subgenerators, regardless of context, is what it's *for*. Without it, the scheduler will have to do quite a bit of extra work to reconstruct sane stack traces. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Tue, Oct 16, 2012 at 1:16 AM, Calvin Spealman wrote:

An example of this is a task which makes multiple requests, but only needs to wait for the results from less-than-all of them before returning. It might still want the other tasks to complete, even if it won't do anything with the results.

yield-from semantics won't allow a called task to continue, if needed, after the calling task itself has completed.

Is there another way these semantics could be expressed?

Sure, did you see my as_completed example? You couldn't use "yield from" for that, you'd need to use an ordinary iterator and an explicit yield in the body of the loop (this is why I disagree with Greg that "yield from" can serve as the one true API - it doesn't handle partial iteration, and it doesn't handle pre- or post- processing around the suspension points while iterating). My preferred way of thinking of "yield from" is as a simple refactoring tool: "Gee, this generator is getting kind of long and unwieldy. I'll move this piece out into a separate generator, and use yield from to invoke it" or "Hmm, I keep using this same sequence of 3 or 4 operations. I guess I'll move them out to a separate generator and use yield from to invoke it in the appropriate places". Compare that with the almost identical equivalents when refactoring a function to call a helper function instead of doing everything inline: "Gee, this function is getting kind of long and unwieldy. I'll move this piece out into a separate function, and call it" or "Hmm, I keep using this same sequence of 3 or 4 operations. I guess I'll move them out to a separate function and call it it in the appropriate places". Just as some operations can't be factored out with simple function calls, hence we have iterators and context managers, so not all operations will be able to be factored out of a coroutine with "yield from" (hence why I consider "yield" to be the more appropriate core primitive, with "yield from" just correctly factoring out the task of complete delegation, which is otherwise hard to do correctly) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Laurens Van Houtven wrote:

On Mon, Oct 15, 2012 at 5:32 PM, Nick Coghlan mailto:ncoghlan@gmail.com> wrote:

My preferred way of thinking of "yield from" is as a simple refactoring tool

I agree. That's how I've used it. Maybe that's just short-sightedness.

And that's exactly how *I* see it as well! Which means some people must be misinterpreting something I'm saying, if they think I see it some other way. -- Greg

Nick Coghlan wrote:

(this is why I disagree with Greg that "yield from" can serve as the one true API - it doesn't handle partial iteration, and it doesn't handle pre- or post- processing around the suspension points while iterating).

I'm aware of the iteration problem, but I'm not convinced that the convolutions necessary to make it possible to use a for-loop for this are worth the bother, as opposed to simply accepting that you can't use the for statement in this situation, and using some other kind of loop. In any case, even if we decide to provide a scheduler instruction to enable using for-loops on suspendable iterators somehow, it doesn't follow that we should use scheduler instructions for anything *else*. I would consider such a scheduler instruction to be a stopgap measure until we can find a better solution -- just as yield-from is a better solution than using "call" and "return" scheduler instructions. -- Greg

On Tue, 16 Oct 2012 17:48:24 +1000 Nick Coghlan wrote:

def _wait_first(futures): # futures must be a set, items will be removed as they complete signal = Future() def chain_result(completed): futures.remove(completed) if completed.cancelled(): signal.cancel() signal.set_running_or_notify_cancel() elif completed.done(): signal.set_result(completed.result()) else: signal.set_exception(completed.exception()) for f in futures: f.add_done_callback(chain_result) return signal

def wait_first(futures): return _wait_first(set(futures))

def as_completed(futures): remaining = set(futures) while 1: if not remaining: break yield _wait_first(remaining)

@task def load_url_async(url) return url, (yield urllib.urlopen_async(url)).read()

@task def example(urls): for get_next_page in as_completed(load_url_async(url) for url in urls): try: url, data = yield get_next_page except Exception as exc: print("Something broke: {}".format(exc)) else: print("Loaded {} bytes from {!r}".format(len(data), url))

Your example looks rather confusing to me. There are a couple of things I don't understand: - why does load_url_async return something instead of yielding it? - how does overlapping of reads happen? you seem to consider that a read() will be non-blocking once the server starts responding to your request, which is only true if the response is small (or you have a very fast connection to the server). - if read() is really non-blocking, why do you yield get_next_page? What does that achieve? Actually, what is yielding a tuple supposed to achieve at all? - where is control transferred over to the scheduler? it seems it's only in get_next_page, while I would expect it to be transferred in as_completed as well. Regards Antoine.

It yields *and* returns, that's the way Guido's API works (as I understand it).

I can't speak for Guido obviously, but you've certainly described what we came up with perfectly (http://pastebin.com/ndS53Cd8, the _Awaiter class starts on line 93).

# The real kicker here? Replace "yield wait_for_page" with "wait_for_page.result()" and you have the equivalent concurrent.futures code.

Basically, the task/tasklet/async decorator aggregates the futures from inside the wrapped method and exposes a single future to the caller. Your example doesn't even need a scheduler or event loop, and we found that all the event loop was really doing was running the callbacks in a certain thread/context/equivalent. And because there is a future coming out of every call, the user can choose when to stop using tasklets and go back to using plain old futures (or whatever subclasses have been used).

I'm sure doing concurrent I/O will require an event loop, unless you use threads under the hood...

Polling I/O will require some sort of loop, yes, but I/O that triggers a callback at the OS level (such as ReadFileEx and WriteFileEx on Windows) doesn't need it. Of course, without an event loop you still need to wait on the future - for polling I/O you could return a subclassed future where waiting starts the polling loop if there isn't a better event loop available. My view is that the most helpful thing to have in the standard is a way for any code to find and interact with an event loop - if we can discover a scheduler/context/loop/whatever and use its commands for "run this callable as soon as you can" and "run this callable when this condition is true" then we can have portable support for polling or event-based I/O (as well as being able to handle other thread-sensitive code such as in UIs). For optimal support, you'll need to have very close coupling between the scheduler and the asynchronous operations. This can be built on top of the portable support, but aiming for optimal support initially is a good way to make this API painful to use and more likely to be ignored.

Well, how do you plan for that callback to execute Python code?

IMO, that is the most important question in all of this discussion. With any I/O some waiting is required - there must be a point where the application is not doing anything other than waiting for the I/O to complete, regardless of whether a loop is used or not. (Ideally the I/O is already complete by the time we start waiting.) The callbacks in the particular examples require a thread to be in an alertable wait state, which is basically equivalent to select(), though a little less discriminatory (as in, ANY I/O callback can interrupt an alertable wait). In my view, these callbacks should be 'leaving a message' for the main program to run a particular function when it next has a chance. Like an interrupt handler, the aim is to do the minimum amount of work and then get out of the way. Having a context (or event loop, message loop or whatever you want to call it) as I described in my last email lets us do the minimum amount of work. I posted our implementation of such a context earlier and Dino posted an example/recipe for using the concept with an existing event loop (Tcl). So while I said we don't _need_ an event loop, that relies on the asynchronous operations being on a separate thread or otherwise not requiring the current thread to pay any attention to them, AND assumes that the continuations are agile and can be run on any thread (or in any process, or whatever granularity you are working at). I believe some way of getting code running back where it started from is essential, and this is most easily done with a loop.

Calvin Spealman wrote:

If we don't kick whatever handles the callback and result immediately, we are essentially re-introducing pre-emptive scheduling. If TaskA is waiting on the result of TaskB, and when TaskB finishes we say "OK, but we need to go let TaskC do something before TaskA is given that result" then we leave room for C to break things, modify state, and generally act in a less-than-determinable way.

I don't see how the risk of this is any higher than the risk that some other task D gets run while task A is waiting and messes something up. Ultimately you have to trust your tasks to behave themselves. -- Greg

What if the event poll was just inside a task, not requiring any loop in the scheduler, or even knowledge by the scheduler, in any way?

I agree, every task can handle all the asynchrony within it and just expose a single 'completed' notification (a Future or similar) to its caller. This is the portable solution - it is going to be less than optimal in some cases, but is much more composable and extensible. As a Python developer, I like the model of "I call this function normally and it gives me a Future to let me know when it's done but I don't really know how it's doing it." (Incidentally, I like it as a C# and C++ developer too.)

On Wed, Oct 17, 2012 at 6:27 AM, Greg Ewing wrote:

Nick Coghlan wrote:

...

# Note that this is an *ordinary iterator*, not a tasklet def as_completed(futures): # We ensure all the operations have started, and get ourselves a set to work with remaining = set(futures) while remaining: # The trick here is that we *don't yield the original futures directly* # Instead, we yield yield _wait_first(remaining)

I've just figured out how your as_completed() thing works, and realised that it's *not* a general solution to the suspendable-iterator problem. You're making use of the fact that you know *how many* values there will be ahead of time, even if you don't know what they are yet.

In general this won't be the case. I don't think there is any trick that will allow a for-loop to be used in the general case, because in order for an iterator to be suspendable, the call to next() would need to be made using yield-from, and it's hidden inside the for-loop implementation.

Yeah, that's what lets me get away with not passing the sent results back down into the iterator (it can figure out from the original arguments when it needs to stop). It gets trickier if you want to terminate the iteration based on the result of an asynchronous operation. For example, here's a very simplistic way you could apply the concept of "yield a future to be handled in the loop body" to the operation of continuously reading binary data from a connection until EOF is received: def read(self): """This knows how to start an IO operation such the future will fire on completion""" future = ... return future # Again, notice this is *not* a tasklet, it's an ordinary iterator that produces Future objects def readall(self): """This can be used in two modes - as an iterator or as a coroutine. As a coroutine: data = yield from conn.readall() As an iterator: for wait_for_chunk in conn.readall(): try: chunk = yield wait_for_chunk except EOFError: break Obviously, the coroutine mode is far more convenient, but you *can* override the default accumulator behaviour if you want/need to by waiting on the individual futures explicitly. However, in this case, you lose the automatic loop termination behaviour, so, you may as well implement the underlying loop explicitly: while 1: try: chunk = yield self.read() except EOFError: break """ output = io.BytesIO() while 1: try: data = yield self.read() except EOFError: break if data: # This check makes iterator mode possible output.write(data) return output.getvalue() Impedance matching in a way that allows the exception handling to be factored out as well as the iteration step is a *lot* trickier, since you need to bring context managers into play if termination is signalled by an exception: # This version produces context managers rather than producing futures directly, and thus can't be # used directly as a coroutine def read_chunks(self): finished = False @contextmanager def handle_chunk(): nonlocal finished data = b'' try: data = yield self.read() except EOFError: finished = True return data while not finished: yield handle_chunk() # Usage for handle_chunk in conn.read_chunks(): with handle_chunk as wait_for_chunk: chunk = yield from wait_for_chunk # We end up doing a final "extra" iteration with chunk = b'' # So we'd likely need to guard with an "if chunk:" or "if not chunk: continue" # which again means we're not getting much value out of using the iterator Using an explicit "add_done_callback" doesn't help much, as you still have to deal with the exception being thrown back in to your generator. I know Guido doesn't want people racing off and designing new syntax for asynchronous iteration, but I'm not sure it's going to be possible to avoid it if we want a clean approach to "forking" the results of asynchronous calls between passing them down into a coroutine (to decide whether or not to terminate iteration) and binding them to a local variable (to allow local processing in the loop body). Compare the arcane incantations above to something like (similar to suggestions previously made by Christian Heimes): def read_chunks(self): """Designed for use as an asynchronous iterator""" while 1: try: yield self.read() except EOFError: break # Usage for chunk in yield from conn.read_chunks(): ... The idea here would be that whereas "for chunk in (yield from conn.read_chunks()):" runs the underlying coroutine to completion and then iterates over the return value, the version without the parentheses would effectively "tee" the values being sent back, *first* sending them to the underlying coroutine (to decide whether or not iteration should continue and to get the value to be yielded at the start of the next iteration) and then, if that doesn't raise StopIteration, binding them to the local variable and proceeding to execution of the loop body. All that said, I still like Guido's concept that the core asynchronous API is *really* future objects, just as it already is in the concurrent.futures module. The @task decorator and yielding future objects to that decorator is then just nice syntactic sugar for hooking generators up to the "add_done_callback" API of future objects. It's completely independent of the underlying event loop and/or asynchronous IO interfaces - those interfaces are about setting things up to invoke the set_* methods of the returned future objects correctly, just as they are with the Executor API in concurrent.futures.

I know you probably weren't intending as_completed() to be a solution to the general suspendable-iterator problem.

Right, I just wanted to be sure that *that particular use case* of waiting for a collection of futures and processing them in completion order could be handled in terms of Guido's API *without* needing any extra magic. The "iterate over data chunks until EOFError is raised" is a better example for highlighting the "how do you write an asynchronous iterator?" problem when it comes to generators-as-coroutines. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Calvin Spealman wrote:

A "sane stack trace" only makes sense if we assume that tasks "call" each other in the same kind of call tree that synchronous code flows in, and I don't think that is necessarily the case.

No, but often it *is* the case, and in those cases we would like to get a traceback that correctly reflects the chain of calls.

There are cases when one task might want to end before tasks it as "called" are complete, and if we use yield-from this is *impossible* but it is very useful.

That depends on what you mean by "use yield-from". It's true that yield-from *on its own* can't achieve the effect of spawning concurrent subtasks; other mechanisms will need to be brought to bear at some point. But there's no reason a solution involving those other mechanisms can't be encapsulated in a library function that you invoke using yield-from. I've posted a couple of examples of how a par() function which does that might be written.

yield-from semantics won't allow a called task to continue, if needed, after the calling task itself has completed.

You seem to be imagining that more is being claimed about the abilities of yield-from than is actually being claimed. Yield-from is just a procedure call; the important thing is what the called procedure does. One of the things it can do is invoke a scheduler primitive that spawns an independent task. In my example scheduler, this is spelled scheduler.schedule(task). This is not a yield-from call, it's just an ordinary call. It adds the given generator to the list of ready tasks, so that it will get run when its chance comes around. Meanwhile, the calling task carries on. -- Greg

Wow, sounds very similar to NDB's approach! Please do check out NDB's tasklets and event loop: http://code.google.com/p/appengine-ndb-experiment/source/browse/ndb/tasklets... On Mon, Oct 15, 2012 at 10:24 AM, Dino Viehland wrote:

I'm still catching up to this thread, but we've been investigating Win 8 support for Python and Win 8 has a very asynchronous API design and so we've been interested in much the same space. We've actually come up with an example of the @task decorator (we called it @async) which is built around using yield + the ability to return from generators added in Python 3.3. Our version of this is also based around futures so that an @async API will return a future. The big difference here might be that we always return a future from a call rather than yielding it up the stack. So our API works with just simple yields rather than yield froms. This is what a simple usage of the API looks like:

from concurrent.futures import ThreadPoolExecutor from urllib.request import urlopen

executor = ThreadPoolExecutor(max_workers=5)

def load_url(url): return urlopen(_url).read()

@async def get_image_async(url): buffer = yield executor.submit(load_url, url) return Image(buffer)

def main(image_uri): img_future = get_image_async(image_uri) # perform other tasks while the image is downloading img = img_future.result()

main("http://www.python.org/images/python-logo.gif")

This example us just using the existing thread pool to run the actual I/O but this will work with anything that will return a future. So inside of an async method anything which is yielded should be a future. The decorator will then attach a callback which will send the result of the future back into the generator, so the "buffer = " line gets the result of the future. Finally the function completes and the future returned from calling get_image_async will have its value set to Image when the StopIteration exception is raised with the return value.

Because we're interested in the GUI side of things here we've also wired this up into Tk so that we can experiment with an existing GUI framework, and I've included the source for the context there. Our thinking here is that different contexts can be created depending upon the framework which you're running in and that the context makes sure the code is running in the right spot, in this case getting back to the GUI thread after an async operation has been completed.

The big outstanding item we're still working through is I/O, but we think the contexts help here too. We're still not quite sure how polling I/O will work, but with the contexts if there's a single thread polling for I/O then the context will get us off the I/O thread and let the polling continue. We are currently thinking that there will need to be a polling thread which handles all of the I/Os, and there could potentially be more than one of these if different libraries aren't cooperating on sharing a single thread.

Here's the code plus the demo Tk app (you'll need your own Holmes.txt file for the sample app to run):

Contexts.py: http://pastebin.com/ndS53Cd8 Tk context: http://pastebin.com/FuZwc1Ur Tk app: http://pastebin.com/Fm5wMXpN Hardwork.py: http://pastebin.com/nMMytdTG

-----Original Message----- From: Python-ideas [mailto:python-ideas-bounces+dinov=microsoft.com@python.org] On Behalf Of Calvin Spealman Sent: Monday, October 15, 2012 7:16 AM To: Nick Coghlan Cc: python-ideas@python.org Subject: Re: [Python-ideas] The async API of the future: yield-from

On Mon, Oct 15, 2012 at 9:48 AM, Nick Coghlan wrote:

...

On Mon, Oct 15, 2012 at 10:31 PM, Calvin Spealman wrote:

...

Currently, "with yield expr:" is not valid syntax, surprisingly.

It's not that surprising, it's the general requirement that yield expressions must be enclosed in parentheses except when used standalone or in a simple assignment statement.

"with (yield expr):" is valid syntax though, so I'm reluctant to endorse doing anything substantially different if the parentheses are omitted.

Silly oversight on my part, and I agree that the parens shouldn't make the difference in meaning.

...

I think the combination of "yield from" to delegate control (including exception handling) completely to a subgenerator and "context manager + for loop + explicit yield" when an operation needs to yield multiple times and the exception handling behaviour should be left to the caller (as in the "as_completed" case) should cover the necessary behaviours.

I'm still -1 on delegating control to subgenerators with yield-from, versus having the scheduler just deal with them directly. I think it is far less flexible.

I would still like to see a less confusing "with yield expr:" by simply allowing it without parens, but no special meaning. I think it would be really useful in coroutines.

with yield collect() as tasks: yield task1() yield task2() results = yield tasks

...

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

-- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

-- --Guido van Rossum (python.org/~guido)

I gave something like this a go a while ago: https://bitbucket.org/anacrolix/green380 "Coroutines" yield events or futures as Nick put them from the top, and the scheduler at the bottom manages events and scheduling. There are a few things I took away from this attempt: 1) Explicit yield a la PEP380 requires syntactical changes *everywhere*. 2) Python's dynamic typing means that neglecting to "yield from" gives you broken code, and Python won't help you here. Add to this that you now have a 380, and "normal synchronous" form of most interfaces and the caller must know what kind is used at all times. 3) Concurrency is nice, but it requires language-level support, and proper parallelism to really shine. The "C way" of doing things is already so heavily ingrained in Python, an entirely new standard library and interpreter that breaks C compatibility is really the only way to proceed, and this certainly isn't worth it just to write code with "yield from" littered on every line.

On Mon, 15 Oct 2012 19:19:29 -0700 Guido van Rossum wrote:

Here you're basically arguing for greenlets/gevent -- you're saying you just don't want to put up with the yields everywhere. But the popularity of Twisted and Tornado show that at least some people are willing to make even bigger sacrifices in order to be able to do async I/O efficiently -- i.e., to solve the C10K problem that Christian Tismer referred to (http://www.kegel.com/c10k.html, http://en.wikipedia.org/wiki/C10k_problem).

To be honest, one of the selling points of Twisted is not that it solves the C10k problem, it's that it's a comprehensive network programming toolkit. I'd bet many users of Twisted don't care that much about the single-thread event-loop approach, and don't have C10k-like problems. Regards Antoine. -- Software development and contracting: http://pro.pitrou.net

That's basically exactly the same as ours (I worked on it with Dino). We assume that yielded objects are futures and wire up the callback. I think the difference is that all the yielded futures are hidden within the decorator, which returns one main future to the caller. This may be slightly inefficient, but it also makes it far easier for end-users. An event loop (in our terminology, a 'context') is only necessary if you need to ensure that callbacks (in this case, the next step in the generator) is run in a certain context (such as a UI thread). Without one, calling an @async method simply gives you back a future that you can wait on. The most important part of PEP 380 for this approach is not yield from, but allowing return <expr> inside a generator. It makes the methods that much more natural. Probably the most important part is that we assume whatever context is available (through contexts.get_current()) has a post() method for scheduling a callback. Basically, we approached this as less of a "how do I run this asynchronously" problem and more of a "how do I run something after this finishes" problem. We also have some ideas about associating properties with futures in a way that lets the caller decide how to run continuations, so you can opt-out of coming back to the calling thread or provide a cancellation token of some sort. These aren't written up yet (obviously), but we've certainly considered it. Cheers, Steve ________________________________________ From: Python-ideas [python-ideas-bounces+steve.dower=microsoft.com@python.org] on behalf of Guido van Rossum [guido@python.org] Sent: Monday, October 15, 2012 6:17 PM To: Dino Viehland Cc: python-ideas@python.org Subject: Re: [Python-ideas] The async API of the future: yield-from On Mon, Oct 15, 2012 at 2:45 PM, Dino Viehland wrote:

They look remarkably similar. The biggest difference I see is that NDB appears to be using an event loop to keep the futures running while we're using add_done_callback (on the yielded futures) to continue stepping the generator function along. So there's not necessary an event loop in our case, and in fact the default context always just executes things synchronously. But frameworks can replace the default context so that work is posted into an event loop of some form.

But do your Futures use threads? NDB doesn't. NDB's event loop doesn't know about Futures; however the @ndb.tasklet decorator does, and the Futures know about the event loop. When you wait for a Future, a callback is added to the Future that will resume the generator when it is done, and in order to run them, the Future passes its callbacks to the event loop to be run. --Guido

-----Original Message----- From: gvanrossum@gmail.com [mailto:gvanrossum@gmail.com] On Behalf Of Guido van Rossum Sent: Monday, October 15, 2012 12:34 PM To: Dino Viehland Cc: ironfroggy@gmail.com; Nick Coghlan; python-ideas@python.org Subject: Re: [Python-ideas] The async API of the future: yield-from

Wow, sounds very similar to NDB's approach! Please do check out NDB's tasklets and event loop: http://code.google.com/p/appengine-ndb-experiment/source/browse/ndb/tasklets...

On Mon, Oct 15, 2012 at 10:24 AM, Dino Viehland wrote:

...

I'm still catching up to this thread, but we've been investigating Win 8 support for Python and Win 8 has a very asynchronous API design and so we've been interested in much the same space. We've actually come up with an example of the @task decorator (we called it @async) which is built around using yield + the ability to return from generators added in Python 3.3. Our version of this is also based around futures so that an @async API will return a future. The big difference here might be that we always return a future from a call rather than yielding it up the stack. So our API works with just simple yields rather than yield froms. This is what a simple usage of the API looks like:

from concurrent.futures import ThreadPoolExecutor from urllib.request import urlopen

executor = ThreadPoolExecutor(max_workers=5)

def load_url(url): return urlopen(_url).read()

@async def get_image_async(url): buffer = yield executor.submit(load_url, url) return Image(buffer)

def main(image_uri): img_future = get_image_async(image_uri) # perform other tasks while the image is downloading img = img_future.result()

main("http://www.python.org/images/python-logo.gif")

This example us just using the existing thread pool to run the actual I/O but this will work with anything that will return a future. So inside of an async method anything which is yielded should be a future. The decorator will then attach a callback which will send the result of the future back into the generator, so the "buffer = " line gets the result of the future. Finally the function completes and the future returned from calling get_image_async will have its value set to Image when the StopIteration exception is raised with the return value.

Because we're interested in the GUI side of things here we've also wired this up into Tk so that we can experiment with an existing GUI framework, and I've included the source for the context there. Our thinking here is that different contexts can be created depending upon the framework which you're running in and that the context makes sure the code is running in the right spot, in this case getting back to the GUI thread after an async operation has been completed.

The big outstanding item we're still working through is I/O, but we think the contexts help here too. We're still not quite sure how polling I/O will work, but with the contexts if there's a single thread polling for I/O then the context will get us off the I/O thread and let the polling continue. We are currently thinking that there will need to be a polling thread which handles all of the I/Os, and there could potentially be more than one of these if different libraries aren't cooperating on sharing a single thread.

Here's the code plus the demo Tk app (you'll need your own Holmes.txt file for the sample app to run):

Contexts.py: http://pastebin.com/ndS53Cd8 Tk context: http://pastebin.com/FuZwc1Ur Tk app: http://pastebin.com/Fm5wMXpN Hardwork.py: http://pastebin.com/nMMytdTG

-----Original Message----- From: Python-ideas [mailto:python-ideas-bounces+dinov=microsoft.com@python.org] On Behalf Of Calvin Spealman Sent: Monday, October 15, 2012 7:16 AM To: Nick Coghlan Cc: python-ideas@python.org Subject: Re: [Python-ideas] The async API of the future: yield-from

On Mon, Oct 15, 2012 at 9:48 AM, Nick Coghlan wrote:

...

On Mon, Oct 15, 2012 at 10:31 PM, Calvin Spealman wrote:

...

Currently, "with yield expr:" is not valid syntax, surprisingly.

It's not that surprising, it's the general requirement that yield expressions must be enclosed in parentheses except when used standalone or in a simple assignment statement.

"with (yield expr):" is valid syntax though, so I'm reluctant to endorse doing anything substantially different if the parentheses are omitted.

Silly oversight on my part, and I agree that the parens shouldn't make the difference in meaning.

...

I think the combination of "yield from" to delegate control (including exception handling) completely to a subgenerator and "context manager + for loop + explicit yield" when an operation needs to yield + multiple times and the exception handling behaviour should be left to the caller (as in the "as_completed" case) should cover the necessary behaviours.

I'm still -1 on delegating control to subgenerators with yield-from, versus having the scheduler just deal with them directly. I think it is far less flexible.

I would still like to see a less confusing "with yield expr:" by simply allowing it without parens, but no special meaning. I think it would be really useful in coroutines.

with yield collect() as tasks: yield task1() yield task2() results = yield tasks

...

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

-- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

-- --Guido van Rossum (python.org/~guido)

-- --Guido van Rossum (python.org/~guido) _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

Calvin Spealman wrote:

I'm still -1 on delegating control to subgenerators with yield-from, versus having the scheduler just deal with them directly.

Do you mean to *disallow* using yield-from for this, or just not to encourage it? I don't see how you *could* disallow it; there's no way for the scheduler to know whether one of the generators it's handling is delegating using yield-from. I also can't see any reason you would want to discourage it. Given that yield-from exists, it's an obvious thing to do. -- Greg

Calvin Spealman wrote:

I think "yield from" is the obvious way to "call" between tasks, but that there are other cases when we want to spawn a task to begin without blocking our task, and that "yield" should be used here.

I've thought of another problem with this. In my scheduler at least, simply spawning a task doesn't immediately allow that task, or any other, to run. Using "yield" to spell this operation gives the impression that it could be a suspension point, when it's actually not. It also forces anything that uses it to be called with "yield from", all the way up, so if you're relying on the presence of yield-froms to warn you of potential suspension points, you'll get false positives. -- Greg

Nick Coghlan wrote:

To me, "yield from" is just a tool that brings generators back to parity with functions when it comes to breaking up a larger algorithm into smaller pieces. Where you would break a function out into subfunctions and call them normally, with a generator you can break out subgenerators and invoke them with yield from.

That's exactly correct. It's the way I intended "yield from" to be thought of right from the beginning. What I'm arguing is that the *public api* for any suspendable operation should be in the form of something called using yield-from, because you never know when the implementation might want to break it down into sub-operations and use yield-from to call *them*.

Any meaningful use of "yield from" in the coroutine context *has* to ultimate devolve to an operation that: 1. Asks the scheduler to schedule another operation 2. Waits for that operation to complete

I don't think I would put it quite that way. In my view of things at least, the scheduler doesn't schedule "operations" (in the sense of "read some bytes from this socket" etc.) Rather, it schedules the running of tasks. So the breakdown is really: 1. Start an operation (this doesn't involve the scheduler) 2. Ask the scheduler to suspend this task until the operation is finished Also, this breakdown is only necessary at the very lowest level, where you want to do something that isn't provided in the form of a generator. Obviously it's *possible* to treat each level of the call chain as its own subtask, that you spawn independently and then wait for it to finish. That's what people have done in the past with their trampoline schedulers that interpret yielded "call" and "return" instructions. But one of the purposes of yield-from is to relieve the scheduler of the need to handle things at that level of granularity. It can treat a generator together with all the subgenerators it might call as a *single* task, the same way that a greenlet is thought of as a single task, however many levels of function calls it might make.

I *thought* Greg's way combined step 1 and step 2 into a single operation: the objects you yield *not only* say what you want to wait for, but also what you want to do.

I don't actually yield objects at all, but...

However, his example par() implementation killed that idea, since it turned out to need to schedule tasks explicitly rather than their being a "execute this in parallel" option.

I don't see how that's a problem. Seems to me it's just as easy for the user to call a par() function as it is to yield a tuple of tasks. And providing this functionality using a function means that different versions or options can be made available for variations such as different ways of handling exceptions. Using yield, you need to pick one of the variations and bless it as being the one that you invoke using special syntax. If you're complaining that the implementation of par() seems too complex, well, that complexity has to occur *somewhere* -- if it's not in the par() function, then it will turn up inside whatever part of the scheduler handles the case that it's given a tuple of tasks.

So now I'm back to think that Greg and Guido are talking about different levels. *Any* scheduling option will be able to be collapsed into an async task invoked by "yield from" by writing:

def simple_async_task(): return yield start_task()

Yes... or another implementation that works some way other than yielding instructions to the scheduler.

I haven't seen anything to suggest that "yield from"'s role should change from what it is in 3.3: a way to factor out generators into multiple pieces with out breaking send() and throw().

I don't think anyone is suggesting that. I'm certainly not. -- Greg

On Mon, Oct 15, 2012 at 10:35 AM, Guido van Rossum wrote:

But, as Christian Tismer wrote, we need to have some kind of idea of what the primitives are that we want to support. Or should we just have async equivalents for everything in threading.py and queue.py? (What about thread-local? Do we need task-local? Shudder.)

Task locals aren't so scary, since they're already the main reason why generators are so handy - task locals are just the frame locals in the generator :) The main primitive I personally want out of an async API is a task-based equivalent to concurrent.futures.as_completed() [1]. This is what I meant about iteration being a bit of a mess: the way the as_completed() works, the suspend/resume channel of the iterator protocol is being used to pass completed future objects back to the calling iterator. That means that channel *can't* be used to talk between the coroutine and the scheduler, so if you decide you need to free it up for that purpose, you're either forced to wait for *all* the futures to be triggered before any of them can be passed to the caller (allowing you to use yield-from and return a container of completed futures) or else you're forced to switch to callback-style programming (this is where Ruby's blocks are a huge advantage - because their for loops essentially *are* callbacks, you have a lot more flexibility in calling back to different places from a single piece of code). However, I can see one why to make it work which is to require the *invoking* code to continue to manage the communication with the scheduler. Using this concept, there would be an "as_completed_async()" primitive that works something like: for get_next_result in as_completed_task(tasks): task, result = yield get_next_result # Process this result, wait for next one The async equivalent of the concurrent.futures example would then look something like: URLS = ['http://www.foxnews.com/', 'http://www.cnn.com/', 'http://europe.wsj.com/', 'http://www.bbc.co.uk/', 'http://some-made-up-domain.com/'] @task def load_url_async(url, timeout): with (yield urlopen_async(url, timeout=timeout)) as handle: return url, handle.read() tasks = (load_url_async(url, 60) for url in URLS) with concurrent.futures.as_completed_async(tasks) as async_results for get_next_result in async_results: try: url, data = yield get_next_result except Exception as exc: print('{!r} generated an exception: {}'.format(url, exc)) else: print('{!r} page is {:d} bytes'.format(url, len(data))) Key parts of this idea: 1. as_completed_async registers the supplied tasks with the main scheduler so they can all start running in parallel 2. as_completed_async is a context manager that will *cancel* all pending jobs on exit 3. as_completed_async is an iterator that produces a special future that fires whenever *any* of the registered tasks has run to completion 4. because there's a separate yield step for each result retrieval, ordinary exception handling mechanisms can be used rather than needing to introspect a future object Cheers, Nick. [1] http://docs.python.org/dev/library/concurrent.futures.html#threadpoolexecuto... -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Mon, Oct 15, 2012 at 1:14 PM, Greg Ewing wrote:

Nick Coghlan wrote:

...

The main primitive I personally want out of an async API is a task-based equivalent to concurrent.futures.as_completed() [1]. This is what I meant about iteration being a bit of a mess: the way the as_completed() works, the suspend/resume channel of the iterator protocol is being used to pass completed future objects back to the calling iterator. That means that channel *can't* be used to talk between the coroutine and the scheduler,

I had to read this a couple of times before I figured out what you're talking about, but I get it now.

This is an instance of a general problem that was noticed back when I was discussing my cofunctions idea: using generator-based coroutines, it's not possible to have a "suspendable iterator", because that would require "yield" to have two conflicting meanings: "suspend this coroutine" on one hand, and "provide a value to my caller" on the other.

Unfortunately, I suspect that a truly elegant solution to this problem will require yet another language addition -- something like

yield for item in subtask(): ...

which would run a slightly different version of the iterator protocol in which values to be yield are wrapped somehow (I haven't figured out all the details yet).

I think I ran into a similar issue with NDB when defining iteration over an asynchronous query. My solution: q = <some query specification> it = q.iter() # Fire off the query to the datastore while (yield it.has_next_async()): # Block until one result emp = it.next() # Get the result that was buffered on the iterator print emp.name, emp.age # Use it -- --Guido van Rossum (python.org/~guido)

On Mon, Oct 15, 2012 at 1:49 AM, Greg Ewing wrote: [...]

No, it can't be as simple as that, because that will just execute the tasks sequentially. It would have to be something like this:

def par(*tasks): n = len(tasks) results = [None] * n for i, task in enumerate(tasks): def thunk(): nonlocal n results[i] = yield from task n -= 1 scheduler.schedule(thunk) while n > 0: yield return results

Not exactly straightforward, but that's why we write it once and put it in the library. :-)

There are two problems with this code. :) The first is a scoping gotcha: every copy of thunk() will attempt run the same task, and assign it to the same index, due to them sharing the "i" and "task" variables. (The closure captures a reference to the outer variable cells, rather than a copy of their values at the time of thunk's definition.) This could be fixed by defining it as "def thunk(i=i, task=task)", to capture copies. The second problem is more serious: the final while loop busy-waits, which will consume all spare CPU time waiting for the underlying tasks to complete. For this to be practical, it must suspend and resume itself more efficiently. Here's my own attempt. I'll assume the following primitive scheduler instructions (see my "generator task protocol" mail for context), but it should be readily adaptable to other primitives: 1. yield tasklib.spawn(task()) instructs the scheduler to spawn a new, independent task. 2. yield tasklib.suspend() suspends the current task. 3. yield tasklib.get_resume() obtains a callable / primitive that can be used to resume the current task later. I'll also expand it to keep track of success and failure by returning a list of (flag, result) tuples, in the style of DeferredList[1]. Code: def par(*tasks): resume = yield tasklib.get_resume() # Prepare to hold the results, and keep track of progress. results = [None] * len(tasks) finished = 0 # Gather the i'th task's result def gather(i, task): nonlocal finished try: r = yield from task except Exception as e: results[i] = (False, e) else: results[i] = (True, r) finished += 1 # If we're the last to complete, resume par() if finished == len(tasks): yield resume() # Spawn subtasks, and wait for completion. for (i, task) in tasks: yield tasklib.spawn(gather(i, task)) yield tasklib.suspend() return results Hopefully, this is easy enough to read: it should be obvious to see how to modify gather() to add support for resuming immediately on the first result or first error. [1] http://twistedmatrix.com/documents/12.1.0/api/twisted.internet.defer.Deferre... -- Piet Delport

I've had some thoughts on why I'm uncomfortable about this kind of pattern: data = yield sock.async_read(1024) The idea here is that sock.async_read() returns a Future or similar object that performs the I/O and waits for the result. However, reading the data isn't necessarily the point at which the suspension actually occurs. If you're using a select-style event loop, the async read operation breaks down into 1. Wait for data to arrive on the socket 2. Read the data So the implementation of sock.async_read() is going to have to create another Future to handle waiting for the socket to become ready. But then the outer Future is an unnecessary complication, because you could get the same effect by defining def async_read(self, length): yield future_to_wait_for_fd(self.fd) return os.read(self.fd, length) and calling it using data = yield from sock.async_read(1024) If Futures are to appear anywhere, they should only be at the very bottom layer, at the transition between generator and non-generator code. And the place where that transition occurs depend on how the lower levels are implemented. If you're using IOCP instead of select, for example, you need to do things the other way around: 1. Start the read operation 2. Wait for it to complete So I feel that all public APIs should be functions called using yield-from, leaving it up to the implementation to decide if and where Futures become involved. -- Greg

On Sun, Oct 14, 2012 at 12:12 AM, Greg Ewing wrote:

I've had some thoughts on why I'm uncomfortable about this kind of pattern:

data = yield sock.async_read(1024)

The idea here is that sock.async_read() returns a Future or similar object that performs the I/O and waits for the result.

However, reading the data isn't necessarily the point at which the suspension actually occurs. If you're using a select-style event loop, the async read operation breaks down into

1. Wait for data to arrive on the socket 2. Read the data

So the implementation of sock.async_read() is going to have to create another Future to handle waiting for the socket to become ready. But then the outer Future is an unnecessary complication, because you could get the same effect by defining

def async_read(self, length): yield future_to_wait_for_fd(self.fd) return os.read(self.fd, length)

and calling it using

data = yield from sock.async_read(1024)

If Futures are to appear anywhere, they should only be at the very bottom layer, at the transition between generator and non-generator code. And the place where that transition occurs depend on how the lower levels are implemented. If you're using IOCP instead of select, for example, you need to do things the other way around:

1. Start the read operation 2. Wait for it to complete

So I feel that all public APIs should be functions called using yield-from, leaving it up to the implementation to decide if and where Futures become involved.

A logical and consistent conclusion. I actually agree: in NDB, where all I have is "yield <future>" I have a similar guideline: all public async APIs return a Future and must be waited on using yield, and only at the lowest level are other types primitives involved (bare App Engine RPCs, callbacks). -- --Guido van Rossum (python.org/~guido)

On Thu, Oct 18, 2012 at 11:46 PM, Jim Jewett wrote:

Is the goal really to provide "The async API of the future", or just to provide "a stdlib module which provides one adequate way to do async"?

I think the yield and yield from solutions all need too much magical scaffolding to be The One True Way, but I don't mind such conventions as much when they're part of a particular example class, such as concurrent.schedulers.YieldScheduler.

To stretch an analogy, generators and context managers are different concepts. Allowing certain generators to be used as context managers (by using the "with" keyword) is fine. But I wouldn't want to give up all the other uses of generators.

If yield starts implying other magical properties that are only useful when communicating with a scheduler, rather than a regular caller ... I'm afraid that muddies the concept up too much for my taste.

I think it is important that this is more than convention. I think that we need our old friend TOOOWTDI (There's Only One Obvious Way To Do It) here more than ever. This stuff is complicated, and following that interoperability of what eventually is written on top of it is going to be complicated. Our focus should be not on providing simple things like "async file read" but crafting an environment where people can continue to write wonderfully expressive and useful libraries that others can combine to their own needs. If we don't provide the layer upon which this disparate pieces cooperate, I fear much of the effort is all for too little gain to be worth the effort.

More specific concerns below:

On 10/12/12, Guido van Rossum wrote:

...

But the only use for send() on a generator is when using it as a coroutine for a concurrent tasks system -- send() really makes no sense for generators used as iterators. And you're claiming, it seems, that you prefer yield-from for concurrent tasks.

But the data doesn't have to be scheduling information; it can be new data, a seed for an algorithm, a command to switch or reset the state ... locking it to the scheduler is part of what worries me.

When a coroutine yields, it yields *to the scheduler* so for whom else should these values be?

...

On Thu, Oct 11, 2012 at 6:32 PM, Greg Ewing

...

...

Keep in mind that a value yielded by a generator being used as part of a coroutine is *not* seen by code calling it with yield-from.

That is part of what bugs me about the yield-from examples.

Until this discussion, I had thought of yield-from as factoring out some code that was still conceptually embedded within the parent generator. This (perhaps correctly) makes it seem more like a temporary replacement, as if the parent were no longer there at all.

But with the yield channel reserved for scheduling overhead, the "generator" can't really generate anything, except through side effects...

Don't forget that yield-from is an expression, not a statement. The value eventually returned from the generator is the result of the yield-from, so the generator still produces a final value. The fact that these are generators is for their ability to suspend, not to iterate.

...

... I feel that "value = yield <something that returns a Future>" is quite a good paradigm,

To me, it seems fine for a particular concrete scheduler, but too strong an assumption for an abstract API.

I can mostly* understand:

YieldScheduler assumes any yielded data is another Task; it will schedule that task, and cause the original (yielding) Task to wait until the new task is completed.

But I wonder what I'm missing with:

Generators should only yield (expressions that create) Futures; the scheduler will automatically unwrap the future and send (or throw) the result back into the parent (or other ancestor) Generator, which will then be resumed.

* "mostly", because if my task is willing to wait for the subtask to complete, then why not just use a blocking call in the first place? Is it just because switching to another task is lighter weight than letting a thread block?

By blocking call do you mean "x = foo()" or "x = yield from foo()"? Blocking call usually means the former, so if you mean that, then you neglect to think of all the other tasks running which are not willing to wait.

What happens if a generator does yield something other than a Future? Will the generator be rescheduled in an already-runnable (as opposed to waiting) state? Will it never be resumed? Will that object be auto-wrapped in a Future for the benefit of whichever other co-routine originally made the request?

I think if the scheduler doesn't know what to do with something, it should be an error. That makes it easier to change things in the future.

Are generators assumed to run to exhaustion, or is some sort of driver needed to keep pumping them?

...

... It would be horrible to require C to create a fake generator.

Would it have to wrap results in a fake Future, so that the scheduler could properly unwrap?

...

...Well, I'm talking about a decorator that you *always* apply, and which does nothing (or very little) when wrapping a generator, but adds generator behavior when wrapping a non-generator function.

Why is an always-applied decorator any less intrusive than a mandatory (mixin) base class?

...

(1) Calling an async operation and waiting for its result, using yield

...

Futures: result = yield some_async_op(args)

I was repeatedly confused over whether "result" would be a Future that still needed resolution, and the example code wasn't always consistent. As I understand it now, the scheduler (not just the particular implementation, but the API) has to automatically treat any yielded data as a future, resolve that future to its result, and then send (or throw) that result (as opposed to the future) back into either the parent task or the least distant ancestor task not to be using "yield from".

...

Yield-from: result = yield from some_async_op(args)

So the generator containing this code suspends itself entirely until some_async_op is exhausted, at which point result will be the StopIteration? (Or None?) Non-Exception results get passed straight to the least-distant ancestor task not using "yield from", but Exceptions propagate through one generation at a time.

The result is not an exception, but the return of some_async_op(args)

...

(2) Setting the result of an async operation

...

Futures: f.set_result(value) # From any callback

PEP 3148 considers set_result private to the executor. Can that always be done from arbitrary callbacks? Can it be done more than once?

I think for the normal case, a task should just return its value, and the Future or the Scheduler should be responsible for calling set_result.

I agree

...

Yield-from: return value # From the outermost generator

Why only the outermost? I'm guessing it is because everything else is suspended, and even if a mid-level generator is explicitly re-added to the task queue, it can't actually continue because of re-entrancy.

...

(3) Handling an exception

Futures: try: result = yield some_async_op(args) except MyException: <handle exception>

So the scheduler does have to unpack the future, and throw rather than send.

...

(4) Raising an exception as the outcome of an async operation

...

Futures: f.set_exception(<Exception instance>)

Again, shouldn't the task itself just raise, and let the future (or the scheduler) call that?

...

Yield-from: raise <Exception instance or class> # From any of the generators

So it doesn't need to be wrapped in a Future, until it needs to cross back over a "schedule this asynchronously" gulf?

...

(5) Having one async operation invoke another async operation

...

Futures: @task def outer(args): res = yield inner(args) return res

...

Yield-from: def outer(args): res = yield from inner(args) return res

Will it ever get to continue processing (under either model) before inner exhausts itself and stops yielding?

...

Note: I'm including this because in the Futures case, each level of yield requires the creation of a separate Future.

Only because of the auto-unboxing. And if the generator suspends itself to wait for the future, then the future will be resolved before control returns to the generator's own parents, so those per-layer Futures won't really add anything.

...

(6) Spawning off multiple async subtasks

Futures: f1 = subtask1(args1) # Note: no yield!!! f2 = subtask2(args2) res1, res2 = yield f1, f2

ah. That makes a bit more sense, though the tuple of futures does complicate the automagic unboxing. (Which containers, to which levels, have to be resolved?)

...

Yield-from: ??????????

*** Greg, can you come up with a good idiom to spell concurrency at this level? Your example only has concurrency in the philosophers example, but it appears to interact directly with the scheduler, and the philosophers don't return values. ***

Why wouldn't this be the same as you already wrote without yield-from? Two subtasks were submitted but not waited for. I suppose you could yield from a generator that submits new subtasks every time it generates something, but that would be solving a more complicated problem. (So it wouldn't be a consequence of the "yield from".)

...

(7) Checking whether an operation is already complete

...

Futures: if f.done(): ...

If f was yielded, it is done, or this code wouldn't be running again to check.

...

Yield-from: ?????????????

And again, if the futures were yielded (even through a yield from) then they're already unboxed; otherwise, you can still check f.done

...

(8) Getting the result of an operation multiple times

Futures:

f = async_op(args) # squirrel away a reference to f somewhere else r = yield f # ... later, elsewhere r = f.result()

Why do you have to squirrel away the reference? Are you assuming that the async scheduler will mess with the locals so that f is no longer valid?

...

Yield-from: ???????????????

This, you cannot reasonably do; the nature of yield-from means that the unresolved futures were never visible within this generator; they were resolved by the scheduler and the results handed straight to the generator's ancestor.

...

(9) Canceling an operation

Futures: f.cancel()

Yield-from: ???????????????

Note: I haven't needed canceling yet, and I believe Devin said that Twisted just got rid of it. However some of the JS Deferred implementations seem to support it.

I think that once you've called "yield from", the generator making that call is suspended until the child generator completes. But a different thread of control could cancel the active (most-descended) generator.

...

(10) Registering additional callbacks

Futures: f.add_done_callback(callback)

Yield-from: ???????

Note: this is used in NDB to trigger "hooks" that should run e.g. when a database write completes. The user's code just writes yield ent.put_async(); the trigger is automatically called by the Future's machinery. This also uses (8).

I think you would have to do add the callbacks within the subgenerator that is spawning f.

That, or un-inline the yield from, and lose the automated send-throw forwarding.

-jJ _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

-- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy

Jim, relax. We're not changing the meaning of yield or generator. We're just making it *possible* to use yield(-from) and generators as coroutines; that's actually a long path that started with PEP 342. No freedom is taken away by PEP 380; it just adds the possibility to do it without managing an explicit stack of coroutine calls in the scheduler. If we believed that there was no advantage to spelling a blocking call as "yield from foo()", we would just spell it as "foo()" and somehow make it work. But (and even Christian Tismer agrees) there is a problem with the shorter spelling -- you lose track of which calls may cause a task-switch. Using yield-from (or yield, for that matter) for this purpose ensures that all callers in the call chain have to explicitly mark the suspension points, and this serves as a useful reminder that after resumption, the world may look differently, because other tasks may have run in the mean time. -- --Guido van Rossum (python.org/~guido)

Calvin Spealman wrote:

I think it is important that this is more than convention. I think that we need our old friend TOOOWTDI (There's Only One Obvious Way To Do It) here more than ever.

This is part of the reason that I don't like the idea of controlling the scheduler by yielding instructions to it. There are a great many ways that such a "scheduler instruction set" could be designed, none of them any more obvious than the others. So rather than single out an arbitrarily chosen set of operations to be regarded as primitives that the scheduler knows about directly, I would rather have *no* such primitives in the public API. -- Greg

Guido van Rossum wrote:

In your current style you write things like this:

block(self.queue) yield

I don't see how this decouples the call site of the primitive from the scheduler any more than if you were to write e.g. this:

yield block(self.queue)

If I wrote a library intended for serious use, the end user probably wouldn't write either of those. Instead he would write something like yield from block(self.queue) and it would be an implementation detail of the library where abouts the 'yield' happened and whether it needed to send a value or not. When I say I don't like scheduler instructions, all I really mean is that they shouldn't be part of the public API. A scheduler can use them internally if it wants, I don't care. -- Greg