The async API of the future: yield-from

[This is the second spin-off thread from "asyncore: included batteries don't fit"] On Thu, Oct 11, 2012 at 6:32 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Guido van Rossum wrote:
It does bother me somehow that you're not using .send() and yield arguments at all. I notice that you have a lot of three-line code blocks like this:
block_for_reading(sock) yield data = sock.recv(1024)
I wouldn't say I have a "lot". In the spamserver, there are really only three -- one for accepting a connection, one for reading from a socket, and one for writing to a socket. These are primitive operations that would be provided by an async socket library.
Hm. In such a small sample program, three near-identical blocks is a lot!
Generally, all the yields would be hidden inside primitives like this. Normally, user code would never need to use 'yield', only 'yield from'.
This probably didn't come through as clearly as it might have in my tutorial. Part of the reason is that at the time I wrote it, I was having to manually expand yield-froms into for-loops, so I was reluctant to use any more of them than I needed to. Also, yield-from was a new and unfamiliar concept, and I didn't want to scare people by overusing it. These considerations led me to push some of the yields slightly further up the layer stack than they could be.
But the fact remains that you can't completely hide these yields -- the best you can do is replace them with a single yield-from.
The general form seems to be:
arrange for a callback when some operation can be done without blocking yield do the operation
This seems to be begging to be collapsed into a single line, e.g.
data = yield sock.recv_async(1024)
I'm not sure how you're imagining that would work, but whatever it is, it's wrong -- that just doesn't make sense.
That's a strong statement! It makes a lot of sense in a world using Futures and a Future-aware trampoline/scheduler, instead of yield-from and bare generators. I can see however that you don't like it in the yield-from world you're envisioning, and how it would be confusing there. I'll get back to this in a bit.
What *would* make sense is
data = yield from sock.recv_async(1024)
with sock.recv_async() being a primitive that encapsulates the block/yield/process triplet.
Right, that's how you would spell it.
(I would also prefer to see the socket wrapped in an object that makes it hard to accidentally block.)
It would be straightforward to make the primitives be methods of a socket wrapper object. I only used functions in the tutorial in the interests of keeping the amount of machinery to a bare minimum.
Understood.
But surely there's still a place for send() and other PEP 342 features?
In the wider world of generator usage, yes. If you have a generator that it makes sense to send() things into, for example, and you want to factor part of it out into another function, the fact that yield-from passes through sent values is useful.
But the only use for send() on a generator is when using it as a coroutine for a concurrent tasks system -- send() really makes no sense for generators used as iterators. And you're claiming, it seems, that you prefer yield-from for concurrent tasks.
But we're talking about a very specialised use of generators here, and so far I haven't thought of a use for sent or yielded values in this context that can't be done in a more straightforward way by other means.
Keep in mind that a value yielded by a generator being used as part of a coroutine is *not* seen by code calling it with yield-from. Rather, it comes out in the inner loop of the scheduler, from the next() call being used to resume the coroutine. Likewise, any send() call would have to be made by the scheduler, not the yield-from caller.
I'm very much aware of that. There is a *huge* difference between yield-from and yield. However, now that I've implemented a substantial library (NDB, which has thousands of users in the App Engine world, if not hundreds of thousands), I feel that "value = yield <something that returns a Future>" is quite a good paradigm, and the only part of PEP 380 I'm really looking forward to embracing (once App Engine supports Python 3.3) is the option to return a value from a generator -- which my users currently have to spell as "raise ndb.Return(<value>)".
So, the send/yield channel is exclusively for communication with the *scheduler* and nothing else. Under the old way of doing generator-based coroutines, this channel was used to simulate a call stack by yielding 'call' and 'return' instructions that the scheduler interpreted. But all that is now taken care of by the yield-from mechanism, and there is nothing left for the send/yield channel to do.
I understand that's the state of the world that you're looking forward to. However I'm slightly worried that in practice there are some issues to be resolved. One is what to do with operations directly implemented in C. It would be horrible to require C to create a fake generator. It would be mildly nasty to have to wrap these all in Python code just so you can use them with yield-from. Fortunately an iterator whose final __next__() raises StopIteration(<value>) works in the latest Python 3.3 (it didn't work in some of the betas IIRC).
my users sometimes want to treat something as a coroutine but they don't have any yields in it
def caller(): data = yield from reader()
def reader(): return 'dummy' yield
works, but if you drop the yield it doesn't work. With a decorator I know how to make it work either way.
If you're talking about a decorator that turns a function into a generator, I can't see anything particularly headachish about that. If you mean something else, you'll have to elaborate.
Well, I'm talking about a decorator that you *always* apply, and which does nothing (or very little) when wrapping a generator, but adds generator behavior when wrapping a non-generator function. Anyway, I am trying to come up with a table comparing Futures and your yield-from-using generators. I'm basing this on a subset of the PEP 3148 API, and I'm not presuming threads -- I'm just looking at the functionality around getting and setting callbacks, results, and exceptions. My reference is actually based on NDB, but the API there differs from PEP 3148 in uninteresting ways, so I'll use the PEP 3148 method names. (1) Calling an async operation and waiting for its result, using yield Futures: result = yield some_async_op(args) Yield-from: result = yield from some_async_op(args) (2) Setting the result of an async operation Futures: f.set_result(value) # From any callback Yield-from: return value # From the outermost generator (3) Handling an exception Futures: try: result = yield some_async_op(args) except MyException: <handle exception> Yield-from: try: result = yield from some_async_op(args) except MyException: <handle exception> Note: with yield-from, the tracebacks for unhandled exceptions are possibly prettier. (4) Raising an exception as the outcome of an async operation Futures: f.set_exception(<Exception instance>) Yield-from: raise <Exception instance or class> # From any of the generators Note: with Futures, the traceback also needs to be stored; in Python 3 it is stored on the Exception instance's __traceback__ attribute. But when letting exceptions bubble through multiple levels of nested calls, you must do something special to ensure the traceback looks right to the end user. (5) Having one async operation invoke another async operation Futures: @task def outer(args): res = yield inner(args) return res Yield-from: def outer(args): res = yield from inner(args) return res Note: I'm including this because in the Futures case, each level of yield requires the creation of a separate Future. In practice this requires decorating all async functions. And also as a lead-in to the next item. (6) Spawning off multiple async subtasks Futures: f1 = subtask1(args1) # Note: no yield!!! f2 = subtask2(args2) res1, res2 = yield f1, f2 Yield-from: ?????????? *** Greg, can you come up with a good idiom to spell concurrency at this level? Your example only has concurrency in the philosophers example, but it appears to interact directly with the scheduler, and the philosophers don't return values. *** (7) Checking whether an operation is already complete Futures: if f.done(): ... Yield-from: ????????????? (8) Getting the result of an operation multiple times Futures: f = async_op(args) # squirrel away a reference to f somewhere else r = yield f # ... later, elsewhere r = f.result() Yield-from: ??????????????? (9) Canceling an operation Futures: f.cancel() Yield-from: ??????????????? Note: I haven't needed canceling yet, and I believe Devin said that Twisted just got rid of it. However some of the JS Deferred implementations seem to support it. (10) Registering additional callbacks Futures: f.add_done_callback(callback) Yield-from: ??????? Note: this is used in NDB to trigger "hooks" that should run e.g. when a database write completes. The user's code just writes yield ent.put_async(); the trigger is automatically called by the Future's machinery. This also uses (8). -- --Guido van Rossum (python.org/~guido)

Guido van Rossum wrote:
But the fact remains that you can't completely hide these yields -- the best you can do is replace them with a single yield-from.
Yes, as things stand, a call to a sub-generator is always going to look different from an ordinary call, all the way up the call chain. I regard that as a wart remaining to be fixed, although opinions seem to differ. I do think it's a bit unfortunate that 'yield from' contains the word 'yield', though, since in this context it's best thought of as a kind of function call rather than a kind of yield.
This seems to be begging to be collapsed into a single line, e.g.
data = yield sock.recv_async(1024)
I'm not sure how you're imagining that would work, but whatever it is, it's wrong -- that just doesn't make sense.
It makes a lot of sense in a world using Futures and a Future-aware trampoline/scheduler, instead of yield-from and bare generators. I can see however that you don't like it in the yield-from world you're envisioning
I don't like it because, to my mind, Futures et al are kludgy workarounds for not having something like yield-from. Now that we do, we shouldn't need them any more. I can see the desirability of being able to interoperate with existing code that uses them, but I'm not convinced that building awareness of them into the guts of the scheduler is the best way to go about it. Why Futures in particular? What if someone wants to use Deferreds instead, or some other similar thing? At some point you need to build adapters. I'd rather see Futures treated on an equal footing with the others, and dealt with by building on the primitive facilities provided by the scheduler.
But the only use for send() on a generator is when using it as a coroutine for a concurrent tasks system... And you're claiming, it seems, that you prefer yield-from for concurrent tasks.
The particular technique of using send() to supply a return value for a simulated sub-generator call is made obsolete by yield-from. I can't rule out the possibility that there may be other uses for send() in a concurrent task system. I just haven't found the need for it in any of the examples I've developed so far.
I feel that "value = yield <something that returns a Future>" is quite a good paradigm,
I feel that it shouldn't be *necessary* to yield any kind of special object in order to suspend a task; just a simple 'yield' should be sufficient. It might make sense to allow this as an *option* for the purpose of interoperating with existing async code. But I would much rather the public API for this was something like value = yield from wait_for_future(a_future) leaving it up to the implementation whether this is achieved by yielding the Future or by some other means. Then we can also have wait_for_deferred(), etc., without giving any one of them special status.
One is what to do with operations directly implemented in C. It would be horrible to require C to create a fake generator. Fortunately an iterator whose final __next__() raises StopIteration(<value>) works in the latest Python 3.3
Well, such an iterator *is* a "fake generator" in all the respects that the scheduler cares about. Especially if the scheduler doesn't rely on send(), so your C object doesn't have to implement a send() method. :-)
Well, I'm talking about a decorator that you *always* apply, and which does nothing (or very little) when wrapping a generator, but adds generator behavior when wrapping a non-generator function.
As long as it's optional, I wouldn't object to the existence of such a decorator, although I would probably choose not to use it most of the time. I would object if it was *required* to make things work properly, because I would worry that this was a symptom of unnecessary complication and inefficiency in the underlying machinery.
(6) Spawning off multiple async subtasks
Futures: f1 = subtask1(args1) # Note: no yield!!! f2 = subtask2(args2) res1, res2 = yield f1, f2
Yield-from: ??????????
*** Greg, can you come up with a good idiom to spell concurrency at this level? Your example only has concurrency in the philosophers example, but it appears to interact directly with the scheduler, and the philosophers don't return values. ***
I don't regard the need to interact directly with the scheduler as a problem. That's because in the world I envisage, there would only be *one* scheduler, for much the same reason that there can really only be one async event handling loop in any given program. It would be part of the standard library and have a well-known API that everyone uses. If you don't want things to be that way, then maybe this is a good use for yielding things to the scheduler. Yielding a generator could mean "spawn this as a concurrent task". You could go further and say that yielding a tuple of generators means to spawn them all concurrently, wait for them all to complete and send back a tuple of the results. The yield-from code would then look pretty much the same as the futures code. However, I'm inclined to think that this is too much functionality to build directly into the scheduler, and that it would be better provided by a class or function that builds on more primitive facilities. So it would look something like Yield-from: task1 = subtask1(args1) task2 = subtask2(args2) res1, res2 = yield from par(task1, task2) where the implementation of par() is left as an exercise for the reader.
(7) Checking whether an operation is already complete
Futures: if f.done(): ...
I'm inclined to think that this is not something the scheduler needs to be directly concerned with. If it's important for one task to know when another task is completed, it's up to those tasks to agree on a way of communicating that information between them. Although... is there a way to non-destructively test whether a generator is exhausted? If so, this could easily be provided as a scheduler primitive.
(8) Getting the result of an operation multiple times
Futures:
f = async_op(args) # squirrel away a reference to f somewhere else r = yield f # ... later, elsewhere r = f.result()
Is this really a big deal? What's wrong with having to store the return value away somewhere if you want to use it multiple times?
(9) Canceling an operation
Futures: f.cancel()
This would be another scheduler primitive. Yield-from: cancel(task) This would remove the task from the ready list or whatever queue it's blocked on, and probably throw an exception into it to give it a chance to clean up.
(10) Registering additional callbacks
Futures: f.add_done_callback(callback)
Another candidate for a higher-level facility, I think. The API might look something like Yield-from: cbt = task_with_callbacks(task) cbt.add_callback(callback) yield from cbt.run() I may have a go at coming up with implementations for some of these things and send them in later posts. -- Greg

On Sat, Oct 13, 2012 at 3:05 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Although... is there a way to non-destructively test whether a generator is exhausted? If so, this could easily be provided as a scheduler primitive.
Yes. Take a look at inspect.getgeneratorstate in 3.2+ (previously, implementations weren't *required* to provide that introspection capability, but now they do in order to support this function in the inspect module). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Fri, Oct 12, 2012 at 10:05 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote: [Long sections snipped, all very clear]
Guido van Rossum wrote:
(6) Spawning off multiple async subtasks
Futures: f1 = subtask1(args1) # Note: no yield!!! f2 = subtask2(args2) res1, res2 = yield f1, f2
Yield-from: ??????????
*** Greg, can you come up with a good idiom to spell concurrency at this level? Your example only has concurrency in the philosophers example, but it appears to interact directly with the scheduler, and the philosophers don't return values. ***
I don't regard the need to interact directly with the scheduler as a problem. That's because in the world I envisage, there would only be *one* scheduler, for much the same reason that there can really only be one async event handling loop in any given program. It would be part of the standard library and have a well-known API that everyone uses.
If you don't want things to be that way, then maybe this is a good use for yielding things to the scheduler. Yielding a generator could mean "spawn this as a concurrent task".
You could go further and say that yielding a tuple of generators means to spawn them all concurrently, wait for them all to complete and send back a tuple of the results. The yield-from code would then look pretty much the same as the futures code.
Sadly it looks that r = yield from (f1(), f2()) ends up interpreting the tuple as the iterator, and you end up with r = (f1(), f2()) (i.e., a tuple of generators) rather than the desired r = ((yield from f1()), (yield from f2()))
However, I'm inclined to think that this is too much functionality to build directly into the scheduler, and that it would be better provided by a class or function that builds on more primitive facilities.
Possibly. In NDB it is actually a very common operation which looks quite elegant. But your solution below is fine (and helps by giving people a specific entry in the documentation they can look up!)
So it would look something like
Yield-from: task1 = subtask1(args1) task2 = subtask2(args2) res1, res2 = yield from par(task1, task2)
where the implementation of par() is left as an exercise for the reader.
So, can par() be as simple as def par(*args): results = [] for task in args: result = yield from task results.append(result) return results ??? Or does it need to interact with the scheduler to ensure fairness? (Not having built one of these, my intuition for how the primitives fit together is still lacking, so excuse me for asking naive questions.) Of course there's the question of what to do when one of the tasks raises an error -- I haven't quite figured that out in NDB either, it runs all the tasks to completion but the caller only sees the first exception. I briefly considered having an "multi-exception" but it felt too weird -- though I'm not married to that decision.
(7) Checking whether an operation is already complete
Futures: if f.done(): ...
I'm inclined to think that this is not something the scheduler needs to be directly concerned with. If it's important for one task to know when another task is completed, it's up to those tasks to agree on a way of communicating that information between them.
Although... is there a way to non-destructively test whether a generator is exhausted? If so, this could easily be provided as a scheduler primitive.
Nick answered this affirmatively.
(8) Getting the result of an operation multiple times
Futures:
f = async_op(args) # squirrel away a reference to f somewhere else r = yield f # ... later, elsewhere r = f.result()
Is this really a big deal? What's wrong with having to store the return value away somewhere if you want to use it multiple times?
I suppose that's okay.
(9) Canceling an operation
Futures: f.cancel()
This would be another scheduler primitive.
Yield-from: cancel(task)
This would remove the task from the ready list or whatever queue it's blocked on, and probably throw an exception into it to give it a chance to clean up.
Ah, of course. (I said I was asking newbie questions. Consider me your first newbie!)
(10) Registering additional callbacks
Futures: f.add_done_callback(callback)
Another candidate for a higher-level facility, I think. The API might look something like
Yield-from: cbt = task_with_callbacks(task) cbt.add_callback(callback) yield from cbt.run()
I may have a go at coming up with implementations for some of these things and send them in later posts.
Or better, add them to the tutorial. (Or an advanced tutorial, "common async patterns". That would actually be a useful collection of use cases for whatever we end up building.) Here's another pattern that I can't quite figure out. It started when Ben Darnell posted a link to Tornado's chat demo (https://github.com/facebook/tornado/blob/master/demos/chat/chatdemo.py). I didn't understand it and asked him offline what it meant. Essentially, it's a barrier pattern where multiple tasks (each representing a different HTTP request, and thus not all starting at the same time) render a partial web page and then block until a new HTTP request comes in that provides the missing info. (For technical reasons they only do this once, and then the browsers re-fetch the URL.) When the missing info is available, it must wake up all blocked task and give then the new info. I wrote a Futures-based version of this -- not the whole thing, but the block-until-more-info-and-wakeup part. Here it is (read 'info' for 'messages'): Each waiter executes this code when it is ready to block: f = Future() # Explicitly create a future! waiters.add(f) messages = yield f <process messages and quit> I'd write a helper for the first two lines: def register(): f = Future() waiters.add(f) return f Then the waiter's code becomes: messages = yield register() <process messages and quit> When new messages become available, the code just sends the same results to all those Futures: def wakeup(messages): for waiter in waiters: waiter.set_result(messages) waiters.clear() (OO sauce left to the reader. :-) If you wonder where the code is that hooks up the waiter.set_result() call with the yield, that's done by the scheduler: when a task yields a Future, it adds a callback to the Future that reschedules the task when the Future's result is set. Edge cases: - Were the waiter to lose interest, it could remove its Future from the list of waiters, but no harm is done leaving it around either. (NDB doesn't have this feature, but if you have a way to remove callbacks, setting the result of a Future that nobody cares about has no ill effect. You could even use a weak set...) - It's possible to broadcast an exception to all waiters by using waiter.set_exception(). -- --Guido van Rossum (python.org/~guido)

On 10/14/2012 10:36 AM, Guido van Rossum wrote:
So, can par() be as simple as
def par(*args): results = [] for task in args: result = yield from task results.append(result) return results
???
Or does it need to interact with the scheduler to ensure fairness? (Not having built one of these, my intuition for how the primitives fit together is still lacking, so excuse me for asking naive questions.)
Of course there's the question of what to do when one of the tasks raises an error -- I haven't quite figured that out in NDB either, it runs all the tasks to completion but the caller only sees the first exception. I briefly considered having an "multi-exception" but it felt too weird -- though I'm not married to that decision.
One answer is to append the exception object to results and let the requesting code sort out what to do. def par(*args): results = [] for task in args: try: result = yield from task results.append(result) except Exception as exc: results.append(exc) return results -- Terry Jan Reedy

On Sun, Oct 14, 2012 at 10:27 AM, Terry Reedy <tjreedy@udel.edu> wrote:
On 10/14/2012 10:36 AM, Guido van Rossum wrote:
So, can par() be as simple as
def par(*args): results = [] for task in args: result = yield from task results.append(result) return results
???
Or does it need to interact with the scheduler to ensure fairness? (Not having built one of these, my intuition for how the primitives fit together is still lacking, so excuse me for asking naive questions.)
Of course there's the question of what to do when one of the tasks raises an error -- I haven't quite figured that out in NDB either, it runs all the tasks to completion but the caller only sees the first exception. I briefly considered having an "multi-exception" but it felt too weird -- though I'm not married to that decision.
One answer is to append the exception object to results and let the requesting code sort out what to do.
def par(*args): results = [] for task in args: try:
result = yield from task results.append(result) except Exception as exc: results.append(exc) return results
But then the caller would have to sort through the results and check for exceptions. I want the caller to be able to use try/except as well. So far the best I've come up with is to recommend that if you care about distinguishing multiple exceptions, use separate yields surrounded by separate try/except blocks. Note that the tasks can still run concurrently, just create all the futures before doing the first yield. -- --Guido van Rossum (python.org/~guido)

On 10/14/2012 1:42 PM, Guido van Rossum wrote:
On Sun, Oct 14, 2012 at 10:27 AM, Terry Reedy <tjreedy@udel.edu> wrote:
On 10/14/2012 10:36 AM, Guido van Rossum wrote:
Of course there's the question of what to do when one of the tasks raises an error -- I haven't quite figured that out in NDB either, it runs all the tasks to completion but the caller only sees the first exception. I briefly considered having an "multi-exception" but it felt too weird -- though I'm not married to that decision.
One answer is to append the exception object to results and let the requesting code sort out what to do.
def par(*args): results = [] for task in args: try:
result = yield from task results.append(result) except Exception as exc: results.append(exc) return results
But then the caller would have to sort through the results and check for exceptions. I want the caller to be able to use try/except as well.
OK. Then ... def par(*args): results = [] exceptions = False for task in args: try: result = yield from task results.append(result) except Exception as exc: results.append(exc) exceptions = True if not exceptions: return results else: exc = MultiXException() exc.results = results raise exc Is this is what you meant by 'multi-exception'? caller: try: results = <whatever> <process results, perhaps by iterating thru them, knowing all represent successed> except MultiXException as exc: errprocess(exc.results)

On Sun, Oct 14, 2012 at 12:38 PM, Terry Reedy <tjreedy@udel.edu> wrote:
On 10/14/2012 1:42 PM, Guido van Rossum wrote:
On Sun, Oct 14, 2012 at 10:27 AM, Terry Reedy <tjreedy@udel.edu> wrote:
On 10/14/2012 10:36 AM, Guido van Rossum wrote:
Of course there's the question of what to do when one of the tasks raises an error -- I haven't quite figured that out in NDB either, it runs all the tasks to completion but the caller only sees the first exception. I briefly considered having an "multi-exception" but it felt too weird -- though I'm not married to that decision.
One answer is to append the exception object to results and let the requesting code sort out what to do.
def par(*args): results = [] for task in args: try:
result = yield from task results.append(result) except Exception as exc: results.append(exc) return results
But then the caller would have to sort through the results and check for exceptions. I want the caller to be able to use try/except as well.
OK. Then ...
def par(*args): results = [] exceptions = False for task in args: try: result = yield from task results.append(result) except Exception as exc: results.append(exc) exceptions = True if not exceptions: return results else: exc = MultiXException() exc.results = results raise exc
Is this is what you meant by 'multi-exception'?
Yes.
caller:
try: results = <whatever> <process results, perhaps by iterating thru them, knowing all represent successed> except MultiXException as exc: errprocess(exc.results)
In NDB I have yet to encounter a situation where I care. -- --Guido van Rossum (python.org/~guido)

On 14.10.12 22:38, Terry Reedy wrote:
On 10/14/2012 1:42 PM, Guido van Rossum wrote:
But then the caller would have to sort through the results and check for exceptions. I want the caller to be able to use try/except as well.
OK. Then ...
def par(*args): results = [] exceptions = False for task in args: try: result = yield from task if exceptions: results.append(StopIteration(result)) else: results.append(result) except Exception as exc: results = [StopIteration(result) for result in results] results.append(exc) exceptions = True if not exceptions: return results else: exc = MultiXException() exc.results = results raise exc

On Sun, Oct 14, 2012 at 7:36 AM, Guido van Rossum <guido@python.org> wrote:
So it would look something like
Yield-from: task1 = subtask1(args1) task2 = subtask2(args2) res1, res2 = yield from par(task1, task2)
where the implementation of par() is left as an exercise for the reader.
So, can par() be as simple as
def par(*args): results = [] for task in args: result = yield from task results.append(result) return results
???
Or does it need to interact with the scheduler to ensure fairness? (Not having built one of these, my intuition for how the primitives fit together is still lacking, so excuse me for asking naive questions.)
It's not just fairness, it needs to interact with the scheduler to get any parallelism at all if the sub-generators have more than one step. Consider: def task1(): print "1A" yield print "1B" yield print "1C" # and so on... def task2(): print "2A" yield print "2B" yield print "2C" def outer(): yield from par(task1(), task2()) Both tasks are started immediately, but can't progress further until they are yielded from to advance the iterator. So with this version of par() you get 1A, 2A, 1B, 1C..., 2B, 2C. To get parallelism I think you have to schedule each sub-generator separately instead of just yielding from them (which negates some of the benefits of yield from like easy error handling). Even if there is a clever version of par() that works more like yield from, you'd need to go back to explicit scheduling if you wanted parallel execution without forcing everything to finish at the same time (which is simple with Futures).
Of course there's the question of what to do when one of the tasks raises an error -- I haven't quite figured that out in NDB either, it runs all the tasks to completion but the caller only sees the first exception. I briefly considered having an "multi-exception" but it felt too weird -- though I'm not married to that decision.
In general for this kind of parallel operation I think it's fine to say that one (unspecified) exception is raised in the outer function and the rest are hidden. With futures, "(r1, r2) = yield (f1, f2)" is just shorthand for "r1 = yield f1; r2 = yield f2", so separating the yields to have separate try/except blocks is no problem. WIth yield from it's not as good because the second operation can't proceed while the outer function is waiting for the first. -Ben

On Sun, Oct 14, 2012 at 3:09 PM, Ben Darnell <ben@bendarnell.com> wrote:
On Sun, Oct 14, 2012 at 7:36 AM, Guido van Rossum <guido@python.org> wrote:
So it would look something like
Yield-from: task1 = subtask1(args1) task2 = subtask2(args2) res1, res2 = yield from par(task1, task2)
where the implementation of par() is left as an exercise for the reader.
So, can par() be as simple as
def par(*args): results = [] for task in args: result = yield from task results.append(result) return results
???
Or does it need to interact with the scheduler to ensure fairness? (Not having built one of these, my intuition for how the primitives fit together is still lacking, so excuse me for asking naive questions.)
It's not just fairness, it needs to interact with the scheduler to get any parallelism at all if the sub-generators have more than one step. Consider:
def task1(): print "1A" yield print "1B" yield print "1C" # and so on...
def task2(): print "2A" yield print "2B" yield print "2C"
def outer(): yield from par(task1(), task2())
Hm, that's a little unrealistic -- in practice you'll rarely see code that yields unless it is also blocking for I/O. I presume that if both tasks immediately block for I/O, the one whose I/O completes first gets the run next; and if it then blocks again, it'll again depend on whose I/O finishes first. (Admittedly this has little to do with fairness now.)
Both tasks are started immediately, but can't progress further until they are yielded from to advance the iterator. So with this version of par() you get 1A, 2A, 1B, 1C..., 2B, 2C.
Really? When you call a generator, it doesn't run until the first yield; it gets suspended before the first bytecode of the body. So if anything, you might get 1A, 1B, 1C, 2A, 2B, 2C. (Which would prove your point just as much of course.) Sadly I don't have a framework lying around where I can test this easily -- I'm pretty sure that the equivalent code in NDB interacts with the scheduler in a way that ensures round-robin scheduling.
To get parallelism I think you have to schedule each sub-generator separately instead of just yielding from them (which negates some of the benefits of yield from like easy error handling).
Honestly I don't mind of the scheduler has to be messy, as long the mess is hidden from the caller.
Even if there is a clever version of par() that works more like yield from, you'd need to go back to explicit scheduling if you wanted parallel execution without forcing everything to finish at the same time (which is simple with Futures).
Why wouldn't all generators that aren't blocked for I/O just run until their next yield, in a round-robin fashion? That's fair enough for me. But as I said, my intuition for how things work in Greg's world is not very good.
Of course there's the question of what to do when one of the tasks raises an error -- I haven't quite figured that out in NDB either, it runs all the tasks to completion but the caller only sees the first exception. I briefly considered having an "multi-exception" but it felt too weird -- though I'm not married to that decision.
In general for this kind of parallel operation I think it's fine to say that one (unspecified) exception is raised in the outer function and the rest are hidden. With futures, "(r1, r2) = yield (f1, f2)" is just shorthand for "r1 = yield f1; r2 = yield f2", so separating the yields to have separate try/except blocks is no problem. With yield from it's not as good because the second operation can't proceed while the outer function is waiting for the first.
Hmmm, I think I see your point. This seems to follow if (as Greg insists) you don't have any decorators on the generators. OTOH I am okay with only getting one of the exceptions. But I think all of the remaining tasks should still be run to completion -- maybe the caller just cared about their side effects. Or maybe this should be an option to par(). -- --Guido van Rossum (python.org/~guido)

On Sun, Oct 14, 2012 at 3:27 PM, Guido van Rossum <guido@python.org> wrote:
On Sun, Oct 14, 2012 at 3:09 PM, Ben Darnell <ben@bendarnell.com> wrote:
On Sun, Oct 14, 2012 at 7:36 AM, Guido van Rossum <guido@python.org> wrote:
So it would look something like
Yield-from: task1 = subtask1(args1) task2 = subtask2(args2) res1, res2 = yield from par(task1, task2)
where the implementation of par() is left as an exercise for the reader.
So, can par() be as simple as
def par(*args): results = [] for task in args: result = yield from task results.append(result) return results
???
Or does it need to interact with the scheduler to ensure fairness? (Not having built one of these, my intuition for how the primitives fit together is still lacking, so excuse me for asking naive questions.)
It's not just fairness, it needs to interact with the scheduler to get any parallelism at all if the sub-generators have more than one step. Consider:
def task1(): print "1A" yield print "1B" yield print "1C" # and so on...
def task2(): print "2A" yield print "2B" yield print "2C"
def outer(): yield from par(task1(), task2())
Hm, that's a little unrealistic -- in practice you'll rarely see code that yields unless it is also blocking for I/O. I presume that if both tasks immediately block for I/O, the one whose I/O completes first gets the run next; and if it then blocks again, it'll again depend on whose I/O finishes first.
(Admittedly this has little to do with fairness now.)
Both tasks are started immediately, but can't progress further until they are yielded from to advance the iterator. So with this version of par() you get 1A, 2A, 1B, 1C..., 2B, 2C.
Really? When you call a generator, it doesn't run until the first yield; it gets suspended before the first bytecode of the body. So if anything, you might get 1A, 1B, 1C, 2A, 2B, 2C. (Which would prove your point just as much of course.)
Ah, OK. I was mistaken about the "first yield" part, but the rest stands. The problem is that as soon as task1 blocks on IO, the entire current task (which includes outer(), par(), and both children) gets unscheduled. no part of task2 gets scheduled until it gets yielded from, because the scheduler can't see it until then.
Sadly I don't have a framework lying around where I can test this easily -- I'm pretty sure that the equivalent code in NDB interacts with the scheduler in a way that ensures round-robin scheduling.
To get parallelism I think you have to schedule each sub-generator separately instead of just yielding from them (which negates some of the benefits of yield from like easy error handling).
Honestly I don't mind of the scheduler has to be messy, as long the mess is hidden from the caller.
Agreed.
Even if there is a clever version of par() that works more like yield from, you'd need to go back to explicit scheduling if you wanted parallel execution without forcing everything to finish at the same time (which is simple with Futures).
Why wouldn't all generators that aren't blocked for I/O just run until their next yield, in a round-robin fashion? That's fair enough for me.
But as I said, my intuition for how things work in Greg's world is not very good.
The good and bad parts of this proposal both stem from the fact that yield from is very similar to just inlining everything together. This gives you the exception handling semantics that you expect from synchronous code, but it means that the scheduler can't distinguish between subtasks; you have to explicitly schedule them as top-level tasks.
Of course there's the question of what to do when one of the tasks raises an error -- I haven't quite figured that out in NDB either, it runs all the tasks to completion but the caller only sees the first exception. I briefly considered having an "multi-exception" but it felt too weird -- though I'm not married to that decision.
In general for this kind of parallel operation I think it's fine to say that one (unspecified) exception is raised in the outer function and the rest are hidden. With futures, "(r1, r2) = yield (f1, f2)" is just shorthand for "r1 = yield f1; r2 = yield f2", so separating the yields to have separate try/except blocks is no problem. With yield from it's not as good because the second operation can't proceed while the outer function is waiting for the first.
Hmmm, I think I see your point. This seems to follow if (as Greg insists) you don't have any decorators on the generators.
OTOH I am okay with only getting one of the exceptions. But I think all of the remaining tasks should still be run to completion -- maybe the caller just cared about their side effects. Or maybe this should be an option to par().
That's probably a good idea. -Ben
-- --Guido van Rossum (python.org/~guido)

On Sun, Oct 14, 2012 at 3:55 PM, Ben Darnell <ben@bendarnell.com> wrote:
Ah, OK. I was mistaken about the "first yield" part, but the rest stands. The problem is that as soon as task1 blocks on IO, the entire current task (which includes outer(), par(), and both children) gets unscheduled. no part of task2 gets scheduled until it gets yielded from, because the scheduler can't see it until then.
Ah, yes. I had forgotten that the whole stack (at least all frames currently blocked in yield-from) is suspended. I really hope that Greg has a working implementation of par(). [...]
The good and bad parts of this proposal both stem from the fact that yield from is very similar to just inlining everything together. This gives you the exception handling semantics that you expect from synchronous code, but it means that the scheduler can't distinguish between subtasks; you have to explicitly schedule them as top-level tasks.
I'm beginning to see that. Thanks for helping me form my intuition about how this stuff works! -- --Guido van Rossum (python.org/~guido)

Ben Darnell wrote: The problem is that as soon as task1 blocks on IO, the entire
current task (which includes outer(), par(), and both children) gets unscheduled. no part of task2 gets scheduled until it gets yielded from, because the scheduler can't see it until then.
The suggested implementation of par() that I posted does explicitly schedule the subtasks. Then it repeatedly yields, giving them a chance to run, until they all complete. -- Greg

Guido van Rossum wrote:
Why wouldn't all generators that aren't blocked for I/O just run until their next yield, in a round-robin fashion? That's fair enough for me.
But as I said, my intuition for how things work in Greg's world is not very good.
That's exactly how my scheduler behaves.
OTOH I am okay with only getting one of the exceptions. But I think all of the remaining tasks should still be run to completion -- maybe the caller just cared about their side effects. Or maybe this should be an option to par().
This is hard to answer without considering real use cases, but my feeling is that if I care enough about the results of the subtasks to wait until they've all completed before continuing, then if anything goes wrong in any of them, I might as well abandon the whole computation. If that's not the case, I'd be happy to wrap each one in a try-except that doesn't propagate the exception to the main task, but just records the information that the subtask failed somewhere, for the main task to check afterwards. Another direction to approach this is to consider that par() ought to be just an optimisation -- the result should be the same as if you'd written sequential code to perform the subtasks one after another. And in that case, an exception in one would prevent any of the following ones from executing, so it's fine if par() behaves like that, too. -- Greg

On Sun, Oct 14, 2012 at 10:58 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Guido van Rossum wrote:
Why wouldn't all generators that aren't blocked for I/O just run until their next yield, in a round-robin fashion? That's fair enough for me.
But as I said, my intuition for how things work in Greg's world is not very good.
That's exactly how my scheduler behaves.
OTOH I am okay with only getting one of the exceptions. But I think all of the remaining tasks should still be run to completion -- maybe the caller just cared about their side effects. Or maybe this should be an option to par().
This is hard to answer without considering real use cases, but my feeling is that if I care enough about the results of the subtasks to wait until they've all completed before continuing, then if anything goes wrong in any of them, I might as well abandon the whole computation.
If that's not the case, I'd be happy to wrap each one in a try-except that doesn't propagate the exception to the main task, but just records the information that the subtask failed somewhere, for the main task to check afterwards.
Another direction to approach this is to consider that par() ought to be just an optimisation -- the result should be the same as if you'd written sequential code to perform the subtasks one after another. And in that case, an exception in one would prevent any of the following ones from executing, so it's fine if par() behaves like that, too.
I'd think of such a par() more as something that saves me typing than as an optimization. Anyway, the key functionality I cannot live without here is to start multiple tasks concurrently. It seems that without par() or some other scheduling primitive, you cannot do that: if I write a = foo_task() # Search google b = bar_task() # Search bing ra = yield from a rb = yield from b # now compare search results the tasks run sequentially. A good par() should run then concurrently. But there needs to be another way to get a task running immediately and concurrently; I believe that would be a = spawn(foo_task()) right? One could then at any later point use ra = yield from a One could also combine these and do e.g. a = spawn(foo_task()) b = spawn(bar_task()) <do more work locally> ra, rb = yield from par(a, b) Have I got the spelling for spawn() right? In many other systems (e.g. threads, greenlets) this kind of operation takes a callable, not the result of calling a function (albeit a generator). If it takes a generator, would it return the same generator or a different one to wait for? -- --Guido van Rossum (python.org/~guido)

On Mon, Oct 15, 2012 at 11:53 AM, Guido van Rossum <guido@python.org> wrote:
On Sun, Oct 14, 2012 at 10:58 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Guido van Rossum wrote:
Why wouldn't all generators that aren't blocked for I/O just run until their next yield, in a round-robin fashion? That's fair enough for me.
But as I said, my intuition for how things work in Greg's world is not very good.
That's exactly how my scheduler behaves.
OTOH I am okay with only getting one of the exceptions. But I think all of the remaining tasks should still be run to completion -- maybe the caller just cared about their side effects. Or maybe this should be an option to par().
This is hard to answer without considering real use cases, but my feeling is that if I care enough about the results of the subtasks to wait until they've all completed before continuing, then if anything goes wrong in any of them, I might as well abandon the whole computation.
If that's not the case, I'd be happy to wrap each one in a try-except that doesn't propagate the exception to the main task, but just records the information that the subtask failed somewhere, for the main task to check afterwards.
Another direction to approach this is to consider that par() ought to be just an optimisation -- the result should be the same as if you'd written sequential code to perform the subtasks one after another. And in that case, an exception in one would prevent any of the following ones from executing, so it's fine if par() behaves like that, too.
I'd think of such a par() more as something that saves me typing than as an optimization. Anyway, the key functionality I cannot live without here is to start multiple tasks concurrently. It seems that without par() or some other scheduling primitive, you cannot do that: if I write
a = foo_task() # Search google b = bar_task() # Search bing ra = yield from a rb = yield from b # now compare search results
the tasks run sequentially. A good par() should run then concurrently. But there needs to be another way to get a task running immediately and concurrently; I believe that would be
a = spawn(foo_task())
right? One could then at any later point use
ra = yield from a
One could also combine these and do e.g.
a = spawn(foo_task()) b = spawn(bar_task()) <do more work locally> ra, rb = yield from par(a, b)
Have I got the spelling for spawn() right? In many other systems (e.g. threads, greenlets) this kind of operation takes a callable, not the result of calling a function (albeit a generator). If it takes a generator, would it return the same generator or a different one to wait for?
I think "start this other async task, but let me continue now" (spawn) is so common and basic an operation it needs to be first class. What if we allow both yield and yield from of a task? If we allow spawn(task()) then we're not getting nice tracebacks anyway, so I think we should allow result1 = yield from task1() # wait for this other task result2 = yield from task2() # wait for this next and future1 = yield task1() # spawn task future2 = yield task2() # spawn other task results = yield future1, future2 I was wrong to say we shouldn't do yield-from task scheduling, I see the benefits now. but I don't think it has to be either or. I think it makes sense to allow both, and that the behavior differences between the two ways to invoke another task would be sensible. Both are primitives we need to support as first-class operation. That is, without some wrapper like spawn().
-- --Guido van Rossum (python.org/~guido) _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
-- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy

Calvin Spealman wrote:
If we allow spawn(task()) then we're not getting nice tracebacks anyway, so I think we should allow
future1 = yield task1() # spawn task future2 = yield task2() # spawn other task
I don't think it's necessary to allow 'yield task' as a method of spawning in order to get nice tracebacks for spawned tasks. In the Task-object-based system I'm thinking about, if an exception reaches the top level of a Task, it gets stored in the Task object until another task wait()s for it, and then it continues to propagate. This makes sense, because the wait() establishes a task-subtask relationship, so the traceback should proceed from the subtask to the waiting task.
Both are primitives we need to support as first-class operation. That is, without some wrapper like spawn().
In my system, spawn() isn't a wrapper -- it *is* the primitive way to create an independent task. And I think it's the only one we need. -- Greg

On Tue, Oct 16, 2012 at 3:45 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Calvin Spealman wrote:
If we allow spawn(task()) then we're not getting nice tracebacks anyway, so I think we should allow
future1 = yield task1() # spawn task future2 = yield task2() # spawn other task
I don't think it's necessary to allow 'yield task' as a method of spawning in order to get nice tracebacks for spawned tasks.
Necessary, no. But I think it feels obvious that you yield things you are waiting on, and so you want to start a task if you yield it. Also, its going to be a common primitive, so I think it should be very easy and clear to write.
In the Task-object-based system I'm thinking about, if an exception reaches the top level of a Task, it gets stored in the Task object until another task wait()s for it, and then it continues to propagate.
This makes sense, because the wait() establishes a task-subtask relationship, so the traceback should proceed from the subtask to the waiting task.
What if two tasks call wait() on the same subtask which raises an error? I think we should let errors propagate through yield-from, primarily. That's what it exists for.
Both are primitives we
need to support as first-class operation. That is, without some wrapper like spawn().
In my system, spawn() isn't a wrapper -- it *is* the primitive way to create an independent task. And I think it's the only one we need.
It has to know what scheduler to talk to, right? We might want to allow multiple schedulers, and tasks shouldn't know who their scheduler is (right?) so that is another advantage of "yield task()"
-- Greg
_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
-- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy

Calvin Spealman wrote:
What if two tasks call wait() on the same subtask which raises an error?
That would be disallowed. A given Task would only be allowed to have its wait() method called once. The reason for this restriction is because of the way tracebacks are attached to exception objects in Python 3, which means that exceptions are effectively single-use now. If it weren't for that, the exception could simply be raised in *both* waiters.
I think we should let errors propagate through yield-from, primarily. That's what it exists for.
Yes, and that's exactly what my wait() mechanism does. You call the wait() method using yield-from. The important idea is that just because you spawn a task, it doesn't necessarily follow that you want to be regarded as the *parent* of that task and receive its exceptions. That only becomes clear when you wait() for it.
In my system, spawn() isn't a wrapper -- it *is* the primitive way to create an independent task. And I think it's the only one we need.
It has to know what scheduler to talk to, right?
Yes, but in my world, there is only *one* scheduler. I understand that not everyone thinks that's a good idea, and I'm thinking about ways to remove that restriction. But I'm not yet sure that it *should* be removed even if it can. It seems to me that having multiple schedulers is inviting many of the same problems as having multiple event loops, and this whole disussion is centred on the idea that there should only be one of those. Just to be clear, I'm not saying there should only be one scheduler *implementation* in existence -- only that there should only be one *instance* of some scheduler implementation in any given program (or thread, if you're using those). And there should be a standard interface for it and an agreed way of finding the instance. What you're saying is that the standard interface should consist of yielded instructions and the instance should be found implicitly using dynamic scoping. This is *very* different from the kind of interface used for everything else in Python, and I'm not yet convinced that such a large amount of weirdness is justified. -- Greg

On Tue, Oct 16, 2012 at 2:14 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
The important idea is that just because you spawn a task, it doesn't necessarily follow that you want to be regarded as the *parent* of that task and receive its exceptions. That only becomes clear when you wait() for it.
Maybe. But the opposite doesn't follow either. It's a toss-up between the spawner and the waiter. -- --Guido van Rossum (python.org/~guido)

Guido van Rossum wrote:
On Tue, Oct 16, 2012 at 2:14 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
The important idea is that just because you spawn a task, it doesn't necessarily follow that you want to be regarded as the *parent* of that task and receive its exceptions.
Maybe. But the opposite doesn't follow either. It's a toss-up between the spawner and the waiter.
So maybe spawn() should have an option indicating that the spawning task is to receive exceptions occuring in the spawned task. -- Greg

On 17.10.12 12:38, Greg Ewing wrote:
Guido van Rossum wrote:
On Tue, Oct 16, 2012 at 2:14 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
The important idea is that just because you spawn a task, it doesn't necessarily follow that you want to be regarded as the *parent* of that task and receive its exceptions.
Maybe. But the opposite doesn't follow either. It's a toss-up between the spawner and the waiter.
So maybe spawn() should have an option indicating that the spawning task is to receive exceptions occuring in the spawned task.
No idea if that helps here, but the same problem occurred for us as well. It is not always clear if an exception should be handled in a certain context, or if it should be passed on and get raised later in the context that is concerned. For that, Stackless has introduced a _bomb_ object that encapsulates an exception, in order to let it pass through the normal call/yield/return interface. It is used to send an exception over a channel, which will explode (raise that exception) when the receiver picks it later up. I could think of something similar as a way to collect very many results in a join construct that collects everything without the need to handle each exception in the very moment it was raised. That would make it possible to collect results efficiently using 'yield from' and inspect the results later. Probably nothing new, just mentioned an idea... -- Christian Tismer :^) <mailto:tismer@stackless.com> Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

On Wed, Oct 17, 2012 at 5:27 AM, Christian Tismer <tismer@stackless.com> wrote:
On 17.10.12 12:38, Greg Ewing wrote:
Guido van Rossum wrote:
On Tue, Oct 16, 2012 at 2:14 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
The important idea is that just because you spawn a task, it doesn't necessarily follow that you want to be regarded as the *parent* of that task and receive its exceptions.
Maybe. But the opposite doesn't follow either. It's a toss-up between the spawner and the waiter.
So maybe spawn() should have an option indicating that the spawning task is to receive exceptions occuring in the spawned task.
No idea if that helps here, but the same problem occurred for us as well. It is not always clear if an exception should be handled in a certain context, or if it should be passed on and get raised later in the context that is concerned.
For that, Stackless has introduced a _bomb_ object that encapsulates an exception, in order to let it pass through the normal call/yield/return interface. It is used to send an exception over a channel, which will explode (raise that exception) when the receiver picks it later up.
Hmm... That sounds a little like your iriginal design for the channel only supported transferring values. At least for NDB, all channels support exceptions and tracebacks as an explicit alternative to the value.
I could think of something similar as a way to collect very many results in a join construct that collects everything without the need to handle each exception in the very moment it was raised.
That would make it possible to collect results efficiently using 'yield from' and inspect the results later.
Probably nothing new, just mentioned an idea...
I do think we're hashing out important ideas... -- --Guido van Rossum (python.org/~guido)

Yes, but in my world, there is only *one* scheduler.
Just to be clear, I'm not saying there should only be one scheduler *implementation* in existence -- only that there should only be one *instance* of some scheduler implementation in any given program (or thread, if you're using those). And there should be a standard interface for it and an agreed way of finding the instance.
I agree with this entirely. There are a lot of optimisations to be had with different scheduler implementations, but the only way this can be portable is with a minimum supported interface and a standard way to find it.

On Tue, Oct 16, 2012 at 5:14 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Calvin Spealman wrote:
What if two tasks call wait() on the same subtask which raises an error?
That would be disallowed. A given Task would only be allowed to have its wait() method called once.
The reason for this restriction is because of the way tracebacks are attached to exception objects in Python 3, which means that exceptions are effectively single-use now. If it weren't for that, the exception could simply be raised in *both* waiters.
I think we should let errors propagate through yield-from, primarily. That's what it exists for.
Yes, and that's exactly what my wait() mechanism does. You call the wait() method using yield-from.
The important idea is that just because you spawn a task, it doesn't necessarily follow that you want to be regarded as the *parent* of that task and receive its exceptions. That only becomes clear when you wait() for it.
In my system, spawn() isn't a wrapper -- it *is* the primitive way to create an independent task. And I think it's the only one we need.
It has to know what scheduler to talk to, right?
Yes, but in my world, there is only *one* scheduler.
Practically speaking, that is nice. But, are there use cases for multiple schedulers we should support? I also like the idea of the scheduler being an iterable, and thus itself being something you can schedule. Turtles all the way down.
I understand that not everyone thinks that's a good idea, and I'm thinking about ways to remove that restriction. But I'm not yet sure that it *should* be removed even if it can. It seems to me that having multiple schedulers is inviting many of the same problems as having multiple event loops, and this whole disussion is centred on the idea that there should only be one of those.
Just to be clear, I'm not saying there should only be one scheduler *implementation* in existence -- only that there should only be one *instance* of some scheduler implementation in any given program (or thread, if you're using those). And there should be a standard interface for it and an agreed way of finding the instance.
What you're saying is that the standard interface should consist of yielded instructions and the instance should be found implicitly using dynamic scoping. This is *very* different from the kind of interface used for everything else in Python, and I'm not yet convinced that such a large amount of weirdness is justified.
I don't follow the part about "found implicitly using dynamic scoping". What do you mean? In my model, the tasks never find the scheduler at all. They don't directly access it at all.
-- Greg _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
-- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy

I wrote:
Just to be clear, I'm not saying there should only be one scheduler *implementation* in existence
But having said that, I can't see much reason why you would need to have more than one scheduler implementation. Multiple event loop implementations are necessary because async I/O needs to be done different ways on different platforms. But the scheduler we're talking about is all pure Python. If the interface is well known and universally used, and there's a good implementation of it in the standard library, why would anyone want another one? -- Greg

But the scheduler we're talking about is all pure Python. If the interface is well known and universally used, and there's a good implementation of it in the standard library, why would anyone want another one?
Probably because they already have another one and can't get rid of it. Whether or not we are trying to include GUI development in this, I can guarantee that people will try and use it with a GUI message loop (to avoid blocking on IO, largely). In this case we'd almost certainly need a different implementation for Wx/Tcl/whatever. "Universally used" is a nice idea, but it will take a long time to get there. A well known interface, especially one that doesn't require the loop itself (i.e. it doesn't have a blocking run() function), lets users write thin wrappers, like the one we did for Tcl: http://pastebin.com/FuZwc1Ur (CallableContext (the 'scheduler') base class is in http://pastebin.com/ndS53Cd8). There needs to be an way to change which one is used at runtime, but there only needs to be one per thread. Cheers, Steve

On Tue, Oct 16, 2012 at 9:45 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
In my system, spawn() isn't a wrapper -- it *is* the primitive way to create an independent task. And I think it's the only one we need.
I think you will at minimum need a way to suspend and resume tasks, in addition to spawn(), as illustrated by the example of par() waiting for not CPU-bound tasks. This could be done either as actual suspend and resume primitives, or by building on a related set of synchronization primitives, such as queues, channels, or condition variables: there are a number of sets of that are mutually co-expressible. Suspending and resuming, in particular, is highly related to the question of how you reify a task as a conventional callback, when the need for that arises. Here's one possible way of approaching this with a combined suspend/resume primitive that might look familiar to people with a FP background: result = yield suspend(lambda resume: ...) (Here, "suspend" could be a scheduler-agnostic instruction object, a la tasklib.suspend(), or a method on a global scheduler.) suspend() would instruct the scheduler to stop running the current task, and call its argument (the lambda in the above example) with a "resume(value)" callable that will arrange to resume the task again with the given value. The body of the lambda (or whatever is passed to suspend()) would be responsible for doing something useful with the resume() callable: e.g. in par() example, it would arrange that the last child task triggers it. In particular, this suspend() could be used to integrate fairly directly with callback-based APIs: for example, if you have a Twisted Deferred, you could do: result = yield suspend(d.addCallback) to suspend the current task and add a callback to d that will resume it again, and receive the Deferred's result. To add support for exceptions, a variation of suspend() could pass two callables, mirroring pairs like send/throw, or callback/errback: result = yield suspend2(lambda resume, throw: ...) result = yield suspend2(d.addCallbacks) -- Piet Delport

Piet Delport wrote:
In particular, this suspend() could be used to integrate fairly directly with callback-based APIs: for example, if you have a Twisted Deferred, you could do:
result = yield suspend(d.addCallback)
I've been thinking about how to express this using the primitives provided by the scheduler in my tutorial. I don't actually have a primitive that simply suspends a task; instead, I have one that moves the current task from the ready list to a specified list: scheduler.block(queue) Similarly, I don't have a primitive that explicitly resumes a particular task[1] -- only one that takes the first task off a specified list and resumes it: scheduler.unblock(queue) I think this is a good idea if we want to be able to cancel tasks, because a cancelled task ought to cleanly disappear from the system, without any risk that something will try to schedule it again. This is achievable if we maintain the invariant that a task always belongs to some queue, and the scheduler knows about that queue. Given these primitives, we can define def wakeup_callback(queue): lambda: scheduler.unblock(queue) and def wait_for_callback(add_callback): q = [] add_callback(wakeup_callback(q)) scheduler.block(q) yield This is starting to look rather like a semaphore. If we assume semaphores as a facility provided by the library, then it becomes very straightforward: def wait_for_callback(add_callback): s = Semaphore() add_callback(s.notify) yield from s.wait() That assumes the callback is single-use. But a semaphore can also handle multi-use callbacks: you just keep the semaphore around and repeatedly wait on it. You will get woken up once for each time the callback is called. s = Semaphore() something.add_callback(s.notify) while we_are_still_interested(): yield from s.wait() ... --- [1] Actually I do, but I'm thinking it shouldn't be exposed as part of the public API for reasons given here. -- Greg

I've converted my tutorial on generator-based tasks for Python 3.3, tidied it up a bit and posted it here: http://www.cosc.canterbury.ac.nz/greg.ewing/python/tasks/ -- Greg

2012/10/18 Greg Ewing <greg.ewing@canterbury.ac.nz>
I've converted my tutorial on generator-based tasks for Python 3.3, tidied it up a bit and posted it here:
I liked it. I was kind of confused about use of yield/yield from in this style of async. Now things seems to be pretty clear. -- Carlo Pires

On Thu, Oct 18, 2012 at 9:49 AM, Greg Ewing <greg.ewing@canterbury.ac.nz>wrote:
I've converted my tutorial on generator-based tasks for Python 3.3, tidied it up a bit and posted it here:
-- Greg
Thanks for writing this. I've used threads all my life so this coroutine/yield-from paradigm is hard for me to grok even after reading this quite a few times. I can't wrap my head around the block and unblock functions. block() removes the current task from the ready_list, but is the current task guaranteed to be my task? If so, then I'd never run again after the yield in acquire(), that is unless a gracious other player unblocks me. block() in acquire() is the philosopher or fork avoiding the scheduler? yield in acquire() is the philosopher relinquishing control or the fork? I think I finally figured it out after staring at it for long enough. I'm not sure it makes sense for scheduler functions to store waiting tasks in a queue owned by the app and invisible from the scheduler. This can cause *invisible deadlocks* such as: schedule(philosopher("Socrates", 8, 3, 1, forks[0], forks[2]), "Socrates") schedule(philosopher("Euclid", 5, 1, 4, forks[2], forks[0]), "Euclid") Which may be really hard to debug. Is there a coroutine strategy for tackling these challenges? Or will I just get better at overcoming them with practice? --Yuval

Yuval Greenfield wrote:
block() removes the current task from the ready_list, but is the current task guaranteed to be my task?
Yes, block() always operates on the currently running task.
If so, then I'd never run again after the yield in acquire(), that is unless a gracious other player unblocks me.
Yes, the unblocking needs to be done by another task, or by something outside the task system such as an I/O callback.
I'm not sure it makes sense for scheduler functions to store waiting tasks in a queue owned by the app and invisible from the scheduler. This can cause *invisible deadlocks* such as:
schedule(philosopher("Socrates", 8, 3, 1, forks[0], forks[2]), "Socrates") schedule(philosopher("Euclid", 5, 1, 4, forks[2], forks[0]), "Euclid")
Deadlocks are a potential problem in any system involving concurrency, and have to be dealt with on a case-by-case basis. Simply having the scheduler know where all the tasks are will not prevent deadlocks. It might make it possible for the scheduler to *detect* deadlocks, but you still have to do something about them. Having said that, I'm thinking about writing a more elaborate version of my scheduler that does keep track of which queue a task is waiting on, mainly so that tasks can be cancelled cleanly.
Is there a coroutine strategy for tackling these challenges? Or will I just get better at overcoming them with practice?
If you've been using threads all your life as you say, then you're probably already pretty good at dealing with them. All of the same techniques apply. -- Greg

On Sat, Oct 20, 2012 at 4:21 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Simply having the scheduler know where all the tasks are will not prevent deadlocks. It might make it possible for the scheduler to *detect* deadlocks, but you still have to do something about them.
Having said that, I'm thinking about writing a more elaborate version of my scheduler that does keep track of which queue a task is waiting on, mainly so that tasks can be cancelled cleanly.
In NDB, I have a feature that detects most deadlocks -- the Future class keeps track of all incomplete instances, and it can dump this list at request. Futures also keep some information about where and for what purpose they were created. Finally, to tie it all together, there's code that detects that you're waiting for something to happen but the event loop is out of things to do (i.e. no pending RPCs, no "call later" callbacks left -- hence, no progress can possibly be made). This feature has caught mostly bugs in NDB itself -- because NDB is primarily a database API, regular NDB users don't normally write code that is likely to deadlock. But in the wider Python 3 world, where regular users would be writing (presumably buggy) protocol implementations and their own servers and clients, I suspect debugging features can make and break a system like this. -- --Guido van Rossum (python.org/~guido)

Guido van Rossum wrote:
But there needs to be another way to get a task running immediately and concurrently; I believe that would be
a = spawn(foo_task())
right? One could then at any later point use
ra = yield from a
Hmmm. I suppose it *could* be made to work that way, but I'm not sure it's a good idea, because it blurs the distinction between invoking a subtask synchronously and waiting for the result of a previously spawned independent task. Recently I've been thinking about an implementation where it would look like this. First you do t = spawn(foo_task()) but what you get back is *not* a generator; rather it's a Task object which wraps a generator and provides various operations. One of them would be r = yield from t.wait() which waits for the task to complete and then returns its value (or if it raised an exception, propagates the exception). Other operations that a Task object might support include t.unblock() # wake up a blocked task t.cancel() # unschedule and clean up the task t.throw(exception) # raise an exception in the task (I haven't included t.block(), because I think that should be a stand-alone function that operates on the current task. Telling some other task to block feels like a dodgy thing to do.)
One could also combine these and do e.g.
a = spawn(foo_task()) b = spawn(bar_task()) <do more work locally> ra, rb = yield from par(a, b)
If you're happy to bail out at the first exception, you wouldn't strictly need a par() function for this, you could just do a = spawn(foo_task()) b = spawn(bar_task()) ra = yield from a.wait() rb = yield from b.wait()
Have I got the spelling for spawn() right? In many other systems (e.g. threads, greenlets) this kind of operation takes a callable, not the result of calling a function (albeit a generator).
That's a result of the fact that a generator doesn't start running as soon as you call it. If you don't like that, the spawn() operation could be defined to take an uncalled generator and make the call for you. But I think it's useful to make the call yourself, because it gives you an opportunity to pass parameters to the task.
If it takes a generator, would it return the same generator or a different one to wait for?
In your version above where you wait for the task simply by calling it with yield-from, spawn() would have to return a generator (or something with the same interface). But it couldn't be the same generator -- it would have to be a wrapper that takes care of blocking until the subtask is finished. -- Greg

On Tue, Oct 16, 2012 at 12:20 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Guido van Rossum wrote:
But there needs to be another way to get a task running immediately and concurrently; I believe that would be
a = spawn(foo_task())
right? One could then at any later point use
ra = yield from a
Hmmm. I suppose it *could* be made to work that way, but I'm not sure it's a good idea, because it blurs the distinction between invoking a subtask synchronously and waiting for the result of a previously spawned independent task.
Are you sure you really want to distinguish between those though? In NDB they are intentionally the same -- invoking some API whose name ends in _async() starts an async subtask and returns a Future; you wait for the subtask by yielding the Future. Starting multiple tasks is just a matter of calling several _async() APIs; then you can wait for any or all of them using yield [future1, future2, ...] *or* by yielding the futures one at a time. This gives users a gentle introduction to concurrency (first they use the synchronous APIs; then they learn to use yield foo_async(); then they learn they can write: f = foo_async() <other work> r = yield f and finally they learn about spawning multiple tasks: f1 = foo_async() f2 = bar_async() rfoo, rbar = yield f1, f2
Recently I've been thinking about an implementation where it would look like this. First you do
t = spawn(foo_task())
but what you get back is *not* a generator; rather it's a Task object which wraps a generator and provides various operations. One of them would be
r = yield from t.wait()
which waits for the task to complete and then returns its value (or if it raised an exception, propagates the exception).
Other operations that a Task object might support include
t.unblock() # wake up a blocked task t.cancel() # unschedule and clean up the task t.throw(exception) # raise an exception in the task
(I haven't included t.block(), because I think that should be a stand-alone function that operates on the current task. Telling some other task to block feels like a dodgy thing to do.)
Right. I'm looking forward to a larger example.
One could also combine these and do e.g.
a = spawn(foo_task()) b = spawn(bar_task()) <do more work locally> ra, rb = yield from par(a, b)
If you're happy to bail out at the first exception, you wouldn't strictly need a par() function for this, you could just do
a = spawn(foo_task()) b = spawn(bar_task()) ra = yield from a.wait() rb = yield from b.wait()
Have I got the spelling for spawn() right? In many other systems (e.g. threads, greenlets) this kind of operation takes a callable, not the result of calling a function (albeit a generator).
That's a result of the fact that a generator doesn't start running as soon as you call it. If you don't like that, the spawn() operation could be defined to take an uncalled generator and make the call for you. But I think it's useful to make the call yourself, because it gives you an opportunity to pass parameters to the task.
Agreed, actually. I was just checking.
If it takes a generator, would it return the same generator or a different one to wait for?
In your version above where you wait for the task simply by calling it with yield-from, spawn() would have to return a generator (or something with the same interface). But it couldn't be the same generator -- it would have to be a wrapper that takes care of blocking until the subtask is finished.
That's fine with me (though Glyph would worry about creating too many objects). -- --Guido van Rossum (python.org/~guido)

Guido van Rossum wrote:
On Tue, Oct 16, 2012 at 12:20 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
it blurs the distinction between invoking a subtask synchronously and waiting for the result of a previously spawned independent task.
Are you sure you really want to distinguish between those though?
I think I do. Partly because I feel that not doing so would make code harder to reason about. Async stuff is difficult enough as it is without hiding the boundaries between one thread of control and another. There are technical reasons as well. If you use 'yield from' to wait for completion of an independent task, then it would seem like you should be able to do this: t1 = task1() t2 = task2() spawn(t1) spawn(t2) r1 = yield from t1 r2 = yield from t2 But that can't work -- the object that you wait on has to be different from the generator instance passed to spawn(). The reason is that if the task finishes before anyone waits on it, the return value needs to be stored somewhere. Having spawn() return an object that deliberately does *not* have the interface of a generator, and having to explicitly wait for it, makes it much less likely that anyone will make that kind of mistake. If you wrote t1 = task1() t2 = task2() spawn(t1) spawn(t2) r1 = yield from t1.wait() r2 = yield from t2.wait() you would quickly get an exception, because generators don't have a wait() method. -- Greg

On Wed, Oct 17, 2012 at 3:16 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Guido van Rossum wrote:
On Tue, Oct 16, 2012 at 12:20 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
it blurs the distinction between invoking a subtask synchronously and waiting for the result of a previously spawned independent task.
Are you sure you really want to distinguish between those though?
I think I do. Partly because I feel that not doing so would make code harder to reason about. Async stuff is difficult enough as it is without hiding the boundaries between one thread of control and another.
There are technical reasons as well. If you use 'yield from' to wait for completion of an independent task, then it would seem like you should be able to do this:
t1 = task1() t2 = task2() spawn(t1) spawn(t2) r1 = yield from t1 r2 = yield from t2
But that can't work -- the object that you wait on has to be different from the generator instance passed to spawn(). The reason is that if the task finishes before anyone waits on it, the return value needs to be stored somewhere.
Having spawn() return an object that deliberately does *not* have the interface of a generator, and having to explicitly wait for it, makes it much less likely that anyone will make that kind of mistake. If you wrote
t1 = task1() t2 = task2() spawn(t1) spawn(t2) r1 = yield from t1.wait() r2 = yield from t2.wait()
you would quickly get an exception, because generators don't have a wait() method.
Ack. I get it. It's like the difference between calling a function vs. running it in an OS thread. -- --Guido van Rossum (python.org/~guido)

Guido van Rossum wrote:
On Fri, Oct 12, 2012 at 10:05 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
You could go further and say that yielding a tuple of generators means to spawn them all concurrently, wait for them all to complete and send back a tuple of the results. The yield-from code would then look pretty much the same as the futures code.
Sadly it looks that
r = yield from (f1(), f2())
ends up interpreting the tuple as the iterator,
That's not yielding a tuple of generators. This is: r = yield (f1(), f2()) Note the absence of 'from'.
So, can par() be as simple as
def par(*args): results = [] for task in args: result = yield from task results.append(result) return results
No, it can't be as simple as that, because that will just execute the tasks sequentially. It would have to be something like this: def par(*tasks): n = len(tasks) results = [None] * n for i, task in enumerate(tasks): def thunk(): nonlocal n results[i] = yield from task n -= 1 scheduler.schedule(thunk) while n > 0: yield return results Not exactly straightforward, but that's why we write it once and put it in the library. :-)
Of course there's the question of what to do when one of the tasks raises an error -- I haven't quite figured that out in NDB either, it runs all the tasks to completion but the caller only sees the first exception. I briefly considered having an "multi-exception" but it felt too weird -- though I'm not married to that decision.
Hmmm. Probably what should happen is that all the other tasks get cancelled and then the exception gets propagated to the caller of par(). If we assume another couple of primitives: scheduler.cancel(task) -- cancels the task scheduler.throw(task, exc) -- raises an exception in the task then we could implement it this way: def par(*tasks): n = len(tasks) results = [None] * n this = scheduler.current_task for i, task in enumerate(tasks): def thunk(): nonlocal n try: results[i] = yield from task except BaseException as e: for t in tasks: scheduler.cancel(t) scheduler.throw(this, e) n -= 1 scheduler.schedule(thunk) while n > 0: yield return results
(10) Registering additional callbacks
While we're at it: class task_with_callbacks(): def __init__(self, task): self.task = task self.callbacks = [] def add_callback(self, cb): self.callbacks.append(cb) def run(self): result = yield from self.task for cb in self.callbacks: cb() return result
Here's another pattern that I can't quite figure out. ... Essentially, it's a barrier pattern where multiple tasks (each representing a different HTTP request, and thus not all starting at the same time) render a partial web page and then block until a new HTTP request comes in that provides the missing info.
This should be fairly straightforward. waiters = [] # Tasks waiting for the event When a task wants to wait: scheduler.block(waiters) When the event occurs: for t in waiters: scheduler.schedule(t) del waiters[:] Incidentally, this is a commonly encountered pattern known as a "condition queue" in IPC parlance. I envisage that the async library would provide encapsulations of this and other standard IPC mechanisms such as mutexes, semaphores, channels, etc. -- Greg

On Sun, Oct 14, 2012 at 4:49 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Guido van Rossum wrote:
On Fri, Oct 12, 2012 at 10:05 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
You could go further and say that yielding a tuple of generators means to spawn them all concurrently, wait for them all to complete and send back a tuple of the results. The yield-from code would then look pretty much the same as the futures code.
Sadly it looks that
r = yield from (f1(), f2())
ends up interpreting the tuple as the iterator,
That's not yielding a tuple of generators. This is:
r = yield (f1(), f2())
Note the absence of 'from'.
That's what I meant -- excuse me for not writing "yield-fromming". :-)
So, can par() be as simple as
def par(*args): results = [] for task in args: result = yield from task results.append(result) return results
No, it can't be as simple as that, because that will just execute the tasks sequentially.
Yeah, Ben just cleared that up for me.
It would have to be something like this:
def par(*tasks): n = len(tasks) results = [None] * n for i, task in enumerate(tasks): def thunk(): nonlocal n results[i] = yield from task n -= 1 scheduler.schedule(thunk) while n > 0: yield return results
Not exactly straightforward, but that's why we write it once and put it in the library. :-)
But, as Christian Tismer wrote, we need to have some kind of idea of what the primitives are that we want to support. Or should we just have async equivalents for everything in threading.py and queue.py? (What about thread-local? Do we need task-local? Shudder.)
Of course there's the question of what to do when one of the tasks raises an error -- I haven't quite figured that out in NDB either, it runs all the tasks to completion but the caller only sees the first exception. I briefly considered having an "multi-exception" but it felt too weird -- though I'm not married to that decision.
Hmmm. Probably what should happen is that all the other tasks get cancelled and then the exception gets propagated to the caller of par().
I think it ought to be at least an option to run them all to completion -- I can easily imagine use cases for that. Also for wanting to receive a list of exceptions. A practical par() may have to grow a few options...
If we assume another couple of primitives:
scheduler.cancel(task) -- cancels the task
scheduler.throw(task, exc) -- raises an exception in the task
then we could implement it this way:
def par(*tasks): n = len(tasks) results = [None] * n this = scheduler.current_task for i, task in enumerate(tasks): def thunk(): nonlocal n try: results[i] = yield from task except BaseException as e: for t in tasks: scheduler.cancel(t) scheduler.throw(this, e) n -= 1 scheduler.schedule(thunk) while n > 0: yield return results
I glazed over here but I trust you.
(10) Registering additional callbacks
While we're at it:
class task_with_callbacks():
def __init__(self, task): self.task = task self.callbacks = []
def add_callback(self, cb): self.callbacks.append(cb)
def run(self): result = yield from self.task for cb in self.callbacks: cb() return result
Nice. (In fact so simple that maybe users can craft this for themselves?)
Here's another pattern that I can't quite figure out. ...
Essentially, it's a barrier pattern where multiple tasks (each representing a different HTTP request, and thus not all starting at the same time) render a partial web page and then block until a new HTTP request comes in that provides the missing info.
This should be fairly straightforward.
waiters = [] # Tasks waiting for the event
When a task wants to wait:
scheduler.block(waiters)
When the event occurs:
for t in waiters: scheduler.schedule(t) del waiters[:]
Incidentally, this is a commonly encountered pattern known as a "condition queue" in IPC parlance. I envisage that the async library would provide encapsulations of this and other standard IPC mechanisms such as mutexes, semaphores, channels, etc.
Maybe you meant condition variable? It looks like threading.Condition with notify_all(). Anyway, I agree we need some primitives like these, but I'm not sure how to choose the set of essentials. -- --Guido van Rossum (python.org/~guido)

Guido van Rossum wrote:
But, as Christian Tismer wrote, we need to have some kind of idea of what the primitives are that we want to support.
Well, I was just responding to your asking what the yield-from equivalent would be to the corresponding thing using Futures. I assumed from the fact that you asked that it was something Futures-using people like to do a lot, so it would be worth putting into a library. There may be other ways to approach it, though. Suppose we had a primitive that just waits for a single task to finish and returns its value. Then we could do this: def par(*tasks): for task in tasks: scheduler.schedule(task) return [yield from scheduler.wait_for(task) for task in tasks] That's straightforward enough that maybe it doesn't even need to be a library function, just a well-known pattern.
Maybe you meant condition variable? It looks like threading.Condition with notify_all().
Something like that -- the terminology probably varies a bit from one library to another. The basic concept is "set of tasks waiting for some condition to become true".
Anyway, I agree we need some primitives like these, but I'm not sure how to choose the set of essentials.
I think that most, maybe all, of the standard synchronisation mechanisms, like mutexes and semaphores, can be built out of the primitives I've already introduced -- essentially just block() and yield. So anything of this kind that we provide will be more in the way of convenience features than essential primitives. -- Greg

On Mon, Oct 15, 2012 at 3:37 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Guido van Rossum wrote:
But, as Christian Tismer wrote, we need to have some kind of idea of what the primitives are that we want to support.
Well, I was just responding to your asking what the yield-from equivalent would be to the corresponding thing using Futures. I assumed from the fact that you asked that it was something Futures-using people like to do a lot, so it would be worth putting into a library.
There may be other ways to approach it, though. Suppose we had a primitive that just waits for a single task to finish and returns its value. Then we could do this:
def par(*tasks): for task in tasks: scheduler.schedule(task) return [yield from scheduler.wait_for(task) for task in tasks]
That's straightforward enough that maybe it doesn't even need to be a library function, just a well-known pattern.
The more I follow this thread the less I understand the point of introducing a new use for yield-from in this discussion. All of this extra work trying to figure how to make yield-from work giving its existing 3.3 semantics could just be avoided if we just allow yielding the tasks directly, and treating them like any other async operation. In the original message yield-from seemed to be suggested, there was no justification, it was just said "so you have to do this" but I don't see that you do. If you allow yielding tasks, then yielding multiple tasks to wait together because trivial: just yield a tuple of them. In fact, I think we should say that yielding any tuple of async operations, whatever those objects actually end of being, should wait for all of them. Maybe we also want to wait on both some http request operation, implemented as a task (a generator), and also a cache hit. def handle_or_cached(request): api_resp, cache_resp = yield request(API_ENDPOINT), cache.get(KEY) if cache_resp: return cache_resp return render(api_resp) Or we could provide wrappers to control the behavior of multiple-wait: def handle_or_cached(request): api_resp, cache_resp = yield first(request(API_ENDPOINT), cache.get(KEY)) if cache_resp: return cache_resp return render(api_resp)
Maybe you meant condition variable? It looks like threading.Condition with notify_all().
Something like that -- the terminology probably varies a bit from one library to another. The basic concept is "set of tasks waiting for some condition to become true".
Anyway, I agree we need some primitives like these, but I'm not sure how to choose the set of essentials.
I think that most, maybe all, of the standard synchronisation mechanisms, like mutexes and semaphores, can be built out of the primitives I've already introduced -- essentially just block() and yield. So anything of this kind that we provide will be more in the way of convenience features than essential primitives.
-- Greg
_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
-- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy

On Mon, Oct 15, 2012 at 8:25 PM, Calvin Spealman <ironfroggy@gmail.com> wrote:
The more I follow this thread the less I understand the point of introducing a new use for yield-from in this discussion.
+1. To me, "yield from" is just a tool that brings generators back to parity with functions when it comes to breaking up a larger algorithm into smaller pieces. Where you would break a function out into subfunctions and call them normally, with a generator you can break out subgenerators and invoke them with yield from. Any meaningful use of "yield from" in the coroutine context *has* to ultimate devolve to an operation that: 1. Asks the scheduler to schedule another operation 2. Waits for that operation to complete Guido's approach to that problem is that step 1 is handled by calling functions that in turn call methods on a thread-local scheduler. These methods return Future objects, which can subsequently be yielded to the scheduler to say "I'm waiting for this future to be set". I *thought* Greg's way combined step 1 and step 2 into a single operation: the objects you yield *not only* say what you want to wait for, but also what you want to do. However, his example par() implementation killed that idea, since it turned out to need to schedule tasks explicitly rather than their being a "execute this in parallel" option. So now I'm back to think that Greg and Guido are talking about different levels. *Any* scheduling option will be able to be collapsed into an async task invoked by "yield from" by writing: def simple_async_task(): return yield start_task() The part that still needs to be figured out is how you turn that suspend/resume communications channel between the lowest level of the task stack and the scheduling loop into something usable, as well as how you handle iteration in a sensible way (I described my preferred approach when writing about the API I'd like to see for an async version of as_completed). I haven't seen anything to suggest that "yield from"'s role should change from what it is in 3.3: a way to factor out generators into multiple pieces with out breaking send() and throw(). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

A thought about more ways we could control groups of tasks, and avoid yield-from, just came to me this morning. def asset_packer(asset_urls): with yield until_all as results: for url in asset_urls: yield http.get(url) return pack(results) or def handle_or_cached(url): with yield first as result: yield http.get(url) yield cache.get(url) return result Currently, "with yield expr:" is not valid syntax, surprisingly. This gives us room to use it for something new. A generator-sensitive context manager. One option is just to allow the syntax directly. The generator yields, and sent value is used as a context manager. This would let the generator tell the scheduler "I'm going to give you a few different async ops, and I want to wait for all of them before I continue." etc. However, it leaves open the question how the scheduler knows the context manager has ended. Could it somehow indicate this to the correct scheduler in __exit__? Another option, if we're adding a new syntax anyway, is to make "with yield expr:" special and yield first the result of __enter__() and then, after the block is done, yield the result of __exit__(), which lets context blocks in the generator talk to the scheduler both before and after. Maybe we don't need the second, nuttier idea. But, I like the general idea. It feels right. On Mon, Oct 15, 2012 at 8:08 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Mon, Oct 15, 2012 at 8:25 PM, Calvin Spealman <ironfroggy@gmail.com> wrote:
The more I follow this thread the less I understand the point of introducing a new use for yield-from in this discussion.
+1. To me, "yield from" is just a tool that brings generators back to parity with functions when it comes to breaking up a larger algorithm into smaller pieces. Where you would break a function out into subfunctions and call them normally, with a generator you can break out subgenerators and invoke them with yield from.
Any meaningful use of "yield from" in the coroutine context *has* to ultimate devolve to an operation that: 1. Asks the scheduler to schedule another operation 2. Waits for that operation to complete
Guido's approach to that problem is that step 1 is handled by calling functions that in turn call methods on a thread-local scheduler. These methods return Future objects, which can subsequently be yielded to the scheduler to say "I'm waiting for this future to be set".
I *thought* Greg's way combined step 1 and step 2 into a single operation: the objects you yield *not only* say what you want to wait for, but also what you want to do. However, his example par() implementation killed that idea, since it turned out to need to schedule tasks explicitly rather than their being a "execute this in parallel" option.
So now I'm back to think that Greg and Guido are talking about different levels. *Any* scheduling option will be able to be collapsed into an async task invoked by "yield from" by writing:
def simple_async_task(): return yield start_task()
The part that still needs to be figured out is how you turn that suspend/resume communications channel between the lowest level of the task stack and the scheduling loop into something usable, as well as how you handle iteration in a sensible way (I described my preferred approach when writing about the API I'd like to see for an async version of as_completed). I haven't seen anything to suggest that "yield from"'s role should change from what it is in 3.3: a way to factor out generators into multiple pieces with out breaking send() and throw().
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia hink there is something wrong with the autolists that are set up to include Premium and Free content.
-- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy

On Mon, Oct 15, 2012 at 10:31 PM, Calvin Spealman <ironfroggy@gmail.com> wrote:
Currently, "with yield expr:" is not valid syntax, surprisingly.
It's not that surprising, it's the general requirement that yield expressions must be enclosed in parentheses except when used standalone or in a simple assignment statement. "with (yield expr):" is valid syntax though, so I'm reluctant to endorse doing anything substantially different if the parentheses are omitted. I think the combination of "yield from" to delegate control (including exception handling) completely to a subgenerator and "context manager + for loop + explicit yield" when an operation needs to yield multiple times and the exception handling behaviour should be left to the caller (as in the "as_completed" case) should cover the necessary behaviours. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Mon, Oct 15, 2012 at 9:48 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Mon, Oct 15, 2012 at 10:31 PM, Calvin Spealman <ironfroggy@gmail.com> wrote:
Currently, "with yield expr:" is not valid syntax, surprisingly.
It's not that surprising, it's the general requirement that yield expressions must be enclosed in parentheses except when used standalone or in a simple assignment statement.
"with (yield expr):" is valid syntax though, so I'm reluctant to endorse doing anything substantially different if the parentheses are omitted.
Silly oversight on my part, and I agree that the parens shouldn't make the difference in meaning.
I think the combination of "yield from" to delegate control (including exception handling) completely to a subgenerator and "context manager + for loop + explicit yield" when an operation needs to yield multiple times and the exception handling behaviour should be left to the caller (as in the "as_completed" case) should cover the necessary behaviours.
I'm still -1 on delegating control to subgenerators with yield-from, versus having the scheduler just deal with them directly. I think it is far less flexible. I would still like to see a less confusing "with yield expr:" by simply allowing it without parens, but no special meaning. I think it would be really useful in coroutines. with yield collect() as tasks: yield task1() yield task2() results = yield tasks
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
-- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy

On Tue, Oct 16, 2012 at 12:16 AM, Calvin Spealman <ironfroggy@gmail.com> wrote:
I'm still -1 on delegating control to subgenerators with yield-from, versus having the scheduler just deal with them directly. I think it is far less flexible.
Um, yield from is to generators as calls are to functions... delegating to subgenerators, regardless of context, is what it's *for*. Without it, the scheduler will have to do quite a bit of extra work to reconstruct sane stack traces. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Mon, Oct 15, 2012 at 10:50 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Tue, Oct 16, 2012 at 12:16 AM, Calvin Spealman <ironfroggy@gmail.com> wrote:
I'm still -1 on delegating control to subgenerators with yield-from, versus having the scheduler just deal with them directly. I think it is far less flexible.
Um, yield from is to generators as calls are to functions... delegating to subgenerators, regardless of context, is what it's *for*. Without it, the scheduler will have to do quite a bit of extra work to reconstruct sane stack traces.
I didn't consider the ease of sane stack traces, that is a good point. I just see all the problems that seem to be harder to do right with yield-from and I wish it could be made simpler by just bypassing them for coroutines. I don't feel they are the same as the original intent of yield-from, but I see the obvious way they match the need now. But, I still want to make my case and will put another hypothetical on the board. A "sane stack trace" only makes sense if we assume that tasks "call" each other in the same kind of call tree that synchronous code flows in, and I don't think that is necessarily the case. There are cases when one task might want to end before tasks it as "called" are complete, and if we use yield-from this is *impossible* but it is very useful. An example of this is a task which makes multiple requests, but only needs to wait for the results from less-than-all of them before returning. It might still want the other tasks to complete, even if it won't do anything with the results. yield-from semantics won't allow a called task to continue, if needed, after the calling task itself has completed. Is there another way these semantics could be expressed?
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
-- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy

On Tue, Oct 16, 2012 at 1:16 AM, Calvin Spealman <ironfroggy@gmail.com> wrote:
An example of this is a task which makes multiple requests, but only needs to wait for the results from less-than-all of them before returning. It might still want the other tasks to complete, even if it won't do anything with the results.
yield-from semantics won't allow a called task to continue, if needed, after the calling task itself has completed.
Is there another way these semantics could be expressed?
Sure, did you see my as_completed example? You couldn't use "yield from" for that, you'd need to use an ordinary iterator and an explicit yield in the body of the loop (this is why I disagree with Greg that "yield from" can serve as the one true API - it doesn't handle partial iteration, and it doesn't handle pre- or post- processing around the suspension points while iterating). My preferred way of thinking of "yield from" is as a simple refactoring tool: "Gee, this generator is getting kind of long and unwieldy. I'll move this piece out into a separate generator, and use yield from to invoke it" or "Hmm, I keep using this same sequence of 3 or 4 operations. I guess I'll move them out to a separate generator and use yield from to invoke it in the appropriate places". Compare that with the almost identical equivalents when refactoring a function to call a helper function instead of doing everything inline: "Gee, this function is getting kind of long and unwieldy. I'll move this piece out into a separate function, and call it" or "Hmm, I keep using this same sequence of 3 or 4 operations. I guess I'll move them out to a separate function and call it it in the appropriate places". Just as some operations can't be factored out with simple function calls, hence we have iterators and context managers, so not all operations will be able to be factored out of a coroutine with "yield from" (hence why I consider "yield" to be the more appropriate core primitive, with "yield from" just correctly factoring out the task of complete delegation, which is otherwise hard to do correctly) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Mon, Oct 15, 2012 at 5:32 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
My preferred way of thinking of "yield from" is as a simple refactoring tool: "Gee, this generator is getting kind of long and unwieldy. I'll move this piece out into a separate generator, and use yield from to invoke it" or "Hmm, I keep using this same sequence of 3 or 4 operations. I guess I'll move them out to a separate generator and use yield from to invoke it in the appropriate places".
I agree. That's how I've used it. Maybe that's just short-sightedness. -- cheers lvh

Laurens Van Houtven wrote:
On Mon, Oct 15, 2012 at 5:32 PM, Nick Coghlan <ncoghlan@gmail.com <mailto:ncoghlan@gmail.com>> wrote:
My preferred way of thinking of "yield from" is as a simple refactoring tool
I agree. That's how I've used it. Maybe that's just short-sightedness.
And that's exactly how *I* see it as well! Which means some people must be misinterpreting something I'm saying, if they think I see it some other way. -- Greg

On Mon, Oct 15, 2012 at 8:32 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
My preferred way of thinking of "yield from" is as a simple refactoring tool: "Gee, this generator is getting kind of long and unwieldy. I'll move this piece out into a separate generator, and use yield from to invoke it" or "Hmm, I keep using this same sequence of 3 or 4 operations. I guess I'll move them out to a separate generator and use yield from to invoke it in the appropriate places".
In the NDB world you would say: "Gee this _tasklet_ is getting kind of long and unwieldy. I'll move this piece out into a separate _tasklet_, and use _yield_ to invoke it." Creating a tasklet is just writing a generator decorated with @ndb.tasklet -- after using this a bit it becomes total second nature (I've seen several coworkers pick it up effortlessly). I'll have to digest your other points about yield vs. yield-from more carefully -- on the one hand I think it would be cool if yield-from could give us an even simpler paradigm to write async code than NDB's version, and that expectation was one of my main reasons to push for PEP 380's acceptance. On the other hand you bring up some good points with the as_completed() example (though I have a feeling Greg will easily sail around it :-). PS. Unrelated, and please don't respond to this or at least change the subject if you feel compelled: there seem to be a lot of bad names in this field. Twisted uses adjectives as nouns (Twisted, Deferred, I read about another one), "add_done_callback" is too longwinded, "as_completed" brings absolutely no useful association with it.. -- --Guido van Rossum (python.org/~guido)

Nick Coghlan wrote:
(this is why I disagree with Greg that "yield from" can serve as the one true API - it doesn't handle partial iteration, and it doesn't handle pre- or post- processing around the suspension points while iterating).
I'm aware of the iteration problem, but I'm not convinced that the convolutions necessary to make it possible to use a for-loop for this are worth the bother, as opposed to simply accepting that you can't use the for statement in this situation, and using some other kind of loop. In any case, even if we decide to provide a scheduler instruction to enable using for-loops on suspendable iterators somehow, it doesn't follow that we should use scheduler instructions for anything *else*. I would consider such a scheduler instruction to be a stopgap measure until we can find a better solution -- just as yield-from is a better solution than using "call" and "return" scheduler instructions. -- Greg

On Tue, Oct 16, 2012 at 3:39 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
In any case, even if we decide to provide a scheduler instruction to enable using for-loops on suspendable iterators somehow, it doesn't follow that we should use scheduler instructions for anything *else*.
The only additional operation needed is an async equivalent to the concurrent.futures.wait() API, which would allow you to provide a set of Futures and say "let me know when one of these operations are done" (http://docs.python.org/py3k/library/concurrent.futures#concurrent.futures.wa...) As it turns out, this shouldn't *need* a new scheduler primitive in Guido's design, since it can be handled by hooking up an appropriate callback to the supplied future objects. Following code isn't tested, but given my understanding of how Guido wants things to work, it should do what I want: def _wait_first(futures): # futures must be a set, items will be removed as they complete signal = Future() def chain_result(completed): futures.remove(completed) if completed.cancelled(): signal.cancel() signal.set_running_or_notify_cancel() elif completed.done(): signal.set_result(completed.result()) else: signal.set_exception(completed.exception()) for f in futures: f.add_done_callback(chain_result) return signal def wait_first(futures): return _wait_first(set(futures)) def as_completed(futures): remaining = set(futures) while 1: if not remaining: break yield _wait_first(remaining) @task def load_url_async(url) return url, (yield urllib.urlopen_async(url)).read() @task def example(urls): for get_next_page in as_completed(load_url_async(url) for url in urls): try: url, data = yield get_next_page except Exception as exc: print("Something broke: {}".format(exc)) else: print("Loaded {} bytes from {!r}".format(len(data), url)) There's no scheduler instruction, there's just Guido's core API concept: the only thing a tasklet is allowed to yield is a Future object, and the step of registering tasks to be run is *always* done via an explicit call to the event loop rather than via the "yield" channel. The yield channel is only used to say "wait for this operation now". What this approach means is that, to get sensible iteration, all you need is an ordinary iterator that produces future objects instead of reporting the results directly. You can then either call this operator with "yield from", in which case the individual results will be ignored and the first failure will abort the iteration, *or* you can invoke it with an explicit for loop, which will be enough to give you control over how exceptions are handled by means of an ordinary try/except block rather than a complicated exception chain. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Tue, 16 Oct 2012 17:48:24 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
def _wait_first(futures): # futures must be a set, items will be removed as they complete signal = Future() def chain_result(completed): futures.remove(completed) if completed.cancelled(): signal.cancel() signal.set_running_or_notify_cancel() elif completed.done(): signal.set_result(completed.result()) else: signal.set_exception(completed.exception()) for f in futures: f.add_done_callback(chain_result) return signal
def wait_first(futures): return _wait_first(set(futures))
def as_completed(futures): remaining = set(futures) while 1: if not remaining: break yield _wait_first(remaining)
@task def load_url_async(url) return url, (yield urllib.urlopen_async(url)).read()
@task def example(urls): for get_next_page in as_completed(load_url_async(url) for url in urls): try: url, data = yield get_next_page except Exception as exc: print("Something broke: {}".format(exc)) else: print("Loaded {} bytes from {!r}".format(len(data), url))
Your example looks rather confusing to me. There are a couple of things I don't understand: - why does load_url_async return something instead of yielding it? - how does overlapping of reads happen? you seem to consider that a read() will be non-blocking once the server starts responding to your request, which is only true if the response is small (or you have a very fast connection to the server). - if read() is really non-blocking, why do you yield get_next_page? What does that achieve? Actually, what is yielding a tuple supposed to achieve at all? - where is control transferred over to the scheduler? it seems it's only in get_next_page, while I would expect it to be transferred in as_completed as well. Regards Antoine.

On Tue, Oct 16, 2012 at 7:43 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Your example looks rather confusing to me. There are a couple of things I don't understand:
- why does load_url_async return something instead of yielding it?
It yields *and* returns, that's the way Guido's API works (as I understand it). However, some of the other stuff was just plain mistakes in my example. Background (again, as I understand it, and I'm sure Guido will correct me if I'm wrong. So, if you think this sounds crazy, *please wait until Guido clarifies* before worrying too much about it): - the "@task" decorator is the part that knows how to interface generators with the event loop (just as @contextmanager adapts between generators and with statements). I believe it handles these things: - when you call it, it creates the generator object and calls next() to advance it to the first yield point - this initial call returns a Future that will fire only when the entire *task* is complete - if a Future is yielded by the underlying generator, the task wrapper adds the appropriate callback to ensure results are pushed back into the underlying generator on completion of the associated operation - when one of these callbacks fires, the generator is advanced and a yielded Future is processed in the same fashion - if at any point the generator finishes instead of yielding another Future, then the callback will call the appropriate notification method on the originally *returned* Future - yielding anything other than a Future from a tasklet is not permitted - it's the IO operations themselves that know how to kick off operations and register the appropriate callbacks with the event loop to get the Future to be triggered - The Future object API is documented in concurrent.futures: http://docs.python.org/py3k/library/concurrent.futures#future-objects I've now posted this example as a gist (https://gist.github.com/3898874), so it should be a easier to read over there. However, I've included it inline below as well. - This first part in my example is a helper function to wait for any one of a set of Futures to be signalled and help keep track of which ones we're still waiting for def _wait_first(futures): # futures must be a set as items will be removed as they complete # we create a signalling future to return to our caller. We will copy # the result of the first future to complete to this signalling future signal = Future() def copy_result(completed): # We ignore every callback after the first one if signal.done(): return # Keep track of which ones have been processed across multiple calls futures.remove(completed) # It would be nice if we could also remove our callback from all the other futures at # this point, but the Future API doesn't currently allow that # Now we pass the result of this future through to our signalling future if completed.cancelled(): signal.cancel() signal.set_running_or_notify_cancel() else: try: result = completed.result() except Exception as exc: signal.set_exception(exc) else: signal.set_result(result) # Here we hook our signalling future up to all our actual operations # If any of them are already complete, then the callback will fire immediately # and we're OK with that for f in futures: f.add_done_callback(copy_result) # And, for our signalling future to be useful, the caller must be able to access it return signal - This is just a public version of the above helper that works with arbitrary iterables: def wait_first(futures): # Helper needs a real set, so we give it one # Also makes sure all operations start immediately when passed a generator return _wait_first(set(futures)) - This is the API I'm most interested in, as it's the async equivalent of http://docs.python.org/py3k/library/concurrent.futures#concurrent.futures.as..., which powers this URL retrieval example: http://docs.python.org/py3k/library/concurrent.futures#threadpoolexecutor-ex... # Note that this is an *ordinary iterator*, not a tasklet def as_completed(futures): # We ensure all the operations have started, and get ourselves a set to work with remaining = set(futures) while remaining: # The trick here is that we *don't yield the original futures directly* # Instead, we yield yield _wait_first(remaining) And now a more complete, heavily commented, version of the example: # First, a tasklet for loading a single page @task def load_url_async(url) # The async URL open operation does three things: # 1. kicks off the connection process # 2. registers a callback with the event handler that will signal a Future object when IO is complete # 3. returns the future object # We then *yield* the Future object, at which point the task decorator takes over and registers a callback # with the *Future* object to resume this generator with the *result* that was passed to the Future object conn = yield urllib.urlopen_async(url) # We assume "conn.read()" is defined in such a way that it allows both "read everything at once" usage *and* a # usage where you read the individual bits of data as they arrive like this: # for wait_for_chunk in conn.read(): # chunk = yield wait_for_chunk # The secret is that conn.read() would be an *ordinary generator* in that case rather than a tasklet. # You could also do a version that *only* supported complete reads, in which case the "from" wouldn't be needed data = yield from conn.read() # We return both the URL *and* the data, so our caller doesn't have to keep track of which url the data is for return url, data # And now the payoff: defining a tasklet to read a bunch of URLs in parallel, processing them in the order of loading rather than the order of requesting them or having to wait until the slowest load completes before doing anything @task def example(urls): # We define the tasks we want to run based on the given URLs # This results in an iterable of Future objects that will fire when # the associated page has been read completely tasks = (load_url_async(url) for url in urls) # And now we use our helper iterable to run things in parallel # and get access to the results as they complete for wait_for_page in as_completed(tasks): try: url, data = yield wait_for_page except Exception as exc: print("Something broke for {!r} ({}: {})".format(url, type(exc), exc)) else: print("Loaded {} bytes from {!r}".format(len(data), url)) # The real kicker here? Replace "yield wait_for_page" with "wait_for_page.result()" and you have the equivalent concurrent.futures code. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

It yields *and* returns, that's the way Guido's API works (as I understand it).
I can't speak for Guido obviously, but you've certainly described what we came up with perfectly (http://pastebin.com/ndS53Cd8, the _Awaiter class starts on line 93).
# The real kicker here? Replace "yield wait_for_page" with "wait_for_page.result()" and you have the equivalent concurrent.futures code.
Basically, the task/tasklet/async decorator aggregates the futures from inside the wrapped method and exposes a single future to the caller. Your example doesn't even need a scheduler or event loop, and we found that all the event loop was really doing was running the callbacks in a certain thread/context/equivalent. And because there is a future coming out of every call, the user can choose when to stop using tasklets and go back to using plain old futures (or whatever subclasses have been used).

On Tue, 16 Oct 2012 14:21:10 +0000 Steve Dower <Steve.Dower@microsoft.com> wrote:
It yields *and* returns, that's the way Guido's API works (as I understand it).
I can't speak for Guido obviously, but you've certainly described what we came up with perfectly (http://pastebin.com/ndS53Cd8, the _Awaiter class starts on line 93).
# The real kicker here? Replace "yield wait_for_page" with "wait_for_page.result()" and you have the equivalent concurrent.futures code.
Basically, the task/tasklet/async decorator aggregates the futures from inside the wrapped method and exposes a single future to the caller. Your example doesn't even need a scheduler or event loop, and we found that all the event loop was really doing was running the callbacks in a certain thread/context/equivalent.
I'm sure doing concurrent I/O will require an event loop, unless you use threads under the hood... Regards Antoine. -- Software development and contracting: http://pro.pitrou.net

I'm sure doing concurrent I/O will require an event loop, unless you use threads under the hood...
Polling I/O will require some sort of loop, yes, but I/O that triggers a callback at the OS level (such as ReadFileEx and WriteFileEx on Windows) doesn't need it. Of course, without an event loop you still need to wait on the future - for polling I/O you could return a subclassed future where waiting starts the polling loop if there isn't a better event loop available. My view is that the most helpful thing to have in the standard is a way for any code to find and interact with an event loop - if we can discover a scheduler/context/loop/whatever and use its commands for "run this callable as soon as you can" and "run this callable when this condition is true" then we can have portable support for polling or event-based I/O (as well as being able to handle other thread-sensitive code such as in UIs). For optimal support, you'll need to have very close coupling between the scheduler and the asynchronous operations. This can be built on top of the portable support, but aiming for optimal support initially is a good way to make this API painful to use and more likely to be ignored.

On Tue, 16 Oct 2012 16:31:55 +0000 Steve Dower <Steve.Dower@microsoft.com> wrote:
I'm sure doing concurrent I/O will require an event loop, unless you use threads under the hood...
Polling I/O will require some sort of loop, yes, but I/O that triggers a callback at the OS level (such as ReadFileEx and WriteFileEx on Windows) doesn't need it.
Well, how do you plan for that callback to execute Python code? Regards Antoine.

Well, how do you plan for that callback to execute Python code?
IMO, that is the most important question in all of this discussion. With any I/O some waiting is required - there must be a point where the application is not doing anything other than waiting for the I/O to complete, regardless of whether a loop is used or not. (Ideally the I/O is already complete by the time we start waiting.) The callbacks in the particular examples require a thread to be in an alertable wait state, which is basically equivalent to select(), though a little less discriminatory (as in, ANY I/O callback can interrupt an alertable wait). In my view, these callbacks should be 'leaving a message' for the main program to run a particular function when it next has a chance. Like an interrupt handler, the aim is to do the minimum amount of work and then get out of the way. Having a context (or event loop, message loop or whatever you want to call it) as I described in my last email lets us do the minimum amount of work. I posted our implementation of such a context earlier and Dino posted an example/recipe for using the concept with an existing event loop (Tcl). So while I said we don't _need_ an event loop, that relies on the asynchronous operations being on a separate thread or otherwise not requiring the current thread to pay any attention to them, AND assumes that the continuations are agile and can be run on any thread (or in any process, or whatever granularity you are working at). I believe some way of getting code running back where it started from is essential, and this is most easily done with a loop.

On Tue, Oct 16, 2012 at 1:04 PM, Steve Dower <Steve.Dower@microsoft.com> wrote:
Well, how do you plan for that callback to execute Python code?
IMO, that is the most important question in all of this discussion.
With any I/O some waiting is required - there must be a point where the application is not doing anything other than waiting for the I/O to complete, regardless of whether a loop is used or not. (Ideally the I/O is already complete by the time we start waiting.) The callbacks in the particular examples require a thread to be in an alertable wait state, which is basically equivalent to select(), though a little less discriminatory (as in, ANY I/O callback can interrupt an alertable wait).
In my view, these callbacks should be 'leaving a message' for the main program to run a particular function when it next has a chance. Like an interrupt handler, the aim is to do the minimum amount of work and then get out of the way.
I like this model as well. However, I recognize some problems with it. If we don't kick whatever handles the callback and result immediately, we are essentially re-introducing pre-emptive scheduling. If TaskA is waiting on the result of TaskB, and when TaskB finishes we say "OK, but we need to go let TaskC do something before TaskA is given that result" then we leave room for C to break things, modify state, and generally act in a less-than-determinable way. I really *like* this model better, I just don't know the best way to reconcile this problem.
Having a context (or event loop, message loop or whatever you want to call it) as I described in my last email lets us do the minimum amount of work. I posted our implementation of such a context earlier and Dino posted an example/recipe for using the concept with an existing event loop (Tcl).
So while I said we don't _need_ an event loop, that relies on the asynchronous operations being on a separate thread or otherwise not requiring the current thread to pay any attention to them, AND assumes that the continuations are agile and can be run on any thread (or in any process, or whatever granularity you are working at). I believe some way of getting code running back where it started from is essential, and this is most easily done with a loop.
_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
-- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy

Calvin Spealman wrote:
If we don't kick whatever handles the callback and result immediately, we are essentially re-introducing pre-emptive scheduling. If TaskA is waiting on the result of TaskB, and when TaskB finishes we say "OK, but we need to go let TaskC do something before TaskA is given that result" then we leave room for C to break things, modify state, and generally act in a less-than-determinable way.
I don't see how the risk of this is any higher than the risk that some other task D gets run while task A is waiting and messes something up. Ultimately you have to trust your tasks to behave themselves. -- Greg

On Tue, Oct 16, 2012 at 12:31 PM, Steve Dower <Steve.Dower@microsoft.com> wrote:
I'm sure doing concurrent I/O will require an event loop, unless you use threads under the hood...
Polling I/O will require some sort of loop, yes, but I/O that triggers a callback at the OS level (such as ReadFileEx and WriteFileEx on Windows) doesn't need it.
Of course, without an event loop you still need to wait on the future - for polling I/O you could return a subclassed future where waiting starts the polling loop if there isn't a better event loop available.
What if the event poll was just inside a task, not requiring any loop in the scheduler, or even knowledge by the scheduler, in any way? An extremely rudimentary version: class Selector(object): def __init__(self): self.r = [] self.w = [] self.x = [] self.futures = {} def add(self, t, fd, future): self.futures[fd] = future getattr(self, t).append(fd) def __iter__(self): return self def __next__(self): r = [fd for fd,future in self.r] w = [fd for fd,future in self.w] x = [fd for fd,future in self.x] r, w, x = select(r, w, x) for fd in chain(r, w, x): self.futures.pop(fd).set_result(fd) for fd in r: self.r.remove(fd) for fd in w: self.w.remove(fd) for fd in x: self.x.remove(fd) This, if even to the scheduler, would handle polling completely outside the scheduler, which makes it easier to mix and match event loops you need to use in a single project. I know I probably got details wrong.
My view is that the most helpful thing to have in the standard is a way for any code to find and interact with an event loop - if we can discover a scheduler/context/loop/whatever and use its commands for "run this callable as soon as you can" and "run this callable when this condition is true" then we can have portable support for polling or event-based I/O (as well as being able to handle other thread-sensitive code such as in UIs).
For optimal support, you'll need to have very close coupling between the scheduler and the asynchronous operations. This can be built on top of the portable support, but aiming for optimal support initially is a good way to make this API painful to use and more likely to be ignored.
_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
-- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy

What if the event poll was just inside a task, not requiring any loop in the scheduler, or even knowledge by the scheduler, in any way?
I agree, every task can handle all the asynchrony within it and just expose a single 'completed' notification (a Future or similar) to its caller. This is the portable solution - it is going to be less than optimal in some cases, but is much more composable and extensible. As a Python developer, I like the model of "I call this function normally and it gives me a Future to let me know when it's done but I don't really know how it's doing it." (Incidentally, I like it as a C# and C++ developer too.)

Nick Coghlan wrote:
# Note that this is an *ordinary iterator*, not a tasklet def as_completed(futures): # We ensure all the operations have started, and get ourselves a set to work with remaining = set(futures) while remaining: # The trick here is that we *don't yield the original futures directly* # Instead, we yield yield _wait_first(remaining)
I've just figured out how your as_completed() thing works, and realised that it's *not* a general solution to the suspendable-iterator problem. You're making use of the fact that you know *how many* values there will be ahead of time, even if you don't know what they are yet. In general this won't be the case. I don't think there is any trick that will allow a for-loop to be used in the general case, because in order for an iterator to be suspendable, the call to next() would need to be made using yield-from, and it's hidden inside the for-loop implementation. I know you probably weren't intending as_completed() to be a solution to the general suspendable-iterator problem. I just wanted to record my thoughts on this. -- Greg

On Wed, Oct 17, 2012 at 6:27 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Nick Coghlan wrote:
# Note that this is an *ordinary iterator*, not a tasklet def as_completed(futures): # We ensure all the operations have started, and get ourselves a set to work with remaining = set(futures) while remaining: # The trick here is that we *don't yield the original futures directly* # Instead, we yield yield _wait_first(remaining)
I've just figured out how your as_completed() thing works, and realised that it's *not* a general solution to the suspendable-iterator problem. You're making use of the fact that you know *how many* values there will be ahead of time, even if you don't know what they are yet.
In general this won't be the case. I don't think there is any trick that will allow a for-loop to be used in the general case, because in order for an iterator to be suspendable, the call to next() would need to be made using yield-from, and it's hidden inside the for-loop implementation.
Yeah, that's what lets me get away with not passing the sent results back down into the iterator (it can figure out from the original arguments when it needs to stop). It gets trickier if you want to terminate the iteration based on the result of an asynchronous operation. For example, here's a very simplistic way you could apply the concept of "yield a future to be handled in the loop body" to the operation of continuously reading binary data from a connection until EOF is received: def read(self): """This knows how to start an IO operation such the future will fire on completion""" future = ... return future # Again, notice this is *not* a tasklet, it's an ordinary iterator that produces Future objects def readall(self): """This can be used in two modes - as an iterator or as a coroutine. As a coroutine: data = yield from conn.readall() As an iterator: for wait_for_chunk in conn.readall(): try: chunk = yield wait_for_chunk except EOFError: break Obviously, the coroutine mode is far more convenient, but you *can* override the default accumulator behaviour if you want/need to by waiting on the individual futures explicitly. However, in this case, you lose the automatic loop termination behaviour, so, you may as well implement the underlying loop explicitly: while 1: try: chunk = yield self.read() except EOFError: break """ output = io.BytesIO() while 1: try: data = yield self.read() except EOFError: break if data: # This check makes iterator mode possible output.write(data) return output.getvalue() Impedance matching in a way that allows the exception handling to be factored out as well as the iteration step is a *lot* trickier, since you need to bring context managers into play if termination is signalled by an exception: # This version produces context managers rather than producing futures directly, and thus can't be # used directly as a coroutine def read_chunks(self): finished = False @contextmanager def handle_chunk(): nonlocal finished data = b'' try: data = yield self.read() except EOFError: finished = True return data while not finished: yield handle_chunk() # Usage for handle_chunk in conn.read_chunks(): with handle_chunk as wait_for_chunk: chunk = yield from wait_for_chunk # We end up doing a final "extra" iteration with chunk = b'' # So we'd likely need to guard with an "if chunk:" or "if not chunk: continue" # which again means we're not getting much value out of using the iterator Using an explicit "add_done_callback" doesn't help much, as you still have to deal with the exception being thrown back in to your generator. I know Guido doesn't want people racing off and designing new syntax for asynchronous iteration, but I'm not sure it's going to be possible to avoid it if we want a clean approach to "forking" the results of asynchronous calls between passing them down into a coroutine (to decide whether or not to terminate iteration) and binding them to a local variable (to allow local processing in the loop body). Compare the arcane incantations above to something like (similar to suggestions previously made by Christian Heimes): def read_chunks(self): """Designed for use as an asynchronous iterator""" while 1: try: yield self.read() except EOFError: break # Usage for chunk in yield from conn.read_chunks(): ... The idea here would be that whereas "for chunk in (yield from conn.read_chunks()):" runs the underlying coroutine to completion and then iterates over the return value, the version without the parentheses would effectively "tee" the values being sent back, *first* sending them to the underlying coroutine (to decide whether or not iteration should continue and to get the value to be yielded at the start of the next iteration) and then, if that doesn't raise StopIteration, binding them to the local variable and proceeding to execution of the loop body. All that said, I still like Guido's concept that the core asynchronous API is *really* future objects, just as it already is in the concurrent.futures module. The @task decorator and yielding future objects to that decorator is then just nice syntactic sugar for hooking generators up to the "add_done_callback" API of future objects. It's completely independent of the underlying event loop and/or asynchronous IO interfaces - those interfaces are about setting things up to invoke the set_* methods of the returned future objects correctly, just as they are with the Executor API in concurrent.futures.
I know you probably weren't intending as_completed() to be a solution to the general suspendable-iterator problem.
Right, I just wanted to be sure that *that particular use case* of waiting for a collection of futures and processing them in completion order could be handled in terms of Guido's API *without* needing any extra magic. The "iterate over data chunks until EOFError is raised" is a better example for highlighting the "how do you write an asynchronous iterator?" problem when it comes to generators-as-coroutines. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Mon, Oct 15, 2012 at 10:39 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Nick Coghlan wrote:
(this is why I disagree with Greg that "yield from" can serve as the one true API - it doesn't handle partial iteration, and it doesn't handle pre- or post- processing around the suspension points while iterating).
I'm aware of the iteration problem, but I'm not convinced that the convolutions necessary to make it possible to use a for-loop for this are worth the bother, as opposed to simply accepting that you can't use the for statement in this situation, and using some other kind of loop.
In any case, even if we decide to provide a scheduler instruction to enable using for-loops on suspendable iterators somehow, it doesn't follow that we should use scheduler instructions for anything *else*.
I don't see how we could ever have a for-loop that yields on every iteration step. The for-loop never uses yield. Thus there can be no direct equivalent to as_completed() in the PEP 380 or PEP 342 coroutine worlds.
I would consider such a scheduler instruction to be a stopgap measure until we can find a better solution -- just as yield-from is a better solution than using "call" and "return" scheduler instructions.
I can already see the armchair language designers race to propose syntax the puts a yield keyword in the for-loop syntax at a point where it is currently not allowed. Let's nip that in the bud and focus on something that can work with Python 3.3. -- --Guido van Rossum (python.org/~guido)

Calvin Spealman wrote:
A "sane stack trace" only makes sense if we assume that tasks "call" each other in the same kind of call tree that synchronous code flows in, and I don't think that is necessarily the case.
No, but often it *is* the case, and in those cases we would like to get a traceback that correctly reflects the chain of calls.
There are cases when one task might want to end before tasks it as "called" are complete, and if we use yield-from this is *impossible* but it is very useful.
That depends on what you mean by "use yield-from". It's true that yield-from *on its own* can't achieve the effect of spawning concurrent subtasks; other mechanisms will need to be brought to bear at some point. But there's no reason a solution involving those other mechanisms can't be encapsulated in a library function that you invoke using yield-from. I've posted a couple of examples of how a par() function which does that might be written.
yield-from semantics won't allow a called task to continue, if needed, after the calling task itself has completed.
You seem to be imagining that more is being claimed about the abilities of yield-from than is actually being claimed. Yield-from is just a procedure call; the important thing is what the called procedure does. One of the things it can do is invoke a scheduler primitive that spawns an independent task. In my example scheduler, this is spelled scheduler.schedule(task). This is not a yield-from call, it's just an ordinary call. It adds the given generator to the list of ready tasks, so that it will get run when its chance comes around. Meanwhile, the calling task carries on. -- Greg

I'm still catching up to this thread, but we've been investigating Win 8 support for Python and Win 8 has a very asynchronous API design and so we've been interested in much the same space. We've actually come up with an example of the @task decorator (we called it @async) which is built around using yield + the ability to return from generators added in Python 3.3. Our version of this is also based around futures so that an @async API will return a future. The big difference here might be that we always return a future from a call rather than yielding it up the stack. So our API works with just simple yields rather than yield froms. This is what a simple usage of the API looks like: from concurrent.futures import ThreadPoolExecutor from urllib.request import urlopen executor = ThreadPoolExecutor(max_workers=5) def load_url(url): return urlopen(_url).read() @async def get_image_async(url): buffer = yield executor.submit(load_url, url) return Image(buffer) def main(image_uri): img_future = get_image_async(image_uri) # perform other tasks while the image is downloading img = img_future.result() main("http://www.python.org/images/python-logo.gif") This example us just using the existing thread pool to run the actual I/O but this will work with anything that will return a future. So inside of an async method anything which is yielded should be a future. The decorator will then attach a callback which will send the result of the future back into the generator, so the "buffer = " line gets the result of the future. Finally the function completes and the future returned from calling get_image_async will have its value set to Image when the StopIteration exception is raised with the return value. Because we're interested in the GUI side of things here we've also wired this up into Tk so that we can experiment with an existing GUI framework, and I've included the source for the context there. Our thinking here is that different contexts can be created depending upon the framework which you're running in and that the context makes sure the code is running in the right spot, in this case getting back to the GUI thread after an async operation has been completed. The big outstanding item we're still working through is I/O, but we think the contexts help here too. We're still not quite sure how polling I/O will work, but with the contexts if there's a single thread polling for I/O then the context will get us off the I/O thread and let the polling continue. We are currently thinking that there will need to be a polling thread which handles all of the I/Os, and there could potentially be more than one of these if different libraries aren't cooperating on sharing a single thread. Here's the code plus the demo Tk app (you'll need your own Holmes.txt file for the sample app to run): Contexts.py: http://pastebin.com/ndS53Cd8 Tk context: http://pastebin.com/FuZwc1Ur Tk app: http://pastebin.com/Fm5wMXpN Hardwork.py: http://pastebin.com/nMMytdTG -----Original Message----- From: Python-ideas [mailto:python-ideas-bounces+dinov=microsoft.com@python.org] On Behalf Of Calvin Spealman Sent: Monday, October 15, 2012 7:16 AM To: Nick Coghlan Cc: python-ideas@python.org Subject: Re: [Python-ideas] The async API of the future: yield-from On Mon, Oct 15, 2012 at 9:48 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Mon, Oct 15, 2012 at 10:31 PM, Calvin Spealman <ironfroggy@gmail.com> wrote:
Currently, "with yield expr:" is not valid syntax, surprisingly.
It's not that surprising, it's the general requirement that yield expressions must be enclosed in parentheses except when used standalone or in a simple assignment statement.
"with (yield expr):" is valid syntax though, so I'm reluctant to endorse doing anything substantially different if the parentheses are omitted.
Silly oversight on my part, and I agree that the parens shouldn't make the difference in meaning.
I think the combination of "yield from" to delegate control (including exception handling) completely to a subgenerator and "context manager + for loop + explicit yield" when an operation needs to yield multiple times and the exception handling behaviour should be left to the caller (as in the "as_completed" case) should cover the necessary behaviours.
I'm still -1 on delegating control to subgenerators with yield-from, versus having the scheduler just deal with them directly. I think it is far less flexible. I would still like to see a less confusing "with yield expr:" by simply allowing it without parens, but no special meaning. I think it would be really useful in coroutines. with yield collect() as tasks: yield task1() yield task2() results = yield tasks
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
-- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

Wow, sounds very similar to NDB's approach! Please do check out NDB's tasklets and event loop: http://code.google.com/p/appengine-ndb-experiment/source/browse/ndb/tasklets... On Mon, Oct 15, 2012 at 10:24 AM, Dino Viehland <dinov@microsoft.com> wrote:
I'm still catching up to this thread, but we've been investigating Win 8 support for Python and Win 8 has a very asynchronous API design and so we've been interested in much the same space. We've actually come up with an example of the @task decorator (we called it @async) which is built around using yield + the ability to return from generators added in Python 3.3. Our version of this is also based around futures so that an @async API will return a future. The big difference here might be that we always return a future from a call rather than yielding it up the stack. So our API works with just simple yields rather than yield froms. This is what a simple usage of the API looks like:
from concurrent.futures import ThreadPoolExecutor from urllib.request import urlopen
executor = ThreadPoolExecutor(max_workers=5)
def load_url(url): return urlopen(_url).read()
@async def get_image_async(url): buffer = yield executor.submit(load_url, url) return Image(buffer)
def main(image_uri): img_future = get_image_async(image_uri) # perform other tasks while the image is downloading img = img_future.result()
main("http://www.python.org/images/python-logo.gif")
This example us just using the existing thread pool to run the actual I/O but this will work with anything that will return a future. So inside of an async method anything which is yielded should be a future. The decorator will then attach a callback which will send the result of the future back into the generator, so the "buffer = " line gets the result of the future. Finally the function completes and the future returned from calling get_image_async will have its value set to Image when the StopIteration exception is raised with the return value.
Because we're interested in the GUI side of things here we've also wired this up into Tk so that we can experiment with an existing GUI framework, and I've included the source for the context there. Our thinking here is that different contexts can be created depending upon the framework which you're running in and that the context makes sure the code is running in the right spot, in this case getting back to the GUI thread after an async operation has been completed.
The big outstanding item we're still working through is I/O, but we think the contexts help here too. We're still not quite sure how polling I/O will work, but with the contexts if there's a single thread polling for I/O then the context will get us off the I/O thread and let the polling continue. We are currently thinking that there will need to be a polling thread which handles all of the I/Os, and there could potentially be more than one of these if different libraries aren't cooperating on sharing a single thread.
Here's the code plus the demo Tk app (you'll need your own Holmes.txt file for the sample app to run):
Contexts.py: http://pastebin.com/ndS53Cd8 Tk context: http://pastebin.com/FuZwc1Ur Tk app: http://pastebin.com/Fm5wMXpN Hardwork.py: http://pastebin.com/nMMytdTG
-----Original Message----- From: Python-ideas [mailto:python-ideas-bounces+dinov=microsoft.com@python.org] On Behalf Of Calvin Spealman Sent: Monday, October 15, 2012 7:16 AM To: Nick Coghlan Cc: python-ideas@python.org Subject: Re: [Python-ideas] The async API of the future: yield-from
On Mon, Oct 15, 2012 at 9:48 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Mon, Oct 15, 2012 at 10:31 PM, Calvin Spealman <ironfroggy@gmail.com> wrote:
Currently, "with yield expr:" is not valid syntax, surprisingly.
It's not that surprising, it's the general requirement that yield expressions must be enclosed in parentheses except when used standalone or in a simple assignment statement.
"with (yield expr):" is valid syntax though, so I'm reluctant to endorse doing anything substantially different if the parentheses are omitted.
Silly oversight on my part, and I agree that the parens shouldn't make the difference in meaning.
I think the combination of "yield from" to delegate control (including exception handling) completely to a subgenerator and "context manager + for loop + explicit yield" when an operation needs to yield multiple times and the exception handling behaviour should be left to the caller (as in the "as_completed" case) should cover the necessary behaviours.
I'm still -1 on delegating control to subgenerators with yield-from, versus having the scheduler just deal with them directly. I think it is far less flexible.
I would still like to see a less confusing "with yield expr:" by simply allowing it without parens, but no special meaning. I think it would be really useful in coroutines.
with yield collect() as tasks: yield task1() yield task2() results = yield tasks
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
-- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
-- --Guido van Rossum (python.org/~guido)

They look remarkably similar. The biggest difference I see is that NDB appears to be using an event loop to keep the futures running while we're using add_done_callback (on the yielded futures) to continue stepping the generator function along. So there's not necessary an event loop in our case, and in fact the default context always just executes things synchronously. But frameworks can replace the default context so that work is posted into an event loop of some form. -----Original Message----- From: gvanrossum@gmail.com [mailto:gvanrossum@gmail.com] On Behalf Of Guido van Rossum Sent: Monday, October 15, 2012 12:34 PM To: Dino Viehland Cc: ironfroggy@gmail.com; Nick Coghlan; python-ideas@python.org Subject: Re: [Python-ideas] The async API of the future: yield-from Wow, sounds very similar to NDB's approach! Please do check out NDB's tasklets and event loop: http://code.google.com/p/appengine-ndb-experiment/source/browse/ndb/tasklets... On Mon, Oct 15, 2012 at 10:24 AM, Dino Viehland <dinov@microsoft.com> wrote:
I'm still catching up to this thread, but we've been investigating Win 8 support for Python and Win 8 has a very asynchronous API design and so we've been interested in much the same space. We've actually come up with an example of the @task decorator (we called it @async) which is built around using yield + the ability to return from generators added in Python 3.3. Our version of this is also based around futures so that an @async API will return a future. The big difference here might be that we always return a future from a call rather than yielding it up the stack. So our API works with just simple yields rather than yield froms. This is what a simple usage of the API looks like:
from concurrent.futures import ThreadPoolExecutor from urllib.request import urlopen
executor = ThreadPoolExecutor(max_workers=5)
def load_url(url): return urlopen(_url).read()
@async def get_image_async(url): buffer = yield executor.submit(load_url, url) return Image(buffer)
def main(image_uri): img_future = get_image_async(image_uri) # perform other tasks while the image is downloading img = img_future.result()
main("http://www.python.org/images/python-logo.gif")
This example us just using the existing thread pool to run the actual I/O but this will work with anything that will return a future. So inside of an async method anything which is yielded should be a future. The decorator will then attach a callback which will send the result of the future back into the generator, so the "buffer = " line gets the result of the future. Finally the function completes and the future returned from calling get_image_async will have its value set to Image when the StopIteration exception is raised with the return value.
Because we're interested in the GUI side of things here we've also wired this up into Tk so that we can experiment with an existing GUI framework, and I've included the source for the context there. Our thinking here is that different contexts can be created depending upon the framework which you're running in and that the context makes sure the code is running in the right spot, in this case getting back to the GUI thread after an async operation has been completed.
The big outstanding item we're still working through is I/O, but we think the contexts help here too. We're still not quite sure how polling I/O will work, but with the contexts if there's a single thread polling for I/O then the context will get us off the I/O thread and let the polling continue. We are currently thinking that there will need to be a polling thread which handles all of the I/Os, and there could potentially be more than one of these if different libraries aren't cooperating on sharing a single thread.
Here's the code plus the demo Tk app (you'll need your own Holmes.txt file for the sample app to run):
Contexts.py: http://pastebin.com/ndS53Cd8 Tk context: http://pastebin.com/FuZwc1Ur Tk app: http://pastebin.com/Fm5wMXpN Hardwork.py: http://pastebin.com/nMMytdTG
-----Original Message----- From: Python-ideas [mailto:python-ideas-bounces+dinov=microsoft.com@python.org] On Behalf Of Calvin Spealman Sent: Monday, October 15, 2012 7:16 AM To: Nick Coghlan Cc: python-ideas@python.org Subject: Re: [Python-ideas] The async API of the future: yield-from
On Mon, Oct 15, 2012 at 9:48 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Mon, Oct 15, 2012 at 10:31 PM, Calvin Spealman <ironfroggy@gmail.com> wrote:
Currently, "with yield expr:" is not valid syntax, surprisingly.
It's not that surprising, it's the general requirement that yield expressions must be enclosed in parentheses except when used standalone or in a simple assignment statement.
"with (yield expr):" is valid syntax though, so I'm reluctant to endorse doing anything substantially different if the parentheses are omitted.
Silly oversight on my part, and I agree that the parens shouldn't make the difference in meaning.
I think the combination of "yield from" to delegate control (including exception handling) completely to a subgenerator and "context manager + for loop + explicit yield" when an operation needs to yield + multiple times and the exception handling behaviour should be left to the caller (as in the "as_completed" case) should cover the necessary behaviours.
I'm still -1 on delegating control to subgenerators with yield-from, versus having the scheduler just deal with them directly. I think it is far less flexible.
I would still like to see a less confusing "with yield expr:" by simply allowing it without parens, but no special meaning. I think it would be really useful in coroutines.
with yield collect() as tasks: yield task1() yield task2() results = yield tasks
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
-- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
-- --Guido van Rossum (python.org/~guido)

I gave something like this a go a while ago: https://bitbucket.org/anacrolix/green380 "Coroutines" yield events or futures as Nick put them from the top, and the scheduler at the bottom manages events and scheduling. There are a few things I took away from this attempt: 1) Explicit yield a la PEP380 requires syntactical changes *everywhere*. 2) Python's dynamic typing means that neglecting to "yield from" gives you broken code, and Python won't help you here. Add to this that you now have a 380, and "normal synchronous" form of most interfaces and the caller must know what kind is used at all times. 3) Concurrency is nice, but it requires language-level support, and proper parallelism to really shine. The "C way" of doing things is already so heavily ingrained in Python, an entirely new standard library and interpreter that breaks C compatibility is really the only way to proceed, and this certainly isn't worth it just to write code with "yield from" littered on every line.

On Mon, Oct 15, 2012 at 4:37 PM, Matt Joiner <anacrolix@gmail.com> wrote:
I gave something like this a go a while ago: https://bitbucket.org/anacrolix/green380
"Coroutines" yield events or futures as Nick put them from the top, and the scheduler at the bottom manages events and scheduling.
There are a few things I took away from this attempt:
1) Explicit yield a la PEP380 requires syntactical changes *everywhere*.
So does using PEP 342 style coroutines (yield Future instead of yield from).
2) Python's dynamic typing means that neglecting to "yield from" gives you broken code, and Python won't help you here. Add to this that you now have a 380, and "normal synchronous" form of most interfaces and the caller must know what kind is used at all times.
In NDB this is alleviated by insisting that the only thing you are allowed to yield is a Future. Anything else raises TypeError. But yes, the first few days when getting used to this style, you end up debugging this a few times.
3) Concurrency is nice, but it requires language-level support, and proper parallelism to really shine. The "C way" of doing things is already so heavily ingrained in Python, an entirely new standard library and interpreter that breaks C compatibility is really the only way to proceed, and this certainly isn't worth it just to write code with "yield from" littered on every line.
Here you're basically arguing for greenlets/gevent -- you're saying you just don't want to put up with the yields everywhere. But the popularity of Twisted and Tornado show that at least some people are willing to make even bigger sacrifices in order to be able to do async I/O efficiently -- i.e., to solve the C10K problem that Christian Tismer referred to (http://www.kegel.com/c10k.html, http://en.wikipedia.org/wiki/C10k_problem). There happen to be several problems with greenlets (even Christian Tismer said so, and included Stackless in the problem). The current effort is striving to help people solve it ith less effort than the async style Twisted and Tornado promote, while avoiding the problems with greenlets. -- --Guido van Rossum (python.org/~guido)

On Mon, 15 Oct 2012 19:19:29 -0700 Guido van Rossum <guido@python.org> wrote:
Here you're basically arguing for greenlets/gevent -- you're saying you just don't want to put up with the yields everywhere. But the popularity of Twisted and Tornado show that at least some people are willing to make even bigger sacrifices in order to be able to do async I/O efficiently -- i.e., to solve the C10K problem that Christian Tismer referred to (http://www.kegel.com/c10k.html, http://en.wikipedia.org/wiki/C10k_problem).
To be honest, one of the selling points of Twisted is not that it solves the C10k problem, it's that it's a comprehensive network programming toolkit. I'd bet many users of Twisted don't care that much about the single-thread event-loop approach, and don't have C10k-like problems. Regards Antoine. -- Software development and contracting: http://pro.pitrou.net

On Mon, Oct 15, 2012 at 2:45 PM, Dino Viehland <dinov@microsoft.com> wrote:
They look remarkably similar. The biggest difference I see is that NDB appears to be using an event loop to keep the futures running while we're using add_done_callback (on the yielded futures) to continue stepping the generator function along. So there's not necessary an event loop in our case, and in fact the default context always just executes things synchronously. But frameworks can replace the default context so that work is posted into an event loop of some form.
But do your Futures use threads? NDB doesn't. NDB's event loop doesn't know about Futures; however the @ndb.tasklet decorator does, and the Futures know about the event loop. When you wait for a Future, a callback is added to the Future that will resume the generator when it is done, and in order to run them, the Future passes its callbacks to the event loop to be run. --Guido
-----Original Message----- From: gvanrossum@gmail.com [mailto:gvanrossum@gmail.com] On Behalf Of Guido van Rossum Sent: Monday, October 15, 2012 12:34 PM To: Dino Viehland Cc: ironfroggy@gmail.com; Nick Coghlan; python-ideas@python.org Subject: Re: [Python-ideas] The async API of the future: yield-from
Wow, sounds very similar to NDB's approach! Please do check out NDB's tasklets and event loop: http://code.google.com/p/appengine-ndb-experiment/source/browse/ndb/tasklets...
On Mon, Oct 15, 2012 at 10:24 AM, Dino Viehland <dinov@microsoft.com> wrote:
I'm still catching up to this thread, but we've been investigating Win 8 support for Python and Win 8 has a very asynchronous API design and so we've been interested in much the same space. We've actually come up with an example of the @task decorator (we called it @async) which is built around using yield + the ability to return from generators added in Python 3.3. Our version of this is also based around futures so that an @async API will return a future. The big difference here might be that we always return a future from a call rather than yielding it up the stack. So our API works with just simple yields rather than yield froms. This is what a simple usage of the API looks like:
from concurrent.futures import ThreadPoolExecutor from urllib.request import urlopen
executor = ThreadPoolExecutor(max_workers=5)
def load_url(url): return urlopen(_url).read()
@async def get_image_async(url): buffer = yield executor.submit(load_url, url) return Image(buffer)
def main(image_uri): img_future = get_image_async(image_uri) # perform other tasks while the image is downloading img = img_future.result()
main("http://www.python.org/images/python-logo.gif")
This example us just using the existing thread pool to run the actual I/O but this will work with anything that will return a future. So inside of an async method anything which is yielded should be a future. The decorator will then attach a callback which will send the result of the future back into the generator, so the "buffer = " line gets the result of the future. Finally the function completes and the future returned from calling get_image_async will have its value set to Image when the StopIteration exception is raised with the return value.
Because we're interested in the GUI side of things here we've also wired this up into Tk so that we can experiment with an existing GUI framework, and I've included the source for the context there. Our thinking here is that different contexts can be created depending upon the framework which you're running in and that the context makes sure the code is running in the right spot, in this case getting back to the GUI thread after an async operation has been completed.
The big outstanding item we're still working through is I/O, but we think the contexts help here too. We're still not quite sure how polling I/O will work, but with the contexts if there's a single thread polling for I/O then the context will get us off the I/O thread and let the polling continue. We are currently thinking that there will need to be a polling thread which handles all of the I/Os, and there could potentially be more than one of these if different libraries aren't cooperating on sharing a single thread.
Here's the code plus the demo Tk app (you'll need your own Holmes.txt file for the sample app to run):
Contexts.py: http://pastebin.com/ndS53Cd8 Tk context: http://pastebin.com/FuZwc1Ur Tk app: http://pastebin.com/Fm5wMXpN Hardwork.py: http://pastebin.com/nMMytdTG
-----Original Message----- From: Python-ideas [mailto:python-ideas-bounces+dinov=microsoft.com@python.org] On Behalf Of Calvin Spealman Sent: Monday, October 15, 2012 7:16 AM To: Nick Coghlan Cc: python-ideas@python.org Subject: Re: [Python-ideas] The async API of the future: yield-from
On Mon, Oct 15, 2012 at 9:48 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Mon, Oct 15, 2012 at 10:31 PM, Calvin Spealman <ironfroggy@gmail.com> wrote:
Currently, "with yield expr:" is not valid syntax, surprisingly.
It's not that surprising, it's the general requirement that yield expressions must be enclosed in parentheses except when used standalone or in a simple assignment statement.
"with (yield expr):" is valid syntax though, so I'm reluctant to endorse doing anything substantially different if the parentheses are omitted.
Silly oversight on my part, and I agree that the parens shouldn't make the difference in meaning.
I think the combination of "yield from" to delegate control (including exception handling) completely to a subgenerator and "context manager + for loop + explicit yield" when an operation needs to yield + multiple times and the exception handling behaviour should be left to the caller (as in the "as_completed" case) should cover the necessary behaviours.
I'm still -1 on delegating control to subgenerators with yield-from, versus having the scheduler just deal with them directly. I think it is far less flexible.
I would still like to see a less confusing "with yield expr:" by simply allowing it without parens, but no special meaning. I think it would be really useful in coroutines.
with yield collect() as tasks: yield task1() yield task2() results = yield tasks
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
-- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
-- --Guido van Rossum (python.org/~guido)
-- --Guido van Rossum (python.org/~guido)

That's basically exactly the same as ours (I worked on it with Dino). We assume that yielded objects are futures and wire up the callback. I think the difference is that all the yielded futures are hidden within the decorator, which returns one main future to the caller. This may be slightly inefficient, but it also makes it far easier for end-users. An event loop (in our terminology, a 'context') is only necessary if you need to ensure that callbacks (in this case, the next step in the generator) is run in a certain context (such as a UI thread). Without one, calling an @async method simply gives you back a future that you can wait on. The most important part of PEP 380 for this approach is not yield from, but allowing return <expr> inside a generator. It makes the methods that much more natural. Probably the most important part is that we assume whatever context is available (through contexts.get_current()) has a post() method for scheduling a callback. Basically, we approached this as less of a "how do I run this asynchronously" problem and more of a "how do I run something after this finishes" problem. We also have some ideas about associating properties with futures in a way that lets the caller decide how to run continuations, so you can opt-out of coming back to the calling thread or provide a cancellation token of some sort. These aren't written up yet (obviously), but we've certainly considered it. Cheers, Steve ________________________________________ From: Python-ideas [python-ideas-bounces+steve.dower=microsoft.com@python.org] on behalf of Guido van Rossum [guido@python.org] Sent: Monday, October 15, 2012 6:17 PM To: Dino Viehland Cc: python-ideas@python.org Subject: Re: [Python-ideas] The async API of the future: yield-from On Mon, Oct 15, 2012 at 2:45 PM, Dino Viehland <dinov@microsoft.com> wrote:
They look remarkably similar. The biggest difference I see is that NDB appears to be using an event loop to keep the futures running while we're using add_done_callback (on the yielded futures) to continue stepping the generator function along. So there's not necessary an event loop in our case, and in fact the default context always just executes things synchronously. But frameworks can replace the default context so that work is posted into an event loop of some form.
But do your Futures use threads? NDB doesn't. NDB's event loop doesn't know about Futures; however the @ndb.tasklet decorator does, and the Futures know about the event loop. When you wait for a Future, a callback is added to the Future that will resume the generator when it is done, and in order to run them, the Future passes its callbacks to the event loop to be run. --Guido
-----Original Message----- From: gvanrossum@gmail.com [mailto:gvanrossum@gmail.com] On Behalf Of Guido van Rossum Sent: Monday, October 15, 2012 12:34 PM To: Dino Viehland Cc: ironfroggy@gmail.com; Nick Coghlan; python-ideas@python.org Subject: Re: [Python-ideas] The async API of the future: yield-from
Wow, sounds very similar to NDB's approach! Please do check out NDB's tasklets and event loop: http://code.google.com/p/appengine-ndb-experiment/source/browse/ndb/tasklets...
On Mon, Oct 15, 2012 at 10:24 AM, Dino Viehland <dinov@microsoft.com> wrote:
I'm still catching up to this thread, but we've been investigating Win 8 support for Python and Win 8 has a very asynchronous API design and so we've been interested in much the same space. We've actually come up with an example of the @task decorator (we called it @async) which is built around using yield + the ability to return from generators added in Python 3.3. Our version of this is also based around futures so that an @async API will return a future. The big difference here might be that we always return a future from a call rather than yielding it up the stack. So our API works with just simple yields rather than yield froms. This is what a simple usage of the API looks like:
from concurrent.futures import ThreadPoolExecutor from urllib.request import urlopen
executor = ThreadPoolExecutor(max_workers=5)
def load_url(url): return urlopen(_url).read()
@async def get_image_async(url): buffer = yield executor.submit(load_url, url) return Image(buffer)
def main(image_uri): img_future = get_image_async(image_uri) # perform other tasks while the image is downloading img = img_future.result()
main("http://www.python.org/images/python-logo.gif")
This example us just using the existing thread pool to run the actual I/O but this will work with anything that will return a future. So inside of an async method anything which is yielded should be a future. The decorator will then attach a callback which will send the result of the future back into the generator, so the "buffer = " line gets the result of the future. Finally the function completes and the future returned from calling get_image_async will have its value set to Image when the StopIteration exception is raised with the return value.
Because we're interested in the GUI side of things here we've also wired this up into Tk so that we can experiment with an existing GUI framework, and I've included the source for the context there. Our thinking here is that different contexts can be created depending upon the framework which you're running in and that the context makes sure the code is running in the right spot, in this case getting back to the GUI thread after an async operation has been completed.
The big outstanding item we're still working through is I/O, but we think the contexts help here too. We're still not quite sure how polling I/O will work, but with the contexts if there's a single thread polling for I/O then the context will get us off the I/O thread and let the polling continue. We are currently thinking that there will need to be a polling thread which handles all of the I/Os, and there could potentially be more than one of these if different libraries aren't cooperating on sharing a single thread.
Here's the code plus the demo Tk app (you'll need your own Holmes.txt file for the sample app to run):
Contexts.py: http://pastebin.com/ndS53Cd8 Tk context: http://pastebin.com/FuZwc1Ur Tk app: http://pastebin.com/Fm5wMXpN Hardwork.py: http://pastebin.com/nMMytdTG
-----Original Message----- From: Python-ideas [mailto:python-ideas-bounces+dinov=microsoft.com@python.org] On Behalf Of Calvin Spealman Sent: Monday, October 15, 2012 7:16 AM To: Nick Coghlan Cc: python-ideas@python.org Subject: Re: [Python-ideas] The async API of the future: yield-from
On Mon, Oct 15, 2012 at 9:48 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Mon, Oct 15, 2012 at 10:31 PM, Calvin Spealman <ironfroggy@gmail.com> wrote:
Currently, "with yield expr:" is not valid syntax, surprisingly.
It's not that surprising, it's the general requirement that yield expressions must be enclosed in parentheses except when used standalone or in a simple assignment statement.
"with (yield expr):" is valid syntax though, so I'm reluctant to endorse doing anything substantially different if the parentheses are omitted.
Silly oversight on my part, and I agree that the parens shouldn't make the difference in meaning.
I think the combination of "yield from" to delegate control (including exception handling) completely to a subgenerator and "context manager + for loop + explicit yield" when an operation needs to yield + multiple times and the exception handling behaviour should be left to the caller (as in the "as_completed" case) should cover the necessary behaviours.
I'm still -1 on delegating control to subgenerators with yield-from, versus having the scheduler just deal with them directly. I think it is far less flexible.
I would still like to see a less confusing "with yield expr:" by simply allowing it without parens, but no special meaning. I think it would be really useful in coroutines.
with yield collect() as tasks: yield task1() yield task2() results = yield tasks
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
-- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
-- --Guido van Rossum (python.org/~guido)
-- --Guido van Rossum (python.org/~guido) _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

Guido wrote:
But do your Futures use threads? NDB doesn't. NDB's event loop doesn't know about Futures; however the @ndb.tasklet decorator does, and the Futures know about the event loop. When you wait for a Future, a callback is added to the Future that will resume the generator when it is done, and in order to run them, the Future passes its callbacks to the event loop to be run.
The decorator and the default context don't do anything w/ threads by default, but once you start combining it w/ other futures threads are likely to be used. For example if you take: @async def get_image_async(url): buffer = yield executor.submit(load_url, url) return Image(buffer) Then the " yield executor.submit(load_url, url)" line is going to yield a future which is running on a thread pool thread. When it completes it's done callback is also going to be delivered on the same thread pool thread. At that point we let the context which was captured when the function was initially called handle resuming the generator. The default context is just going to synchronously continue to the function, so the generator would then resume running on the thread pool thread. But if you're running in a GUI app which sets up its own context then the context will post an event into the UI event loop and execution will continue on the UI thread. Likewise if there were a bunch of async I/O routines then this would combine with them in a similar way - async I/O would result in a future, the futures would signal that they're done on some worker thread, and then the async methods will get to continue running on that worker thread unless the current context wants to do something different.

Calvin Spealman wrote:
I'm still -1 on delegating control to subgenerators with yield-from, versus having the scheduler just deal with them directly.
Do you mean to *disallow* using yield-from for this, or just not to encourage it? I don't see how you *could* disallow it; there's no way for the scheduler to know whether one of the generators it's handling is delegating using yield-from. I also can't see any reason you would want to discourage it. Given that yield-from exists, it's an obvious thing to do. -- Greg

On Mon, Oct 15, 2012 at 8:55 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Calvin Spealman wrote:
I'm still -1 on delegating control to subgenerators with yield-from, versus having the scheduler just deal with them directly.
Do you mean to *disallow* using yield-from for this, or just not to encourage it?
I don't see how you *could* disallow it; there's no way for the scheduler to know whether one of the generators it's handling is delegating using yield-from.
I also can't see any reason you would want to discourage it. Given that yield-from exists, it's an obvious thing to do.
I have since changed my position slightly. I don't want to disallow it, no. I don't want to discourage, no. But, I do think *both* are useful. I think "yield from" is the obvious way to "call" between tasks, but that there are other cases when we want to spawn a task to begin without blocking our task, and that "yield" should be used here. We should be table to simply yield a task to tell the scheduler to start it, possibly getting a Future in return which we can use to get the eventual result. This may make it easier to do multiple sub-tasks together. We might yield N tasks, and then "yield from wait(futures)" to wait for them all to complete.
-- Greg _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
-- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy

Calvin Spealman wrote:
I think "yield from" is the obvious way to "call" between tasks, but that there are other cases when we want to spawn a task to begin without blocking our task, and that "yield" should be used here.
I've thought of another problem with this. In my scheduler at least, simply spawning a task doesn't immediately allow that task, or any other, to run. Using "yield" to spell this operation gives the impression that it could be a suspension point, when it's actually not. It also forces anything that uses it to be called with "yield from", all the way up, so if you're relying on the presence of yield-froms to warn you of potential suspension points, you'll get false positives. -- Greg

On Tue, Oct 16, 2012 at 4:48 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Calvin Spealman wrote:
I think "yield from" is the obvious way to "call" between tasks, but that there are other cases when we want to spawn a task to begin without blocking our task, and that "yield" should be used here.
I've thought of another problem with this. In my scheduler at least, simply spawning a task doesn't immediately allow that task, or any other, to run. Using "yield" to spell this operation gives the impression that it could be a suspension point, when it's actually not.
While i still like the feeling, I must concede this point. I could see them being yielded and forgotten... assuming they would suspend. Dang.
It also forces anything that uses it to be called with "yield from", all the way up, so if you're relying on the presence of yield-froms to warn you of potential suspension points, you'll get false positives.
-- Greg _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
-- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy

Nick Coghlan wrote:
To me, "yield from" is just a tool that brings generators back to parity with functions when it comes to breaking up a larger algorithm into smaller pieces. Where you would break a function out into subfunctions and call them normally, with a generator you can break out subgenerators and invoke them with yield from.
That's exactly correct. It's the way I intended "yield from" to be thought of right from the beginning. What I'm arguing is that the *public api* for any suspendable operation should be in the form of something called using yield-from, because you never know when the implementation might want to break it down into sub-operations and use yield-from to call *them*.
Any meaningful use of "yield from" in the coroutine context *has* to ultimate devolve to an operation that: 1. Asks the scheduler to schedule another operation 2. Waits for that operation to complete
I don't think I would put it quite that way. In my view of things at least, the scheduler doesn't schedule "operations" (in the sense of "read some bytes from this socket" etc.) Rather, it schedules the running of tasks. So the breakdown is really: 1. Start an operation (this doesn't involve the scheduler) 2. Ask the scheduler to suspend this task until the operation is finished Also, this breakdown is only necessary at the very lowest level, where you want to do something that isn't provided in the form of a generator. Obviously it's *possible* to treat each level of the call chain as its own subtask, that you spawn independently and then wait for it to finish. That's what people have done in the past with their trampoline schedulers that interpret yielded "call" and "return" instructions. But one of the purposes of yield-from is to relieve the scheduler of the need to handle things at that level of granularity. It can treat a generator together with all the subgenerators it might call as a *single* task, the same way that a greenlet is thought of as a single task, however many levels of function calls it might make.
I *thought* Greg's way combined step 1 and step 2 into a single operation: the objects you yield *not only* say what you want to wait for, but also what you want to do.
I don't actually yield objects at all, but...
However, his example par() implementation killed that idea, since it turned out to need to schedule tasks explicitly rather than their being a "execute this in parallel" option.
I don't see how that's a problem. Seems to me it's just as easy for the user to call a par() function as it is to yield a tuple of tasks. And providing this functionality using a function means that different versions or options can be made available for variations such as different ways of handling exceptions. Using yield, you need to pick one of the variations and bless it as being the one that you invoke using special syntax. If you're complaining that the implementation of par() seems too complex, well, that complexity has to occur *somewhere* -- if it's not in the par() function, then it will turn up inside whatever part of the scheduler handles the case that it's given a tuple of tasks.
So now I'm back to think that Greg and Guido are talking about different levels. *Any* scheduling option will be able to be collapsed into an async task invoked by "yield from" by writing:
def simple_async_task(): return yield start_task()
Yes... or another implementation that works some way other than yielding instructions to the scheduler.
I haven't seen anything to suggest that "yield from"'s role should change from what it is in 3.3: a way to factor out generators into multiple pieces with out breaking send() and throw().
I don't think anyone is suggesting that. I'm certainly not. -- Greg

I've just had another thought I'd like to mention concerning the way we think about subtasks. There's actually a subtle difference between invoking a subgenerator using yield-from on the one hand, and spawning it as a separate task and then waiting for it on the other. When you call a subgenerator using yield-from, a switch to another task can't occur until that subgenerator or something it calls reaches a yield. But (at least the way my scheduler currently works), if you spawn it as a separate task and then block waiting for it to complete, other tasks can run immediately, before the subtask has even started. If you're relying on knowing where the yields can occur, this difference could be important. So I think the distinction between calling and spawning subtasks needs to be maintained. This means that spawning must be something you request explicitly in some way. -- Greg

On Mon, Oct 15, 2012 at 10:35 AM, Guido van Rossum <guido@python.org> wrote:
But, as Christian Tismer wrote, we need to have some kind of idea of what the primitives are that we want to support. Or should we just have async equivalents for everything in threading.py and queue.py? (What about thread-local? Do we need task-local? Shudder.)
Task locals aren't so scary, since they're already the main reason why generators are so handy - task locals are just the frame locals in the generator :) The main primitive I personally want out of an async API is a task-based equivalent to concurrent.futures.as_completed() [1]. This is what I meant about iteration being a bit of a mess: the way the as_completed() works, the suspend/resume channel of the iterator protocol is being used to pass completed future objects back to the calling iterator. That means that channel *can't* be used to talk between the coroutine and the scheduler, so if you decide you need to free it up for that purpose, you're either forced to wait for *all* the futures to be triggered before any of them can be passed to the caller (allowing you to use yield-from and return a container of completed futures) or else you're forced to switch to callback-style programming (this is where Ruby's blocks are a huge advantage - because their for loops essentially *are* callbacks, you have a lot more flexibility in calling back to different places from a single piece of code). However, I can see one why to make it work which is to require the *invoking* code to continue to manage the communication with the scheduler. Using this concept, there would be an "as_completed_async()" primitive that works something like: for get_next_result in as_completed_task(tasks): task, result = yield get_next_result # Process this result, wait for next one The async equivalent of the concurrent.futures example would then look something like: URLS = ['http://www.foxnews.com/', 'http://www.cnn.com/', 'http://europe.wsj.com/', 'http://www.bbc.co.uk/', 'http://some-made-up-domain.com/'] @task def load_url_async(url, timeout): with (yield urlopen_async(url, timeout=timeout)) as handle: return url, handle.read() tasks = (load_url_async(url, 60) for url in URLS) with concurrent.futures.as_completed_async(tasks) as async_results for get_next_result in async_results: try: url, data = yield get_next_result except Exception as exc: print('{!r} generated an exception: {}'.format(url, exc)) else: print('{!r} page is {:d} bytes'.format(url, len(data))) Key parts of this idea: 1. as_completed_async registers the supplied tasks with the main scheduler so they can all start running in parallel 2. as_completed_async is a context manager that will *cancel* all pending jobs on exit 3. as_completed_async is an iterator that produces a special future that fires whenever *any* of the registered tasks has run to completion 4. because there's a separate yield step for each result retrieval, ordinary exception handling mechanisms can be used rather than needing to introspect a future object Cheers, Nick. [1] http://docs.python.org/dev/library/concurrent.futures.html#threadpoolexecuto... -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Nick Coghlan wrote:
The main primitive I personally want out of an async API is a task-based equivalent to concurrent.futures.as_completed() [1]. This is what I meant about iteration being a bit of a mess: the way the as_completed() works, the suspend/resume channel of the iterator protocol is being used to pass completed future objects back to the calling iterator. That means that channel *can't* be used to talk between the coroutine and the scheduler,
I had to read this a couple of times before I figured out what you're talking about, but I get it now. This is an instance of a general problem that was noticed back when I was discussing my cofunctions idea: using generator-based coroutines, it's not possible to have a "suspendable iterator", because that would require "yield" to have two conflicting meanings: "suspend this coroutine" on one hand, and "provide a value to my caller" on the other. Unfortunately, I suspect that a truly elegant solution to this problem will require yet another language addition -- something like yield for item in subtask(): ... which would run a slightly different version of the iterator protocol in which values to be yield are wrapped somehow (I haven't figured out all the details yet). -- Greg

On Mon, Oct 15, 2012 at 1:14 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Nick Coghlan wrote:
The main primitive I personally want out of an async API is a task-based equivalent to concurrent.futures.as_completed() [1]. This is what I meant about iteration being a bit of a mess: the way the as_completed() works, the suspend/resume channel of the iterator protocol is being used to pass completed future objects back to the calling iterator. That means that channel *can't* be used to talk between the coroutine and the scheduler,
I had to read this a couple of times before I figured out what you're talking about, but I get it now.
This is an instance of a general problem that was noticed back when I was discussing my cofunctions idea: using generator-based coroutines, it's not possible to have a "suspendable iterator", because that would require "yield" to have two conflicting meanings: "suspend this coroutine" on one hand, and "provide a value to my caller" on the other.
Unfortunately, I suspect that a truly elegant solution to this problem will require yet another language addition -- something like
yield for item in subtask(): ...
which would run a slightly different version of the iterator protocol in which values to be yield are wrapped somehow (I haven't figured out all the details yet).
I think I ran into a similar issue with NDB when defining iteration over an asynchronous query. My solution: q = <some query specification> it = q.iter() # Fire off the query to the datastore while (yield it.has_next_async()): # Block until one result emp = it.next() # Get the result that was buffered on the iterator print emp.name, emp.age # Use it -- --Guido van Rossum (python.org/~guido)

On Mon, Oct 15, 2012 at 10:10 PM, Guido van Rossum <guido@python.org> wrote:
On Mon, Oct 15, 2012 at 1:14 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Nick Coghlan wrote:
The main primitive I personally want out of an async API is a task-based equivalent to concurrent.futures.as_completed() [1]. This is what I meant about iteration being a bit of a mess: the way the as_completed() works, the suspend/resume channel of the iterator protocol is being used to pass completed future objects back to the calling iterator. That means that channel *can't* be used to talk between the coroutine and the scheduler,
I had to read this a couple of times before I figured out what you're talking about, but I get it now.
This is an instance of a general problem that was noticed back when I was discussing my cofunctions idea: using generator-based coroutines, it's not possible to have a "suspendable iterator", because that would require "yield" to have two conflicting meanings: "suspend this coroutine" on one hand, and "provide a value to my caller" on the other.
Unfortunately, I suspect that a truly elegant solution to this problem will require yet another language addition -- something like
yield for item in subtask(): ...
which would run a slightly different version of the iterator protocol in which values to be yield are wrapped somehow (I haven't figured out all the details yet).
I think I ran into a similar issue with NDB when defining iteration over an asynchronous query. My solution:
q = <some query specification> it = q.iter() # Fire off the query to the datastore while (yield it.has_next_async()): # Block until one result emp = it.next() # Get the result that was buffered on the iterator print emp.name, emp.age # Use it
Crazy Idea I Probably Don't Actually Want: for yield emp in q: print emp.name, emp.age Turns into something like: _it = iter(q) for _emp in _it: emp = yield _emp print emp.name, emp.age
-- --Guido van Rossum (python.org/~guido) _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
-- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy

On Mon, Oct 15, 2012 at 1:49 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote: [...]
No, it can't be as simple as that, because that will just execute the tasks sequentially. It would have to be something like this:
def par(*tasks): n = len(tasks) results = [None] * n for i, task in enumerate(tasks): def thunk(): nonlocal n results[i] = yield from task n -= 1 scheduler.schedule(thunk) while n > 0: yield return results
Not exactly straightforward, but that's why we write it once and put it in the library. :-)
There are two problems with this code. :) The first is a scoping gotcha: every copy of thunk() will attempt run the same task, and assign it to the same index, due to them sharing the "i" and "task" variables. (The closure captures a reference to the outer variable cells, rather than a copy of their values at the time of thunk's definition.) This could be fixed by defining it as "def thunk(i=i, task=task)", to capture copies. The second problem is more serious: the final while loop busy-waits, which will consume all spare CPU time waiting for the underlying tasks to complete. For this to be practical, it must suspend and resume itself more efficiently. Here's my own attempt. I'll assume the following primitive scheduler instructions (see my "generator task protocol" mail for context), but it should be readily adaptable to other primitives: 1. yield tasklib.spawn(task()) instructs the scheduler to spawn a new, independent task. 2. yield tasklib.suspend() suspends the current task. 3. yield tasklib.get_resume() obtains a callable / primitive that can be used to resume the current task later. I'll also expand it to keep track of success and failure by returning a list of (flag, result) tuples, in the style of DeferredList[1]. Code: def par(*tasks): resume = yield tasklib.get_resume() # Prepare to hold the results, and keep track of progress. results = [None] * len(tasks) finished = 0 # Gather the i'th task's result def gather(i, task): nonlocal finished try: r = yield from task except Exception as e: results[i] = (False, e) else: results[i] = (True, r) finished += 1 # If we're the last to complete, resume par() if finished == len(tasks): yield resume() # Spawn subtasks, and wait for completion. for (i, task) in tasks: yield tasklib.spawn(gather(i, task)) yield tasklib.suspend() return results Hopefully, this is easy enough to read: it should be obvious to see how to modify gather() to add support for resuming immediately on the first result or first error. [1] http://twistedmatrix.com/documents/12.1.0/api/twisted.internet.defer.Deferre... -- Piet Delport

My general thought on this is that "yield from generator" is the coroutine equivalent of a function call, while "yield future" would be the way the lowest level of the generator talked to the standard event loop. -- Sent from my phone, thus the relative brevity :)

I've had some thoughts on why I'm uncomfortable about this kind of pattern: data = yield sock.async_read(1024) The idea here is that sock.async_read() returns a Future or similar object that performs the I/O and waits for the result. However, reading the data isn't necessarily the point at which the suspension actually occurs. If you're using a select-style event loop, the async read operation breaks down into 1. Wait for data to arrive on the socket 2. Read the data So the implementation of sock.async_read() is going to have to create another Future to handle waiting for the socket to become ready. But then the outer Future is an unnecessary complication, because you could get the same effect by defining def async_read(self, length): yield future_to_wait_for_fd(self.fd) return os.read(self.fd, length) and calling it using data = yield from sock.async_read(1024) If Futures are to appear anywhere, they should only be at the very bottom layer, at the transition between generator and non-generator code. And the place where that transition occurs depend on how the lower levels are implemented. If you're using IOCP instead of select, for example, you need to do things the other way around: 1. Start the read operation 2. Wait for it to complete So I feel that all public APIs should be functions called using yield-from, leaving it up to the implementation to decide if and where Futures become involved. -- Greg

On Sun, 14 Oct 2012 20:12:04 +1300 Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
So the implementation of sock.async_read() is going to have to create another Future to handle waiting for the socket to become ready. But then the outer Future is an unnecessary complication, because you could get the same effect by defining
def async_read(self, length): yield future_to_wait_for_fd(self.fd) return os.read(self.fd, length)
read() may fail even if select() returned successfully. See http://bugs.python.org/issue9090 What this means is that your select-style event loop should probably also handle actually reading the data. Besides, this will make its API more easily ported to something like IOCP. Regards Antoine. -- Software development and contracting: http://pro.pitrou.net

On Sun, Oct 14, 2012 at 12:12 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
I've had some thoughts on why I'm uncomfortable about this kind of pattern:
data = yield sock.async_read(1024)
The idea here is that sock.async_read() returns a Future or similar object that performs the I/O and waits for the result.
However, reading the data isn't necessarily the point at which the suspension actually occurs. If you're using a select-style event loop, the async read operation breaks down into
1. Wait for data to arrive on the socket 2. Read the data
So the implementation of sock.async_read() is going to have to create another Future to handle waiting for the socket to become ready. But then the outer Future is an unnecessary complication, because you could get the same effect by defining
def async_read(self, length): yield future_to_wait_for_fd(self.fd) return os.read(self.fd, length)
and calling it using
data = yield from sock.async_read(1024)
If Futures are to appear anywhere, they should only be at the very bottom layer, at the transition between generator and non-generator code. And the place where that transition occurs depend on how the lower levels are implemented. If you're using IOCP instead of select, for example, you need to do things the other way around:
1. Start the read operation 2. Wait for it to complete
So I feel that all public APIs should be functions called using yield-from, leaving it up to the implementation to decide if and where Futures become involved.
A logical and consistent conclusion. I actually agree: in NDB, where all I have is "yield <future>" I have a similar guideline: all public async APIs return a Future and must be waited on using yield, and only at the lowest level are other types primitives involved (bare App Engine RPCs, callbacks). -- --Guido van Rossum (python.org/~guido)

Is the goal really to provide "The async API of the future", or just to provide "a stdlib module which provides one adequate way to do async"? I think the yield and yield from solutions all need too much magical scaffolding to be The One True Way, but I don't mind such conventions as much when they're part of a particular example class, such as concurrent.schedulers.YieldScheduler. To stretch an analogy, generators and context managers are different concepts. Allowing certain generators to be used as context managers (by using the "with" keyword) is fine. But I wouldn't want to give up all the other uses of generators. If yield starts implying other magical properties that are only useful when communicating with a scheduler, rather than a regular caller ... I'm afraid that muddies the concept up too much for my taste. More specific concerns below: On 10/12/12, Guido van Rossum <guido@python.org> wrote:
But the only use for send() on a generator is when using it as a coroutine for a concurrent tasks system -- send() really makes no sense for generators used as iterators. And you're claiming, it seems, that you prefer yield-from for concurrent tasks.
But the data doesn't have to be scheduling information; it can be new data, a seed for an algorithm, a command to switch or reset the state ... locking it to the scheduler is part of what worries me.
On Thu, Oct 11, 2012 at 6:32 PM, Greg Ewing <greg.ewing@canterbury.ac.nz>
Keep in mind that a value yielded by a generator being used as part of a coroutine is *not* seen by code calling it with yield-from.
That is part of what bugs me about the yield-from examples. Until this discussion, I had thought of yield-from as factoring out some code that was still conceptually embedded within the parent generator. This (perhaps correctly) makes it seem more like a temporary replacement, as if the parent were no longer there at all. But with the yield channel reserved for scheduling overhead, the "generator" can't really generate anything, except through side effects...
... I feel that "value = yield <something that returns a Future>" is quite a good paradigm,
To me, it seems fine for a particular concrete scheduler, but too strong an assumption for an abstract API. I can mostly* understand: YieldScheduler assumes any yielded data is another Task; it will schedule that task, and cause the original (yielding) Task to wait until the new task is completed. But I wonder what I'm missing with: Generators should only yield (expressions that create) Futures; the scheduler will automatically unwrap the future and send (or throw) the result back into the parent (or other ancestor) Generator, which will then be resumed. * "mostly", because if my task is willing to wait for the subtask to complete, then why not just use a blocking call in the first place? Is it just because switching to another task is lighter weight than letting a thread block? What happens if a generator does yield something other than a Future? Will the generator be rescheduled in an already-runnable (as opposed to waiting) state? Will it never be resumed? Will that object be auto-wrapped in a Future for the benefit of whichever other co-routine originally made the request? Are generators assumed to run to exhaustion, or is some sort of driver needed to keep pumping them?
... It would be horrible to require C to create a fake generator.
Would it have to wrap results in a fake Future, so that the scheduler could properly unwrap?
...Well, I'm talking about a decorator that you *always* apply, and which does nothing (or very little) when wrapping a generator, but adds generator behavior when wrapping a non-generator function.
Why is an always-applied decorator any less intrusive than a mandatory (mixin) base class?
(1) Calling an async operation and waiting for its result, using yield
Futures: result = yield some_async_op(args)
I was repeatedly confused over whether "result" would be a Future that still needed resolution, and the example code wasn't always consistent. As I understand it now, the scheduler (not just the particular implementation, but the API) has to automatically treat any yielded data as a future, resolve that future to its result, and then send (or throw) that result (as opposed to the future) back into either the parent task or the least distant ancestor task not to be using "yield from".
Yield-from: result = yield from some_async_op(args)
So the generator containing this code suspends itself entirely until some_async_op is exhausted, at which point result will be the StopIteration? (Or None?) Non-Exception results get passed straight to the least-distant ancestor task not using "yield from", but Exceptions propagate through one generation at a time.
(2) Setting the result of an async operation
Futures: f.set_result(value) # From any callback
PEP 3148 considers set_result private to the executor. Can that always be done from arbitrary callbacks? Can it be done more than once? I think for the normal case, a task should just return its value, and the Future or the Scheduler should be responsible for calling set_result.
Yield-from: return value # From the outermost generator
Why only the outermost? I'm guessing it is because everything else is suspended, and even if a mid-level generator is explicitly re-added to the task queue, it can't actually continue because of re-entrancy.
(3) Handling an exception
Futures: try: result = yield some_async_op(args) except MyException: <handle exception>
So the scheduler does have to unpack the future, and throw rather than send.
(4) Raising an exception as the outcome of an async operation
Futures: f.set_exception(<Exception instance>)
Again, shouldn't the task itself just raise, and let the future (or the scheduler) call that?
Yield-from: raise <Exception instance or class> # From any of the generators
So it doesn't need to be wrapped in a Future, until it needs to cross back over a "schedule this asynchronously" gulf?
(5) Having one async operation invoke another async operation
Futures: @task def outer(args): res = yield inner(args) return res
Yield-from: def outer(args): res = yield from inner(args) return res
Will it ever get to continue processing (under either model) before inner exhausts itself and stops yielding?
Note: I'm including this because in the Futures case, each level of yield requires the creation of a separate Future.
Only because of the auto-unboxing. And if the generator suspends itself to wait for the future, then the future will be resolved before control returns to the generator's own parents, so those per-layer Futures won't really add anything.
(6) Spawning off multiple async subtasks
Futures: f1 = subtask1(args1) # Note: no yield!!! f2 = subtask2(args2) res1, res2 = yield f1, f2
ah. That makes a bit more sense, though the tuple of futures does complicate the automagic unboxing. (Which containers, to which levels, have to be resolved?)
Yield-from: ??????????
*** Greg, can you come up with a good idiom to spell concurrency at this level? Your example only has concurrency in the philosophers example, but it appears to interact directly with the scheduler, and the philosophers don't return values. ***
Why wouldn't this be the same as you already wrote without yield-from? Two subtasks were submitted but not waited for. I suppose you could yield from a generator that submits new subtasks every time it generates something, but that would be solving a more complicated problem. (So it wouldn't be a consequence of the "yield from".)
(7) Checking whether an operation is already complete
Futures: if f.done(): ...
If f was yielded, it is done, or this code wouldn't be running again to check.
Yield-from: ?????????????
And again, if the futures were yielded (even through a yield from) then they're already unboxed; otherwise, you can still check f.done
(8) Getting the result of an operation multiple times
Futures:
f = async_op(args) # squirrel away a reference to f somewhere else r = yield f # ... later, elsewhere r = f.result()
Why do you have to squirrel away the reference? Are you assuming that the async scheduler will mess with the locals so that f is no longer valid?
Yield-from: ???????????????
This, you cannot reasonably do; the nature of yield-from means that the unresolved futures were never visible within this generator; they were resolved by the scheduler and the results handed straight to the generator's ancestor.
(9) Canceling an operation
Futures: f.cancel()
Yield-from: ???????????????
Note: I haven't needed canceling yet, and I believe Devin said that Twisted just got rid of it. However some of the JS Deferred implementations seem to support it.
I think that once you've called "yield from", the generator making that call is suspended until the child generator completes. But a different thread of control could cancel the active (most-descended) generator.
(10) Registering additional callbacks
Futures: f.add_done_callback(callback)
Yield-from: ???????
Note: this is used in NDB to trigger "hooks" that should run e.g. when a database write completes. The user's code just writes yield ent.put_async(); the trigger is automatically called by the Future's machinery. This also uses (8).
I think you would have to do add the callbacks within the subgenerator that is spawning f. That, or un-inline the yield from, and lose the automated send-throw forwarding. -jJ

On Thu, Oct 18, 2012 at 11:46 PM, Jim Jewett <jimjjewett@gmail.com> wrote:
Is the goal really to provide "The async API of the future", or just to provide "a stdlib module which provides one adequate way to do async"?
I think the yield and yield from solutions all need too much magical scaffolding to be The One True Way, but I don't mind such conventions as much when they're part of a particular example class, such as concurrent.schedulers.YieldScheduler.
To stretch an analogy, generators and context managers are different concepts. Allowing certain generators to be used as context managers (by using the "with" keyword) is fine. But I wouldn't want to give up all the other uses of generators.
If yield starts implying other magical properties that are only useful when communicating with a scheduler, rather than a regular caller ... I'm afraid that muddies the concept up too much for my taste.
I think it is important that this is more than convention. I think that we need our old friend TOOOWTDI (There's Only One Obvious Way To Do It) here more than ever. This stuff is complicated, and following that interoperability of what eventually is written on top of it is going to be complicated. Our focus should be not on providing simple things like "async file read" but crafting an environment where people can continue to write wonderfully expressive and useful libraries that others can combine to their own needs. If we don't provide the layer upon which this disparate pieces cooperate, I fear much of the effort is all for too little gain to be worth the effort.
More specific concerns below:
On 10/12/12, Guido van Rossum <guido@python.org> wrote:
But the only use for send() on a generator is when using it as a coroutine for a concurrent tasks system -- send() really makes no sense for generators used as iterators. And you're claiming, it seems, that you prefer yield-from for concurrent tasks.
But the data doesn't have to be scheduling information; it can be new data, a seed for an algorithm, a command to switch or reset the state ... locking it to the scheduler is part of what worries me.
When a coroutine yields, it yields *to the scheduler* so for whom else should these values be?
On Thu, Oct 11, 2012 at 6:32 PM, Greg Ewing <greg.ewing@canterbury.ac.nz>
Keep in mind that a value yielded by a generator being used as part of a coroutine is *not* seen by code calling it with yield-from.
That is part of what bugs me about the yield-from examples.
Until this discussion, I had thought of yield-from as factoring out some code that was still conceptually embedded within the parent generator. This (perhaps correctly) makes it seem more like a temporary replacement, as if the parent were no longer there at all.
But with the yield channel reserved for scheduling overhead, the "generator" can't really generate anything, except through side effects...
Don't forget that yield-from is an expression, not a statement. The value eventually returned from the generator is the result of the yield-from, so the generator still produces a final value. The fact that these are generators is for their ability to suspend, not to iterate.
... I feel that "value = yield <something that returns a Future>" is quite a good paradigm,
To me, it seems fine for a particular concrete scheduler, but too strong an assumption for an abstract API.
I can mostly* understand:
YieldScheduler assumes any yielded data is another Task; it will schedule that task, and cause the original (yielding) Task to wait until the new task is completed.
But I wonder what I'm missing with:
Generators should only yield (expressions that create) Futures; the scheduler will automatically unwrap the future and send (or throw) the result back into the parent (or other ancestor) Generator, which will then be resumed.
* "mostly", because if my task is willing to wait for the subtask to complete, then why not just use a blocking call in the first place? Is it just because switching to another task is lighter weight than letting a thread block?
By blocking call do you mean "x = foo()" or "x = yield from foo()"? Blocking call usually means the former, so if you mean that, then you neglect to think of all the other tasks running which are not willing to wait.
What happens if a generator does yield something other than a Future? Will the generator be rescheduled in an already-runnable (as opposed to waiting) state? Will it never be resumed? Will that object be auto-wrapped in a Future for the benefit of whichever other co-routine originally made the request?
I think if the scheduler doesn't know what to do with something, it should be an error. That makes it easier to change things in the future.
Are generators assumed to run to exhaustion, or is some sort of driver needed to keep pumping them?
... It would be horrible to require C to create a fake generator.
Would it have to wrap results in a fake Future, so that the scheduler could properly unwrap?
...Well, I'm talking about a decorator that you *always* apply, and which does nothing (or very little) when wrapping a generator, but adds generator behavior when wrapping a non-generator function.
Why is an always-applied decorator any less intrusive than a mandatory (mixin) base class?
(1) Calling an async operation and waiting for its result, using yield
Futures: result = yield some_async_op(args)
I was repeatedly confused over whether "result" would be a Future that still needed resolution, and the example code wasn't always consistent. As I understand it now, the scheduler (not just the particular implementation, but the API) has to automatically treat any yielded data as a future, resolve that future to its result, and then send (or throw) that result (as opposed to the future) back into either the parent task or the least distant ancestor task not to be using "yield from".
Yield-from: result = yield from some_async_op(args)
So the generator containing this code suspends itself entirely until some_async_op is exhausted, at which point result will be the StopIteration? (Or None?) Non-Exception results get passed straight to the least-distant ancestor task not using "yield from", but Exceptions propagate through one generation at a time.
The result is not an exception, but the return of some_async_op(args)
(2) Setting the result of an async operation
Futures: f.set_result(value) # From any callback
PEP 3148 considers set_result private to the executor. Can that always be done from arbitrary callbacks? Can it be done more than once?
I think for the normal case, a task should just return its value, and the Future or the Scheduler should be responsible for calling set_result.
I agree
Yield-from: return value # From the outermost generator
Why only the outermost? I'm guessing it is because everything else is suspended, and even if a mid-level generator is explicitly re-added to the task queue, it can't actually continue because of re-entrancy.
(3) Handling an exception
Futures: try: result = yield some_async_op(args) except MyException: <handle exception>
So the scheduler does have to unpack the future, and throw rather than send.
(4) Raising an exception as the outcome of an async operation
Futures: f.set_exception(<Exception instance>)
Again, shouldn't the task itself just raise, and let the future (or the scheduler) call that?
Yield-from: raise <Exception instance or class> # From any of the generators
So it doesn't need to be wrapped in a Future, until it needs to cross back over a "schedule this asynchronously" gulf?
(5) Having one async operation invoke another async operation
Futures: @task def outer(args): res = yield inner(args) return res
Yield-from: def outer(args): res = yield from inner(args) return res
Will it ever get to continue processing (under either model) before inner exhausts itself and stops yielding?
Note: I'm including this because in the Futures case, each level of yield requires the creation of a separate Future.
Only because of the auto-unboxing. And if the generator suspends itself to wait for the future, then the future will be resolved before control returns to the generator's own parents, so those per-layer Futures won't really add anything.
(6) Spawning off multiple async subtasks
Futures: f1 = subtask1(args1) # Note: no yield!!! f2 = subtask2(args2) res1, res2 = yield f1, f2
ah. That makes a bit more sense, though the tuple of futures does complicate the automagic unboxing. (Which containers, to which levels, have to be resolved?)
Yield-from: ??????????
*** Greg, can you come up with a good idiom to spell concurrency at this level? Your example only has concurrency in the philosophers example, but it appears to interact directly with the scheduler, and the philosophers don't return values. ***
Why wouldn't this be the same as you already wrote without yield-from? Two subtasks were submitted but not waited for. I suppose you could yield from a generator that submits new subtasks every time it generates something, but that would be solving a more complicated problem. (So it wouldn't be a consequence of the "yield from".)
(7) Checking whether an operation is already complete
Futures: if f.done(): ...
If f was yielded, it is done, or this code wouldn't be running again to check.
Yield-from: ?????????????
And again, if the futures were yielded (even through a yield from) then they're already unboxed; otherwise, you can still check f.done
(8) Getting the result of an operation multiple times
Futures:
f = async_op(args) # squirrel away a reference to f somewhere else r = yield f # ... later, elsewhere r = f.result()
Why do you have to squirrel away the reference? Are you assuming that the async scheduler will mess with the locals so that f is no longer valid?
Yield-from: ???????????????
This, you cannot reasonably do; the nature of yield-from means that the unresolved futures were never visible within this generator; they were resolved by the scheduler and the results handed straight to the generator's ancestor.
(9) Canceling an operation
Futures: f.cancel()
Yield-from: ???????????????
Note: I haven't needed canceling yet, and I believe Devin said that Twisted just got rid of it. However some of the JS Deferred implementations seem to support it.
I think that once you've called "yield from", the generator making that call is suspended until the child generator completes. But a different thread of control could cancel the active (most-descended) generator.
(10) Registering additional callbacks
Futures: f.add_done_callback(callback)
Yield-from: ???????
Note: this is used in NDB to trigger "hooks" that should run e.g. when a database write completes. The user's code just writes yield ent.put_async(); the trigger is automatically called by the Future's machinery. This also uses (8).
I think you would have to do add the callbacks within the subgenerator that is spawning f.
That, or un-inline the yield from, and lose the automated send-throw forwarding.
-jJ _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
-- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy

On 10/19/12, Calvin Spealman <ironfroggy@gmail.com> wrote:
On Thu, Oct 18, 2012 at 11:46 PM, Jim Jewett <jimjjewett@gmail.com> wrote:
[I think the yield solutions are (too magic)/(prematurely lock too much policy) to be "The" API, but work fine as "an example API"]
I think it is important that this is more than convention. ... Our focus should be not on providing simple things like "async file read" but crafting an environment where people can continue to write wonderfully expressive and useful libraries that others can combine to their own needs.
And I think that adding (requirements for generator usage) / (implied meaning of yield) prevents that.
On 10/12/12, Guido van Rossum <guido@python.org> wrote:
But the only use for send() on a generator is when using it as a coroutine for a concurrent tasks system -- send() really makes no sense for generators used as iterators.
But the data doesn't have to be scheduling information; it can be new data, a seed for an algorithm, a command to switch or reset the state ... locking it to the scheduler is part of what worries me.
When a coroutine yields, it yields *to the scheduler* so for whom else should these values be?
Who says that there has to be a scheduler? Or at least a single scheduler? To me, the "obvious" solution is that each co-routine is "scheduled" only by its own caller, and runs on its own micro-thread. The caller thread may or may not wait for a result to be yielded, but would not normally wait for the entire generator to be exhausted forever (the "return"). The next call to the co-routine may well be from an entirely different caller, particularly if the co-routine is a generic source or sink. There may well be several other co-routines (as opposed to a single scheduler) that enforce policy, and may send messages about things like "switch to that source of randomness", "start using this other database instance as a backup", "stop listening on that port". They would certainly want to use throw, and perhaps send as well. In practice, creating a full thread for each such co-routine probably won't work well under current threading systems, because an OS thread (let alone an OS process) is too heavy-weight. And without OS support, python has to do some internal scheduling. But I'm not convinced that the current situation will last forever, so I don't want to muddy up the *abstraction* just to coddle temporary limitations.
But with the yield channel reserved for scheduling overhead, the "generator" can't really generate anything, except through side effects...
Don't forget that yield-from is an expression, not a statement. The value eventually returned from the generator is the result of the yield-from, so the generator still produces a final value.
Assuming it terminates, then yes. But that isn't (conceptually) a generator; it is an ordinary function call.
The fact that these are generators is for their ability to suspend, not to iterate.
So "generator" is not really the right term. Abusing that for one module is acceptable, but I'm reluctant to bake that change into an officially sanctioned API, let alone one important enough that it might eventually be seen as the primary definition.
* "mostly", because if my task is willing to wait for the subtask to complete, then why not just use a blocking call in the first place? Is it just because switching to another task is lighter weight than letting a thread block?
By blocking call do you mean "x = foo()" or "x = yield from foo()"? Blocking call usually means the former, so if you mean that, then you neglect to think of all the other tasks running which are not willing to wait.
Exactly. From my own code's perspective, is there any difference between those two? (Well, besides the fact that the second is wordier, and puts more constraints on what I can use for foo.) So why not just use the first spelling, let the (possibly OS-level) scheduler notice that I'm blocked (if I happen to be), and let it suspend my thread waiting on foo? Is it just that *current* ways to suspend a thread of execution are expensive, and we hope to do it more cheaply? If so, that is a perfectly sensible justification for conventions within a single stdlib module. But since the trade-offs may change with time, the current costs shouldn't drive decisions about the async API, let alone changes to the meaning of "yield" or "generator".
[Questions about generators that do not follow the new constraints]
I think if the scheduler doesn't know what to do with something, it should be an error. That makes it easier to change things in the future.
Those were all things that could reasonably happen simply by reusing correct existing code. For a specific implementation, even a stdlib module, it is OK to treat them as errors; a specific module can always be viewed as incomplete. But for "the asynchronous API of the future", undefined behavior just guarantees warts. We may eventually decide that the warts are in the existing legacy code, but there would still be warts. -jJ

Jim, relax. We're not changing the meaning of yield or generator. We're just making it *possible* to use yield(-from) and generators as coroutines; that's actually a long path that started with PEP 342. No freedom is taken away by PEP 380; it just adds the possibility to do it without managing an explicit stack of coroutine calls in the scheduler. If we believed that there was no advantage to spelling a blocking call as "yield from foo()", we would just spell it as "foo()" and somehow make it work. But (and even Christian Tismer agrees) there is a problem with the shorter spelling -- you lose track of which calls may cause a task-switch. Using yield-from (or yield, for that matter) for this purpose ensures that all callers in the call chain have to explicitly mark the suspension points, and this serves as a useful reminder that after resumption, the world may look differently, because other tasks may have run in the mean time. -- --Guido van Rossum (python.org/~guido)

Jim Jewett wrote:
Who says that there has to be a scheduler? Or at least a single scheduler?
To me, the "obvious" solution is that each co-routine is "scheduled" only by its own caller, and runs on its own micro-thread.
I think you may be confused about what we mean by a "scheduler". The scheduler is not something that you tell which task should run next. Rather, the scheduler decides which task to run next when the current task says "I'm waiting for something, let someone else have a turn." The task that gets run will very often be one that the suspending task knows nothing about. It's for that reason -- not all the tasks know about each other -- that I think it's best to have only one scheduler in any given system, so that it can make the best decision about what to run next. -- Greg

Calvin Spealman wrote:
I think it is important that this is more than convention. I think that we need our old friend TOOOWTDI (There's Only One Obvious Way To Do It) here more than ever.
This is part of the reason that I don't like the idea of controlling the scheduler by yielding instructions to it. There are a great many ways that such a "scheduler instruction set" could be designed, none of them any more obvious than the others. So rather than single out an arbitrarily chosen set of operations to be regarded as primitives that the scheduler knows about directly, I would rather have *no* such primitives in the public API. -- Greg

On Fri, Oct 19, 2012 at 4:29 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Calvin Spealman wrote:
I think it is important that this is more than convention. I think that we need our old friend TOOOWTDI (There's Only One Obvious Way To Do It) here more than ever.
This is part of the reason that I don't like the idea of controlling the scheduler by yielding instructions to it. There are a great many ways that such a "scheduler instruction set" could be designed, none of them any more obvious than the others.
So rather than single out an arbitrarily chosen set of operations to be regarded as primitives that the scheduler knows about directly, I would rather have *no* such primitives in the public API.
But you have that problem anyway. In your current style you write things like this: block(self.queue) yield I don't see how this decouples the call site of the primitive from the scheduler any more than if you were to write e.g. this: yield block(self.queue) In fact, you can write it in your current framework and it would have the exact same effect! That's because block() returns None, so it comes down to calling block(self.queue) and then yielding None, which is exactly what happens in the first form as well. And even if block() were to return a value, since the scheduler ignores the return value from next(), it still works the same way. Not that I recommend doing this just because it works -- but if we liked the second form better, we could easily implement block() in such a way that you'd *have* to write it like that. So, I don't see what we gain by writing it the first way. -- --Guido van Rossum (python.org/~guido)

Guido van Rossum wrote:
In your current style you write things like this:
block(self.queue) yield
I don't see how this decouples the call site of the primitive from the scheduler any more than if you were to write e.g. this:
yield block(self.queue)
If I wrote a library intended for serious use, the end user probably wouldn't write either of those. Instead he would write something like yield from block(self.queue) and it would be an implementation detail of the library where abouts the 'yield' happened and whether it needed to send a value or not. When I say I don't like scheduler instructions, all I really mean is that they shouldn't be part of the public API. A scheduler can use them internally if it wants, I don't care. -- Greg
participants (17)
-
Antoine Pitrou
-
Ben Darnell
-
Calvin Spealman
-
Carlo Pires
-
Christian Tismer
-
Dino Viehland
-
Greg Ewing
-
Guido van Rossum
-
Jim Jewett
-
Laurens Van Houtven
-
Matt Joiner
-
Nick Coghlan
-
Piet Delport
-
Serhiy Storchaka
-
Steve Dower
-
Terry Reedy
-
Yuval Greenfield