[Python-checkins] peps: Add section on coroutines and the scheduler.
guido.van.rossum
python-checkins at python.org
Fri Dec 14 02:47:45 CET 2012
http://hg.python.org/peps/rev/dcf4e647f4d2
changeset: 4608:dcf4e647f4d2
user: Guido van Rossum <guido at google.com>
date: Thu Dec 13 17:47:40 2012 -0800
summary:
Add section on coroutines and the scheduler.
files:
pep-3156.txt | 135 +++++++++++++++++++++++++++++++++++---
1 files changed, 122 insertions(+), 13 deletions(-)
diff --git a/pep-3156.txt b/pep-3156.txt
--- a/pep-3156.txt
+++ b/pep-3156.txt
@@ -16,7 +16,7 @@
Python 3.3. Consider this the concrete proposal that is missing from
PEP 3153. The proposal includes a pluggable event loop API, transport
and protocol abstractions similar to those in Twisted, and a
-higher-level scheduler based on yield-from (PEP 380). A reference
+higher-level scheduler based on ``yield from`` (PEP 380). A reference
implementation is in the works under the code name tulip.
@@ -50,7 +50,7 @@
An event loop implementation may provide additional methods and
guarantees.
-The event loop interface does not depend on yield-from. Rather, it
+The event loop interface does not depend on ``yield from``. Rather, it
uses a combination of callbacks, additional interfaces (transports and
protocols), and Futures. The latter are similar to those defined in
PEP 3148, but have a different implementation and are not tied to
@@ -59,7 +59,7 @@
For users (like myself) who don't like using callbacks, a scheduler is
provided for writing asynchronous I/O code as coroutines using the PEP
-380 yield-from expressions. The scheduler is not pluggable;
+380 ``yield from`` expressions. The scheduler is not pluggable;
pluggability occurs at the event loop level, and the scheduler should
work with any conforming event loop implementation.
@@ -406,9 +406,10 @@
A ``tulip.Future`` object is not acceptable to the ``wait()`` and
``as_completed()`` functions in the ``concurrent.futures`` package.
-A ``tulip.Future`` object is acceptable to a yield-from expression
-when used in a coroutine. See the section "Coroutines and the
-Scheduler" below.
+A ``tulip.Future`` object is acceptable to a ``yield from`` expression
+when used in a coroutine. This is implemented through the
+``__iter__()`` interface on the Future. See the section "Coroutines
+and the Scheduler" below.
Transports
----------
@@ -551,11 +552,6 @@
initiated by the other end? Does this call connection_lost()? Is the
protocol then allowed to write more? (I think it should be!)
-Coroutines and the Scheduler
-----------------------------
-
-TBD.
-
Callback Style
--------------
@@ -583,6 +579,106 @@
policy.)
+Coroutines and the Scheduler
+============================
+
+This is a separate toplevel section because its status is different
+from the event loop interface. Usage of coroutines is optional, and
+it is perfectly fine to write code using callbacks only. On the other
+hand, there is only one implementation of the scheduler/coroutine API,
+and if you're using coroutines, that's the one you're using.
+
+A coroutine is a generator that follows certain conventions. For
+documentation purposes, all coroutines should be decorated with
+``@tulip.coroutine``, but this cannot be strictly enforced.
+
+Coroutines use the ``yield from`` syntax introduced in PEP 380,
+instead of the original ``yield`` syntax.
+
+Unfortunately, the word "coroutine", like the word "generator", is
+used for two different (though related) concepts:
+
+- The function that defines a coroutine (a function definition
+ decorated with ``tulip.coroutine``). If disambiguation is needed,
+ we call this a *coroutine function*.
+
+- The object obtained by calling a coroutine function. This object
+ represents a computation or an I/O operation (usually a combination)
+ that will complete eventually. For disambiguation we call it a
+ *coroutine object*.
+
+Things a coroutine can do:
+
+- ``result = yield from future`` -- suspends the coroutine until the
+ future is done, then returns the future's result, or raises its
+ exception, which will be propagated.
+
+- ``result = yield from coroutine`` -- wait for another coroutine to
+ produce a result (or raise an exception, which will be propagated).
+ The ``coroutine`` expression must be a *call* to another coroutine.
+
+- ``results = yield from tulip.par(futures_and_coroutines)`` -- Wait
+ for a list of futures and/or coroutines to complete and return a
+ list of their results. If one of the futures or coroutines raises
+ an exception, that exception is propagated, after attempting to
+ cancel all other futures and coroutines in the list.
+
+- ``return result`` -- produce a result to the coroutine that is
+ waiting for this one using ``yield from``.
+
+- ``raise exception`` -- raise an exception in the coroutine that is
+ waiting for this one using ``yield from``.
+
+Calling a coroutine does not start its code running -- it is just a
+generator, and the coroutine object returned by the call is really a
+generator object, which doesn't do anything until you iterate over it.
+In the case of a coroutine object, there are two basic ways to start
+it running: call ``yield from coroutine`` from another coroutine
+(assuming the other coroutine is already running!), or convert it to a
+Task.
+
+Coroutines can only run when the event loop is running.
+
+Tasks
+-----
+
+A Task is an object that manages an independently running coroutine.
+The Task interface is the same as the Future interface. The task
+becomes done when its coroutine returns or raises an exception; if it
+returns a result, that becomes the task's result, if it raises an
+exception, that becomes the task's exception.
+
+Canceling a task that's not done yet prevents its coroutine from
+completing; in this case an exception is thrown into the coroutine
+that it may catch to further handle cancelation, but it doesn't have
+to (this is done using the standard ``close()`` method on generators,
+described in PEP 342).
+
+The ``par()`` function described above runs coroutines in parallel by
+converting them to Tasks. (Arguments that are already Tasks or
+Futures are not converted.)
+
+Tasks are also useful for interoperating between coroutines and
+callback-based frameworks like Twisted. After converting a coroutine
+into a Task, callbacks can be added to the Task.
+
+You may ask, why not convert all coroutines to Tasks? The
+``@tulip.coroutine`` decorator could do this. This would slow things
+down considerably in the case where one coroutine calls another (and
+so on), as waiting for a "bare" coroutine has much less overhead than
+waiting for a Future.
+
+The Scheduler
+-------------
+
+The scheduler has no public interface. You interact with it by using
+``yield from future`` and ``yield from task``. In fact, there is no
+single object representing the scheduler -- its behavior is
+implemented by the ``Task`` and ``Future`` classes using only the
+public interface of the event loop, so it will work with third-party
+event loop implementations, too.
+
+
Open Issues
===========
@@ -617,19 +713,32 @@
yield from sch.block_future(f)
res = f.result()
+- Do we need a larger vocabulary of operations for combining
+ coroutines and/or futures? E.g. in addition to par() we could have
+ a way to run several coroutines sequentially (returning all results
+ or passing the result of one to the next and returning the final
+ result?). We might also introduce explicit locks (though these will
+ be a bit of a pain to use, as we can't use the ``with lock: block``
+ syntax). Anyway, I think all of these are easy enough to write
+ using ``Task``.
+
+- Priorities?
+
Acknowledgments
===============
Apart from PEP 3153, influences include PEP 380 and Greg Ewing's
-tutorial for yield-from, Twisted, Tornado, ZeroMQ, pyftpdlib, tulip
+tutorial for ``yield from``, Twisted, Tornado, ZeroMQ, pyftpdlib, tulip
(the author's attempts at synthesis of all these), wattle (Steve
Dower's counter-proposal), numerous discussions on python-ideas from
September through December 2012, a Skype session with Steve Dower and
Dino Viehland, email exchanges with Ben Darnell, an audience with
Niels Provos (original author of libevent), and two in-person meetings
with several Twisted developers, including Glyph, Brian Warner, David
-Reid, and Duncan McGreggor.
+Reid, and Duncan McGreggor. Also, the author's previous work on async
+support in the NDB library for Google App Engine was an important
+influence.
Copyright
--
Repository URL: http://hg.python.org/peps
More information about the Python-checkins
mailing list