[Python-checkins] python/nondist/peps pep-0342.txt,1.4,1.5
pje@users.sourceforge.net
pje at users.sourceforge.net
Sun Jun 26 04:18:43 CEST 2005
Update of /cvsroot/python/python/nondist/peps
In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv16919
Modified Files:
pep-0342.txt
Log Message:
PEP 342: Update and assume co-authorship, as directed by Guido in private
email, merging the implemented features from PEP 343 and dropping most of
the PEP 340 holdover features such as __next__, the next() builtin, and
'continue EXPR'. Also, added more motivation and examples, giving lots
of credit to PEPs 288 and 325, where most of these ideas were first thought
of and initially fleshed out.
Index: pep-0342.txt
===================================================================
RCS file: /cvsroot/python/python/nondist/peps/pep-0342.txt,v
retrieving revision 1.4
retrieving revision 1.5
diff -u -d -r1.4 -r1.5
--- pep-0342.txt 14 Jun 2005 15:14:01 -0000 1.4
+++ pep-0342.txt 26 Jun 2005 02:18:40 -0000 1.5
@@ -1,8 +1,8 @@
PEP: 342
-Title: Coroutines via Enhanced Iterators
+Title: Coroutines via Enhanced Generators
Version: $Revision$
Last-Modified: $Date$
-Author: Guido van Rossum
+Author: Guido van Rossum, Phillip J. Eby
Status: Draft
Type: Standards Track
Content-Type: text/plain
@@ -11,118 +11,150 @@
Introduction
- This PEP proposes a new iterator API that allows values to be
- passed into an iterator using "continue EXPR". These values are
- received in the iterator as an argument to the new __next__
- method, and can be accessed in a generator with a
- yield-expression.
-
- The content of this PEP is derived from the original content of
- PEP 340, broken off into its own PEP as the new iterator API is
- pretty much orthogonal from the anonymous block statement
- discussion. Thanks to Steven Bethard for doing the editing.
-
- Update: at this point I'm leaning towards preferring next() over
- __next__() again, but I've no time to update the PEP right now.
- I've changed the title to Coroutines via Enhanced Iterators at
- Timothy Delaney's suggestion.
-
-Motivation and Summary
+ This PEP proposes some enhancements to the API and syntax of
+ generators, to make them usable as simple coroutines. It is
+ basically a combination of ideas from these two PEPs, which
+ may be considered redundant if this PEP is accepted:
- TBD.
+ - PEP 288, Generators Attributes and Exceptions. The current PEP
+ covers its second half, generator exceptions (in fact the
+ throw() method name was taken from PEP 288). PEP 342 replaces
+ generator attributes, however, with a concept from an earlier
+ revision of PEP 288, the "yield expression".
-Use Cases
+ - PEP 325, Resource-Release Support for Generators. PEP 342
+ ties up a few loose ends in the PEP 325 spec, to make it suitable
+ for actual implementation.
- See the Examples section near the end.
+Motivation
-Specification: the __next__() Method
+ Coroutines are a natural way of expressing many algorithms, such as
+ simulations, games, asynchronous I/O, and other forms of event-
+ driven programming or co-operative multitasking. Python's generator
+ functions are almost coroutines -- but not quite -- in that they
+ allow pausing execution to produce a value, but do not provide for
+ values or exceptions to be passed in when execution resumes. They
+ also do not allow execution to be paused within the "try" portion of
+ try/finally blocks, and therefore make it difficult for an aborted
+ coroutine to clean up after itself.
- A new method for iterators is proposed, called __next__(). It
- takes one optional argument, which defaults to None. Calling the
- __next__() method without argument or with None is equivalent to
- using the old iterator API, next(). For backwards compatibility,
- it is recommended that iterators also implement a next() method as
- an alias for calling the __next__() method without an argument.
+ Generators also cannot yield control while other functions are
+ executing, unless those functions are also expressed as generators,
+ and the outer generator is written to yield in response to values
+ yielded by the inner generator, which complicates the implementation
+ of even relatively simple use cases like asynchronous
+ communications, because calling any functions either requires the
+ generator to "block" (i.e. be unable to yield control), or else a
+ lot of boilerplate looping code around every needed function call.
- The argument to the __next__() method may be used by the iterator
- as a hint on what to do next.
+ However, if it were possible to pass values or exceptions into a
+ generator at the point where it was suspended, a simple co-routine
+ scheduler or "trampoline function" would let coroutines "call" each
+ other without blocking -- a tremendous boon for asynchronous
+ applications. Such applications could then write co-routines to
+ do non-blocking socket I/O by yielding control to an I/O scheduler
+ until data has been sent or becomes available. Meanwhile, code that
+ performs the I/O would simply do something like:
-Specification: the next() Built-in Function
+ data = (yield nonblocking_read(my_socket, nbytes))
- This is a built-in function defined as follows:
+ In order to pause execution until the nonblocking_read() coroutine
+ produced a value.
- def next(itr, arg=None):
- nxt = getattr(itr, "__next__", None)
- if nxt is not None:
- return nxt(arg)
- if arg is None:
- return itr.next()
- raise TypeError("next() with arg for old-style iterator")
+ In other words, with a few relatively minor enhancements to the
+ language and the implementation of the generator-iterator type,
+ Python will be able to support writing asynchronous applications
+ without needing to write entire applications as a series of
+ callbacks, and without requiring the use resource-intensive threads
+ for programs that need hundreds or even thousands of co-operatively
+ multitasking pseudothreads. In a sense, these enhancements will
+ give Python many of the benefits of the Stackless Python fork,
+ without requiring any significant modification to the CPython core
+ or its APIs. In addition, these enhancements should be readily
+ implementable by any Python implementation (such as Jython) that
+ already supports generators.
- This function is proposed because there is often a need to call
- the next() method outside a for-loop; the new API, and the
- backwards compatibility code, is too ugly to have to repeat in
- user code.
-Specification: a Change to the 'for' Loop
+Specification Summary
- A small change in the translation of the for-loop is proposed.
- The statement
+ By adding a few simple methods to the generator-iterator type, and
+ with two minor syntax adjustments, Python developers will be able
+ to use generator functions to implement co-routines and other forms
+ of co-operative multitasking. These method and adjustments are:
- for VAR1 in EXPR1:
- BLOCK1
- else:
- BLOCK2
+ 1. Redefine "yield" to be an expression, rather than a statement.
+ The current yield statement would become a yield expression
+ whose value is thrown away. A yield expression's value is
+ None if the generator is resumed by a normal next() call.
- will be translated as follows:
+ 2. Add a new send() method for generator-iterators, which resumes
+ the generator and "sends" a value that becomes the result of the
+ current "yield expression". The send() method returns the next
+ value yielded by the generator, or raises StopIteration if the
+ generator exits without yielding another value.
- itr = iter(EXPR1)
- arg = None # Set by "continue EXPR2", see below
- brk = False
- while True:
- try:
- VAR1 = next(itr, arg)
- except StopIteration:
- brk = True
- break
- arg = None
- BLOCK1
- if brk:
- BLOCK2
+ 3. Add a new throw() method for generator-iterators, which raises
+ an exception at the point where the generator was paused, and
+ which returns the next value yielded by the generator, raising
+ StopIteration if the generator exits without yielding another
+ value. (If the generator does not catch the passed-in exception,
+ or raises a different exception, then that exception propagates
+ to the caller.
- (However, the variables 'itr' etc. are not user-visible and the
- built-in names used cannot be overridden by the user.)
+ 4. Add a close() method for generator-iterators, which raises
+ GeneratorExit at the point where the generator was paused. If
+ the generator then raises StopIteration (by exiting normally, or
+ due to already being closed) or GeneratorExit (by not catching
+ the exception), close() returns to its caller. If the generator
+ yields a value, a RuntimeError is raised. If the generator
+ raises any other exception, it is propagated to the caller.
+ close() does nothing if the generator has already exited due to
+ an exception or normal exit.
-Specification: the Extended 'continue' Statement
+ 5. Adding support to ensure that close() is called when a generator
+ iterator is garbage-collected.
- In the translation of the for-loop, inside BLOCK1, the new syntax
+ 6. Allowing "yield" to be used in try/finally blocks, since garbage
+ collection or an explicit close() call allows the finally clause
+ to execute.
- continue EXPR2
+ A prototype patch implementing all of these changes against the
+ current Python CVS HEAD is available as SourceForge patch #1223381
+ (http://python.org/sf/1223381).
- is legal and is translated into
- arg = EXPR2
- continue
+Specification: Sending Values into Generators
- (Where 'arg' references the corresponding hidden variable from the
- previous section.)
+ New generator method: send(value)
- This is also the case in the body of the block-statement proposed
- below.
+ A new method for generator-iterators is proposed, called send(). It
+ takes exactly one argument, which is the value that should be "sent
+ in" to the generator. Calling send(None) is exactly equivalent to
+ calling a generator's next() method. Calling send() with any other
+ value is the same, except that the value produced by the generator's
+ current yield expression will be different.
- EXPR2 may contain commas; "continue 1, 2, 3" is equivalent to
- "continue (1, 2, 3)".
+ Because generator-iterators begin execution at the top of the
+ generator's function body, there is no yield expression to receive
+ a value when the generator has just been created. Therefore,
+ calling send() with a non-None argument is prohibited when the
+ generator iterator has just started, and a TypeError is raised if
+ this occurs (presumably due to a logic error of some kind). Thus,
+ before you can communicate with a coroutine you must call first
+ call next() or send(None) to advance its execution to its first
+ yield expression.
-Specification: Generators and Yield-Expressions
+ As with the next() method, the send() method returns the next value
+ yielded by the generator-iterator, or raises StopIteration if the
+ generator exits normally, or has already exited. If the generator
+ raises an exception, it is propagated to send()'s caller.
- Generators will implement the new __next__() method API, as well
- as the old argument-less next() method which becomes an alias for
- calling __next__() without an argument.
+ New syntax: Yield Expressions
The yield-statement will be allowed to be used on the right-hand
side of an assignment; in that case it is referred to as
yield-expression. The value of this yield-expression is None
- unless __next__() was called with an argument; see below.
+ unless send() was called with a non-None argument; see below.
A yield-expression must always be parenthesized except when it
occurs at the top-level expression on the right-hand side of an
@@ -151,57 +183,318 @@
yield without passing an explicit value ("yield" is of course
equivalent to "yield None").
- When __next__() is called with an argument that is not None, the
- yield-expression that it resumes will return the argument. If it
- resumes a yield-statement, the value is ignored (this is similar
- to ignoring the value returned by a function call). When the
- *initial* call to __next__() receives an argument that is not
- None, TypeError is raised; this is likely caused by some logic
- error. When __next__() is called without an argument or with None
- as argument, and a yield-expression is resumed, the
- yield-expression returns None.
+ When send(value) is called, the yield-expression that it resumes
+ will return the passed-in value. When next() is called, the resumed
+ yield-expression will return None. If the yield-expression is a
+ yield-statement, this returned value is ignored, similar to ignoring
+ the value returned by a function call used as a statement.
+
+ In effect, a yield-expression is like an inverted function call; the
+ argument to yield is in fact returned (yielded) from the currently
+ executing function, and the "return value" of yield is the argument
+ passed in via send().
Note: the syntactic extensions to yield make its use very similar
to that in Ruby. This is intentional. Do note that in Python the
- block passes a value to the generator using "continue EXPR" rather
+ block passes a value to the generator using "send(EXPR)" rather
than "return EXPR", and the underlying mechanism whereby control
is passed between the generator and the block is completely
different. Blocks in Python are not compiled into thunks; rather,
yield suspends execution of the generator's frame. Some edge
cases work differently; in Python, you cannot save the block for
later use, and you cannot test whether there is a block or not.
+ (XXX - this stuff about blocks seems out of place now, perhaps
+ Guido can edit to clarify.)
-Alternative
+Specification: Exceptions and Cleanup
- An alternative proposal is still under consideration, where
- instead of adding a __next__() method, the existing next() method
- is given an optional argument. The next() built-in function is
- then unnecessary. The only line that changes in the translation is
- the line
+ Let a generator object be the iterator produced by calling a
+ generator function. Below, 'g' always refers to a generator
+ object.
- VAR1 = next(itr, arg)
+ New syntax: yield allowed inside try-finally
- which will be replaced by this
+ The syntax for generator functions is extended to allow a
+ yield-statement inside a try-finally statement.
- if arg is None:
- VAR1 = itr.next()
- else:
- VAR1 = itr.next(arg)
+ New generator method: throw(type, value=None, traceback=None)
- If "continue EXPR2" is used and EXPR2 does not evaluate to None,
- and the iterator's next() method does not support the optional
- argument, a TypeError exception will be raised, which is the same
- behavior as above.
+ g.throw(type, value, traceback) causes the specified exception to
+ be thrown at the point where the generator g is currently
+ suspended (i.e. at a yield-statement, or at the start of its
+ function body if next() has not been called yet). If the
+ generator catches the exception and yields another value, that is
+ the return value of g.throw(). If it doesn't catch the exception,
+ the throw() appears to raise the same exception passed it (it
+ "falls through"). If the generator raises another exception (this
+ includes the StopIteration produced when it returns) that
+ exception is raised by the throw() call. In summary, throw()
+ behaves like next() or send(), except it raises an exception at the
+ suspension point. If the generator is already in the closed
+ state, throw() just raises the exception it was passed without
+ executing any of the generator's code.
- This proposal is more compatible (no new method name, no new
- built-in needed) but less future-proof; in some sense it was a
- mistake to call this method next() instead of __next__(), since
- *all* other operations corresponding to function pointers in the C
- type structure have names with leading and trailing underscores.
+ The effect of raising the exception is exactly as if the
+ statement:
+
+ raise type, value, traceback
+
+ was executed at the suspension point. The type argument must
+ not be None, and the type and value must be compatible. If the
+ value is not an instance of the type, a new exception instance
+ is created, with the value passed in as argument(s), following
+ the same rules that the raise statement uses to create an
+ exception instance. The traceback, if supplied, must be a valid
+ Python traceback object, or a TypeError occurs.
+
+ New standard exception: GeneratorExit
+
+ A new standard exception is defined, GeneratorExit, inheriting
+ from Exception. A generator should handle this by re-raising it
+ or by raising StopIteration.
+
+ New generator method: close()
+
+ g.close() is defined by the following pseudo-code:
+
+ def close(self):
+ try:
+ self.throw(GeneratorExit)
+ except (GeneratorExit, StopIteration):
+ pass
+ else:
+ raise RuntimeError("generator ignored GeneratorExit")
+ # Other exceptions are not caught
+
+ New generator method: __del__()
+
+ g.__del__() is a wrapper for g.close(). This will be called when
+ the generator object is garbage-collected (in CPython, this is
+ when its reference count goes to zero). If close() raises an
+ exception, a traceback for the exception is printed to sys.stderr
+ and further ignored; it is not propagated back to the place that
+ triggered the garbage collection. This is consistent with the
+ handling of exceptions in __del__() methods on class instances.
+
+ If the generator object participates in a cycle, g.__del__() may
+ not be called. This is the behavior of CPython's current garbage
+ collector. The reason for the restriction is that the GC code
+ needs to "break" a cycle at an arbitrary point in order to collect
+ it, and from then on no Python code should be allowed to see the
+ objects that formed the cycle, as they may be in an invalid state.
+ Objects "hanging off" a cycle are not subject to this restriction.
+ Note that it is unlikely to see a generator object participate in
+ a cycle in practice. However, storing a generator object in a
+ global variable creates a cycle via the generator frame's
+ f_globals pointer. Another way to create a cycle would be to
+ store a reference to the generator object in a data structure that
+ is passed to the generator as an argument. Neither of these cases
+ are very likely given the typical pattern of generator use.
+
+ Also, in the CPython implementation of this PEP, the frame object
+ used by the generator should be released whenever its execution is
+ terminated due to an error or normal exit. This will ensure that
+ generators that cannot be resumed do not remain part of an
+ uncollectable reference cycle. This allows other code to
+ potentially use close() in a try/finally or "with" block (per PEP
+ 343) to ensure that a given generator is properly finalized.
+
+Optional Extensions
+
+ The Extended 'continue' Statement
+
+ An earlier draft of this PEP proposed a new "continue EXPR"
+ syntax for use in for-loops (carried over from PEP 340), that
+ would pass the value of EXPR into the iterator being looped over.
+ This feature has been withdrawn for the time being, because the
+ scope of this PEP has been narrowed to focus only on passing values
+ into generator-iterators, and not other kinds of iterators. It
+ was also felt by some on the Python-Dev list that adding new syntax
+ for this particular feature would be premature at best.
+
+Open Issues
+
+ Discussion on python-dev has revealed some open issues. I list
+ them here, with my preferred resolution and its motivation. The
+ PEP as currently written reflects this preferred resolution.
+
+ 1. What exception should be raised by close() when the generator
+ yields another value as a response to the GeneratorExit
+ exception?
+
+ I originally chose TypeError because it represents gross
+ misbehavior of the generator function, which should be fixed by
+ changing the code. But the with_template decorator class in
+ PEP 343 uses RuntimeError for similar offenses. Arguably they
+ should all use the same exception. I'd rather not introduce a
+ new exception class just for this purpose, since it's not an
+ exception that I want people to catch: I want it to turn into a
+ traceback which is seen by the programmer who then fixes the
+ code. So now I believe they should both raise RuntimeError.
+ There are some precedents for that: it's raised by the core
+ Python code in situations where endless recursion is detected,
+ and for uninitialized objects (and for a variety of
+ miscellaneous conditions).
+
+ 2. Oren Tirosh has proposed renaming the send() method to feed(),
+ for compatibility with the "consumer interface" (see
+ http://effbot.org/zone/consumer.htm for the specification.)
+
+ However, looking more closely at the consumer interface, it seems
+ that the desired semantics for feed() are different than for
+ send(), because send() can't be meaningfully called on a just-
+ started generator. Also, the consumer interface as currently
+ defined doesn't include handling for StopIteration.
+
+ Therefore, it seems like it would probably be more useful to
+ create a simple decorator that wraps a generator function to make
+ it conform to the consumer interface. For example, it could
+ "warm up" the generator with an initial next() call, trap
+ StopIteration, and perhaps even provide reset() by re-invoking
+ the generator function.
+
+Examples
+
+ 1. A simple co-routine scheduler or "trampoline" that lets
+ coroutines "call" other coroutines by yielding the coroutine
+ they wish to invoke. Any non-generator value yielded by
+ a coroutine is returned to the coroutine that "called" the
+ one yielding the value. Similarly, if a coroutine raises an
+ exception, the exception is propagated to its "caller". In
+ effect, this example emulates simple tasklets as are used
+ in Stackless Python, as long as you use a yield expression to
+ invoke routines that would otherwise "block". This is only
+ a very simple example, and far more sophisticated schedulers
+ are possible. (For example, the existing GTasklet framework
+ for Python (http://www.gnome.org/~gjc/gtasklet/gtasklets.html)
+ and the peak.events framework (http://peak.telecommunity.com/)
+ already implement similar scheduling capabilities, but must
+ currently use awkward workarounds for the inability to pass
+ values or exceptions into generators.)
+
+ import collections
+
+ class Trampoline:
+ """Manage communications between coroutines until
+
+ running = False
+
+ def __init__(self):
+ self.queue = collections.deque
+
+ def add(self, coroutine):
+ """Request that a coroutine be executed"""
+ self.schedule(coroutine)
+
+ def run(self):
+ result = None
+ self.running = True
+ try:
+ while self.running and self.queue:
+ func = self.queue.popleft()
+ result = func()
+ return result
+ finally:
+ self.running = False
+
+ def stop(self):
+ self.running = False
+
+ def schedule(self, coroutine, stack=(), value=None, *exc):
+ def resume():
+ try:
+ if exc:
+ value = coroutine.throw(value,*exc)
+ else:
+ value = coroutine.send(value)
+ except:
+ if stack:
+ # send the error back to the "caller"
+ self.schedule(
+ stack[0], stack[1], *sys.exc_info()
+ )
+ else:
+ # Nothing left in this pseudothread to
+ # handle it, let it propagate to the
+ # run loop
+ raise
+
+ if isinstance(value, types.GeneratorType):
+ # Yielded to a specific coroutine, push the
+ # current one on the stack, and call the new
+ # one with no args
+ self.schedule(value, (coroutine,stack))
+
+ elif stack:
+ # Yielded a result, pop the stack and send the
+ # value to the caller
+ self.schedule(stack[0], stack[1], value)
+
+ # else: this pseudothread has ended
+
+ self.queue.append(resume)
+
+ 2. A simple "echo" server, and code to run it using a trampoline
+ (presumes the existence of "nonblocking_read",
+ "nonblocking_write", and other I/O coroutines, that e.g. raise
+ ConnectionLost if the connection is closed):
+
+ # coroutine function that echos data back on a connected
+ # socket
+ #
+ def echo_handler(sock):
+ while True:
+ try:
+ data = yield nonblocking_read(sock)
+ yield nonblocking_write(sock, data)
+ except ConnectionLost:
+ pass # exit normally if connection lost
+
+ # coroutine function that listens for connections on a
+ # socket, and then launches a service "handler" coroutine
+ # to service the connection
+ #
+ def listen_on(trampoline, sock, handler):
+ while True:
+ # get the next incoming connection
+ connected_socket = yield nonblocking_accept(sock)
+
+ # start another coroutine to handle the connection
+ trampoline.add( handler(connected_socket) )
+
+ # Create a scheduler to manage all our coroutines
+ t = Trampoline()
+
+ # Create a coroutine instance to run the echo_handler on
+ # incoming connections
+ #
+ server = listen_on(
+ t, listening_socket("localhost","echo"), echo_handler
+ )
+
+ # Add the coroutine to the scheduler
+ t.add(server)
+
+ # loop forever, accepting connections and servicing them
+ # "in parallel"
+ #
+ t.run()
+
+
+Reference Implementation
+
+ A prototype patch implementing all of the features described in this
+ PEP is available as SourceForge patch #1223381
+ (http://python.org/sf/1223381).
Acknowledgements
- See Acknowledgements of PEP 340.
+ Raymond Hettinger (PEP 288) and Samuele Pedroni (PEP 325) first
+ formally proposed the ideas of communicating values or exceptions
+ into generators, and the ability to "close" generators. Timothy
+ Delaney suggested the title of this PEP, and Steven Bethard helped
+ edit a previous version. See also the Acknowledgements section
+ of PEP 340.
References
More information about the Python-checkins
mailing list