[Python-checkins] r85965 - peps/trunk/pep-0380.txt
georg.brandl
python-checkins at python.org
Sat Oct 30 08:52:39 CEST 2010
Author: georg.brandl
Date: Sat Oct 30 08:52:39 2010
New Revision: 85965
Log:
PEP 380 update from Greg.
Modified:
peps/trunk/pep-0380.txt
Modified: peps/trunk/pep-0380.txt
==============================================================================
--- peps/trunk/pep-0380.txt (original)
+++ peps/trunk/pep-0380.txt Sat Oct 30 08:52:39 2010
@@ -15,10 +15,10 @@
========
A syntax is proposed for a generator to delegate part of its
-operations to another generator. This allows a section of code
-containing 'yield' to be factored out and placed in another
-generator. Additionally, the subgenerator is allowed to return with a
-value, and the value is made available to the delegating generator.
+operations to another generator. This allows a section of code
+containing 'yield' to be factored out and placed in another generator.
+Additionally, the subgenerator is allowed to return with a value, and
+the value is made available to the delegating generator.
The new syntax also opens up some opportunities for optimisation when
one generator re-yields values produced by another.
@@ -35,8 +35,8 @@
it is necessary to explicitly iterate over this second generator and
re-yield any values that it produces.
-If yielding of values is the only concern, this can be performed without
-much difficulty using a loop such as
+If yielding of values is the only concern, this can be performed
+without much difficulty using a loop such as
::
@@ -44,15 +44,15 @@
yield v
However, if the subgenerator is to interact properly with the caller
-in the case of calls to ``send()``, ``throw()`` and ``close()``, things
-become considerably more difficult. As will be seen later, the necessary
-code is very complicated, and it is tricky to handle all the corner cases
-correctly.
-
-A new syntax will be proposed to address this issue. In the simplest
-use cases, it will be equivalent to the above for-loop, but it will also
-handle the full range of generator behaviour, and allow generator code
-to be refactored in a simple and straightforward way.
+in the case of calls to ``send()``, ``throw()`` and ``close()``,
+things become considerably more difficult. As will be seen later, the
+necessary code is very complicated, and it is tricky to handle all the
+corner cases correctly.
+
+A new syntax will be proposed to address this issue. In the simplest
+use cases, it will be equivalent to the above for-loop, but it will
+also handle the full range of generator behaviour, and allow generator
+code to be refactored in a simple and straightforward way.
Proposal
@@ -81,27 +81,29 @@
* Any values that the iterator yields are passed directly to the
caller.
- * Any values sent to the delegating generator using ``send()``
- are passed directly to the iterator. If the sent value is None,
- the iterator's ``next()`` method is called. If the sent value is
- not None, the iterator's ``send()`` method is called. If the call
- raises StopIteration, the delegating generator is resumed. Any other
- exception is propagated to the delegating generator.
+ * Any values sent to the delegating generator using ``send()`` are
+ passed directly to the iterator. If the sent value is None, the
+ iterator's ``__next__()`` method is called. If the sent value
+ is not None, the iterator's ``send()`` method is called. If the
+ call raises StopIteration, the delegating generator is resumed.
+ Any other exception is propagated to the delegating generator.
* Exceptions other than GeneratorExit thrown into the delegating
generator are passed to the ``throw()`` method of the iterator.
- If the call raises StopIteration, the delegating generator is resumed.
- Any other exception is propagated to the delegating generator.
-
- * If a GeneratorExit exception is thrown into the delegating generator,
- or the ``close()`` method of the delegating generator is called, then
- the ``close()`` method of the iterator is called if it has one. If this
- call results in an exception, it is propagated to the delegating generator.
- Otherwise, GeneratorExit is raised in the delegating generator.
+ If the call raises StopIteration, the delegating generator is
+ resumed. Any other exception is propagated to the delegating
+ generator.
+
+ * If a GeneratorExit exception is thrown into the delegating
+ generator, or the ``close()`` method of the delegating generator
+ is called, then the ``close()`` method of the iterator is called
+ if it has one. If this call results in an exception, it is
+ propagated to the delegating generator. Otherwise,
+ GeneratorExit is raised in the delegating generator.
* The value of the ``yield from`` expression is the first argument
- to the ``StopIteration`` exception raised by the iterator when it
- terminates.
+ to the ``StopIteration`` exception raised by the iterator when
+ it terminates.
* ``return expr`` in a generator causes ``StopIteration(expr)`` to
be raised upon exit from the generator.
@@ -120,15 +122,11 @@
Python 3 syntax is used in this section.
-1. The statement
-
-::
+1. The statement ::
RESULT = yield from EXPR
-is semantically equivalent to
-
-::
+is semantically equivalent to ::
_i = iter(EXPR)
try:
@@ -171,24 +169,18 @@
RESULT = _r
-2. In a generator, the statement
-
-::
+2. In a generator, the statement ::
return value
-is semantically equivalent to
-
-::
+is semantically equivalent to ::
raise StopIteration(value)
-except that, as currently, the exception cannot be caught by ``except``
-clauses within the returning generator.
+except that, as currently, the exception cannot be caught by
+``except`` clauses within the returning generator.
-3. The StopIteration exception behaves as though defined thusly:
-
-::
+3. The StopIteration exception behaves as though defined thusly::
class StopIteration(Exception):
@@ -207,22 +199,23 @@
-------------------------
The rationale behind most of the semantics presented above stems from
-the desire to be able to refactor generator code. It should be possible
-to take an section of code containing one or more ``yield`` expressions,
-move it into a separate function (using the usual techniques to deal
-with references to variables in the surrounding scope, etc.), and
-call the new function using a ``yield from`` expression.
+the desire to be able to refactor generator code. It should be
+possible to take an section of code containing one or more ``yield``
+expressions, move it into a separate function (using the usual
+techniques to deal with references to variables in the surrounding
+scope, etc.), and call the new function using a ``yield from``
+expression.
The behaviour of the resulting compound generator should be, as far as
reasonably practicable, the same as the original unfactored generator
-in all situations, including calls to ``next()``, ``send()``,
+in all situations, including calls to ``__next__()``, ``send()``,
``throw()`` and ``close()``.
The semantics in cases of subiterators other than generators has been
chosen as a reasonable generalization of the generator case.
-The proposed semantics have the following limitations with regard
-to refactoring:
+The proposed semantics have the following limitations with regard to
+refactoring:
* A block of code that catches GeneratorExit without subsequently
re-raising it cannot be factored out while retaining exactly the
@@ -238,22 +231,22 @@
Finalization
------------
-There was some debate as to whether explicitly finalizing the delegating
-generator by calling its ``close()`` method while it is suspended at a
-``yield from`` should also finalize the subiterator. An argument against
-doing so is that it would result in premature finalization of the
-subiterator if references to it exist elsewhere.
+There was some debate as to whether explicitly finalizing the
+delegating generator by calling its ``close()`` method while it is
+suspended at a ``yield from`` should also finalize the subiterator.
+An argument against doing so is that it would result in premature
+finalization of the subiterator if references to it exist elsewhere.
Consideration of non-refcounting Python implementations led to the
decision that this explicit finalization should be performed, so that
explicitly closing a factored generator has the same effect as doing
so to an unfactored one in all Python implementations.
-The assumption made is that, in the majority of use cases, the subiterator
-will not be shared. The rare case of a shared subiterator can be
-accommodated by means of a wrapper that blocks ``throw()`` and ``close()``
-calls, or by using a means other than ``yield from`` to call the
-subiterator.
+The assumption made is that, in the majority of use cases, the
+subiterator will not be shared. The rare case of a shared subiterator
+can be accommodated by means of a wrapper that blocks ``throw()`` and
+``close()`` calls, or by using a means other than ``yield from`` to
+call the subiterator.
Generators as Threads
@@ -267,38 +260,34 @@
ordinary function, passing it parameters and receiving a returned
value.
-Using the proposed syntax, a statement such as
-
-::
+Using the proposed syntax, a statement such as ::
y = f(x)
where f is an ordinary function, can be transformed into a delegation
-call
-
-::
+call ::
y = yield from g(x)
-where g is a generator. One can reason about the behaviour of the
+where g is a generator. One can reason about the behaviour of the
resulting code by thinking of g as an ordinary function that can be
suspended using a ``yield`` statement.
When using generators as threads in this way, typically one is not
interested in the values being passed in or out of the yields.
However, there are use cases for this as well, where the thread is
-seen as a producer or consumer of items. The ``yield from``
-expression allows the logic of the thread to be spread over as
-many functions as desired, with the production or consumption of
-items occuring in any subfunction, and the items are automatically
-routed to or from their ultimate source or destination.
+seen as a producer or consumer of items. The ``yield from``
+expression allows the logic of the thread to be spread over as many
+functions as desired, with the production or consumption of items
+occuring in any subfunction, and the items are automatically routed to
+or from their ultimate source or destination.
Concerning ``throw()`` and ``close()``, it is reasonable to expect
that if an exception is thrown into the thread from outside, it should
-first be raised in the innermost generator where the thread is suspended,
-and propagate outwards from there; and that if the thread is terminated
-from outside by calling ``close()``, the chain of active generators
-should be finalised from the innermost outwards.
+first be raised in the innermost generator where the thread is
+suspended, and propagate outwards from there; and that if the thread
+is terminated from outside by calling ``close()``, the chain of active
+generators should be finalised from the innermost outwards.
Syntax
@@ -315,32 +304,32 @@
Using a specialised syntax opens up possibilities for optimisation
when there is a long chain of generators. Such chains can arise, for
instance, when recursively traversing a tree structure. The overhead
-of passing ``next()`` calls and yielded values down and up the chain
-can cause what ought to be an O(n) operation to become, in the worst
-case, O(n\*\*2).
+of passing ``__next__()`` calls and yielded values down and up the
+chain can cause what ought to be an O(n) operation to become, in the
+worst case, O(n\*\*2).
A possible strategy is to add a slot to generator objects to hold a
-generator being delegated to. When a ``next()`` or ``send()`` call is
-made on the generator, this slot is checked first, and if it is
-nonempty, the generator that it references is resumed instead. If it
-raises StopIteration, the slot is cleared and the main generator is
+generator being delegated to. When a ``__next__()`` or ``send()``
+call is made on the generator, this slot is checked first, and if it
+is nonempty, the generator that it references is resumed instead. If
+it raises StopIteration, the slot is cleared and the main generator is
resumed.
This would reduce the delegation overhead to a chain of C function
-calls involving no Python code execution. A possible enhancement would
-be to traverse the whole chain of generators in a loop and directly
-resume the one at the end, although the handling of StopIteration is
-more complicated then.
+calls involving no Python code execution. A possible enhancement
+would be to traverse the whole chain of generators in a loop and
+directly resume the one at the end, although the handling of
+StopIteration is more complicated then.
Use of StopIteration to return values
-------------------------------------
There are a variety of ways that the return value from the generator
-could be passed back. Some alternatives include storing it as an
+could be passed back. Some alternatives include storing it as an
attribute of the generator-iterator object, or returning it as the
-value of the ``close()`` call to the subgenerator. However, the proposed
-mechanism is attractive for a couple of reasons:
+value of the ``close()`` call to the subgenerator. However, the
+proposed mechanism is attractive for a couple of reasons:
* Using a generalization of the StopIteration exception makes it easy
for other kinds of iterators to participate in the protocol without
@@ -348,7 +337,7 @@
* It simplifies the implementation, because the point at which the
return value from the subgenerator becomes available is the same
- point at which the exception is raised. Delaying until any later
+ point at which the exception is raised. Delaying until any later
time would require storing the return value somewhere.
@@ -358,9 +347,9 @@
Some ideas were discussed but rejected.
Suggestion: There should be some way to prevent the initial call to
-next(), or substitute it with a send() call with a specified value,
-the intention being to support the use of generators wrapped so that
-the initial next() is performed automatically.
+__next__(), or substitute it with a send() call with a specified
+value, the intention being to support the use of generators wrapped so
+that the initial __next__() is performed automatically.
Resolution: Outside the scope of the proposal. Such generators should
not be used with ``yield from``.
@@ -369,11 +358,27 @@
value, return that value from the ``close()`` call to the delegating
generator.
-Resolution: Undesirable for a number of reasons. The purpose of closing
-a generator is to ensure proper cleanup, not to obtain a meaningful
-return value. Also, it would be unreliable unless the return value were
-stored so as to be available to subsequent close calls, which would
-cause it to persist for longer than expected.
+The motivation for this feature is so that the end of a stream of
+values being sent to a generator can be signalled by closing the
+generator. The generator would catch GeneratorExit, finish its
+computation and return a result, which would then become the return
+value of the close() call.
+
+Resolution: This usage of close() and GeneratorExit would be
+incompatible with their current role as a bail-out and clean-up
+mechanism. It would require that when closing a delegating generator,
+after the subgenerator is closed, the delegating generator be resumed
+instead of re-raising GeneratorExit. But this is not acceptable,
+because it would fail to ensure that the delegating generator is
+finalised properly in the case where close() is being called for
+cleanup purposes.
+
+Signalling the end of values to a consumer is better addressed by
+other means, such as sending in a sentinel value or throwing in an
+exception agreed upon by the producer and consumer. The consumer can
+then detect the sentinel or exception and respond by finishing its
+computation and returning normally. Such a scheme behaves correctly
+in the presence of delegation.
Suggestion: If ``close()`` is not to return a value, then raise an
exception if StopIteration with a non-None value occurs.
@@ -385,26 +390,36 @@
Criticisms
==========
-Under this proposal, the value of a ``yield from`` expression would
-be derived in a very different way from that of an ordinary ``yield``
-expression. This suggests that some other syntax not containing the
-word ``yield`` might be more appropriate, but no acceptable alternative
-has so far been proposed. Rejected alternatives include ``call``,
-``delegate`` and ``gcall``.
-
-It has been suggested that some mechanism other than ``return`` in
-the subgenerator should be used to establish the value returned by
-the ``yield from`` expression. However, this would interfere with
-the goal of being able to think of the subgenerator as a suspendable
+Under this proposal, the value of a ``yield from`` expression would be
+derived in a very different way from that of an ordinary ``yield``
+expression. This suggests that some other syntax not containing the
+word ``yield`` might be more appropriate, but no acceptable
+alternative has so far been proposed. Rejected alternatives include
+``call``, ``delegate`` and ``gcall``.
+
+It has been suggested that some mechanism other than ``return`` in the
+subgenerator should be used to establish the value returned by the
+``yield from`` expression. However, this would interfere with the
+goal of being able to think of the subgenerator as a suspendable
function, since it would not be able to return values in the same way
as other functions.
The use of an exception to pass the return value has been criticised
as an "abuse of exceptions", without any concrete justification of
-this claim. In any case, this is only one suggested implementation;
+this claim. In any case, this is only one suggested implementation;
another mechanism could be used without losing any essential features
of the proposal.
+It has been suggested that a different exception, such as
+GeneratorReturn, should be used instead of StopIteration to return a
+value. However, no convincing practical reason for this has been put
+forward, and the addition of a ``value`` attribute to StopIteration
+mitigates any difficulties in extracting a return value from a
+StopIteration exception that may or may not have one. Also, using a
+different exception would mean that, unlike ordinary functions,
+'return' without a value in a generator would not be equivalent to
+'return None'.
+
Alternative Proposals
=====================
@@ -418,19 +433,21 @@
To the author's knowledge, previous proposals have focused only on
yielding values, and thereby suffered from the criticism that the
two-line for-loop they replace is not sufficiently tiresome to write
-to justify a new syntax. By dealing with the full generator
-protocol, this proposal provides considerably more benefit.
+to justify a new syntax. By dealing with the full generator protocol,
+this proposal provides considerably more benefit.
Additional Material
===================
-Some examples of the use of the proposed syntax are available, and also a
-prototype implementation based on the first optimisation outlined above.
+Some examples of the use of the proposed syntax are available, and
+also a prototype implementation based on the first optimisation
+outlined above.
`Examples and Implementation`_
-.. _Examples and Implementation: http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/
+.. _Examples and Implementation:
+ http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/
Copyright
More information about the Python-checkins
mailing list