PEP 255: Simple Generators

[Terry Reedy]

When I first read the PEP, I assumed that def'ed generators would somehow
return the successive yielded values on successive calls (wrong).  IE,
yield would be like return but without tossing the in-process information
(partly right, partly wrong).  As I now understand it, putting 'yield'
somewhere within a function body will *also* magically transform the
function definition into syntactic sugar for an iteration factory function.
IE

def squares(n):
  for i in range(n):
    yield i*i

will be an abbreviation for

def squares(n):
  return make_iterator('squares', '''
for i in range(n):
  yield i*i
''')
where make_iterator adds 'raise StopIteration' where appropriate.

[Or even much more verbosely, it abbreviates (in this particular example)
something like

class squares:
  def __init__(self, n):
    self.i = 0
    self.n = n
  def next(self):
     i = self.i
    if i < self.n: return i*i
    else raise StopIteration

But note that simulating 'for' is slower and more error-prone.]

The existing separation of function and code objects makes this magic
behind-the-scenes swap neatly possible.  Compile normally (with the
addition of 'yield') up to the last step, but instead of attaching the
compiled code object to the compiled function object, substitute a standard
make_iterator code object that when called will attach the compiled code
object to an instance of a standard iterator object as a next method.

This *is* a bit mind-bending.  I predict that at least some newcomers will
react to it much the same way that even experienced Pythoneers react to
metaclasses.  Careful explanation will be necessary.  Examples like that
above might help.

Question: am I correct in thinking that separating iterator-generation from
iterator-first-use makes it possible to have multiple iterators from the
same generator in simultaneous use, as in

sq10 = squares(10)
sq20 = squares(20)
...
?

On def versus gen(erator) as the keyword:  regardless of the word, the
magic swap takes some thinking about.  The bad of gen: it introduces
another keyword, and one for a process that is identical to def up to the
magic swap.  The good of gen: it announces the swap right away.
Alternative warning mechanism: a simple comment convention, as in

def squares(n): # generator for squares from 0 to (n-1)**2
--or--
def squares(n): # make iterator for squares from 0 to (n-1)**2

On body lines having global effects: the proposed yield 'wart' is similar
to the assignment-makes-local 'wart', as in

i = 387
def nada:
  print i
... (many lines)
  i = 'haha'

Lesson: function bodies are scanned globally  to determine their final
compilation and processing.


Terry J. Reedy