[ python-Bugs-955772 ] Nested generator terminates prematurely

[SourceForge.net]

Bugs item #955772, was opened at 2004-05-18 13:02
Message generated for change (Comment added) made by ygale
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=955772&group_id=5470

Category: Python Interpreter Core
Group: None
Status: Closed
Resolution: Invalid
Priority: 5
Submitted By: Yitz Gale (ygale)
Assigned to: Nobody/Anonymous (nobody)
Summary: Nested generator terminates prematurely

Initial Comment:
def g(x, y):
  for i in x:
    for j in y:
      yield i, j

r2 = (0, 1)
[e for e in g(r2, g(r2, r2))]

Expected result:

[(0, (0, 0)), (0, (0, 1)), (0, (1, 0)), (0, (1, 1)),
 (1, (0, 0)), (1, (0, 1)), (1, (1, 0)), (1, (1, 1))]

Actual result:

[(0, (0, 0)), (0, (0, 1)), (0, (1, 0)), (0, (1, 1))]

----------------------------------------------------------------------

>Comment By: Yitz Gale (ygale)
Date: 2004-05-20 00:33

Message:
Logged In: YES 
user_id=1033539

OK. I can get the semantics I want using the following:

def g(x, y):
  for i in x:
    for j in y:
      yield i, j
g = restartable(g)

where I have defined:

class restartable:
  def __init__(self, genfn):
    self.genfn = genfn
  def __call__(self, *args):
    return restartable_generator(self.genfn, *args)

class restartable_generator:
  def __init__(self, genfn, *args):
    self.genfn = genfn
    self.args = args
  def __iter__(self):
    return self.genfn(*self.args)


----------------------------------------------------------------------

Comment By: Raymond Hettinger (rhettinger)
Date: 2004-05-19 19:36

Message:
Logged In: YES 
user_id=80475

Marking this as invalid and closing.

Sorry, non-re-iterability is documented fact of life in the
world of generators and iterators.

The work arounds include making the inner generator into a
list or re-instantiating a new generator on every loop:
 
def g(x, y):
    for i in x
        for j in g(x, y)
             yield i, j


----------------------------------------------------------------------

Comment By: Armin Rigo (arigo)
Date: 2004-05-19 15:55

Message:
Logged In: YES 
user_id=4771

Your issue is that you only create a total of two generator
instances.  The 'inner' one is immediately exhausted. 
Afterwards, this same instance is used again on other 'for'
loops but this has no effect, as it has already been exhausted.

The difference is the same as between r2 and iter(r2).  If
you do that:

it = iter(r2)
for x in it: print x
for x in it: print x

the second loop will be empty beause the first loop has
exhausted the iterator.  Generators are iterators (like it)
and not sequences (like r2).

Using the same iterator on several 'for' loops is useful,
though (e.g. if the first loop can be interrupted with
'break'), so there is no way your code could raise an
exception, short of saying that it is not allowed to call
next() on already-exhausted iterators -- this would be too
big a change.

----------------------------------------------------------------------

Comment By: Yitz Gale (ygale)
Date: 2004-05-19 14:59

Message:
Logged In: YES 
user_id=1033539

Python functions can be called recursively
in general. They know how to save their
local namespace in a separate
frame for each call. That includes arguments,
since the arguments live in the local namespace
of the function.

Generator functions also seem to be supported,
as my example shows.

There is a restriction on a generator object
that you may not call its next() method again
while a previous call to next() is still running.

But this is definitely not a case of that
restriction - we have two separate generator
instances, and each ought to have its own frame.

If there is some other restriction, I think it ought
to be documented. And if possible, it should
raise an exception, like the other restriction.

This smells like a bug to me, though.

----------------------------------------------------------------------

Comment By: Michael Hudson (mwh)
Date: 2004-05-19 13:59

Message:
Logged In: YES 
user_id=6656

Well, it's impossible in general.  You'd have to store any
arguments the generator took somewhere too, wouldn't you?

What about things like:

def foo(aList):
      while aList:
           yield aList.pop()

?

----------------------------------------------------------------------

Comment By: Yitz Gale (ygale)
Date: 2004-05-19 13:57

Message:
Logged In: YES 
user_id=1033539

Too bad. What exactly is the restriction?
I didn't find anything in the docs. And things
like this often do work and are useful.

For example:

def primes():
  yield 2
  for n in count(3):
    for p in primes():
      if p > sqrt(n):
        yield n
        break
      if n % p == 0:
        break


----------------------------------------------------------------------

Comment By: Michael Hudson (mwh)
Date: 2004-05-18 14:24

Message:
Logged In: YES 
user_id=6656

Um.  I think the answer to this is "generators are not
reiterable".

----------------------------------------------------------------------

Comment By: Yitz Gale (ygale)
Date: 2004-05-18 13:04

Message:
Logged In: YES 
user_id=1033539

Trying again to get the indentation correct:

def g(x, y):
  for i in x:
    for j in y:
      yield i, j


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=955772&group_id=5470