RE: [Python-Dev] PEP 289 - Generator Expressions
Try a concrete example?
t1 = (1,2,3) t2 = ("a", "b", "c") t3 = ("first", "second", "third") (x,y,z for x in t1 for y in t2 for z in t3) Do What I Mean is the crossproduct, 27 elements long, which you get with a list comprehension. This will happen if at least (all but one) generators are pre-evaluated. It will also happen if at least (all but one) generators are restarted after they raise StopIterator. Doing strictly what happens today gets only (1,'a',"first"), (1,'a',"second"), (1,'a',"third") because t3 has raised StopIteration by the time t1 or t2 advance. Effectively, only the last iterator matters. Keeping the last value of an iterator also adds (1, 'b', "third"), (1, 'c', "third"), (2, 'c', "third"), (3, '3', "third") for seven values -- but perhaps not a well-chosen seven. Pre-evaluating only the leftmost gets all combinations of (1,2,3) and ("first", "second", "third"), but still misses items like (2, 'b', "second"). Since these are in the middle of the series, they are the most likely to be forgotten during testing. Restarting the second (and later) iterators for each value of the first iterator does the right thing, without requiring any magical pre-evaluation. (Unless the iterator can't be restarted -- in which case there is no right thing, and the user should have decided what to cache.) Thus my suggestion of (x,y,z for x in t1 for y in t2 for z in t3) <==> for x in t1.__iter__(): for y in t2.__iter__(): for z in t3.__iter__(): yield x,y,z For the example data, this creates 3 separate iterators on the letters and nine iterators on the words - but produces the expected result, without having to store or pre-evaluate anything. -jJ
[Jewett, Jim J]
t1 = (1,2,3) t2 = ("a", "b", "c") t3 = ("first", "second", "third")
(x,y,z for x in t1 for y in t2 for z in t3)
Do What I Mean is the crossproduct, 27 elements long, which you get with a list comprehension.
Well, that expression by itself will result in a generator-iterator getting created, with no actual results. If you put it in a context that drives generation, then it will deliver 27 results, and regardless of whether early-binding, late-binding, or mostly-late binding is used (although the latter two are sensitive to changes in the bindings for t2 and t3, if any such occur between the time the genexp is created and the time the genexp is driven). For mostly-late binding, it's the same as this Python program today (stuffing the genexp in a list() call to force it to do something): """ t1 = (1,2,3) t2 = ("a", "b", "c") t3 = ("first", "second", "third") def __g(primary): for x in primary: for y in t2: for z in t3: yield x, y, z print list(__g(t1)) """ Try running that, and see what you get.
This will happen if at least (all but one) generators are pre-evaluated.
There are no generators in your example, apart from the one created by the genexp itself. You may be confusing generators with iterable objects.
It will also happen if at least (all but one) generators are restarted after they raise StopIterator.
Doing strictly what happens today gets only
(1,'a',"first"), (1,'a',"second"), (1,'a',"third")
No -- run the above. You'll get the 27 you expect.
because t3 has raised StopIteration by the time t1 or t2 advance.
The innermost "for" loop swallows StopIteration each of the 9 times it occurs, and the outer for loops never see it. They keep going. This is how "for" loops always work (nothing is changing wrt that).
participants (2)
-
Jewett, Jim J -
Tim Peters