On Sat, Oct 6, 2012 at 4:10 PM, Serhiy Storchaka wrote:

As StopIteration now have value, this value is lost when using functions which works with iterators/generators (map, filter, itertools). Therefore, wrapping the iterator, which preserved its semantics in versions before 3.3, no longer preserves it:

map(lambda x: x, iterator) filter(lambda x: True, iterator) itertools.accumulate(iterator, lambda x, y: y) itertools.chain(iterator) itertools.compress(iterator, itertools.cycle([True])) itertools.dropwhile(lambda x: False, iterator) itertools.filterfalse(lambda x: False, iterator) next(itertools.groupby(iterator, lambda x: None))[1] itertools.takewhile(lambda x: True, iterator) itertools.tee(iterator, 1)[0]

Perhaps it would be worth to propagate original exception (or at least it's value) in functions for which it makes sense.

Can you provide an example of a time when you want to use such a value with a generator on which you want to use one of these so I can better understand why this is necessary? the times I'm familiar with wanting this value I'd usually be manually stepping through my generator. Mike

On Sat, Oct 6, 2012 at 6:09 PM, Steven D'Aprano wrote:

A concrete example would be useful for those who don't know about the (new?) StopIteration.value attribute. I think you are referring to this:

py> def myiter(): ... yield 1 ... raise StopIteration("spam") ... py> it = map(lambda x:x, myiter()) py> next(it) 1 py> next(it) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration

The argument given to StopIteration is eaten by map.

But this is not *new* to 3.3, it goes back to at least 2.4, so I'm not sure if you are talking about this or something different.

What's new in 3.3 (due to PEP 380) is that instead of the rather awkward and uncommon raise StopIteration("spam") you can now write return "spam" with exactly the same effect. But yes, this was all considered and accepted when PEP 380 was debated (endlessly :-), and I see no reason to change anything about this. "Don't do that" is the best I can say about it -- there are a zillion other situations in Python where that's the only sensible motto. -- --Guido van Rossum (python.org/~guido)

On 06.10.12 23:47, Mike Graham wrote:

Can you provide an example of a time when you want to use such a value with a generator on which you want to use one of these so I can better understand why this is necessary? the times I'm familiar with wanting this value I'd usually be manually stepping through my generator.

There are no many uses yet because it's a new feature. Python 3.3 just released. For example see the proposed patch for http://bugs.python.org/issue16009. In general case `yield from` returns such a value.

On 07/10/2012 8:06pm, Serhiy Storchaka wrote:

I propose make it equivalent to:

def chain(*iterables): value = None for it in iterables: value = yield from it return value

That means that all but the last return value is ignored. Why is the last return value any more important than the earlier ones? ISTM it would make just as much sense to do def chain(*iterables): values = [] for it in iterables: values.append(yield from it) return values But I don't see any point in changing the current behaviour. Richard

On Sun, Oct 7, 2012 at 12:30 PM, Serhiy Storchaka wrote:

On 07.10.12 04:45, Guido van Rossum wrote:

...

But yes, this was all considered and accepted when PEP 380 was debated (endlessly :-), and I see no reason to change anything about this.

The reason is that when someone uses StopIteration.value for some purposes, he will lose this value if the iterator will be wrapped into itertools.chain (quite often used technique) or into other standard iterator wrapper.

If this is just about iterator.chain() I may see some value in it (but TBH the discussion so far mostly confuses -- please spend some more time coming up with good examples that show actually useful use cases rather than f() and g() or foo() and bar()) OTOH yield from is not primarily for iterators -- it is for coroutines. I suspect most of the itertools functionality just doesn't work with coroutines.

...

"Don't do that" is the best I can say about it -- there are a zillion other situations in Python where that's the only sensible motto.

The problem is that two different authors can use two legal techniques (using values returned by "yield from" and wrap iterators with itertools.chain) which do not work in combination. The conflict easily solved if instead of standard itertools.chain to use handwriten code. It looks as bug in itertools.chain.

Okay, so please do work out a complete, useful use case. We may yet see the light. -- --Guido van Rossum (python.org/~guido)

On 7 October 2012 21:19, Guido van Rossum wrote:

On Sun, Oct 7, 2012 at 12:30 PM, Serhiy Storchaka wrote:

...

On 07.10.12 04:45, Guido van Rossum wrote:

...

But yes, this was all considered and accepted when PEP 380 was debated (endlessly :-), and I see no reason to change anything about this.

The reason is that when someone uses StopIteration.value for some purposes, he will lose this value if the iterator will be wrapped into itertools.chain (quite often used technique) or into other standard iterator wrapper.

If this is just about iterator.chain() I may see some value in it (but TBH the discussion so far mostly confuses -- please spend some more time coming up with good examples that show actually useful use cases rather than f() and g() or foo() and bar())

OTOH yield from is not primarily for iterators -- it is for coroutines. I suspect most of the itertools functionality just doesn't work with coroutines.

I think what Serhiy is saying is that although pep 380 mainly discusses generator functions it has effectively changed the definition of what it means to be an iterator for all iterators: previously an iterator was just something that yielded values but now it also returns a value. Since the meaning of an iterator has changed, functions that work with iterators need to be updated. Before pep 380 filter(lambda x: True, obj) returned an object that was the same kind of iterator as obj (it would yield the same values). Now the "kind of iterator" that obj is depends not only on the values that it yields but also on the value that it returns. Since filter does not pass on the same return value, filter(lambda x: True, obj) is no longer the same kind of iterator as obj. The same considerations apply to many other functions such as map, itertools.groupby, itertools.dropwhile. Cases like itertools.chain and zip are trickier since they each act on multiple underlying iterables. Probably chain should return a tuple of the return values from each of its iterables. This feature was new in Python 3.3 which was released a week ago so it is not widely used but it has uses that are not anything to do with coroutines. As an example of how you could use it, consider parsing a file that can contains #include statements. When the #include statement is encountered we need to insert the contents of the included file. This is easy to do with a recursive generator. The example uses the return value of the generator to keep track of which line is being parsed in relation to the flattened output file: def parse(filename, output_lineno=0): with open(filename) as fin: for input_lineno, line in enumerate(fin): if line.startswith('#include '): subfilename = line.split()[1] output_lineno = yield from parse(subfilename, output_lineno) else: try: yield parse_line(line) except ParseLineError: raise ParseError(filename, input_lineno, output_lineno) output_lineno += 1 return output_lineno When writing code like the above that depends on being able to get the value returned from an iterator, it is no longer possible to freely mix utilities like filter, map, zip, itertools.chain with the iterators returned by parse() as they no longer act as transparent wrappers over the underlying iterators (by not propagating the value attached to StopIteration). Hopefully, I've understood Serhiy and the docs correctly (I don't have access to Python 3.3 right now to test any of this). Oscar

Serhiy Storchaka wrote:

The conflict easily solved if instead of standard itertools.chain to use handwriten code. It looks as bug in itertools.chain.

Don't underestimate the value of handwritten code. It makes the intent clear to the reader, whereas relying on some arbitrary default behaviour of a function doesn't. -- Greg

On 10/7/2012 7:30 PM, Greg Ewing wrote:

Oscar Benjamin wrote:

...

Before pep 380 filter(lambda x: True, obj) returned an object that was the same kind of iterator as obj (it would yield the same values). Now the "kind of iterator" that obj is depends not only on the values that it yields but also on the value that it returns. Since filter does not pass on the same return value, filter(lambda x: True, obj) is no longer the same kind of iterator as obj.

Something like this has happened before, when the ability to send() values into a generator was added. If you wrap a generator with filter, you likewise don't get the same kind of object -- you don't get the ability to send() things into your filtered generator.

So, "provide the same kind of iterator" is not currently part of the contract of these functions.

Iterators are Python's generic sequential access device. They do that one thing and do it well. The iterator protocol is intentionally and properly minimal. An iterator class *must* have appropriate .__iter__ and .__next__ methods. It *may* also have any other method and any data attribute. Indeed, any iterator much have some specific internal data. But these are ignored in generic iterator (or iterable) functions. If one does not want that, one should write more specific code. For instance, file objects are iterators. Wrappers such as filter(lambda line: line[0] != '\n', open('somefile')) do not have any of the many other file methods and attributes. No one expects otherwise. If one needs access to the other attributes of the file object, one keeps a direct reference to the file object. Hence, the recommended idiom is to use a with statement. Generators are another class of objects that are both iterators (and hence iterables) and something more. When they are used as input arguments to generic functions of iterables, the other behaviors are ignored, and should be ignored, just as with file objects and any other iterator+ objects. Serhily, if you want a module of *generator* specific functions ('gentools' ?), you should write one and submit it to pypi for testing. -- Terry Jan Reedy

On 08.10.12 01:43, Oscar Benjamin wrote:

Hopefully, I've understood Serhiy and the docs correctly (I don't have access to Python 3.3 right now to test any of this).

Thank you for explanation and example, Oscar.

On 07.10.12 22:18, Richard Oudkerk wrote:

That means that all but the last return value is ignored. Why is the last return value any more important than the earlier ones?

Because I think the last return value more useful for idiom lookahead = next(iterator) process(lookahead) iterator = itertools.chain([lookahead], iterator)

ISTM it would make just as much sense to do

def chain(*iterables): values = [] for it in iterables: values.append(yield from it) return values

It changes the behavior for iterators. And now more difficult to get a generator which yields and returns the same values as the original. We need yet one wrapper. def lastvalue(generator): return (yield from generator)[-1] iterator = lastvalue(itertools.chain([lookahead], iterator)) Yes, it can work.

On 8 October 2012 00:36, Guido van Rossum wrote:

On Sun, Oct 7, 2012 at 3:43 PM, Oscar Benjamin wrote:

...

I think what Serhiy is saying is that although pep 380 mainly discusses generator functions it has effectively changed the definition of what it means to be an iterator for all iterators: previously an iterator was just something that yielded values but now it also returns a value. Since the meaning of an iterator has changed, functions that work with iterators need to be updated.

I think there are different philosophical viewpoints possible on that issue. My own perspective is that there is no change in the definition of iterator -- only in the definition of generator. Note that the *ability* to attach a value to StopIteration is not new at all.

I guess I'm viewing it from the perspective that an ordinary iterator is simply an iterator that happens to return None just like a function that doesn't bother to return anything. If I understand correctly, though, it is possible for any iterator to return a value that yield from would propagate, so the feature (returning a value) is not specific to generators.

...

This feature was new in Python 3.3 which was released a week ago

It's been in alpha/beta/candidate for a long time, and PEP 380 was first discussed in 2009.

...

so it is not widely used but it has uses that are not anything to do with coroutines.

Yes, as a shortcut for "for x in <iterator>: yield x". Note that the for-loop ignores the value in the StopIteration -- would you want to change that too?

Not really. I thought about how it could be changed. Once APIs are available that use this feature to communicate important information, use cases will arise for using the same APIs outside of a coroutine context. I'm not really sure how you could get the value from a for loop. I guess it would have to be tied to the else clause in some way.

...

As an example of how you could use it, consider parsing a file that can contains #include statements. When the #include statement is encountered we need to insert the contents of the included file. This is easy to do with a recursive generator. The example uses the return value of the generator to keep track of which line is being parsed in relation to the flattened output file:

def parse(filename, output_lineno=0): with open(filename) as fin: for input_lineno, line in enumerate(fin): if line.startswith('#include '): subfilename = line.split()[1] output_lineno = yield from parse(subfilename, output_lineno) else: try: yield parse_line(line) except ParseLineError: raise ParseError(filename, input_lineno, output_lineno) output_lineno += 1 return output_lineno

Hm. This example looks constructed to prove your point... It would be easier to count the output lines in the caller. Or you could use a class to hold that state. I think it's just a bad habit to start using the return value for this purpose. Please use the same approach as you would before 3.3, using "yield from" just as the shortcut I mentione above.

I'll admit that the example is contrived but it's to think about how to use the new feature rather than to prove a point (Otherwise I would have contrived a reason for wanting to use filter()). I just wanted to demonstrate that people can (and will) use this outside of a coroutine context. Also I envisage something like this being a common use case. The 'yield from' expression can only provide information to its immediate caller by returning a value attached to StopIteration or be raising a different type of exception. There will be many cases where people want to get some information about what was yielded/done by 'yield from' at the point where it is used. Oscar

On Mon, Oct 8, 2012 at 4:24 PM, Oscar Benjamin wrote:

On 8 October 2012 00:36, Guido van Rossum wrote:

...

On Sun, Oct 7, 2012 at 3:43 PM, Oscar Benjamin wrote:

...

I think what Serhiy is saying is that although pep 380 mainly discusses generator functions it has effectively changed the definition of what it means to be an iterator for all iterators: previously an iterator was just something that yielded values but now it also returns a value. Since the meaning of an iterator has changed, functions that work with iterators need to be updated.

I think there are different philosophical viewpoints possible on that issue. My own perspective is that there is no change in the definition of iterator -- only in the definition of generator. Note that the *ability* to attach a value to StopIteration is not new at all.

I guess I'm viewing it from the perspective that an ordinary iterator is simply an iterator that happens to return None just like a function that doesn't bother to return anything. If I understand correctly, though, it is possible for any iterator to return a value that yield from would propagate, so the feature (returning a value) is not specific to generators.

Substitute "pass a value via StopIteration" and I'll agree that it is *possible*. I still don't think it is all that useful, nor that it should be encouraged (outside the use case of coroutines).

...

...

This feature was new in Python 3.3 which was released a week ago

It's been in alpha/beta/candidate for a long time, and PEP 380 was first discussed in 2009.

...

so it is not widely used but it has uses that are not anything to do with coroutines.

Yes, as a shortcut for "for x in <iterator>: yield x". Note that the for-loop ignores the value in the StopIteration -- would you want to change that too?

Not really. I thought about how it could be changed. Once APIs are available that use this feature to communicate important information, use cases will arise for using the same APIs outside of a coroutine context. I'm not really sure how you could get the value from a for loop. I guess it would have to be tied to the else clause in some way.

Given the elusive nature of StopIteration (many operations catch and ignore it, and that's the main intended use) I don't think it should be used to pass along *important* information except for the specific case of coroutines, where the normal use case is to use .send() instead of .__next__() and to catch the StopIteration exception.

...

...

As an example of how you could use it, consider parsing a file that can contains #include statements. When the #include statement is encountered we need to insert the contents of the included file. This is easy to do with a recursive generator. The example uses the return value of the generator to keep track of which line is being parsed in relation to the flattened output file:

def parse(filename, output_lineno=0): with open(filename) as fin: for input_lineno, line in enumerate(fin): if line.startswith('#include '): subfilename = line.split()[1] output_lineno = yield from parse(subfilename, output_lineno) else: try: yield parse_line(line) except ParseLineError: raise ParseError(filename, input_lineno, output_lineno) output_lineno += 1 return output_lineno

Hm. This example looks constructed to prove your point... It would be easier to count the output lines in the caller. Or you could use a class to hold that state. I think it's just a bad habit to start using the return value for this purpose. Please use the same approach as you would before 3.3, using "yield from" just as the shortcut I mentione above.

I'll admit that the example is contrived but it's to think about how to use the new feature rather than to prove a point (Otherwise I would have contrived a reason for wanting to use filter()). I just wanted to demonstrate that people can (and will) use this outside of a coroutine context.

Just that they will use it doesn't make it a good idea. I claim it's a bad idea and I don't think you're close to convincing me otherwise.

Also I envisage something like this being a common use case. The 'yield from' expression can only provide information to its immediate caller by returning a value attached to StopIteration or be raising a different type of exception. There will be many cases where people want to get some information about what was yielded/done by 'yield from' at the point where it is used.

Maybe. But I think we should wait a few years before we conclude that we made a mistake. The story of iterators and generators has evolved in many small steps, each informed by how the previous step turned out. It's way too soon to say that the existence of yield-from requires us to change all the other iterator algebra to preserve the value from StopIteration. I'll happily take this discussion up again after we've used it for a couple of years though! -- --Guido van Rossum (python.org/~guido)

On 10/8/2012 5:12 PM, Serhiy Storchaka wrote:

On 08.10.12 05:40, Terry Reedy wrote:

...

Serhily, if you want a module of *generator* specific functions ('gentools' ?), you should write one and submit it to pypi for testing.

In http://bugs.python.org/issue16150 there is proposed extending of itertools.chain to support generators (send(), throw() and close() methods). Is it wrong?

Yes -- Terry Jan Reedy

On 7 October 2012 23:43, Oscar Benjamin wrote:

Before pep 380 filter(lambda x: True, obj) returned an object that was the same kind of iterator as obj (it would yield the same values). Now the "kind of iterator" that obj is depends not only on the values that it yields but also on the value that it returns. Since filter does not pass on the same return value, filter(lambda x: True, obj) is no longer the same kind of iterator as obj. The same considerations apply to many other functions such as map, itertools.groupby, itertools.dropwhile.

I really should have checked this before posting but I didn't have Python 3.3 available: Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:55:48) [MSC v.1600 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information.

...

...

import itertools

def f(): ... return 'Returned from generator!' ... yield ... next(filter(lambda x:True, f())) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: Returned from generator!

So filter does propagate the same StopIteration instance. However map does not:

...

...

next(map(None, f())) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration

The itertools module is inconsistent in this respect as well. As already mentioned itertools.chain() hides the value:

...

...

next(itertools.chain(f(), f())) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration next(itertools.chain(f())) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration

Other functions may or may not:

...

...

next(itertools.dropwhile(lambda x:True, f())) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: Returned from generator! next(itertools.groupby(f())) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration

These next two seem wrong since there are two iterables (but I don't think they can be done differently):

...

...

def g(): ... return 'From the other generator...' ... yield ... next(itertools.compress(f(), g())) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: Returned from generator! next(zip(f(), g())) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: Returned from generator!

I guess this should be treated as undefined behaviour? Perhaps it should be documented as such so that anyone who chooses to rely on it was warned. Also some of the itertools documentation is ambiguous in relation to returning vs yielding values from an iterator. Those on the builtin functions page are defined carefully: http://docs.python.org/py3k/library/functions.html#filter filter(function, iterable) Construct an iterator from those elements of iterable for which function returns true. http://docs.python.org/py3k/library/functions.html#map map(function, iterable, ...) Return an iterator that applies function to every item of iterable, yielding the results. But some places in the itertools module use 'return' in place of 'yield': http://docs.python.org/py3k/library/itertools.html#itertools.filterfalse itertools.filterfalse(predicate, iterable) Make an iterator that filters elements from iterable returning only those for which the predicate is False. If predicate is None, return the items that are false. http://docs.python.org/py3k/library/itertools.html#itertools.groupby itertools.groupby(iterable, key=None) Make an iterator that returns consecutive keys and groups from the iterable. The key is a function computing a key value for each element. If not specified or is None, key defaults to an identity function and returns the element unchanged. Oscar

On 09.10.12 10:51, Greg Ewing wrote:

Where we seem to disagree is on whether returning a value with StopIteration is part of the iterator protocol or the generator protocol.

Is a generator expression work with the iterator protocol or the generator protocol? A generator expression eats a value with StopIteration:

...

...

def G(): ... return 42 ... yield ... next(x for x in G()) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration

Is it a bug?

On 10/9/2012 11:34 AM, Serhiy Storchaka wrote:

On 09.10.12 10:51, Greg Ewing wrote:

...

Where we seem to disagree is on whether returning a value with StopIteration is part of the iterator protocol or the generator protocol.

There is a generator class but no 'generator protocol'. Adding the extra generator methods to another iterator class will not give its instances the suspend/resume behavior of generators. That requires the special bytecodes and flags resulting from the presence of 'yield' in the generator function whose call produces the generator.

Is a generator expression work with the iterator protocol or the generator protocol?

A generator expression produces a generator, which implements the iterator protocol and has the extra generator methods and suspend/resume behavior. Part of the iterator protocol is that .__next__ methods raise StopIteration to signal that no more objects will be yielded. A value can be attached to StopIteration, but it is irrelevant to it use as a 'done' signal. Any iterator .__next__ method. can raise or pass along StopIteration(something). Whether 'something' is even seen or not is a different question. The main users of iterators, for statements, ignore anything extra.

A generator expression eats a value with StopIteration:

...

...

...

def G(): ... return 42 ... yield ... next(x for x in G()) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration

Is it a bug?

Of course not. A generator expression is an abbreviation for a def statement defining a generator function followed by a call to that generator function. (x for x in G()) is roughly equivalent to def __(): for x in G(): yield x # when execution reaches here, None is returned, as usual _ = __() del __ _ # IE, _ is the value of the expression A for loop stops when it catches (and swallows) a StopIteration instance. That instance has served it function as a signal. The for mechanism ignores any attributes thereof. The generator .__next__ method that wraps the generator code object (the compiled body of the generator function) raises StopIteration if the code object ends by returning None. So the StopIteration printed in the traceback above is a different StopIteration instance and come from a different callable than the one from G that stopped the for loop in the generator. There is no sensible way to connect the two. Note that a generator can iterate through multiple iterators, like map and chain do. If the generator stops by raising StopIteration instead of returning None, *that* StopIteration instance is passed along by the .__next__ wrapper. (This may be an implementation detail, but it is currently true.)

...

...

def g2(): SI = StopIteration('g2') print(SI, id(SI)) raise SI yield 1

...

...

try: next(g2()) except StopIteration as SI: print(SI, id(SI))

g2 52759816 g2 52759816 If you want any iterator to raise or propagate a value-laden StopIteration, you must do it explicitly or avoid swallowing one.

...

...

def G(): return 42; yield

...

...

def g3(): # replacement for your generator expression it = iter(G()) while True: yield next(it)

...

...

next(g3()) Traceback (most recent call last): File "", line 1, in <module> next(g3()) File "", line 4, in g3 yield next(it) StopIteration: 42 # now you see the value

Since filter takes a single iterable, it can be written like g3 and not catch the StopIteration of the corresponding iterator. def filter(pred, iterable): it = iter(iterable) while True: item = next(it) if pred(item): yield item # never reaches here, never returns None Map takes multiple iterables. In 2.x, map extended short iterables with None to match the longest. So it had to swallow StopIteration until it had collected one for each iterator. In 3.x, map stops at the first StopIteration, so it probably could be rewritten to not catch it. Whether it makes sense to do that is another question. -- Terry Jan Reedy

Serhiy Storchaka wrote:

Is a generator expression work with the iterator protocol or the generator protocol?

Iterator protocol, I think. There is no way to explicitly return a value from a generator expression, and I don't think it should implicitly return one either. Keep in mind that there can be more than one iterable involved in a genexp, so it's not clear what the return value should be in general. -- Greg

The goal described by the OP could be reached with a wrapper generator -- something like this: [snip]

PS. A more convenient version (you don't need to repeat yourself): import collections def genfrom(iter_factory, *args): final_value = None def catching(iterable): subiter = iter(iterable) nonlocal final_value try: while True: yield next(subiter) except StopIteration as exc: if exc.args: final_value = exc.args[0] raise args = [catching(arg.iterable) if isinstance(arg, genfrom.this) else arg for arg in args] yield from iter_factory(*args) return final_value genfrom.this = collections.namedtuple('propagate_from_this', 'iterable') Some examples: >>> import itertools >>> def f(): ... yield 'g' ... return 10000000 ... >>> my_chain = genfrom(itertools.chain, 'abc', 'def', genfrom.this(f())) >>> while True: ... print(next(my_chain)) ... a b c d e f g Traceback (most recent call last): File "<stdin>", line 2, in <module> StopIteration: 10000000 >>> my_filter = genfrom(filter, lambda x: True, genfrom.this(f())) >>> next(my_filter) 'g' >>> next(my_filter) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: 10000000