Propagating StopIteration value
As StopIteration now have value, this value is lost when using functions which works with iterators/generators (map, filter, itertools). Therefore, wrapping the iterator, which preserved its semantics in versions before 3.3, no longer preserves it: map(lambda x: x, iterator) filter(lambda x: True, iterator) itertools.accumulate(iterator, lambda x, y: y) itertools.chain(iterator) itertools.compress(iterator, itertools.cycle([True])) itertools.dropwhile(lambda x: False, iterator) itertools.filterfalse(lambda x: False, iterator) next(itertools.groupby(iterator, lambda x: None))[1] itertools.takewhile(lambda x: True, iterator) itertools.tee(iterator, 1)[0] Perhaps it would be worth to propagate original exception (or at least it's value) in functions for which it makes sense.
On Sat, Oct 6, 2012 at 4:10 PM, Serhiy Storchaka <storchaka@gmail.com> wrote:
As StopIteration now have value, this value is lost when using functions which works with iterators/generators (map, filter, itertools). Therefore, wrapping the iterator, which preserved its semantics in versions before 3.3, no longer preserves it:
map(lambda x: x, iterator) filter(lambda x: True, iterator) itertools.accumulate(iterator, lambda x, y: y) itertools.chain(iterator) itertools.compress(iterator, itertools.cycle([True])) itertools.dropwhile(lambda x: False, iterator) itertools.filterfalse(lambda x: False, iterator) next(itertools.groupby(iterator, lambda x: None))[1] itertools.takewhile(lambda x: True, iterator) itertools.tee(iterator, 1)[0]
Perhaps it would be worth to propagate original exception (or at least it's value) in functions for which it makes sense.
Can you provide an example of a time when you want to use such a value with a generator on which you want to use one of these so I can better understand why this is necessary? the times I'm familiar with wanting this value I'd usually be manually stepping through my generator. Mike
On 06.10.12 23:47, Mike Graham wrote:
Can you provide an example of a time when you want to use such a value with a generator on which you want to use one of these so I can better understand why this is necessary? the times I'm familiar with wanting this value I'd usually be manually stepping through my generator.
There are no many uses yet because it's a new feature. Python 3.3 just released. For example see the proposed patch for http://bugs.python.org/issue16009. In general case `yield from` returns such a value.
On 07/10/12 07:10, Serhiy Storchaka wrote:
As StopIteration now have value, this value is lost when using functions which works with iterators/generators (map, filter, itertools). Therefore, wrapping the iterator, which preserved its semantics in versions before 3.3, no longer preserves it: [...] Perhaps it would be worth to propagate original exception (or at least it's value) in functions for which it makes sense.
A concrete example would be useful for those who don't know about the (new?) StopIteration.value attribute. I think you are referring to this: py> def myiter(): ... yield 1 ... raise StopIteration("spam") ... py> it = map(lambda x:x, myiter()) py> next(it) 1 py> next(it) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration The argument given to StopIteration is eaten by map. But this is not *new* to 3.3, it goes back to at least 2.4, so I'm not sure if you are talking about this or something different. -- Steven
On Sat, Oct 6, 2012 at 6:09 PM, Steven D'Aprano <steve@pearwood.info> wrote:
A concrete example would be useful for those who don't know about the (new?) StopIteration.value attribute. I think you are referring to this:
py> def myiter(): ... yield 1 ... raise StopIteration("spam") ... py> it = map(lambda x:x, myiter()) py> next(it) 1 py> next(it) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration
The argument given to StopIteration is eaten by map.
But this is not *new* to 3.3, it goes back to at least 2.4, so I'm not sure if you are talking about this or something different.
What's new in 3.3 (due to PEP 380) is that instead of the rather awkward and uncommon raise StopIteration("spam") you can now write return "spam" with exactly the same effect. But yes, this was all considered and accepted when PEP 380 was debated (endlessly :-), and I see no reason to change anything about this. "Don't do that" is the best I can say about it -- there are a zillion other situations in Python where that's the only sensible motto. -- --Guido van Rossum (python.org/~guido)
On 07.10.12 04:45, Guido van Rossum wrote:
But yes, this was all considered and accepted when PEP 380 was debated (endlessly :-), and I see no reason to change anything about this.
The reason is that when someone uses StopIteration.value for some purposes, he will lose this value if the iterator will be wrapped into itertools.chain (quite often used technique) or into other standard iterator wrapper.
"Don't do that" is the best I can say about it -- there are a zillion other situations in Python where that's the only sensible motto.
The problem is that two different authors can use two legal techniques (using values returned by "yield from" and wrap iterators with itertools.chain) which do not work in combination. The conflict easily solved if instead of standard itertools.chain to use handwriten code. It looks as bug in itertools.chain.
On Sun, Oct 7, 2012 at 12:30 PM, Serhiy Storchaka <storchaka@gmail.com> wrote:
On 07.10.12 04:45, Guido van Rossum wrote:
But yes, this was all considered and accepted when PEP 380 was debated (endlessly :-), and I see no reason to change anything about this.
The reason is that when someone uses StopIteration.value for some purposes, he will lose this value if the iterator will be wrapped into itertools.chain (quite often used technique) or into other standard iterator wrapper.
If this is just about iterator.chain() I may see some value in it (but TBH the discussion so far mostly confuses -- please spend some more time coming up with good examples that show actually useful use cases rather than f() and g() or foo() and bar()) OTOH yield from is not primarily for iterators -- it is for coroutines. I suspect most of the itertools functionality just doesn't work with coroutines.
"Don't do that" is the best I can say about it -- there are a zillion other situations in Python where that's the only sensible motto.
The problem is that two different authors can use two legal techniques (using values returned by "yield from" and wrap iterators with itertools.chain) which do not work in combination. The conflict easily solved if instead of standard itertools.chain to use handwriten code. It looks as bug in itertools.chain.
Okay, so please do work out a complete, useful use case. We may yet see the light. -- --Guido van Rossum (python.org/~guido)
On 7 October 2012 21:19, Guido van Rossum <guido@python.org> wrote:
On Sun, Oct 7, 2012 at 12:30 PM, Serhiy Storchaka <storchaka@gmail.com> wrote:
On 07.10.12 04:45, Guido van Rossum wrote:
But yes, this was all considered and accepted when PEP 380 was debated (endlessly :-), and I see no reason to change anything about this.
The reason is that when someone uses StopIteration.value for some purposes, he will lose this value if the iterator will be wrapped into itertools.chain (quite often used technique) or into other standard iterator wrapper.
If this is just about iterator.chain() I may see some value in it (but TBH the discussion so far mostly confuses -- please spend some more time coming up with good examples that show actually useful use cases rather than f() and g() or foo() and bar())
OTOH yield from is not primarily for iterators -- it is for coroutines. I suspect most of the itertools functionality just doesn't work with coroutines.
I think what Serhiy is saying is that although pep 380 mainly discusses generator functions it has effectively changed the definition of what it means to be an iterator for all iterators: previously an iterator was just something that yielded values but now it also returns a value. Since the meaning of an iterator has changed, functions that work with iterators need to be updated. Before pep 380 filter(lambda x: True, obj) returned an object that was the same kind of iterator as obj (it would yield the same values). Now the "kind of iterator" that obj is depends not only on the values that it yields but also on the value that it returns. Since filter does not pass on the same return value, filter(lambda x: True, obj) is no longer the same kind of iterator as obj. The same considerations apply to many other functions such as map, itertools.groupby, itertools.dropwhile. Cases like itertools.chain and zip are trickier since they each act on multiple underlying iterables. Probably chain should return a tuple of the return values from each of its iterables. This feature was new in Python 3.3 which was released a week ago so it is not widely used but it has uses that are not anything to do with coroutines. As an example of how you could use it, consider parsing a file that can contains #include statements. When the #include statement is encountered we need to insert the contents of the included file. This is easy to do with a recursive generator. The example uses the return value of the generator to keep track of which line is being parsed in relation to the flattened output file: def parse(filename, output_lineno=0): with open(filename) as fin: for input_lineno, line in enumerate(fin): if line.startswith('#include '): subfilename = line.split()[1] output_lineno = yield from parse(subfilename, output_lineno) else: try: yield parse_line(line) except ParseLineError: raise ParseError(filename, input_lineno, output_lineno) output_lineno += 1 return output_lineno When writing code like the above that depends on being able to get the value returned from an iterator, it is no longer possible to freely mix utilities like filter, map, zip, itertools.chain with the iterators returned by parse() as they no longer act as transparent wrappers over the underlying iterators (by not propagating the value attached to StopIteration). Hopefully, I've understood Serhiy and the docs correctly (I don't have access to Python 3.3 right now to test any of this). Oscar
Oscar Benjamin wrote:
Before pep 380 filter(lambda x: True, obj) returned an object that was the same kind of iterator as obj (it would yield the same values). Now the "kind of iterator" that obj is depends not only on the values that it yields but also on the value that it returns. Since filter does not pass on the same return value, filter(lambda x: True, obj) is no longer the same kind of iterator as obj.
Something like this has happened before, when the ability to send() values into a generator was added. If you wrap a generator with filter, you likewise don't get the same kind of object -- you don't get the ability to send() things into your filtered generator. So, "provide the same kind of iterator" is not currently part of the contract of these functions.
When writing code like the above that depends on being able to get the value returned from an iterator, it is no longer possible to freely mix utilities like filter, map, zip, itertools.chain with the iterators returned by parse() as they no longer act as transparent wrappers over the underlying iterators (by not propagating the value attached to StopIteration).
In many cases they *can't* act as transparent wrappers with respect to the return value, because there is more than one return value to deal with. There's also the added complication that sometimes not all of the sub-iterators are run to completion -- e.g. izip() stops as soon as one of them reaches the end. -- Greg
On 10/7/2012 7:30 PM, Greg Ewing wrote:
Oscar Benjamin wrote:
Before pep 380 filter(lambda x: True, obj) returned an object that was the same kind of iterator as obj (it would yield the same values). Now the "kind of iterator" that obj is depends not only on the values that it yields but also on the value that it returns. Since filter does not pass on the same return value, filter(lambda x: True, obj) is no longer the same kind of iterator as obj.
Something like this has happened before, when the ability to send() values into a generator was added. If you wrap a generator with filter, you likewise don't get the same kind of object -- you don't get the ability to send() things into your filtered generator.
So, "provide the same kind of iterator" is not currently part of the contract of these functions.
Iterators are Python's generic sequential access device. They do that one thing and do it well. The iterator protocol is intentionally and properly minimal. An iterator class *must* have appropriate .__iter__ and .__next__ methods. It *may* also have any other method and any data attribute. Indeed, any iterator much have some specific internal data. But these are ignored in generic iterator (or iterable) functions. If one does not want that, one should write more specific code. For instance, file objects are iterators. Wrappers such as filter(lambda line: line[0] != '\n', open('somefile')) do not have any of the many other file methods and attributes. No one expects otherwise. If one needs access to the other attributes of the file object, one keeps a direct reference to the file object. Hence, the recommended idiom is to use a with statement. Generators are another class of objects that are both iterators (and hence iterables) and something more. When they are used as input arguments to generic functions of iterables, the other behaviors are ignored, and should be ignored, just as with file objects and any other iterator+ objects. Serhily, if you want a module of *generator* specific functions ('gentools' ?), you should write one and submit it to pypi for testing. -- Terry Jan Reedy
On 08.10.12 05:40, Terry Reedy wrote:
Serhily, if you want a module of *generator* specific functions ('gentools' ?), you should write one and submit it to pypi for testing.
In http://bugs.python.org/issue16150 there is proposed extending of itertools.chain to support generators (send(), throw() and close() methods). Is it wrong?
On 10/8/2012 5:12 PM, Serhiy Storchaka wrote:
On 08.10.12 05:40, Terry Reedy wrote:
Serhily, if you want a module of *generator* specific functions ('gentools' ?), you should write one and submit it to pypi for testing.
In http://bugs.python.org/issue16150 there is proposed extending of itertools.chain to support generators (send(), throw() and close() methods). Is it wrong?
Yes -- Terry Jan Reedy
On 8 October 2012 03:40, Terry Reedy <tjreedy@udel.edu> wrote:
On 10/7/2012 7:30 PM, Greg Ewing wrote:
Oscar Benjamin wrote:
Before pep 380 filter(lambda x: True, obj) returned an object that was the same kind of iterator as obj (it would yield the same values). Now the "kind of iterator" that obj is depends not only on the values that it yields but also on the value that it returns. Since filter does not pass on the same return value, filter(lambda x: True, obj) is no longer the same kind of iterator as obj.
Something like this has happened before, when the ability to send() values into a generator was added. If you wrap a generator with filter, you likewise don't get the same kind of object -- you don't get the ability to send() things into your filtered generator.
So, "provide the same kind of iterator" is not currently part of the contract of these functions.
They do provide the same kind of iterator in the sense that they reproduce the properties of the object *in so far as it is an iterator* by yielding the same values. I probably should have compared filter(lambda x: True, obj) with iter(obj) rather than obj. In most cases iter(obj) has a more limited interface. send() is clearly specific to generators: user defined iterator classes can provide any number of state-changing methods (usually with more relevant names) but this is difficult for generators so a generic mechanism is needed. The return value attached to StopIteration "feels" more fundamental to me since there is now specific language syntax both for extracting it and for returning it in generator functions.
Iterators are Python's generic sequential access device. They do that one thing and do it well.
The iterator protocol is intentionally and properly minimal. An iterator class *must* have appropriate .__iter__ and .__next__ methods. It *may* also have any other method and any data attribute. Indeed, any iterator much have some specific internal data. But these are ignored in generic iterator (or iterable) functions. If one does not want that, one should write more specific code.
Generalising the concept of an iterator this way is entirely backwards compatible with existing iterators and does not place any additional burden on defining iterators: most iterators can simply be iterators that return None. The feature is optional for any iterator but this thread is about whether it should be optional for a generic processor of iterators.
Serhily, if you want a module of *generator* specific functions ('gentools' ?), you should write one and submit it to pypi for testing.
This is probably the right idea. As the feature gains use cases the best way to handle it will become clearer. Oscar
Oscar Benjamin wrote:
They do provide the same kind of iterator in the sense that they reproduce the properties of the object *in so far as it is an iterator* by yielding the same values.
I think we agree on that. Where we seem to disagree is on whether returning a value with StopIteration is part of the iterator protocol or the generator protocol. To my mind it's part of the generator protocol, and as such, itertools functions are not under any obligation to support it. -- Greg
On 09.10.12 10:51, Greg Ewing wrote:
Where we seem to disagree is on whether returning a value with StopIteration is part of the iterator protocol or the generator protocol.
Is a generator expression work with the iterator protocol or the generator protocol? A generator expression eats a value with StopIteration:
def G(): ... return 42 ... yield ... next(x for x in G()) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration
Is it a bug?
On 10/9/2012 11:34 AM, Serhiy Storchaka wrote:
On 09.10.12 10:51, Greg Ewing wrote:
Where we seem to disagree is on whether returning a value with StopIteration is part of the iterator protocol or the generator protocol.
There is a generator class but no 'generator protocol'. Adding the extra generator methods to another iterator class will not give its instances the suspend/resume behavior of generators. That requires the special bytecodes and flags resulting from the presence of 'yield' in the generator function whose call produces the generator.
Is a generator expression work with the iterator protocol or the generator protocol?
A generator expression produces a generator, which implements the iterator protocol and has the extra generator methods and suspend/resume behavior. Part of the iterator protocol is that .__next__ methods raise StopIteration to signal that no more objects will be yielded. A value can be attached to StopIteration, but it is irrelevant to it use as a 'done' signal. Any iterator .__next__ method. can raise or pass along StopIteration(something). Whether 'something' is even seen or not is a different question. The main users of iterators, for statements, ignore anything extra.
A generator expression eats a value with StopIteration:
def G(): ... return 42 ... yield ... next(x for x in G()) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration
Is it a bug?
Of course not. A generator expression is an abbreviation for a def statement defining a generator function followed by a call to that generator function. (x for x in G()) is roughly equivalent to def __(): for x in G(): yield x # when execution reaches here, None is returned, as usual _ = __() del __ _ # IE, _ is the value of the expression A for loop stops when it catches (and swallows) a StopIteration instance. That instance has served it function as a signal. The for mechanism ignores any attributes thereof. The generator .__next__ method that wraps the generator code object (the compiled body of the generator function) raises StopIteration if the code object ends by returning None. So the StopIteration printed in the traceback above is a different StopIteration instance and come from a different callable than the one from G that stopped the for loop in the generator. There is no sensible way to connect the two. Note that a generator can iterate through multiple iterators, like map and chain do. If the generator stops by raising StopIteration instead of returning None, *that* StopIteration instance is passed along by the .__next__ wrapper. (This may be an implementation detail, but it is currently true.)
def g2(): SI = StopIteration('g2') print(SI, id(SI)) raise SI yield 1
try: next(g2()) except StopIteration as SI: print(SI, id(SI))
g2 52759816 g2 52759816 If you want any iterator to raise or propagate a value-laden StopIteration, you must do it explicitly or avoid swallowing one.
def G(): return 42; yield
def g3(): # replacement for your generator expression it = iter(G()) while True: yield next(it)
next(g3()) Traceback (most recent call last): File "<pyshell#29>", line 1, in <module> next(g3()) File "<pyshell#28>", line 4, in g3 yield next(it) StopIteration: 42 # now you see the value
Since filter takes a single iterable, it can be written like g3 and not catch the StopIteration of the corresponding iterator. def filter(pred, iterable): it = iter(iterable) while True: item = next(it) if pred(item): yield item # never reaches here, never returns None Map takes multiple iterables. In 2.x, map extended short iterables with None to match the longest. So it had to swallow StopIteration until it had collected one for each iterator. In 3.x, map stops at the first StopIteration, so it probably could be rewritten to not catch it. Whether it makes sense to do that is another question. -- Terry Jan Reedy
On 9 October 2012 22:37, Terry Reedy <tjreedy@udel.edu> wrote:
On 10/9/2012 11:34 AM, Serhiy Storchaka wrote:
On 09.10.12 10:51, Greg Ewing wrote:
Where we seem to disagree is on whether returning a value with StopIteration is part of the iterator protocol or the generator protocol.
Correct.
Part of the iterator protocol is that .__next__ methods raise StopIteration to signal that no more objects will be yielded. A value can be attached to StopIteration, but it is irrelevant to it use as a 'done' signal. Any iterator .__next__ method. can raise or pass along StopIteration(something). Whether 'something' is even seen or not is a different question. The main users of iterators, for statements, ignore anything extra.
I know this isn't going anywhere right now but since it might one day I thought I'd say that I considered how it could be different and the best I came up with was: def f(): return 42 yield for x in f(): pass else return_value: # return_value = 42 if we get here
If the generator stops by raising StopIteration instead of returning None, *that* StopIteration instance is passed along by the .__next__ wrapper. (This may be an implementation detail, but it is currently true.)
I'm wondering whether propagating or not propagating the StopIteration should be a documented feature of some iterator-based functions or should always be considered an implementation detail (i.e. undefined language behaviour). Currently in Python 3.3 I guess that it is always an implementation detail since the behaviour probably results from an implementation that was written under the assumption that StopIteration instances are interchangeable.
Since filter takes a single iterable, it can be written like g3 and not catch the StopIteration of the corresponding iterator.
def filter(pred, iterable): it = iter(iterable) while True: item = next(it) if pred(item): yield item # never reaches here, never returns None
Map takes multiple iterables. In 2.x, map extended short iterables with None to match the longest. So it had to swallow StopIteration until it had collected one for each iterator. In 3.x, map stops at the first StopIteration, so it probably could be rewritten to not catch it. Whether it makes sense to do that is another question.
Thanks. That makes more sense now as I hadn't considered this behaviour of map before. Oscar
Serhiy Storchaka wrote:
Is a generator expression work with the iterator protocol or the generator protocol?
Iterator protocol, I think. There is no way to explicitly return a value from a generator expression, and I don't think it should implicitly return one either. Keep in mind that there can be more than one iterable involved in a genexp, so it's not clear what the return value should be in general. -- Greg
On Sun, Oct 7, 2012 at 3:43 PM, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
On 7 October 2012 21:19, Guido van Rossum <guido@python.org> wrote:
On Sun, Oct 7, 2012 at 12:30 PM, Serhiy Storchaka <storchaka@gmail.com> wrote:
On 07.10.12 04:45, Guido van Rossum wrote:
But yes, this was all considered and accepted when PEP 380 was debated (endlessly :-), and I see no reason to change anything about this.
The reason is that when someone uses StopIteration.value for some purposes, he will lose this value if the iterator will be wrapped into itertools.chain (quite often used technique) or into other standard iterator wrapper.
If this is just about iterator.chain() I may see some value in it (but TBH the discussion so far mostly confuses -- please spend some more time coming up with good examples that show actually useful use cases rather than f() and g() or foo() and bar())
OTOH yield from is not primarily for iterators -- it is for coroutines. I suspect most of the itertools functionality just doesn't work with coroutines.
I think what Serhiy is saying is that although pep 380 mainly discusses generator functions it has effectively changed the definition of what it means to be an iterator for all iterators: previously an iterator was just something that yielded values but now it also returns a value. Since the meaning of an iterator has changed, functions that work with iterators need to be updated.
I think there are different philosophical viewpoints possible on that issue. My own perspective is that there is no change in the definition of iterator -- only in the definition of generator. Note that the *ability* to attach a value to StopIteration is not new at all.
Before pep 380 filter(lambda x: True, obj) returned an object that was the same kind of iterator as obj (it would yield the same values). Now the "kind of iterator" that obj is depends not only on the values that it yields but also on the value that it returns. Since filter does not pass on the same return value, filter(lambda x: True, obj) is no longer the same kind of iterator as obj. The same considerations apply to many other functions such as map, itertools.groupby, itertools.dropwhile.
There are other differences between iterators and generators that are not preserved by the various forms of "iterator algebra" that can be applied -- in particular, non-generator iterators don't support send(). I think it's perfectly valid to view generators as a kind of special iterators with properties that aren't preserved by applying generic iterator operations to them (like itertools or filter()).
Cases like itertools.chain and zip are trickier since they each act on multiple underlying iterables. Probably chain should return a tuple of the return values from each of its iterables.
That's one possible interpretation, but I doubt it's the most useful one.
This feature was new in Python 3.3 which was released a week ago
It's been in alpha/beta/candidate for a long time, and PEP 380 was first discussed in 2009.
so it is not widely used but it has uses that are not anything to do with coroutines.
Yes, as a shortcut for "for x in <iterator>: yield x". Note that the for-loop ignores the value in the StopIteration -- would you want to change that too?
As an example of how you could use it, consider parsing a file that can contains #include statements. When the #include statement is encountered we need to insert the contents of the included file. This is easy to do with a recursive generator. The example uses the return value of the generator to keep track of which line is being parsed in relation to the flattened output file:
def parse(filename, output_lineno=0): with open(filename) as fin: for input_lineno, line in enumerate(fin): if line.startswith('#include '): subfilename = line.split()[1] output_lineno = yield from parse(subfilename, output_lineno) else: try: yield parse_line(line) except ParseLineError: raise ParseError(filename, input_lineno, output_lineno) output_lineno += 1 return output_lineno
Hm. This example looks constructed to prove your point... It would be easier to count the output lines in the caller. Or you could use a class to hold that state. I think it's just a bad habit to start using the return value for this purpose. Please use the same approach as you would before 3.3, using "yield from" just as the shortcut I mentione above.
When writing code like the above that depends on being able to get the value returned from an iterator, it is no longer possible to freely mix utilities like filter, map, zip, itertools.chain with the iterators returned by parse() as they no longer act as transparent wrappers over the underlying iterators (by not propagating the value attached to StopIteration).
I see that as one more argument for not using the return value here...
Hopefully, I've understood Serhiy and the docs correctly (I don't have access to Python 3.3 right now to test any of this).
I don't doubt it. But I think you're fighting windmills. -- --Guido van Rossum (python.org/~guido)
On 8 October 2012 00:36, Guido van Rossum <guido@python.org> wrote:
On Sun, Oct 7, 2012 at 3:43 PM, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
I think what Serhiy is saying is that although pep 380 mainly discusses generator functions it has effectively changed the definition of what it means to be an iterator for all iterators: previously an iterator was just something that yielded values but now it also returns a value. Since the meaning of an iterator has changed, functions that work with iterators need to be updated.
I think there are different philosophical viewpoints possible on that issue. My own perspective is that there is no change in the definition of iterator -- only in the definition of generator. Note that the *ability* to attach a value to StopIteration is not new at all.
I guess I'm viewing it from the perspective that an ordinary iterator is simply an iterator that happens to return None just like a function that doesn't bother to return anything. If I understand correctly, though, it is possible for any iterator to return a value that yield from would propagate, so the feature (returning a value) is not specific to generators.
This feature was new in Python 3.3 which was released a week ago
It's been in alpha/beta/candidate for a long time, and PEP 380 was first discussed in 2009.
so it is not widely used but it has uses that are not anything to do with coroutines.
Yes, as a shortcut for "for x in <iterator>: yield x". Note that the for-loop ignores the value in the StopIteration -- would you want to change that too?
Not really. I thought about how it could be changed. Once APIs are available that use this feature to communicate important information, use cases will arise for using the same APIs outside of a coroutine context. I'm not really sure how you could get the value from a for loop. I guess it would have to be tied to the else clause in some way.
As an example of how you could use it, consider parsing a file that can contains #include statements. When the #include statement is encountered we need to insert the contents of the included file. This is easy to do with a recursive generator. The example uses the return value of the generator to keep track of which line is being parsed in relation to the flattened output file:
def parse(filename, output_lineno=0): with open(filename) as fin: for input_lineno, line in enumerate(fin): if line.startswith('#include '): subfilename = line.split()[1] output_lineno = yield from parse(subfilename, output_lineno) else: try: yield parse_line(line) except ParseLineError: raise ParseError(filename, input_lineno, output_lineno) output_lineno += 1 return output_lineno
Hm. This example looks constructed to prove your point... It would be easier to count the output lines in the caller. Or you could use a class to hold that state. I think it's just a bad habit to start using the return value for this purpose. Please use the same approach as you would before 3.3, using "yield from" just as the shortcut I mentione above.
I'll admit that the example is contrived but it's to think about how to use the new feature rather than to prove a point (Otherwise I would have contrived a reason for wanting to use filter()). I just wanted to demonstrate that people can (and will) use this outside of a coroutine context. Also I envisage something like this being a common use case. The 'yield from' expression can only provide information to its immediate caller by returning a value attached to StopIteration or be raising a different type of exception. There will be many cases where people want to get some information about what was yielded/done by 'yield from' at the point where it is used. Oscar
On Mon, Oct 8, 2012 at 4:24 PM, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
On 8 October 2012 00:36, Guido van Rossum <guido@python.org> wrote:
On Sun, Oct 7, 2012 at 3:43 PM, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
I think what Serhiy is saying is that although pep 380 mainly discusses generator functions it has effectively changed the definition of what it means to be an iterator for all iterators: previously an iterator was just something that yielded values but now it also returns a value. Since the meaning of an iterator has changed, functions that work with iterators need to be updated.
I think there are different philosophical viewpoints possible on that issue. My own perspective is that there is no change in the definition of iterator -- only in the definition of generator. Note that the *ability* to attach a value to StopIteration is not new at all.
I guess I'm viewing it from the perspective that an ordinary iterator is simply an iterator that happens to return None just like a function that doesn't bother to return anything. If I understand correctly, though, it is possible for any iterator to return a value that yield from would propagate, so the feature (returning a value) is not specific to generators.
Substitute "pass a value via StopIteration" and I'll agree that it is *possible*. I still don't think it is all that useful, nor that it should be encouraged (outside the use case of coroutines).
This feature was new in Python 3.3 which was released a week ago
It's been in alpha/beta/candidate for a long time, and PEP 380 was first discussed in 2009.
so it is not widely used but it has uses that are not anything to do with coroutines.
Yes, as a shortcut for "for x in <iterator>: yield x". Note that the for-loop ignores the value in the StopIteration -- would you want to change that too?
Not really. I thought about how it could be changed. Once APIs are available that use this feature to communicate important information, use cases will arise for using the same APIs outside of a coroutine context. I'm not really sure how you could get the value from a for loop. I guess it would have to be tied to the else clause in some way.
Given the elusive nature of StopIteration (many operations catch and ignore it, and that's the main intended use) I don't think it should be used to pass along *important* information except for the specific case of coroutines, where the normal use case is to use .send() instead of .__next__() and to catch the StopIteration exception.
As an example of how you could use it, consider parsing a file that can contains #include statements. When the #include statement is encountered we need to insert the contents of the included file. This is easy to do with a recursive generator. The example uses the return value of the generator to keep track of which line is being parsed in relation to the flattened output file:
def parse(filename, output_lineno=0): with open(filename) as fin: for input_lineno, line in enumerate(fin): if line.startswith('#include '): subfilename = line.split()[1] output_lineno = yield from parse(subfilename, output_lineno) else: try: yield parse_line(line) except ParseLineError: raise ParseError(filename, input_lineno, output_lineno) output_lineno += 1 return output_lineno
Hm. This example looks constructed to prove your point... It would be easier to count the output lines in the caller. Or you could use a class to hold that state. I think it's just a bad habit to start using the return value for this purpose. Please use the same approach as you would before 3.3, using "yield from" just as the shortcut I mentione above.
I'll admit that the example is contrived but it's to think about how to use the new feature rather than to prove a point (Otherwise I would have contrived a reason for wanting to use filter()). I just wanted to demonstrate that people can (and will) use this outside of a coroutine context.
Just that they will use it doesn't make it a good idea. I claim it's a bad idea and I don't think you're close to convincing me otherwise.
Also I envisage something like this being a common use case. The 'yield from' expression can only provide information to its immediate caller by returning a value attached to StopIteration or be raising a different type of exception. There will be many cases where people want to get some information about what was yielded/done by 'yield from' at the point where it is used.
Maybe. But I think we should wait a few years before we conclude that we made a mistake. The story of iterators and generators has evolved in many small steps, each informed by how the previous step turned out. It's way too soon to say that the existence of yield-from requires us to change all the other iterator algebra to preserve the value from StopIteration. I'll happily take this discussion up again after we've used it for a couple of years though! -- --Guido van Rossum (python.org/~guido)
On 9 October 2012 00:47, Guido van Rossum <guido@python.org> wrote:
Given the elusive nature of StopIteration (many operations catch and ignore it, and that's the main intended use) I don't think it should be used to pass along *important* information except for the specific case of coroutines, where the normal use case is to use .send() instead of .__next__() and to catch the StopIteration exception.
It certainly is elusive! I caught a bug a few weeks ago where StopIteration was generated from a call to next and caught by a for loop several frames above. I couldn't work out why the loop was terminating early (since there was no attempt to catch any exceptions anywhere in the code) and it took about 20 minutes of processing to reproduce. With no traceback and no way to catch the exception with pdb it had me stumped for a while. Oscar
On 7 October 2012 23:43, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
Before pep 380 filter(lambda x: True, obj) returned an object that was the same kind of iterator as obj (it would yield the same values). Now the "kind of iterator" that obj is depends not only on the values that it yields but also on the value that it returns. Since filter does not pass on the same return value, filter(lambda x: True, obj) is no longer the same kind of iterator as obj. The same considerations apply to many other functions such as map, itertools.groupby, itertools.dropwhile.
I really should have checked this before posting but I didn't have Python 3.3 available: Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:55:48) [MSC v.1600 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information.
import itertools
def f(): ... return 'Returned from generator!' ... yield ... next(filter(lambda x:True, f())) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: Returned from generator!
So filter does propagate the same StopIteration instance. However map does not:
next(map(None, f())) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration
The itertools module is inconsistent in this respect as well. As already mentioned itertools.chain() hides the value:
next(itertools.chain(f(), f())) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration next(itertools.chain(f())) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration
Other functions may or may not:
next(itertools.dropwhile(lambda x:True, f())) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: Returned from generator! next(itertools.groupby(f())) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration
These next two seem wrong since there are two iterables (but I don't think they can be done differently):
def g(): ... return 'From the other generator...' ... yield ... next(itertools.compress(f(), g())) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: Returned from generator! next(zip(f(), g())) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: Returned from generator!
I guess this should be treated as undefined behaviour? Perhaps it should be documented as such so that anyone who chooses to rely on it was warned. Also some of the itertools documentation is ambiguous in relation to returning vs yielding values from an iterator. Those on the builtin functions page are defined carefully: http://docs.python.org/py3k/library/functions.html#filter filter(function, iterable) Construct an iterator from those elements of iterable for which function returns true. http://docs.python.org/py3k/library/functions.html#map map(function, iterable, ...) Return an iterator that applies function to every item of iterable, yielding the results. But some places in the itertools module use 'return' in place of 'yield': http://docs.python.org/py3k/library/itertools.html#itertools.filterfalse itertools.filterfalse(predicate, iterable) Make an iterator that filters elements from iterable returning only those for which the predicate is False. If predicate is None, return the items that are false. http://docs.python.org/py3k/library/itertools.html#itertools.groupby itertools.groupby(iterable, key=None) Make an iterator that returns consecutive keys and groups from the iterable. The key is a function computing a key value for each element. If not specified or is None, key defaults to an identity function and returns the element unchanged. Oscar
On 09.10.12 16:07, Oscar Benjamin wrote:
I really should have checked this before posting but I didn't have Python 3.3 available:
Generator expression also eats the StopIteration value:
next(x for x in f()) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration
These next two seem wrong since there are two iterables (but I don't think they can be done differently):
def g(): .... return 'From the other generator...' .... yield .... next(itertools.compress(f(), g())) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: Returned from generator! next(zip(f(), g())) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: Returned from generator!
def h(): ... yield 42 ... return 'From the another generator...' ... next(zip(f(), h())) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: Returned from generator! next(zip(h(), f())) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: Returned from generator!
This is logical. Value returned from the first exhausted iterator.
I guess this should be treated as undefined behaviour? Perhaps it should be documented as such so that anyone who chooses to rely on it was warned.
This should be treated as implementation details now.
Hello .* On 09.10.2012 17:28, Serhiy Storchaka wrote:
On 09.10.12 16:07, Oscar Benjamin wrote:
I really should have checked this before posting but I didn't have Python 3.3 available:
Generator expression also eats the StopIteration value:
next(x for x in f()) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration [snip]
1. Why shouldn't it "eat" that value? The full-generator equivalent, even with `yield from`, will "eat" it also: >>> def _make_this_gen_expr_equivalent(): >>> yield from f() # or: for x in f(): yield x ... >>> g = _make_this_gen_expr_equivalent() >>> next(g) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration After all, any generator will "eat" the StopIteration argument unless: * explicitly propagates it (with `return arg` or `raise StopIteration(arg)`), or * iterates over the subiterator "by hand" using the next() builtin or the __next__() method and does not catch StopIteration. 2. I believe that the new `yield from...` feature changes nothing in the *iterator* protocol. What it adds are only two things that can be placed in the code of a *generator*: 1) a return statement with a value -- finishing execution of the generator + raising StopIteration instantiated with that value passed as the only argument, 2) a `yield from subiterator` expression which propagates the items generated by the subiterator (not necessarily a generator) + does all that dance with __next__/send/throw/close (see PEP 380...) + caches StopIteration and returns the value passed with this exception (if any value has been passed). Not less, not more. Especially, the `yield from subiterator` expression itself* does not propagate* its value outside the generator. 3. The goal described by the OP could be reached with a wrapper generator -- something like this: def preservefrom(iter_factory, *args, which): final_value = None subiter = iter(which) def catching_gen(): nonlocal final_value try: while True: yield next(subiter) except StopIteration as exc: if exc.args: final_value = exc.args[0] raise args = [arg if arg is not which else catching_gen() for arg in args] yield from iter_factory(*args) return final_value Example usage: >>> import itertools >>> def f(): ... yield 'g' ... return 1000000 ... >>> my_gen = f() >>> my_chain = preservefrom(itertools.chain, 'abc', 'def', my_gen, which=my_gen) >>> while True: ... print(next(my_chain)) ... a b c d e f g Traceback (most recent call last): File "<stdin>", line 2, in <module> StopIteration: 1000000 >>> my_gen = f() >>> my_filter = preservefrom(filter, lambda x: True, my_gen, which=my_gen) >>> next(my_filter) 'g' >>> next(my_filter) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: 1000000 Best regards. *j
The goal described by the OP could be reached with a wrapper generator -- something like this: [snip]
PS. A more convenient version (you don't need to repeat yourself): import collections def genfrom(iter_factory, *args): final_value = None def catching(iterable): subiter = iter(iterable) nonlocal final_value try: while True: yield next(subiter) except StopIteration as exc: if exc.args: final_value = exc.args[0] raise args = [catching(arg.iterable) if isinstance(arg, genfrom.this) else arg for arg in args] yield from iter_factory(*args) return final_value genfrom.this = collections.namedtuple('propagate_from_this', 'iterable') Some examples: >>> import itertools >>> def f(): ... yield 'g' ... return 10000000 ... >>> my_chain = genfrom(itertools.chain, 'abc', 'def', genfrom.this(f())) >>> while True: ... print(next(my_chain)) ... a b c d e f g Traceback (most recent call last): File "<stdin>", line 2, in <module> StopIteration: 10000000 >>> my_filter = genfrom(filter, lambda x: True, genfrom.this(f())) >>> next(my_filter) 'g' >>> next(my_filter) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: 10000000
W dniu 11.10.2012 01:29, Jan Kaliszewski napisał(a):
The goal described by the OP could be reached with a wrapper generator -- something like this: [snip]
PS. A more convenient version (you don't need to repeat yourself): [snip].
PS2. Sorry for flooding, obviously it can be simpler: import collections def genfrom(iter_factory, *args): final_value = None def catching(iterable): nonlocal final_value final_value = yield from iterable args = [catching(arg.iterable) if isinstance(arg, genfrom.this) else arg for arg in args] yield from iter_factory(*args) return final_value genfrom.this = collections.namedtuple('propagate_from_this', 'iterable') Cheers. *j
On 07.10.12 23:19, Guido van Rossum wrote:
If this is just about iterator.chain() I may see some value in it (but TBH the discussion so far mostly confuses -- please spend some more time coming up with good examples that show actually useful use cases rather than f() and g() or foo() and bar())
Not I was the first one who showed an example with f() and g(). ;) I only showed that it was wrong analogy. Yes, first of all I think about itertools.chain(). But then I found all other iterator tools which also can be extended to better generators support. Perhaps. I have only one imperfect example for use of StopIterator's value from generator (my patch for issue16009). It is difficult to find examples for feature, which appeared only recently. But I think I can find them before 3.4 feature freezing.
OTOH yield from is not primarily for iterators -- it is for coroutines. I suspect most of the itertools functionality just doesn't work with coroutines.
Indeed. But they work with subset of generators, and this subset can be extended. Please look at http://bugs.python.org/issue16150 (Implement generator interface in itertools.chain). Does it make sense?
On Mon, Oct 8, 2012 at 2:02 PM, Serhiy Storchaka <storchaka@gmail.com> wrote:
On 07.10.12 23:19, Guido van Rossum wrote:
If this is just about iterator.chain() I may see some value in it (but TBH the discussion so far mostly confuses -- please spend some more time coming up with good examples that show actually useful use cases rather than f() and g() or foo() and bar())
Not I was the first one who showed an example with f() and g(). ;) I only showed that it was wrong analogy.
Yes, first of all I think about itertools.chain(). But then I found all other iterator tools which also can be extended to better generators support. Perhaps.
I have only one imperfect example for use of StopIterator's value from generator (my patch for issue16009).
I don't understand that code at all, and it seems to be undocumented (no docstrings, no mention in the external docs). Why is it using StopIteration at all? There isn't an iterator or generator in sight. AFAICT it should just use a different exception. But even if you did use StopIteration -- why would you care about itertools here? AFAICT it's just being used as a private communication channel between scan_once() and its caller. Where is the possibility to wrap anything in itertools at all?
It is difficult to find examples for feature, which appeared only recently. But I think I can find them before 3.4 feature freezing.
I think you're going at this from the wrong direction. You shouldn't be using this feature in circumstances where you're at all likely to run into this "problem".
OTOH yield from is not primarily for iterators -- it is for coroutines. I suspect most of the itertools functionality just doesn't work with coroutines.>
Indeed. But they work with subset of generators, and this subset can be extended. Please look at http://bugs.python.org/issue16150 (Implement generator interface in itertools.chain). Does it make sense?
But that just seems to perpetuate the idea that you have, which IMO is wrongheaded. Itertools is for iterators, and all the extra generator features make no sense for it. -- --Guido van Rossum (python.org/~guido)
On 09.10.12 01:44, Guido van Rossum wrote:
I don't understand that code at all, and it seems to be undocumented (no docstrings, no mention in the external docs). Why is it using StopIteration at all? There isn't an iterator or generator in sight. AFAICT it should just use a different exception.
I agree with you. StopIteration is not needed here (or I don't understand that code), ValueError can be raised instead it. Perhaps the author was going to use it for the iterative parsing. This is a bad example, but it is only real example which I have. I have also the artificial (but also imperfect) example: def file_reader(f): while not f.eof: yield f.read(0x2000) def zlib_decompressor(input): d = zlib.decompressobj() while not d.eof: yield d.decompress(d.unconsumed_tail or next(input)) return d.unused_data def bzip2_decompressor(input): decomp = bz2.BZ2Decompressor() while not decomp.eof: yield decomp.decompress(next(input)) return decomp.unused_data def detect_decompressor(input): data = b'' while len(data) < 5: data += next(input) if data.startswith('deflt'): decompressor = zlib_decompressor data = data[5:] elif data.startswith('bzip2'): decompressor = bzip2_decompressor data = data[5:] else: decompressor = None input = itertools.chain([data], input) return decompressor, input def multi_stream_decompressor(input): while True: decompressor, input = detect_decompressor(input) if decompressor is None: return input unused_data = yield from decompressor(input) if not unused_data: return input input = itertools.chain([unused_data], input) Of cause this can be implemented without generators, using a class to hold a state.
I think you're going at this from the wrong direction. You shouldn't be using this feature in circumstances where you're at all likely to run into this "problem".
I think that the new language features (as returning value from generators/iterators) will generated new methods of solving problems. And for these new methods will be useful to expand the existing tools. But now I see that it is still too early to talk about it.
Itertools is for iterators, and all the extra generator features make no sense for it.
As said Greg, the question is whether returning a value with StopIteration is part of the iterator protocol or the generator protocol.
Serhiy Storchaka wrote:
The conflict easily solved if instead of standard itertools.chain to use handwriten code. It looks as bug in itertools.chain.
Don't underestimate the value of handwritten code. It makes the intent clear to the reader, whereas relying on some arbitrary default behaviour of a function doesn't. -- Greg
Steven D'Aprano wrote:
py> def myiter(): ... yield 1 ... raise StopIteration("spam") ... py> it = map(lambda x:x, myiter()) py> next(it) 1 py> next(it) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration
The argument given to StopIteration is eaten by map.
It's highly debatable whether this is even wrong. The purpose of StopIteration(value) is for a generator to return a value to its immediate caller when invoked using yield-from. The value is not intended to propagate any further than that. A non-iterator analogy would be def f(): return 42 def g(): f() Would you expect g() to return 42 here? -- Greg
On 07.10.12 05:11, Greg Ewing wrote:
It's highly debatable whether this is even wrong. The purpose of StopIteration(value) is for a generator to return a value to its immediate caller when invoked using yield-from. The value is not intended to propagate any further than that.
If immediate caller can propagate generated values with the help of "yield from", why it can not propagate returned from "yield from" value?
A non-iterator analogy would be
def f(): return 42
def g(): f()
No, a non-iterator analogy would be g = functools.partial(f) or g = functools.lru_cache()(f) I expect g() to return 42 here. And it will be expected and useful if yield from itertools.chain([prefix], iterator) will return the same value as yield from iterator Now chain equivalent to: def chain(*iterables): for it in iterables: yield from it I propose make it equivalent to: def chain(*iterables): value = None for it in iterables: value = yield from it return value
On 07/10/2012 8:06pm, Serhiy Storchaka wrote:
I propose make it equivalent to:
def chain(*iterables): value = None for it in iterables: value = yield from it return value
That means that all but the last return value is ignored. Why is the last return value any more important than the earlier ones? ISTM it would make just as much sense to do def chain(*iterables): values = [] for it in iterables: values.append(yield from it) return values But I don't see any point in changing the current behaviour. Richard
On 07.10.12 22:18, Richard Oudkerk wrote:
That means that all but the last return value is ignored. Why is the last return value any more important than the earlier ones?
Because I think the last return value more useful for idiom lookahead = next(iterator) process(lookahead) iterator = itertools.chain([lookahead], iterator)
ISTM it would make just as much sense to do
def chain(*iterables): values = [] for it in iterables: values.append(yield from it) return values
It changes the behavior for iterators. And now more difficult to get a generator which yields and returns the same values as the original. We need yet one wrapper. def lastvalue(generator): return (yield from generator)[-1] iterator = lastvalue(itertools.chain([lookahead], iterator)) Yes, it can work.
On Sun, Oct 7, 2012 at 3:06 PM, Serhiy Storchaka <storchaka@gmail.com> wrote:
On 07.10.12 05:11, Greg Ewing wrote:
It's highly debatable whether this is even wrong. The purpose of StopIteration(value) is for a generator to return a value to its immediate caller when invoked using yield-from. The value is not intended to propagate any further than that.
If immediate caller can propagate generated values with the help of "yield from", why it can not propagate returned from "yield from" value?
A non-iterator analogy would be
def f(): return 42
def g(): f()
No, a non-iterator analogy would be
g = functools.partial(f)
or
g = functools.lru_cache()(f)
I expect g() to return 42 here.
Rather than speaking in analogies, can we be concrete? I can't imagine doing map(f, x) where x is a generator whose return value I cared about. Can you show us a concrete example of something that looks like practical code? Mike
participants (9)
-
Greg Ewing -
Guido van Rossum -
Jan Kaliszewski -
Mike Graham -
Oscar Benjamin -
Richard Oudkerk -
Serhiy Storchaka -
Steven D'Aprano -
Terry Reedy