Somehow during the discussion of PEP 479 it was concluded that generators will become distinct from iterators. A few people have pointed out that this is a harmful distinction (complicates the language hard to explain etc). I think it is simply an invalid distinction and should be abolished. PEP 479 proposes to change the way that generators handle StopIteration fall-through by default. However the iterator protocol has never formally specified that an iterator should allow StopIteration to fall through to a grandparent consumer. I considered the fact that StopIteration fall-through was possible to be a convenient design feature but it was never a required part of the definition of an iterator. Many iterators perform other kinds of exception handling without allowing that to fall through and many iterators catch StopIteration from child iterators without reraising it. This has never lead anyone to suggest that such iterators are not true iterators in any way. Under PEP 479 generators will handle a StopIteration arising from the executing frame differently. The generator will still be an object with __iter__ and __next__ that exposes StopIteration at the appropriate time to its parent iterator-consumer. In other words it will still satisfy the definition of an iterator. The PEP says: """ Under this proposal, generators and iterators would be distinct, but related, concepts. Like the mixing of text and bytes in Python 2, the mixing of generators and iterators has resulted in certain perceived conveniences, but proper separation will make bugs more visible. """" This is just plain false. Under the proposal generators will still be iterators. Mixing generators and iterators is nothing like mixing text and bytes and never has been. Mixing iterators and iterables is a bit like mixing text and the bytes in the sense that it can seem to work but then sometimes silently do the wrong thing. Mixing generators and iterators is like mixing sets and containers: there is no mixing since it is simply a false distinction. AFAICT from reading through the discussions this idea has (implicitly) followed from the following fallacious argument: 1) Bare next creates a problem for generators. 2) Therefore we fix it by changing generators. 3) Therefore generators are not iterators any more. Anyone with experience in propositional logic can see multiple fallacies in that argument but actually the biggest mistake is simply in the opening premise: bare next is a problem for all iterators. Generators are affected precisely because they are iterators. Generators were introduced as "a kind of Python iterator, but of an especially powerful kind" (PEP 255). The coroutine idea has since turned generators into something of a Frankenstein concept but the fundamental fact that they are iterators remains unchanged. And regardless of how much discussion coroutines generate the fact remains that 99% of generators are used purely for iteration. I propose to abolish this notion that generators are not iterators and to amend the text of the PEP to unambiguously state that generators are iterators regardless of any changes to the way they propagate StopIteration from the executing frame. Oscar
On Wed, Dec 10, 2014 at 12:36:59PM +0000, Oscar Benjamin wrote:
I certainly hope so. Currently, generators are iterators: they obey the iterator protocol and the Iterator ABC correctly registers them as instances. py> def gen(): ... yield 1 ... py> it = gen() py> iter(it) is it # Iterator protocol is obeyed. True py> hasattr(it, '__next__') True py> from collections.abc import Iterator py> isinstance(it, Iterator) # And registered with the ABC. True Surely this isn't going to change? If it does, I expect that's going to break an enormous amount of code. If generators are to cease to be iterators, what will they be?
Sounds reasonable to me. -- Steven
On 11 December 2014 at 00:18, Steven D'Aprano <steve@pearwood.info> wrote:
If generators are to cease to be iterators, what will they be?
What the PEP is trying to say is that generators are not the same thing as __next__ method implementations (and I agree that shortening the latter to "iterators" is incorrect). __next__ method paths for leaving the frame: * "return value" = produce next value * "raise StopIteration" = end of iteration Current generator paths for leaving the frame: * "yield value" = produce next value * "return" = end of iteration * "raise StopIteration" = end of iteration PEP 479 generator paths for leaving the frame: * "yield value" = produce next value * "return" = end of iteration * "raise StopIteration" = RuntimeError The only change is to eliminate the redundant method of leaving the frame, as that behaviour means that adding a "yield" to a function *implicitly* suppresses StopIteration exceptions raised elsewhere in that frame. That implicit side effect on exception handling behaviour doesn't exist for __next__ method implementations, or for ordinary functions used in iteration operations that accept arbitrary callables, as ordinary functions won't include any yield expressions (by definition). Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thu, Dec 11, 2014 at 3:21 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Current wording: (at least, I think this is the only bit that's questionable) """ Under this proposal, generators and iterators would be distinct, but related, concepts. Like the mixing of text and bytes in Python 2, the mixing of generators and iterators has resulted in certain perceived conveniences, but proper separation will make bugs more visible. """ Would it be better to clarify that as "generator functions"? Maybe something like this: """ Under this proposal, generator functions and iterators would be distinct, but related, concepts. Like the mixing of text and bytes in Python 2, the mixing of generators and iterators has resulted in certain perceived conveniences, but proper separation will make bugs more visible. The distinction is simple: A generator function returns a generator object. The latter is an iterator, having proper __iter__ and __next__ methods, while the former has neither and does not follow iterator protocol. """ (Most of the text ignores __iter__, even though technically it's an important part of iterator protocol for it to exist and return self. I don't think anyone will be confused by the omission, as it perfectly follows __next__, but I could add text if people think it's important.) ChrisA
On 10 Dec 2014 16:38, "Chris Angelico" <rosuav@gmail.com> wrote:
I find this more confusing than the original, actually, because now it sounds like you're saying that the distinction between a generator function and a generator instance is something new that this PEP is adding, in order to fix all the problems that are being caused by people writing 'for x in genfunc: ...'. Which doesn't make much sense. Like Nick said, it seems to me that the key distinction to emphasize is the distinction between generator function bodies and __next__ method bodies. -n
I have replaced the confusing paragraph with this: """ The proposal does not change the relationship between generators and iterators: a generator object is still an iterator, and not all iterators are generators. Generators have additional methods that iterators don't have, like ``send`` and ``throw``. All this is unchanged. Nothing changes for generator users -- only authors of generator functions may have to learn something new. """ Let us know if there is still confusion about what the PEP means. --Guido On Wed, Dec 10, 2014 at 10:13 AM, Guido van Rossum <guido@python.org> wrote:
-- --Guido van Rossum (python.org/~guido)
On 10 December 2014 at 20:46, Guido van Rossum <guido@python.org> wrote:
Generators are affected and this therefore affects authors of both generator functions and generator expressions. In the case of a generator function you can achieve the old behaviour post-PEP 479 simply by wrapping the entire body of the function with: try: # body except StopIteration as e: return e.value The only difference is that the generated StopIteration has a different traceback attached so you can't see where it was originally "raised". Generator expressions would need to be converted to generator functions to achieve the same effect.
Let us know if there is still confusion about what the PEP means.
The PEP is still very confused. Primarily the fact is that the PEP is attempting to address an issue which affects all iterators but proposing a solution which is only about generators. The PEP seems to suggest that there is something special about generators in this respect when there really isn't. For example: """ The interaction of generators and StopIteration is currently somewhat surprising, and can conceal obscure bugs. An unexpected exception should not result in subtly altered behaviour, but should cause a noisy and easily-debugged traceback. Currently, StopIteration can be absorbed by the generator construct. """ There is no interaction between generators and StopIteration. The issue isn't about generators it is about the iterator protocol. StopIteration cannot be absorbed by the generator construct. I think the PEP would be clearer if it properly acknowledged that the problem is a problem for all iterators. The question then is why the fix is only targeted at generators and what should be done about the same problem that occurs in many other forms. The PEP rationale avoids these issues by falsely claiming that generators are special.
On 12/11/2014 09:14 AM, Oscar Benjamin wrote:
It seems to me, the main benefit is for writers of nested generators and co-routines. It makes those cases easier to debug, and may help generators in general be more reusable, and reliable, in the context of nested generators and co-routines. Which if true could contribute to better, and more dependable, multi-thread, multi-process, or even multi-core, software designs. That is the general impression I get from the conversations about the changes being made. But I'm not sure I can point to any one message stating it like that. Possibly someone can confirm if any of those points are valid and why. Or it could be I'm just a bit too optimistic. ;-) Cheers, Ron
On 12/11/2014 10:14 AM, Oscar Benjamin wrote:
On 10 December 2014 at 20:46, Guido van Rossum <guido@python.org> wrote:
What is very confusing is the ambiguous use of 'generator' to mean either 'generator function' or 'generator class instance'. Since I do not know what either of you mean by 'generator' in each instance above, I do not know exactly what either of you meant. If the PEP and discussants consistency used 'generator' to mean "instance of the internal generator class, initialized with a suspended instance of a *user-written* generator function body", then I think some of the confusion would dissipate. The PEP is about the conjunction of the following facts (and a few more, but I think these are the most salient): a) generator function bodies follow a different protocol than iterator __next__ methods, one that does not require them to *ever* raise StopIteration; b) if a g.f. body does raise StopIteration, it might be a substitute for 'return', but it might be a bug -- and apparently bugs do happen in real code; and c) generator.__next__ currently trusts that *user-written* g.f.s are never buggy. The PEP proposal is, either literally or in effect, that generator.__next should stop trusting StopIteration from a g.f., thereby disallowing the sometimes convenient but optional substitution. The justification is that bugs should not pass silently, and now they do.
Guido today add the following, apparently in response to the above. +When implementing a regular ``__next__()`` method, the only way to +indicate the end of the iteration is to raise ``StopIteration``. So +catching ``StopIteration`` here and converting it to ``RuntimeError`` +would defeat the purpose. This is a reminder of the special status of +generator functions: in a generator function, raising +``StopIteration`` is redundant since the iteration can be terminated +by a simple ``return``. I would put this answer slightly differently. A __next__ method is supposed to raise StopIteration if *and only if* there are no more items to return. The __next__ method author is responsible for fulfilling the contract. Core developers are responsible for generator.__next__; we are not responsible for iterators that others write. Anyone who writes an iterator class whose instances are initialized with 3rd party non-iterator code that is executed in .__next__, should think about what to do if that code raises StopIteration. It would be possible for the PEP to recommend that other .__next__ methods executing external code might do something similar to what generator.__next__ will do. try: execute(self.external_code) except StopIteration as e: raise RuntimeError(<whatever>) -- Terry Jan Reedy
On 12 December 2014 at 01:14, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
I believe you're misunderstanding the problem being solved. The specific problem deemed worthy of being fixed is that the presence of "yield" in a function body can implicitly suppress StopIteration exceptions raised elsewhere in that function body (or in functions it calls). The difference in behaviour between comprehensions and generator expressions when it comes to embedded function calls that trigger StopIteration is a special case of that more general difference. This is a problem unique to generators, it does not affect any other iterator (since explicit __next__ method implementations do not use yield). The change in the PEP is to change that side effect such that those exceptions are converted to RuntimeError rather than silently suppressed - making generator function bodies behave less like __next__ method implementations. For special methods themselves, the problem of unexpected signalling exceptions from other called functions being interpreted according to their defined meaning in the relevant protocol is inherent in their design - they only have two "normal" outcomes (returning from the frame, or raising the signalling exception). That limitation applies to StopIteration & __next__ in the same way that it applies to KeyError/IndexError and __getitem__, or AttributeError and the various methods in the attribute access protocols. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 12 December 2014 at 10:34, Nick Coghlan <ncoghlan@gmail.com> wrote:
And I believe you are too. :)
The yield causes the function to become a generator function. The frame for a generator function (like for any other function) will allow uncaught exceptions to propagate to the frame above. The difference between generator functions and other functions is that the code in the body of a generator function is executed when someone calls the generator's __next__ method. Since the caller (the iterator consumer) is expecting StopIteration it is treated as the signalling the end of iteration. The yield suppresses nothing; it is the iterator consumer e.g. the for-loop or the list() function etc. which catches the StopIteration and treats it as termination.
I don't know what you mean by this.
Incorrect. The problem is not unique to generators and the yield is irrelevant. The problem (insofar as it is) is a problem for all iterators since all iterators interact with iterator-consumers and it is the iterator-consumer that catches the StopIteration. Here is an example using map:
Ordinarily map should terminate when the underlying iterator raises StopIteration. Since it simply allows a StopIteration raised anywhere to bubble up it will also allow the StopIteration raised by func to bubble up. Implicitly the effect of that is that map agrees to cooperatively terminate iteration when any part of the code that it executes raises StopIteration. Note that it is the list() function (the iterator-consumer) that actually catches the StopIteration and terminates just as it would be if map were a generator:
In principle that could be a useful feature in some cases. In practise it is more likely to be something that masks a bug manifesting in a stray StopIteration e.g.:
In this case it is the for-loop (the iterator-consumer) that catches the StopIteration. Whether or not map is a generator or some other iterator is irrelevant.
Fine but generator function bodies are __next__ method implementations. You can claim that the generator function implements __iter__ but its body implements the __next__ method of the generator. It is precisely because it is a __next__ method that it finds itself being called by an iterator consumer and so finds that a loose StopIteration is suppressed by the consumer. This is the same for all iterators and there is nothing special about generators in this regard (although the PEP will change that a little). To summarise succinctly: Loose StopIteration can have an unintended interaction with the iterator protocol and more specifically with iterator-consumers. The effect is as if the iterator had been stopped since the consumer will stop iteration. Because generators are iterators they (like all iterators) are affected by this. Oscar
On 12 December 2014 at 22:42, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
Oscar, this is where you're *just plain wrong* from a technical perspective. The conversion to a generator function also introduces an additional layer of indirection at iteration time through the generator-iterator's __next__ method implementation. That indirection layer is the one that converts an ordinary return from the generator frame into a StopIteration exception. Historically, it also allowed StopIteration from within the frame to propagate out, make it impossible to tell from outside the generator iterator whether the termination was due to the generator returning normally, or due to StopIteraton being raised. The introduction of "yield from" in Python 3.3, and its use in the coroutine design for the asyncio library and similar contexts *makes that distinction more important than it used to be*. It even managed to introduce obscure bugs around the use of generators to write context managers, where factoring out part of the context manager to a subgenerator may implicitly suppress a StopIteration exception raised in the body of the with statement. The problem you think PEP 479 is trying to solve *is* the one where the discussion started, but it is *not* the one where it ended when Guido accepted PEP 479. The problems PEP 479 solves are generator specific - they have nothing to do with the iterator protocol in general. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 12 December 2014 at 13:14, Nick Coghlan <ncoghlan@gmail.com> wrote:
Which of the above sentences is technically incorrect?
I understand all of this.
Okay well this is not clear from the PEP (and bear in mind that I did take the time to read through all of the lengthy threads on this subject before posting here). I haven't read the most recent changes but when I did read the PEP the justification appeared to stem from claims that were confusing at best and plain false at worst e.g. "generators are not iterators". Oscar
On Fri, Dec 12, 2014 at 5:58 AM, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
I don't think there is much technical confusion here, but there seems to be, so, so that highlights that it's very important to write vary carefully about all this. I"m going to try this: 1) It's not clear what is meant by an "iterator" -- when we use the term is seems like we are talking about a type, when we are really talking about "anything that conforms to the iterator protocol", so I'm going to introduce a new term: TIIP (Type Implementing the Iterator Protocol) -- maybe that's too cute, but role with it for now. 2) there are a number of TIPPS in the standard library: things like tupleiterator and friends, the range object, etc. 3) there are essentially two (and a half) ways for python code authors to write their own, custom TIIP: 1) write a class that has a __iter__ and __next__ methods that "do the right thing" 2a) write a generator function 2b) write a generator expression The latter create a generator type, which is a specific TIIP that come built in to the standard library. Generator functions are a particularly nifty syntax for creating TIIPs -- i.e. I'm pretty sure that there is nothing you can do with a generator function that you can't do by writing a custom TIIP class. But generator functions provide really nifty syntax that "does a lot of the bookeeping for you" This is pretty much how I've taught this stuff in the past. But what this PEP does is change generators a bit, so that they do even more of the book keeping for you -- that sure seems reasonable enough when phrased that way. So I agree with Oscar (and others) here the issue of what happens to a StopIteration that is raised inside a TIIP be another call is an issue that effects not only generators, but also custom TIIP classes. I think is wold be nice if this were addressed in both cases the same way, but that has been proposed and rejected for what I, at least, think are sound technical reasons. And the PEP has been accepted so the only thing left to bike-shed about is how it discussed. So why it it worth changing the behavior of generators? 1) we can -- custom classes are, well, custom, so are free to do what author wants them to do. Generators are built-in classes and can therefor be controlled by python itself. 2) generator function have a different, cleaner, more compact way to express intent. while they create generators that conform to the iteration protocol -- i.e. raise a StopIteration when done, the author does not have to explicitly do that -- you can write a generator function without any awareness of StopIteration ( I might argue that you won't get far if you don't understand all that, but...) So having generators automatically "do the right thing" when they call another TIIP inside themselves conforms well to the spirit of generator functions: clean compact code that does the book keeping for you. 3) when you write a custom TIIP class, you have complete control over what it does, and have to be thinking about when StopIteration is raise. So you are more likely to think about how you want to handle a StopIteration that may be raised by another call to a TIP inside your code. You also want to be able to control that depending on circumstances, so probably wouldn't want magic Exception mangling anyway. Not sure if this helps the PEP writing, but I've got some good notes for the next time I teach this... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Fri, Dec 12, 2014 at 3:06 PM, Alexander Belopolsky < alexander.belopolsky@gmail.com> wrote:
The same logic applies to generators and I think this misconception is at the root of the confusion in this thread. When you are writing a generator function, you are not creating a custom TIIP. You are writing a code object that Python knows how to convert into an instance of a built-in TIIP (the generator type).
On Fri, Dec 12, 2014 at 12:14 PM, Alexander Belopolsky < alexander.belopolsky@gmail.com> wrote:
The point was supposed to be that you want a TIIP with particular behavior that is not built-in (hence custom). Yes, in terms of type, a generator function, when called, will return the a built-in type, but its behavior is customized. Indeed the same as writing a class with __getitem__ and no __iter__ or __next__ will give you an instance a built-in iterator type with customized behavior. I guess my take is that the user shouldn't have to know exactly what the type of TIIP is produced is -- only what its behavior will be. But maybe that IS the source of much of this confusion -- the PEP authors make the point that generators are something special. I want tend to think of them as an implementation detail. This PEP makes them more special, and that's really the point of the contentious part of the PEP. So maybe that's the the point we are trying to make -- while we can argue about whether generators ARE iterators, or whether they have been "conflated" -- generators are special, and might as well be more special ;-) -Chris
-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Fri, Dec 12, 2014 at 12:16 PM, Chris Barker <chris.barker@noaa.gov> wrote:
2) there are a number of TIPPS in the standard library: things like tupleiterator and friends, the range object, etc.
The range object is not an iterator
isinstance(range(1), collections.Iterator) False
and does not implement the iterator protocol
Please don't confuse iterators and iterables:
isinstance(range(1), collections.Iterable) True
On Fri, Dec 12, 2014 at 11:41 AM, Alexander Belopolsky < alexander.belopolsky@gmail.com> wrote:
indeed -- sorry about that -- the "rangeiterator" object, that is. The point here is that there are a bunch of builtin types that are TIIPs
Please don't confuse iterators and iterables:
indeed, that is a distinction that should be kept clear. Thanks -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Dec 12, 2014, at 9:16, Chris Barker <chris.barker@noaa.gov> wrote:
When you're writing actual code, you can test this with isinstance(collections.abc.Iterator). Of course ABCs don't _perfectly_ cover their related protocols (in this case, if you create a type that has __iter__ and __next__ but __iter__ doesn't return self--or, of course, if you just register some random type with Iterator explicitly--you've created a subclass of Iterator that isn't a subtype of the iterator protocol). But I think it still makes sense to use Iterator as shorthand for "type that subclasses Iterator" with the implied "is an actual subtype of the semantic type intended by the Iterator class". This kind of shorthand is used regularly in discussing types in every OO language, not just Python. You don't see people writing about "an instance of a Type Implementing IEnumerable [or NSFastEnumeration or random_access_iterator]). You just see "an IEnumerable". And to discuss the type itself, you just say "an IEnumerable type". The fact that IEnumerable is itself defined as a type is rarely a source of confusion--and when it is, you say "an IEnumerable subtype". In Python, the term "Iterator" is just as consistent and meaningful as in all these other languages. The fact that some people confuse iterables and iterators isn't a reason to abandon this simplicity. Especially since it doesn't even help to solve the problem--as we've just seen in this very message I'm replying to, it's just as easy for someone to mistakenly use TIIP to describe "range" as it is for them to mistakenly use "Iterator". (In fact, it's arguably worse, because what would you call an Iterable under the same naming scheme but Type Implementing Iterable Protocol, or TIIP?) The things we need to be clear about here are the things that _dont't_ have an official name. In particular, the thing that's built from calling a generator function or evaluating a generator expression and used by the generator.__next__ method is not a generator, a generator function, a generator function body, an iterator, or anything else with a name in the language. And that's what's confusing people. They want to call it one of the things that has a name. Renaming any of those things that it's not doesn't help to give the thing that it is a name.
On Fri, Dec 12, 2014 at 3:22 PM, Andrew Barnert <abarnert@yahoo.com> wrote:
this issue here is that inteh PEP, and in this list, it seems "iterator" was used to describe classes with __init__ and __next__ methods, AS APPOSED to the things returned from functions with a yield in them... and: In [12]: g = (i for i in range(10)) In [13]: isinstance(it, collections.abc.Iterator) Out[13]: True and In [15]: def gf(): ....: yield 1 ....: yield 2 In [16]: gen = gf() In [17]: isinstance(gen, collections.abc.Iterator) Out[17]: True So generators ARE iterators, but his definition and yet the contrast was being made....
This kind of shorthand is used regularly in discussing types in every OO language, not just Python.
sure, but Python is dynamically types, so "conforms to a protocol" not at all teh same is "isinstance", even though ABCS have made this closer...
actually, the confusion between iterables and iterators want the issue I as trying to address -- but the confusion with a sentence like "Under this proposal, generators and iterators would be distinct, but related, concepts" That has been removed from the PEP -- I think it's more clean now in any case. But yes, I could have written all that and stuck with plain old "iterator" instead of making up "TIIP" Especially since it doesn't even help to solve the problem--as we've just
My bad -- typing too fast! And this was never about the itterable vs iterator distinction anyway.
I was not trying to mingle iterable and iterator -- really I wan't :-)
now I am confused: In [18]: g = ( i for i in range(10) ) In [19]: type(g) Out[19]: generator how is that not a "generator"? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Dec 12, 2014, at 16:40, Chris Barker <chris.barker@noaa.gov> wrote:
But the thing returned by a function without yield in it is an instance of the built-in class generator, which is a subtype of Iterator. Are you trying to make a distinction between iterator _types_ and iterator type _instances_? If so, TIIP doesn't seem to help very much.
Yes, there was some unfortunate wording in the earlier version that Guido already acknowledged and promised to fix well before your message.
This kind of shorthand is used regularly in discussing types in every OO language, not just Python.
sure, but Python is dynamically types, so "conforms to a protocol" not at all teh same is "isinstance", even though ABCS have made this closer...
The whole point of ABCs is to allow you to define protocols in-language. Other dynamic OO languages like Smalltalk and ObjC have similar concepts.
You've skipped the "used by the generator" part of that sentence. Of course g is an instance of generator (and therefore also an iterator). But you're not defining the generator type here--that already exists as a builtin type. What you're defining is a thing made up of a code object and a frame, which is used by that generator instance to do its generating. That thing is not a generator, or a generator type (or an iterator instance or type), or a __next__ method, or anything else with a name. (I don't know if that's the thing Guido was trying to describe as not an iterator, but I don't think that's too important, since he's already acknowledged that the distinction he was trying to make isn't important anyway.)
On Dec 12, 2014, at 17:32, Ethan Furman <ethan@stoneleaf.us> wrote:
Sorry, autocorrect typo. The thing returned by a function *with a* (not *without*) yield in it is a generator. And generator is a class with __iter__ and __next__. Of course an individual generator *instance* isn't such a type, but that's no different from the fact that 2 isn't a number type.
On Fri, Dec 12, 2014 at 03:22:32PM -0800, Andrew Barnert wrote:
+1 to this. But:
I don't know what thing you are referring to. If I write this: py> def gen(): ... yield 1 ... py> it = gen() then `gen` is a function. It's a specific kind of function that uses "yield", and the name for that is a generator function. Informally, sometimes people call it a "generator", which is a bad habit due to the possibility of confusing `gen` with `it`, but since it is quite rare to be in a position where such confusion can occur (PEP 479 and related discussions not withstanding), we can often get away with such sloppy terminology. Also, `it` is an iterator. It's also a generator: py> type(it) <class 'generator'> The same applies to generator expressions: py> type(c for c in "aaa") <class 'generator'> So I'm not sure which mysterious object you are referring to that doesn't have a standard name. Can you give a concrete example? -- Steven
On Dec 13, 2014, at 5:36, Steven D'Aprano <steve@pearwood.info> wrote:
Chris Barker just asked exactly the same question yesterday, except that he trimmed the quote more and used "g" instead of "it" for his example. So I'll just paste the same reply, and you can manually s/g/it/. Of course g is an instance of generator (and therefore also an iterator). But you're not defining the generator type here--that already exists as a builtin type. What you're defining is a thing made up of a code object and a frame, which is used by that generator instance to do its generating. That thing is not a generator, or a generator type (or an iterator instance or type), or a __next__ method, or anything else with a name. (I don't know if that's the thing Guido was trying to describe as not an iterator, but I don't think that's too important, since he's already acknowledged that the distinction he was trying to make isn't important anyway.)
On 12/13/2014 01:37 PM, Andrew Barnert wrote:
It's a function with a flag set.
gen.__code__.co_flags 99
Where a function's flags would be this...
So, it the answer is yes and no. It's a function object, with an altered behaviour. It seems like the flag may be checked at call time, and then a new function generator instance is returned..
And the original function body is moved to it's gi_code attribute.
I think the send() method, a builtin C routine, handles all the particulars of running gi_code with gi_frame. (I'd have to look to be sure.) I think if function definitons with yield in them created generator_instance_factory objects instead of function objects, things would be easier to explain. What it comes down to, is they aren't normal functions and it's a bit confusing to think of them that way. Cheers, Ron
On Sun, Dec 14, 2014 at 7:19 AM, Ron Adam <ron3200@gmail.com> wrote:
They're no more different than, say, a decorated function that returns a different function. But if you want a different way to explain them, here's a similar way to craft an iterator: class seq_iter: def __init__(self, func, *args, **kw): self.func = func self.args = args self.kw = kw self.state = 0 def __iter__(self): return self def __next__(self): if self.state == -1: raise StopIteration ret = self.func(self.state, self, *self.args, **self.kw) self.state += 1 if ret is not self: return ret self.state = -1 raise StopIteration def sequential_iterator(func): @functools.wraps(func) def inner(*args, **kw): return seq_iter(func, *args, **kw) return inner ##### Example usage ##### @sequential_iterator def ten_squares(idx, finished): if idx == 10: return finished return idx*idx The function returned by the decorator isn't the one decorated, but it's clearly still a function. No reason for it to be a unique type. This is similar to the relationship between generator functions and the iterators returned when you call them. It's an alternative way to write an iterator class, dealing with lots of the bookkeeping for you. ChrisA
On 12/13/2014 03:27 PM, Chris Angelico wrote:
They're no more different than, say, a decorated function that returns a different function.
But that is different, because the decorator isn't visible. You can also offer examples of other objects in terms of other objects, they are still other objects. So it raises questions like this ... When should we make a distinction of one object from another? And when shouldn't we? Usually it has more to do with weather or not it makes writing some program easier. I'm not sure changing the type in this case would be useful in that way.
But if you want a different way to explain them, here's a similar way to craft an iterator:
I think it's what the average python user expects that matters. The term generator_function was use quite a lot in the PEP discussion in order to distinguish it from a regular function, and also from the generator returned from it. So it seems to me, that maybe there is something to that. Currently (especially when discussing things here), I think it's important to try to have an accurate model of what is being discussed. Comparisons to similar code is also useful, but for checking the implementation works correctly and consistently. They don't replace the accurate model. That model could be pseudo-code, or python, or a diagram, but it helps to keep in in mind. So, we have a rule we need to remember: If a function has "yield" in it, it will return a generator-object. Ok. So a generator is: a function + that rule. We have a case where we differentiate it in our minds, but the code, type(g) indicates it is not different, but the code actually does do something a bit magical. Yes we can visually examine the body of the function and tell that it's different, but that isn't full proof. That is some of what leads me to think it should maybe be a different type. Possibly a sub-class of a function class, would be the only option. Fortunately generators are not new, so most everyone does understands how to define and use them, just not how the implementation is actually done. But maybe that's ok. <shrug> Cheers, Ron
On Sun, Dec 14, 2014 at 12:33 PM, Ron Adam <ron3200@gmail.com> wrote:
Many things can be distinguished that are not distinct types. A generator function is a function that has a 'yield' in it. A factory function is a function that constructs and returns a new object. A recursive function is a function which calls itself. All three are functions in every way. ChrisA
On 12/13/2014 07:38 PM, Chris Angelico wrote:
I see only two things... functions that do exactly what's in the defined body when called, and generator_functions which does something other than what is defined in the body when called. That seems like quite a difference to me. Cheers, Ron
On Sat, Dec 13, 2014 at 11:37:23AM -0800, Andrew Barnert wrote:
I read your reply to Chris last night, and it didn't make sense to me then and it still doesn't make sense to me now :-( Hence my request for a concrete example. I am still unclear as to what this mystery unnamed thing is. It's not a generator function, because that is called a generator function. It's not the thing you get when you call a generator function, because that is called a generator or an iterator, depending on context. Surely it's not the generator type, because that has a name: "generator". We always use the same name for the class and its instances, e.g. we talk about "int" the class as well as saying that 123 is an int. The class "int" is not itself an int instance, but we still call it "int". And likewise for "generator".
Of course g is an instance of generator (and therefore also an iterator).
Agreed. But giving examples of things which aren't the mystery entity doesn't really help explain which mystery entity you are talking about :-)
Sure the thing you are defining when you write "def gen() ..." is a function? I thought we were in agreement about that. It's a function with a flag set, a.k.a. a generator function. If you're referring to something else, a concrete example will help. Because as we stand now, I don't know whether we're talking about something important or not, or if we are in agreement, disagreement, or merely confusion :-) -- Steven
On Dec 13, 2014, at 17:17, Steven D'Aprano <steve@pearwood.info> wrote:
You didn't ask for a concrete example, you provided the exact same answer as Chris but with different names.
Again, it's the thing used by the generator instance's __next__ method: the frame and code objects.
I don't know how else to explain it. Look at the members stored by the generator type: a frame and a code object. So, going back to the original point: Generator instances are not different from iterators (except that it's a subtype, of course). The builtin generator type is likewise no different than a custom iterator class. The generator body is a normal function body. The only thing that isn't like an iterator is the frame. Which is not an important distinction. I think Chris was making a different distinction, claiming that generators are not an Iterator type (which they aren't, but only because they're _instances_ of an Iterator type, the builtin type generator), and I was pointing out that this can't be the distinction Guido was trying to make. But, again, since Guido has already said that sentence was mistaken, this really isn't important.
I already said from the start that this is almost certainly not important.
In an effort to try to keep this already too-long thread a bit shorter, I'm going to trim drastically. On Sat, Dec 13, 2014 at 09:43:02PM -0800, Andrew Barnert wrote:
On Dec 13, 2014, at 17:17, Steven D'Aprano <steve@pearwood.info> wrote:
My first post in this sub-thread ended with: "So I'm not sure which mysterious object you are referring to that doesn't have a standard name. Can you give a concrete example?" But moving along:
Aha, now the penny drops. Given a generator object, is has attributes holding a frame object and a code object (which it shares with the generator function that created it): py> def gen(): yield 1 ... py> it = gen() py> it.gi_frame <frame object at 0xb7b8f95c> py> it.gi_code <code object gen at 0xb7b6e660, file "<stdin>", line 1> py> it.gi_code is gen.__code__ True Both have standard names: "frame object" and "code object" respectively.
I don't know how else to explain it. Look at the members stored by the generator type: a frame and a code object.
That's exactly the concrete example I was looking for.
Well, there are a few other differences as well. `generator` is a concrete type, `Iterator` is an abstract type, and `iterator` can refer to either a specific iterator type: py> class X: ... def __getitem__(self, i): pass ... py> iter(X()) <iterator object at 0xb7af4b0c> or speaking more generically, to anything which obeys the iterator protocol. More importantly, generators have abilities that other iterators don't have: you can send data into a generator, not just get data out of them.
I'm not sure what Chris was trying to say, but I've seen various other people maintain that generators aren't iterators. As you say, that is wrong: py> from collections.abc import Iterator py> isinstance(gen(), Iterator) True The *class* `generator` is an Iterator (sub-)type, but not an Iterator itself: py> issubclass(type(gen()), Iterator) True py> isinstance(type(gen()), Iterator) False which is no different from instances of int being numbers, but int itself (the class) not being a number. Thank you for helping me understand what you were trying to say. -- Steven
On Dec 14, 2014, at 0:58, Steven D'Aprano <steve@pearwood.info> wrote:
Sure, but the problem is that people have been writing things that come "from the generator's internals", and then trying to explain how those things "make generators different from iterators" (or "... from explicit __next__ methods"). Because of the wording that was being used, people were looking for some way in which generators, or the generator type, or generator functions, are different from iterators, from other iterator types, or whatever, when none of that is at issue; it's the suspended generator frame, which doesn't have any parallel in other iterator implementations. I was thinking that if we had some term like "generator state" to refer to the frame-and-code (or just the frame, because the code isn't really necessary), the confusion wouldn't have arisen. Now I notice that Nick Coghlan has been referring to "the generator frame", which is a nice way to avoid that confusion. What I was basically suggesting is that someone should rewrite the explanations exactly the way Nick already had been rewriting them. :)
That was in the message you replied to. I think the problem isn't that I didn't give the example you were looking for, it's that (at least in that message) I didn't explain why you (or Chris or anyone else) should care about it, so you passed right over it. Sorry for not making that part clear.
On Sun, Dec 14, 2014 at 7:58 PM, Andrew Barnert < abarnert@yahoo.com.dmarc.invalid> wrote:
I never was too much into using an exception to signal than an iterator had finished its job. Either intuition or experience said that it would be a source of obscure problems. I'm not proposing a solution. Not even proposing a change. I do propose that since this thread has been long and going long, we take the opportunity to consider the base assumptions. Most newcomers to programming take a while to grok complex control flows, and what's happening here is too complex even for Python contributors. I suggest we revisit "import this" for a little while, just in case we made a mistake along the way. Sorry for the rambling! Regards, -- Juancarlo *Añez*
Juancarlo Añez writes:
As I understand Raymond's point, what is happening here is *not* particularly complex (I don't see it as complex myself, anyway). It's simply an obscure (because rare) and obvious (once you know about it) consequence of two simple design decisions: "exceptions bubble" and "iterator termination is signaled with an exception". That obscurity is compounded by the fact that a programmer error will pass silently. (There must be a theorem that says that in any Turing complete language some errors will pass silently!) In the case of *generators* silently ignoring an error is unnecessary, and PEP 479 fixes that one case, at a cost of breaking a useful idiom in some functions. For iterators in general, I suppose we could add a flag to checkers to look for function calls inside iterators, and warn the user to check for the potential for spuriously raising StopIteration. (I doubt that's a great idea, but it could be done.)
On 12/12/2014 7:42 AM, Oscar Benjamin wrote:> On 12 December 2014 at 10:34, Nick Coghlan <ncoghlan@gmail.com> wrote:
This makes no sense to me either.
Neither do I
The parenthetical comment is true. However, I think part of the problem is unique to generator functions, and part is more general -- that loose usage of StopIteration creates problems when a .__next__ method calls external code.
Incorrect. The problem is not unique to generators and the yield is irrelevant.
Nick (and Guido) are looking at the unique part and Oscar is looking at he general part.
This function violates the implicit guideline, which I think should be more explicit in the doc, that the only functions that should expose StopIteration to the outside world are iterator.__next__ functions and the corresponding builtin next(). It is not surprising that this violation causes map to misbehave. However, changing the .__next__ methods of map and filter iterators should be a new thread, which I will write after posting this. The g.f. PEP says that generator functions should also follow this guideline and not raise StopIteration (but should always return when not yielding). ...
This is about as wrong, or at least as confusing, as saying that the function passed to map is a __next__ method implementation. There is only one generator.__next__, just as there is only one map.__next__. Both .__next__ methods execute the user function (or function frame) passed in to the corresponding .__init__ to calculate the next value to return. So the external functions assist the __next__ methods but are not __next__ methods themselves. In particular, they should not raise StopIteration. -- Terry Jan Reedy
On 12 December 2014 at 20:06, Terry Reedy <tjreedy@udel.edu> wrote:
I'm wondering if Nick is referring to two different things at once: On the one hand "yield from" which is an iterator-consumer suppresses StopIteration from a child iterator and on the other hand every generator interacts with a parent iterator-consumer that will catch any StopIteration it raises. When generator function f "yield from"s to generator function g both things occur in tandem in two different frames. <snip>
The function was just supposed to generate the effect. More plausible examples can be made. I tripped over this myself once when doing something with iterators that required me to extract the first item before processing the rest. A simple demonstration: def sum(iterable): iterator = iter(iterable) total = next(iterator) for item in iterator: total += item return total results = list(map(sum, data_sources))
The g.f. PEP says that generator functions should also follow this guideline and not raise StopIteration (but should always return when not yielding).
The idea is that it happens by accident. Oscar
On 12/12/2014 4:10 PM, Oscar Benjamin wrote:
Examples like this has been posted on python-list, probably more than once. The answer has been to catch StopIteration (which should only come from next/__next__) and either return a default value or raise ValueError('cannot sum an empty iterable').
results = list(map(sum, data_sources))
-- Terry Jan Reedy
On 12 December 2014 at 22:54, Terry Reedy <tjreedy@udel.edu> wrote:
I know what the solution is. For my purposes simply raising an exception (any exception that wasn't StopIteration) would have been sufficient. I'm just saying that it's a mistake that can easily be made. Since I made it I'm now super-suspicious of next() so I probably wouldn't make it again. This mistake is the root of all of the problems that PEP 479 attempts to solve. If you don't believe it is a problem that someone might make this mistake then it follows that PEP 479 is a massive mistake.
On 13 December 2014 at 09:15, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
Oscar, StopIteration unexpectedly escaping from __next__ methods is *not* the problem PEP 479 is trying to solve, and that misunderstanding appears to be the fundamental error causing your concern. PEP 479 is aiming to solve a specific problem with *writing generator functions*, not the more general problem with invoking iterators that bothers you. There is nothing to be done about StopIteration escaping from __next__ methods in general - it's an inherent limitation of the iterator protocol. StopIteration escaping from generator frames is a different story, as the generator-iterator __next__ method implementation has the chance to intercept them and convert them to something else. So let me try again from the pedagogical angle in a post PEP 380 Python world. If you are teaching someone to write explicit __next__ methods, here are the things you *absolutely* need to teach them: 1. Values returned from the method are produced as values in the iterator 2. Raising StopIteration terminates the iterator 3. If invoked via "yield from", the argument to StopIteration is the result of the "yield from" expression (default: None) 4. If delegating to a function via a function call, points 2 & 3 also apply to that subfunction (this includes the builtin next() function) If you are instead teaching someone to write generator functions, here are the things you *absolutely* need to teach them: 1. Values passed to yield expressions are produced as values in the iterator 2. Returning from the frame terminates the iterator 3. If invoked via "yield from", the return value of the generator frame is the result of the "yield from" expression (default: None) 4. If delegating to a subgenerator (rather than to an arbitrary iterator) via a "yield from" expression, then next(), send() and throw() are passed to that subgenerator until it terminates, at which point execution resumes in the parent generator frame (either producing a value from the yield from expression, or raising an exception if the subgenerator failed) With just those 8 points (and their existing knowledge regarding how to *use* iterators and generators), a developer is now equipped to write full featured __next__ method and generator function implementations, including delegation to subiterators and subgenerators via "yield from" in the latter case. Unfortunately, in both Python 3.3 and 3.4, there's a problem with these instructions: they leave something out. The thing they leave out is the fact that points 2-4 from the section on __next__ methods *also* apply to writing generator functions. Even though it's completely unnecessary, and usually not the right approach if only supporting modern versions of Python, new users still need to be taught it, because it can have unexpected side effects on the way their generators work. This redundancy increases the cognitive complexity of generator functions: new users need to be taught *two* ways of doing something, and then told "don't use the second way, we're only teaching it to you because someone else might use it in code you end up reading or maintaining, or because you might hit it accidentally and be struggling to figure out why your generator based iteration is unexpectedly finishing early". *That's* the problem PEP 479 solves: it's takes those 3 bullet points regarding the behaviour of StopIteration inside __next__ method implementations and makes them *no longer apply* to writing generator functions. If folks do run into them, they'll get a RuntimError and have the opportunity to debug it, and figure out where that came from. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 12/14/2014 10:02 AM, Nick Coghlan wrote:
I had forgotten this apparently important point. After re-reading the somewhat confusing 'Yield expressions' doc, I came up with this example. class subc: def __iter__(self): return self def __init__(self): self.i, self.it = -1, (0,1,2) def __next__(self): if self.i < 2: self.i += 1 return self.it[self.i] else: raise StopIteration('subc done') def subg(): for i in range(3): yield i return 'sub done' def sup(sub): print((yield from sub())) print(list(sup(subc))) print(list(sup(subg))) it = subg() result = [] try: while True: result.append(next(it)) except StopIteration as e: print(e.args[0]) print(result)
Pep 380 not only adds 'yield from', but also a mechanism for *any* iterator to optionally pass a terminal message back to *any* direct user that looks for the message. I don't know how I might use this feature, which is why I more or less ignored it at the time it was introduced, but it is part of asyncio's use of 'yield from'. For statements ignore the return message, so this is one possible reason to use while and explicit next inside try.
Or the return value can be accessed by catching the resulting StopIteration of next().
-- Terry Jan Reedy
On 14 December 2014 at 15:02, Nick Coghlan <ncoghlan@gmail.com> wrote:
Fair enough. I understand your point that the intention of the PEP is only to solve this for generators. I also understand that these threads are not working towards anything so this is my last post on the subject. I see nothing wrong with the rationale that "it's easier to to address this problem for generators" but that's very different from the "this problem is unique to generators" argument. Since the start of this thread the PEP is now modified so that it no longer explicitly makes the latter argument (although it still does not explicitly state the former).
I'm not sure why you single out 3.3 and 3.4 as it happens without yield from. In 2.7 (and anything since 2.2):
I will leave this subject now. Oscar
On Wed, Dec 10, 2014 at 8:35 AM, Chris Angelico <rosuav@gmail.com> wrote:
agreed that this is moistly semantics, so yes, "generator functions" is probably better. Also, IIUC, there is no such thing (or at least not one thing) as an "iterator" -- rather there are various things that conform to the iterator protocol. The most common of these is a class that implements __iter__ and __next__ methods (that behave the way specified) Should we call that an "iterator class"? Whereas there IS such a thing as a "generator" -- it is the type that is returned by a generator function or generator expression (comprehension?) And generators are one of the types that conform to the iterator protocol So, and here is where I am a bit confused, does this PEP change somethign about the generator type, or something about the syntax of for writing generator functions? (or both?) If it's really changing the behavior of the type, then the PEP should use "generator"
I think this confuses things, too: generator functions and iterators have always been completely different -- again, there is no such thing as an iterator at all, there are only things that conform to the protocol, and as I think Oscar pointed out, the behavior of StopIteration _inside_ such a type is not specified by the protocol. We could define an "iterator" as anything that conforms to the protocol -- probably how it is being used for the most part already. In which case. generators are a type of iterator custom classes with __iter__ and __next__ are another type of iterator some build-in objects are also iterators -- range objects, files, what have you. So how to write the above paragraph? How about something like: """ Under this proposal, generator functions and custom iterator classes would be more distinct than they currently are. They are two ways of writing an iterator, but with not only different syntax but different behavior with respect to the handling of StopIteration within the object itself. But generator functions have always been a compact way to write an iterator that "does some of the book keeping for you" -- with this PEP, they will do a bit more of the book keeping. """ """ Like the mixing of text and bytes in Python 2, the mixing of generators and iterators has resulted in certain perceived conveniences, """ I don't know that generators and iterators have ever been "mixed" in the way that text and bytes were. So I think this analogy just confused things. """ The distinction is simple: A generator function returns generator object. The latter is an iterator, having proper __iter__ and __next__ methods, while the former has neither and does not follow iterator protocol. """ I'm not sure there is confusion about that either -- at least not the confusion that is being addressed here. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
Chris Angelico wrote:
No, that's still too confused -- all of that was true before as well. Generator functions have never been iterators themselves. I think Nick has it nailed -- the difference is that code inside a generator function implementing a __next__ method will behave differently from that inside an ordinary function implementing a __next__ method. -- Greg
On Wed, Dec 10, 2014 at 1:34 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
That's hardly news though. *Except* for the possibility of raising StopIteration to end the iteration, the inside of a generator has always been completely different from the inside of a __next__ method implementation. -- --Guido van Rossum (python.org/~guido)
On 12/10/2014 4:37 PM, Guido van Rossum wrote:
Perhaps some Python equivalent of the C code will make it clearer what will and (mostly) will not change. Correct me if I am wrong, but I believe the function __call__ method must do something more or less like the following. class function: ... def __call__(self, *arg, **kwargs): "Setup execution frame and either run it or pass to generator." frame = execution_frame(self, args, kwargs) # several attributes of self are used to setup the frame if no_yield(self.code): run(frame) # until it stops return top(frame.stack) else: return generator(self.code, frame) The generator must look something like the following. class generator: def __iter__(self): return self def __init__(self, code, frame): self.gi_code = code self.gi_frame = frame self.gi_running = False def __next__(self): "Adapt gen-func yield, return to iterator return, raise S.I." self.gi_running = True run(self.gi_frame) if yielded(): return top(self.gi_frame.stack) else: # returned raise StopIteration() self.gi_running = False ... (close, send, and throw methods, not relevant here) (I probably have some details of gi_running wrong, and ignored how exhausted generator calls are handled.) Ignoring gi_code, which I believe is only there for introspection (such as gi_code.__name__, which I use for error-reporting), the link between the generator function and the generator's pre-written __next__ method is the suspended generator-function frame, stored as gi_frame. This is what make one generator instance different from another. Currently, a StopIteration raised during 'run(self.gi_frame)' is passed on like any other exception, but treated specially by the generator caller. The change is to wrap the frame execution or do the equivalent thereof. try: run(self.gi_frame) except StopIteration() raise RuntimeError All else remains the same. Do I have this right? The rationale for the change is the bulk of the PEP. However, the PEP rationale and its acceptance are based on the existing relationship between the execution frame of the gf body and the generator.__next__ method that translates the 'gf protocol' to the iterator protocol. Guido, Nick, and Greg have all pointed to this, but for me, thinking in terms of Python equivalents makes it clearer. -- Terry Jan Reedy
On Thu, Dec 11, 2014 at 8:49 PM, Terry Reedy <tjreedy@udel.edu> wrote:
That's pretty much how the proof-of-concept patch works, yes. (Though the parentheses after StopIteration seem wrong I think?) In theory, it should chain the exceptions, though that's not always currently working. Improvements to the patch welcomed :) ChrisA
On Wed, Dec 10, 2014 at 4:34 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
I think the problem is with the term "generator function" because what we call "generator function" is neither a generator nor a function. It is a callable that returns a generator. To our excuse, this abuse of language is not uncommon. People often say "integer function" when they mean a function with integer domain and range (or even just range).
On Thu, Dec 11, 2014 at 9:11 AM, Alexander Belopolsky <alexander.belopolsky@gmail.com> wrote:
Interesting. Are you saying the 'def' statement may produce a thing that isn't a function, even though it calls itself a function?
It's a function that returns an iterator, same as this is:
A generator function is a function that returns a generator object. Is this problematic? ChrisA
On Dec 10, 2014, at 14:22, Chris Angelico <rosuav@gmail.com> wrote:
def whatisthis(): return gen() That's not a generator function, but it is a function that returns a generator. (That's a bit of a silly case, but there's plenty of real-life code like this. In fact, IIRC, a couple of the functions in itertools are generator functions in the pure Python implementation but regular functions that return iterators in the C implementation.)
On Wed, Dec 10, 2014 at 05:11:43PM -0500, Alexander Belopolsky wrote:
I think the problem is with the term "generator function" because what we call "generator function" is neither a generator nor a function.
I'm afraid that is just wrong. "Generator functions" are functions. py> def gen(): ... yield 1 ... py> type(gen) <class 'function'> They are distinguishable from other functions by the presence of a flag on the __code__ object. The ``isgeneratorfunction`` function in the inspect module is short enough to reproduce here: def isgeneratorfunction(object): """Return true if the object is a user-defined generator function. Generator function objects provides same attributes as functions. See help(isfunction) for attributes listing.""" return bool((isfunction(object) or ismethod(object)) and object.__code__.co_flags & CO_GENERATOR) Generator functions are essentially syntactic sugar for building iterators from coroutines. The specific type of iterator they return is a generator, which is a built-in type (but not a built-in name). -- Steven
On Wed, Dec 10, 2014, at 07:36, Oscar Benjamin wrote:
The problem isn't what they do when the child iterator raises StopIteration. That's always been well-defined as "treat that particular child iterator as having ended" and behave however the function wants to treat that case (e.g. chain moves on to the next argument). The question is what they should do when *one of their other inputs* (such as a map or filter function) raises StopIteration.
On Wed, Dec 10, 2014 at 12:36:59PM +0000, Oscar Benjamin wrote:
I certainly hope so. Currently, generators are iterators: they obey the iterator protocol and the Iterator ABC correctly registers them as instances. py> def gen(): ... yield 1 ... py> it = gen() py> iter(it) is it # Iterator protocol is obeyed. True py> hasattr(it, '__next__') True py> from collections.abc import Iterator py> isinstance(it, Iterator) # And registered with the ABC. True Surely this isn't going to change? If it does, I expect that's going to break an enormous amount of code. If generators are to cease to be iterators, what will they be?
Sounds reasonable to me. -- Steven
On 11 December 2014 at 00:18, Steven D'Aprano <steve@pearwood.info> wrote:
If generators are to cease to be iterators, what will they be?
What the PEP is trying to say is that generators are not the same thing as __next__ method implementations (and I agree that shortening the latter to "iterators" is incorrect). __next__ method paths for leaving the frame: * "return value" = produce next value * "raise StopIteration" = end of iteration Current generator paths for leaving the frame: * "yield value" = produce next value * "return" = end of iteration * "raise StopIteration" = end of iteration PEP 479 generator paths for leaving the frame: * "yield value" = produce next value * "return" = end of iteration * "raise StopIteration" = RuntimeError The only change is to eliminate the redundant method of leaving the frame, as that behaviour means that adding a "yield" to a function *implicitly* suppresses StopIteration exceptions raised elsewhere in that frame. That implicit side effect on exception handling behaviour doesn't exist for __next__ method implementations, or for ordinary functions used in iteration operations that accept arbitrary callables, as ordinary functions won't include any yield expressions (by definition). Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thu, Dec 11, 2014 at 3:21 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Current wording: (at least, I think this is the only bit that's questionable) """ Under this proposal, generators and iterators would be distinct, but related, concepts. Like the mixing of text and bytes in Python 2, the mixing of generators and iterators has resulted in certain perceived conveniences, but proper separation will make bugs more visible. """ Would it be better to clarify that as "generator functions"? Maybe something like this: """ Under this proposal, generator functions and iterators would be distinct, but related, concepts. Like the mixing of text and bytes in Python 2, the mixing of generators and iterators has resulted in certain perceived conveniences, but proper separation will make bugs more visible. The distinction is simple: A generator function returns a generator object. The latter is an iterator, having proper __iter__ and __next__ methods, while the former has neither and does not follow iterator protocol. """ (Most of the text ignores __iter__, even though technically it's an important part of iterator protocol for it to exist and return self. I don't think anyone will be confused by the omission, as it perfectly follows __next__, but I could add text if people think it's important.) ChrisA
On 10 Dec 2014 16:38, "Chris Angelico" <rosuav@gmail.com> wrote:
I find this more confusing than the original, actually, because now it sounds like you're saying that the distinction between a generator function and a generator instance is something new that this PEP is adding, in order to fix all the problems that are being caused by people writing 'for x in genfunc: ...'. Which doesn't make much sense. Like Nick said, it seems to me that the key distinction to emphasize is the distinction between generator function bodies and __next__ method bodies. -n
I have replaced the confusing paragraph with this: """ The proposal does not change the relationship between generators and iterators: a generator object is still an iterator, and not all iterators are generators. Generators have additional methods that iterators don't have, like ``send`` and ``throw``. All this is unchanged. Nothing changes for generator users -- only authors of generator functions may have to learn something new. """ Let us know if there is still confusion about what the PEP means. --Guido On Wed, Dec 10, 2014 at 10:13 AM, Guido van Rossum <guido@python.org> wrote:
-- --Guido van Rossum (python.org/~guido)
On 10 December 2014 at 20:46, Guido van Rossum <guido@python.org> wrote:
Generators are affected and this therefore affects authors of both generator functions and generator expressions. In the case of a generator function you can achieve the old behaviour post-PEP 479 simply by wrapping the entire body of the function with: try: # body except StopIteration as e: return e.value The only difference is that the generated StopIteration has a different traceback attached so you can't see where it was originally "raised". Generator expressions would need to be converted to generator functions to achieve the same effect.
Let us know if there is still confusion about what the PEP means.
The PEP is still very confused. Primarily the fact is that the PEP is attempting to address an issue which affects all iterators but proposing a solution which is only about generators. The PEP seems to suggest that there is something special about generators in this respect when there really isn't. For example: """ The interaction of generators and StopIteration is currently somewhat surprising, and can conceal obscure bugs. An unexpected exception should not result in subtly altered behaviour, but should cause a noisy and easily-debugged traceback. Currently, StopIteration can be absorbed by the generator construct. """ There is no interaction between generators and StopIteration. The issue isn't about generators it is about the iterator protocol. StopIteration cannot be absorbed by the generator construct. I think the PEP would be clearer if it properly acknowledged that the problem is a problem for all iterators. The question then is why the fix is only targeted at generators and what should be done about the same problem that occurs in many other forms. The PEP rationale avoids these issues by falsely claiming that generators are special.
On 12/11/2014 09:14 AM, Oscar Benjamin wrote:
It seems to me, the main benefit is for writers of nested generators and co-routines. It makes those cases easier to debug, and may help generators in general be more reusable, and reliable, in the context of nested generators and co-routines. Which if true could contribute to better, and more dependable, multi-thread, multi-process, or even multi-core, software designs. That is the general impression I get from the conversations about the changes being made. But I'm not sure I can point to any one message stating it like that. Possibly someone can confirm if any of those points are valid and why. Or it could be I'm just a bit too optimistic. ;-) Cheers, Ron
On 12/11/2014 10:14 AM, Oscar Benjamin wrote:
On 10 December 2014 at 20:46, Guido van Rossum <guido@python.org> wrote:
What is very confusing is the ambiguous use of 'generator' to mean either 'generator function' or 'generator class instance'. Since I do not know what either of you mean by 'generator' in each instance above, I do not know exactly what either of you meant. If the PEP and discussants consistency used 'generator' to mean "instance of the internal generator class, initialized with a suspended instance of a *user-written* generator function body", then I think some of the confusion would dissipate. The PEP is about the conjunction of the following facts (and a few more, but I think these are the most salient): a) generator function bodies follow a different protocol than iterator __next__ methods, one that does not require them to *ever* raise StopIteration; b) if a g.f. body does raise StopIteration, it might be a substitute for 'return', but it might be a bug -- and apparently bugs do happen in real code; and c) generator.__next__ currently trusts that *user-written* g.f.s are never buggy. The PEP proposal is, either literally or in effect, that generator.__next should stop trusting StopIteration from a g.f., thereby disallowing the sometimes convenient but optional substitution. The justification is that bugs should not pass silently, and now they do.
Guido today add the following, apparently in response to the above. +When implementing a regular ``__next__()`` method, the only way to +indicate the end of the iteration is to raise ``StopIteration``. So +catching ``StopIteration`` here and converting it to ``RuntimeError`` +would defeat the purpose. This is a reminder of the special status of +generator functions: in a generator function, raising +``StopIteration`` is redundant since the iteration can be terminated +by a simple ``return``. I would put this answer slightly differently. A __next__ method is supposed to raise StopIteration if *and only if* there are no more items to return. The __next__ method author is responsible for fulfilling the contract. Core developers are responsible for generator.__next__; we are not responsible for iterators that others write. Anyone who writes an iterator class whose instances are initialized with 3rd party non-iterator code that is executed in .__next__, should think about what to do if that code raises StopIteration. It would be possible for the PEP to recommend that other .__next__ methods executing external code might do something similar to what generator.__next__ will do. try: execute(self.external_code) except StopIteration as e: raise RuntimeError(<whatever>) -- Terry Jan Reedy
On 12 December 2014 at 01:14, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
I believe you're misunderstanding the problem being solved. The specific problem deemed worthy of being fixed is that the presence of "yield" in a function body can implicitly suppress StopIteration exceptions raised elsewhere in that function body (or in functions it calls). The difference in behaviour between comprehensions and generator expressions when it comes to embedded function calls that trigger StopIteration is a special case of that more general difference. This is a problem unique to generators, it does not affect any other iterator (since explicit __next__ method implementations do not use yield). The change in the PEP is to change that side effect such that those exceptions are converted to RuntimeError rather than silently suppressed - making generator function bodies behave less like __next__ method implementations. For special methods themselves, the problem of unexpected signalling exceptions from other called functions being interpreted according to their defined meaning in the relevant protocol is inherent in their design - they only have two "normal" outcomes (returning from the frame, or raising the signalling exception). That limitation applies to StopIteration & __next__ in the same way that it applies to KeyError/IndexError and __getitem__, or AttributeError and the various methods in the attribute access protocols. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 12 December 2014 at 10:34, Nick Coghlan <ncoghlan@gmail.com> wrote:
And I believe you are too. :)
The yield causes the function to become a generator function. The frame for a generator function (like for any other function) will allow uncaught exceptions to propagate to the frame above. The difference between generator functions and other functions is that the code in the body of a generator function is executed when someone calls the generator's __next__ method. Since the caller (the iterator consumer) is expecting StopIteration it is treated as the signalling the end of iteration. The yield suppresses nothing; it is the iterator consumer e.g. the for-loop or the list() function etc. which catches the StopIteration and treats it as termination.
I don't know what you mean by this.
Incorrect. The problem is not unique to generators and the yield is irrelevant. The problem (insofar as it is) is a problem for all iterators since all iterators interact with iterator-consumers and it is the iterator-consumer that catches the StopIteration. Here is an example using map:
Ordinarily map should terminate when the underlying iterator raises StopIteration. Since it simply allows a StopIteration raised anywhere to bubble up it will also allow the StopIteration raised by func to bubble up. Implicitly the effect of that is that map agrees to cooperatively terminate iteration when any part of the code that it executes raises StopIteration. Note that it is the list() function (the iterator-consumer) that actually catches the StopIteration and terminates just as it would be if map were a generator:
In principle that could be a useful feature in some cases. In practise it is more likely to be something that masks a bug manifesting in a stray StopIteration e.g.:
In this case it is the for-loop (the iterator-consumer) that catches the StopIteration. Whether or not map is a generator or some other iterator is irrelevant.
Fine but generator function bodies are __next__ method implementations. You can claim that the generator function implements __iter__ but its body implements the __next__ method of the generator. It is precisely because it is a __next__ method that it finds itself being called by an iterator consumer and so finds that a loose StopIteration is suppressed by the consumer. This is the same for all iterators and there is nothing special about generators in this regard (although the PEP will change that a little). To summarise succinctly: Loose StopIteration can have an unintended interaction with the iterator protocol and more specifically with iterator-consumers. The effect is as if the iterator had been stopped since the consumer will stop iteration. Because generators are iterators they (like all iterators) are affected by this. Oscar
On 12 December 2014 at 22:42, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
Oscar, this is where you're *just plain wrong* from a technical perspective. The conversion to a generator function also introduces an additional layer of indirection at iteration time through the generator-iterator's __next__ method implementation. That indirection layer is the one that converts an ordinary return from the generator frame into a StopIteration exception. Historically, it also allowed StopIteration from within the frame to propagate out, make it impossible to tell from outside the generator iterator whether the termination was due to the generator returning normally, or due to StopIteraton being raised. The introduction of "yield from" in Python 3.3, and its use in the coroutine design for the asyncio library and similar contexts *makes that distinction more important than it used to be*. It even managed to introduce obscure bugs around the use of generators to write context managers, where factoring out part of the context manager to a subgenerator may implicitly suppress a StopIteration exception raised in the body of the with statement. The problem you think PEP 479 is trying to solve *is* the one where the discussion started, but it is *not* the one where it ended when Guido accepted PEP 479. The problems PEP 479 solves are generator specific - they have nothing to do with the iterator protocol in general. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 12 December 2014 at 13:14, Nick Coghlan <ncoghlan@gmail.com> wrote:
Which of the above sentences is technically incorrect?
I understand all of this.
Okay well this is not clear from the PEP (and bear in mind that I did take the time to read through all of the lengthy threads on this subject before posting here). I haven't read the most recent changes but when I did read the PEP the justification appeared to stem from claims that were confusing at best and plain false at worst e.g. "generators are not iterators". Oscar
On Fri, Dec 12, 2014 at 5:58 AM, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
I don't think there is much technical confusion here, but there seems to be, so, so that highlights that it's very important to write vary carefully about all this. I"m going to try this: 1) It's not clear what is meant by an "iterator" -- when we use the term is seems like we are talking about a type, when we are really talking about "anything that conforms to the iterator protocol", so I'm going to introduce a new term: TIIP (Type Implementing the Iterator Protocol) -- maybe that's too cute, but role with it for now. 2) there are a number of TIPPS in the standard library: things like tupleiterator and friends, the range object, etc. 3) there are essentially two (and a half) ways for python code authors to write their own, custom TIIP: 1) write a class that has a __iter__ and __next__ methods that "do the right thing" 2a) write a generator function 2b) write a generator expression The latter create a generator type, which is a specific TIIP that come built in to the standard library. Generator functions are a particularly nifty syntax for creating TIIPs -- i.e. I'm pretty sure that there is nothing you can do with a generator function that you can't do by writing a custom TIIP class. But generator functions provide really nifty syntax that "does a lot of the bookeeping for you" This is pretty much how I've taught this stuff in the past. But what this PEP does is change generators a bit, so that they do even more of the book keeping for you -- that sure seems reasonable enough when phrased that way. So I agree with Oscar (and others) here the issue of what happens to a StopIteration that is raised inside a TIIP be another call is an issue that effects not only generators, but also custom TIIP classes. I think is wold be nice if this were addressed in both cases the same way, but that has been proposed and rejected for what I, at least, think are sound technical reasons. And the PEP has been accepted so the only thing left to bike-shed about is how it discussed. So why it it worth changing the behavior of generators? 1) we can -- custom classes are, well, custom, so are free to do what author wants them to do. Generators are built-in classes and can therefor be controlled by python itself. 2) generator function have a different, cleaner, more compact way to express intent. while they create generators that conform to the iteration protocol -- i.e. raise a StopIteration when done, the author does not have to explicitly do that -- you can write a generator function without any awareness of StopIteration ( I might argue that you won't get far if you don't understand all that, but...) So having generators automatically "do the right thing" when they call another TIIP inside themselves conforms well to the spirit of generator functions: clean compact code that does the book keeping for you. 3) when you write a custom TIIP class, you have complete control over what it does, and have to be thinking about when StopIteration is raise. So you are more likely to think about how you want to handle a StopIteration that may be raised by another call to a TIP inside your code. You also want to be able to control that depending on circumstances, so probably wouldn't want magic Exception mangling anyway. Not sure if this helps the PEP writing, but I've got some good notes for the next time I teach this... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Fri, Dec 12, 2014 at 3:06 PM, Alexander Belopolsky < alexander.belopolsky@gmail.com> wrote:
The same logic applies to generators and I think this misconception is at the root of the confusion in this thread. When you are writing a generator function, you are not creating a custom TIIP. You are writing a code object that Python knows how to convert into an instance of a built-in TIIP (the generator type).
On Fri, Dec 12, 2014 at 12:14 PM, Alexander Belopolsky < alexander.belopolsky@gmail.com> wrote:
The point was supposed to be that you want a TIIP with particular behavior that is not built-in (hence custom). Yes, in terms of type, a generator function, when called, will return the a built-in type, but its behavior is customized. Indeed the same as writing a class with __getitem__ and no __iter__ or __next__ will give you an instance a built-in iterator type with customized behavior. I guess my take is that the user shouldn't have to know exactly what the type of TIIP is produced is -- only what its behavior will be. But maybe that IS the source of much of this confusion -- the PEP authors make the point that generators are something special. I want tend to think of them as an implementation detail. This PEP makes them more special, and that's really the point of the contentious part of the PEP. So maybe that's the the point we are trying to make -- while we can argue about whether generators ARE iterators, or whether they have been "conflated" -- generators are special, and might as well be more special ;-) -Chris
-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Fri, Dec 12, 2014 at 12:16 PM, Chris Barker <chris.barker@noaa.gov> wrote:
2) there are a number of TIPPS in the standard library: things like tupleiterator and friends, the range object, etc.
The range object is not an iterator
isinstance(range(1), collections.Iterator) False
and does not implement the iterator protocol
Please don't confuse iterators and iterables:
isinstance(range(1), collections.Iterable) True
On Fri, Dec 12, 2014 at 11:41 AM, Alexander Belopolsky < alexander.belopolsky@gmail.com> wrote:
indeed -- sorry about that -- the "rangeiterator" object, that is. The point here is that there are a bunch of builtin types that are TIIPs
Please don't confuse iterators and iterables:
indeed, that is a distinction that should be kept clear. Thanks -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Dec 12, 2014, at 9:16, Chris Barker <chris.barker@noaa.gov> wrote:
When you're writing actual code, you can test this with isinstance(collections.abc.Iterator). Of course ABCs don't _perfectly_ cover their related protocols (in this case, if you create a type that has __iter__ and __next__ but __iter__ doesn't return self--or, of course, if you just register some random type with Iterator explicitly--you've created a subclass of Iterator that isn't a subtype of the iterator protocol). But I think it still makes sense to use Iterator as shorthand for "type that subclasses Iterator" with the implied "is an actual subtype of the semantic type intended by the Iterator class". This kind of shorthand is used regularly in discussing types in every OO language, not just Python. You don't see people writing about "an instance of a Type Implementing IEnumerable [or NSFastEnumeration or random_access_iterator]). You just see "an IEnumerable". And to discuss the type itself, you just say "an IEnumerable type". The fact that IEnumerable is itself defined as a type is rarely a source of confusion--and when it is, you say "an IEnumerable subtype". In Python, the term "Iterator" is just as consistent and meaningful as in all these other languages. The fact that some people confuse iterables and iterators isn't a reason to abandon this simplicity. Especially since it doesn't even help to solve the problem--as we've just seen in this very message I'm replying to, it's just as easy for someone to mistakenly use TIIP to describe "range" as it is for them to mistakenly use "Iterator". (In fact, it's arguably worse, because what would you call an Iterable under the same naming scheme but Type Implementing Iterable Protocol, or TIIP?) The things we need to be clear about here are the things that _dont't_ have an official name. In particular, the thing that's built from calling a generator function or evaluating a generator expression and used by the generator.__next__ method is not a generator, a generator function, a generator function body, an iterator, or anything else with a name in the language. And that's what's confusing people. They want to call it one of the things that has a name. Renaming any of those things that it's not doesn't help to give the thing that it is a name.
On Fri, Dec 12, 2014 at 3:22 PM, Andrew Barnert <abarnert@yahoo.com> wrote:
this issue here is that inteh PEP, and in this list, it seems "iterator" was used to describe classes with __init__ and __next__ methods, AS APPOSED to the things returned from functions with a yield in them... and: In [12]: g = (i for i in range(10)) In [13]: isinstance(it, collections.abc.Iterator) Out[13]: True and In [15]: def gf(): ....: yield 1 ....: yield 2 In [16]: gen = gf() In [17]: isinstance(gen, collections.abc.Iterator) Out[17]: True So generators ARE iterators, but his definition and yet the contrast was being made....
This kind of shorthand is used regularly in discussing types in every OO language, not just Python.
sure, but Python is dynamically types, so "conforms to a protocol" not at all teh same is "isinstance", even though ABCS have made this closer...
actually, the confusion between iterables and iterators want the issue I as trying to address -- but the confusion with a sentence like "Under this proposal, generators and iterators would be distinct, but related, concepts" That has been removed from the PEP -- I think it's more clean now in any case. But yes, I could have written all that and stuck with plain old "iterator" instead of making up "TIIP" Especially since it doesn't even help to solve the problem--as we've just
My bad -- typing too fast! And this was never about the itterable vs iterator distinction anyway.
I was not trying to mingle iterable and iterator -- really I wan't :-)
now I am confused: In [18]: g = ( i for i in range(10) ) In [19]: type(g) Out[19]: generator how is that not a "generator"? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Dec 12, 2014, at 16:40, Chris Barker <chris.barker@noaa.gov> wrote:
But the thing returned by a function without yield in it is an instance of the built-in class generator, which is a subtype of Iterator. Are you trying to make a distinction between iterator _types_ and iterator type _instances_? If so, TIIP doesn't seem to help very much.
Yes, there was some unfortunate wording in the earlier version that Guido already acknowledged and promised to fix well before your message.
This kind of shorthand is used regularly in discussing types in every OO language, not just Python.
sure, but Python is dynamically types, so "conforms to a protocol" not at all teh same is "isinstance", even though ABCS have made this closer...
The whole point of ABCs is to allow you to define protocols in-language. Other dynamic OO languages like Smalltalk and ObjC have similar concepts.
You've skipped the "used by the generator" part of that sentence. Of course g is an instance of generator (and therefore also an iterator). But you're not defining the generator type here--that already exists as a builtin type. What you're defining is a thing made up of a code object and a frame, which is used by that generator instance to do its generating. That thing is not a generator, or a generator type (or an iterator instance or type), or a __next__ method, or anything else with a name. (I don't know if that's the thing Guido was trying to describe as not an iterator, but I don't think that's too important, since he's already acknowledged that the distinction he was trying to make isn't important anyway.)
On Dec 12, 2014, at 17:32, Ethan Furman <ethan@stoneleaf.us> wrote:
Sorry, autocorrect typo. The thing returned by a function *with a* (not *without*) yield in it is a generator. And generator is a class with __iter__ and __next__. Of course an individual generator *instance* isn't such a type, but that's no different from the fact that 2 isn't a number type.
On Fri, Dec 12, 2014 at 03:22:32PM -0800, Andrew Barnert wrote:
+1 to this. But:
I don't know what thing you are referring to. If I write this: py> def gen(): ... yield 1 ... py> it = gen() then `gen` is a function. It's a specific kind of function that uses "yield", and the name for that is a generator function. Informally, sometimes people call it a "generator", which is a bad habit due to the possibility of confusing `gen` with `it`, but since it is quite rare to be in a position where such confusion can occur (PEP 479 and related discussions not withstanding), we can often get away with such sloppy terminology. Also, `it` is an iterator. It's also a generator: py> type(it) <class 'generator'> The same applies to generator expressions: py> type(c for c in "aaa") <class 'generator'> So I'm not sure which mysterious object you are referring to that doesn't have a standard name. Can you give a concrete example? -- Steven
On Dec 13, 2014, at 5:36, Steven D'Aprano <steve@pearwood.info> wrote:
Chris Barker just asked exactly the same question yesterday, except that he trimmed the quote more and used "g" instead of "it" for his example. So I'll just paste the same reply, and you can manually s/g/it/. Of course g is an instance of generator (and therefore also an iterator). But you're not defining the generator type here--that already exists as a builtin type. What you're defining is a thing made up of a code object and a frame, which is used by that generator instance to do its generating. That thing is not a generator, or a generator type (or an iterator instance or type), or a __next__ method, or anything else with a name. (I don't know if that's the thing Guido was trying to describe as not an iterator, but I don't think that's too important, since he's already acknowledged that the distinction he was trying to make isn't important anyway.)
On 12/13/2014 01:37 PM, Andrew Barnert wrote:
It's a function with a flag set.
gen.__code__.co_flags 99
Where a function's flags would be this...
So, it the answer is yes and no. It's a function object, with an altered behaviour. It seems like the flag may be checked at call time, and then a new function generator instance is returned..
And the original function body is moved to it's gi_code attribute.
I think the send() method, a builtin C routine, handles all the particulars of running gi_code with gi_frame. (I'd have to look to be sure.) I think if function definitons with yield in them created generator_instance_factory objects instead of function objects, things would be easier to explain. What it comes down to, is they aren't normal functions and it's a bit confusing to think of them that way. Cheers, Ron
On Sun, Dec 14, 2014 at 7:19 AM, Ron Adam <ron3200@gmail.com> wrote:
They're no more different than, say, a decorated function that returns a different function. But if you want a different way to explain them, here's a similar way to craft an iterator: class seq_iter: def __init__(self, func, *args, **kw): self.func = func self.args = args self.kw = kw self.state = 0 def __iter__(self): return self def __next__(self): if self.state == -1: raise StopIteration ret = self.func(self.state, self, *self.args, **self.kw) self.state += 1 if ret is not self: return ret self.state = -1 raise StopIteration def sequential_iterator(func): @functools.wraps(func) def inner(*args, **kw): return seq_iter(func, *args, **kw) return inner ##### Example usage ##### @sequential_iterator def ten_squares(idx, finished): if idx == 10: return finished return idx*idx The function returned by the decorator isn't the one decorated, but it's clearly still a function. No reason for it to be a unique type. This is similar to the relationship between generator functions and the iterators returned when you call them. It's an alternative way to write an iterator class, dealing with lots of the bookkeeping for you. ChrisA
On 12/13/2014 03:27 PM, Chris Angelico wrote:
They're no more different than, say, a decorated function that returns a different function.
But that is different, because the decorator isn't visible. You can also offer examples of other objects in terms of other objects, they are still other objects. So it raises questions like this ... When should we make a distinction of one object from another? And when shouldn't we? Usually it has more to do with weather or not it makes writing some program easier. I'm not sure changing the type in this case would be useful in that way.
But if you want a different way to explain them, here's a similar way to craft an iterator:
I think it's what the average python user expects that matters. The term generator_function was use quite a lot in the PEP discussion in order to distinguish it from a regular function, and also from the generator returned from it. So it seems to me, that maybe there is something to that. Currently (especially when discussing things here), I think it's important to try to have an accurate model of what is being discussed. Comparisons to similar code is also useful, but for checking the implementation works correctly and consistently. They don't replace the accurate model. That model could be pseudo-code, or python, or a diagram, but it helps to keep in in mind. So, we have a rule we need to remember: If a function has "yield" in it, it will return a generator-object. Ok. So a generator is: a function + that rule. We have a case where we differentiate it in our minds, but the code, type(g) indicates it is not different, but the code actually does do something a bit magical. Yes we can visually examine the body of the function and tell that it's different, but that isn't full proof. That is some of what leads me to think it should maybe be a different type. Possibly a sub-class of a function class, would be the only option. Fortunately generators are not new, so most everyone does understands how to define and use them, just not how the implementation is actually done. But maybe that's ok. <shrug> Cheers, Ron
On Sun, Dec 14, 2014 at 12:33 PM, Ron Adam <ron3200@gmail.com> wrote:
Many things can be distinguished that are not distinct types. A generator function is a function that has a 'yield' in it. A factory function is a function that constructs and returns a new object. A recursive function is a function which calls itself. All three are functions in every way. ChrisA
On 12/13/2014 07:38 PM, Chris Angelico wrote:
I see only two things... functions that do exactly what's in the defined body when called, and generator_functions which does something other than what is defined in the body when called. That seems like quite a difference to me. Cheers, Ron
On Sat, Dec 13, 2014 at 11:37:23AM -0800, Andrew Barnert wrote:
I read your reply to Chris last night, and it didn't make sense to me then and it still doesn't make sense to me now :-( Hence my request for a concrete example. I am still unclear as to what this mystery unnamed thing is. It's not a generator function, because that is called a generator function. It's not the thing you get when you call a generator function, because that is called a generator or an iterator, depending on context. Surely it's not the generator type, because that has a name: "generator". We always use the same name for the class and its instances, e.g. we talk about "int" the class as well as saying that 123 is an int. The class "int" is not itself an int instance, but we still call it "int". And likewise for "generator".
Of course g is an instance of generator (and therefore also an iterator).
Agreed. But giving examples of things which aren't the mystery entity doesn't really help explain which mystery entity you are talking about :-)
Sure the thing you are defining when you write "def gen() ..." is a function? I thought we were in agreement about that. It's a function with a flag set, a.k.a. a generator function. If you're referring to something else, a concrete example will help. Because as we stand now, I don't know whether we're talking about something important or not, or if we are in agreement, disagreement, or merely confusion :-) -- Steven
On Dec 13, 2014, at 17:17, Steven D'Aprano <steve@pearwood.info> wrote:
You didn't ask for a concrete example, you provided the exact same answer as Chris but with different names.
Again, it's the thing used by the generator instance's __next__ method: the frame and code objects.
I don't know how else to explain it. Look at the members stored by the generator type: a frame and a code object. So, going back to the original point: Generator instances are not different from iterators (except that it's a subtype, of course). The builtin generator type is likewise no different than a custom iterator class. The generator body is a normal function body. The only thing that isn't like an iterator is the frame. Which is not an important distinction. I think Chris was making a different distinction, claiming that generators are not an Iterator type (which they aren't, but only because they're _instances_ of an Iterator type, the builtin type generator), and I was pointing out that this can't be the distinction Guido was trying to make. But, again, since Guido has already said that sentence was mistaken, this really isn't important.
I already said from the start that this is almost certainly not important.
In an effort to try to keep this already too-long thread a bit shorter, I'm going to trim drastically. On Sat, Dec 13, 2014 at 09:43:02PM -0800, Andrew Barnert wrote:
On Dec 13, 2014, at 17:17, Steven D'Aprano <steve@pearwood.info> wrote:
My first post in this sub-thread ended with: "So I'm not sure which mysterious object you are referring to that doesn't have a standard name. Can you give a concrete example?" But moving along:
Aha, now the penny drops. Given a generator object, is has attributes holding a frame object and a code object (which it shares with the generator function that created it): py> def gen(): yield 1 ... py> it = gen() py> it.gi_frame <frame object at 0xb7b8f95c> py> it.gi_code <code object gen at 0xb7b6e660, file "<stdin>", line 1> py> it.gi_code is gen.__code__ True Both have standard names: "frame object" and "code object" respectively.
I don't know how else to explain it. Look at the members stored by the generator type: a frame and a code object.
That's exactly the concrete example I was looking for.
Well, there are a few other differences as well. `generator` is a concrete type, `Iterator` is an abstract type, and `iterator` can refer to either a specific iterator type: py> class X: ... def __getitem__(self, i): pass ... py> iter(X()) <iterator object at 0xb7af4b0c> or speaking more generically, to anything which obeys the iterator protocol. More importantly, generators have abilities that other iterators don't have: you can send data into a generator, not just get data out of them.
I'm not sure what Chris was trying to say, but I've seen various other people maintain that generators aren't iterators. As you say, that is wrong: py> from collections.abc import Iterator py> isinstance(gen(), Iterator) True The *class* `generator` is an Iterator (sub-)type, but not an Iterator itself: py> issubclass(type(gen()), Iterator) True py> isinstance(type(gen()), Iterator) False which is no different from instances of int being numbers, but int itself (the class) not being a number. Thank you for helping me understand what you were trying to say. -- Steven
On Dec 14, 2014, at 0:58, Steven D'Aprano <steve@pearwood.info> wrote:
Sure, but the problem is that people have been writing things that come "from the generator's internals", and then trying to explain how those things "make generators different from iterators" (or "... from explicit __next__ methods"). Because of the wording that was being used, people were looking for some way in which generators, or the generator type, or generator functions, are different from iterators, from other iterator types, or whatever, when none of that is at issue; it's the suspended generator frame, which doesn't have any parallel in other iterator implementations. I was thinking that if we had some term like "generator state" to refer to the frame-and-code (or just the frame, because the code isn't really necessary), the confusion wouldn't have arisen. Now I notice that Nick Coghlan has been referring to "the generator frame", which is a nice way to avoid that confusion. What I was basically suggesting is that someone should rewrite the explanations exactly the way Nick already had been rewriting them. :)
That was in the message you replied to. I think the problem isn't that I didn't give the example you were looking for, it's that (at least in that message) I didn't explain why you (or Chris or anyone else) should care about it, so you passed right over it. Sorry for not making that part clear.
On Sun, Dec 14, 2014 at 7:58 PM, Andrew Barnert < abarnert@yahoo.com.dmarc.invalid> wrote:
I never was too much into using an exception to signal than an iterator had finished its job. Either intuition or experience said that it would be a source of obscure problems. I'm not proposing a solution. Not even proposing a change. I do propose that since this thread has been long and going long, we take the opportunity to consider the base assumptions. Most newcomers to programming take a while to grok complex control flows, and what's happening here is too complex even for Python contributors. I suggest we revisit "import this" for a little while, just in case we made a mistake along the way. Sorry for the rambling! Regards, -- Juancarlo *Añez*
Juancarlo Añez writes:
As I understand Raymond's point, what is happening here is *not* particularly complex (I don't see it as complex myself, anyway). It's simply an obscure (because rare) and obvious (once you know about it) consequence of two simple design decisions: "exceptions bubble" and "iterator termination is signaled with an exception". That obscurity is compounded by the fact that a programmer error will pass silently. (There must be a theorem that says that in any Turing complete language some errors will pass silently!) In the case of *generators* silently ignoring an error is unnecessary, and PEP 479 fixes that one case, at a cost of breaking a useful idiom in some functions. For iterators in general, I suppose we could add a flag to checkers to look for function calls inside iterators, and warn the user to check for the potential for spuriously raising StopIteration. (I doubt that's a great idea, but it could be done.)
On 12/12/2014 7:42 AM, Oscar Benjamin wrote:> On 12 December 2014 at 10:34, Nick Coghlan <ncoghlan@gmail.com> wrote:
This makes no sense to me either.
Neither do I
The parenthetical comment is true. However, I think part of the problem is unique to generator functions, and part is more general -- that loose usage of StopIteration creates problems when a .__next__ method calls external code.
Incorrect. The problem is not unique to generators and the yield is irrelevant.
Nick (and Guido) are looking at the unique part and Oscar is looking at he general part.
This function violates the implicit guideline, which I think should be more explicit in the doc, that the only functions that should expose StopIteration to the outside world are iterator.__next__ functions and the corresponding builtin next(). It is not surprising that this violation causes map to misbehave. However, changing the .__next__ methods of map and filter iterators should be a new thread, which I will write after posting this. The g.f. PEP says that generator functions should also follow this guideline and not raise StopIteration (but should always return when not yielding). ...
This is about as wrong, or at least as confusing, as saying that the function passed to map is a __next__ method implementation. There is only one generator.__next__, just as there is only one map.__next__. Both .__next__ methods execute the user function (or function frame) passed in to the corresponding .__init__ to calculate the next value to return. So the external functions assist the __next__ methods but are not __next__ methods themselves. In particular, they should not raise StopIteration. -- Terry Jan Reedy
On 12 December 2014 at 20:06, Terry Reedy <tjreedy@udel.edu> wrote:
I'm wondering if Nick is referring to two different things at once: On the one hand "yield from" which is an iterator-consumer suppresses StopIteration from a child iterator and on the other hand every generator interacts with a parent iterator-consumer that will catch any StopIteration it raises. When generator function f "yield from"s to generator function g both things occur in tandem in two different frames. <snip>
The function was just supposed to generate the effect. More plausible examples can be made. I tripped over this myself once when doing something with iterators that required me to extract the first item before processing the rest. A simple demonstration: def sum(iterable): iterator = iter(iterable) total = next(iterator) for item in iterator: total += item return total results = list(map(sum, data_sources))
The g.f. PEP says that generator functions should also follow this guideline and not raise StopIteration (but should always return when not yielding).
The idea is that it happens by accident. Oscar
On 12/12/2014 4:10 PM, Oscar Benjamin wrote:
Examples like this has been posted on python-list, probably more than once. The answer has been to catch StopIteration (which should only come from next/__next__) and either return a default value or raise ValueError('cannot sum an empty iterable').
results = list(map(sum, data_sources))
-- Terry Jan Reedy
On 12 December 2014 at 22:54, Terry Reedy <tjreedy@udel.edu> wrote:
I know what the solution is. For my purposes simply raising an exception (any exception that wasn't StopIteration) would have been sufficient. I'm just saying that it's a mistake that can easily be made. Since I made it I'm now super-suspicious of next() so I probably wouldn't make it again. This mistake is the root of all of the problems that PEP 479 attempts to solve. If you don't believe it is a problem that someone might make this mistake then it follows that PEP 479 is a massive mistake.
On 13 December 2014 at 09:15, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
Oscar, StopIteration unexpectedly escaping from __next__ methods is *not* the problem PEP 479 is trying to solve, and that misunderstanding appears to be the fundamental error causing your concern. PEP 479 is aiming to solve a specific problem with *writing generator functions*, not the more general problem with invoking iterators that bothers you. There is nothing to be done about StopIteration escaping from __next__ methods in general - it's an inherent limitation of the iterator protocol. StopIteration escaping from generator frames is a different story, as the generator-iterator __next__ method implementation has the chance to intercept them and convert them to something else. So let me try again from the pedagogical angle in a post PEP 380 Python world. If you are teaching someone to write explicit __next__ methods, here are the things you *absolutely* need to teach them: 1. Values returned from the method are produced as values in the iterator 2. Raising StopIteration terminates the iterator 3. If invoked via "yield from", the argument to StopIteration is the result of the "yield from" expression (default: None) 4. If delegating to a function via a function call, points 2 & 3 also apply to that subfunction (this includes the builtin next() function) If you are instead teaching someone to write generator functions, here are the things you *absolutely* need to teach them: 1. Values passed to yield expressions are produced as values in the iterator 2. Returning from the frame terminates the iterator 3. If invoked via "yield from", the return value of the generator frame is the result of the "yield from" expression (default: None) 4. If delegating to a subgenerator (rather than to an arbitrary iterator) via a "yield from" expression, then next(), send() and throw() are passed to that subgenerator until it terminates, at which point execution resumes in the parent generator frame (either producing a value from the yield from expression, or raising an exception if the subgenerator failed) With just those 8 points (and their existing knowledge regarding how to *use* iterators and generators), a developer is now equipped to write full featured __next__ method and generator function implementations, including delegation to subiterators and subgenerators via "yield from" in the latter case. Unfortunately, in both Python 3.3 and 3.4, there's a problem with these instructions: they leave something out. The thing they leave out is the fact that points 2-4 from the section on __next__ methods *also* apply to writing generator functions. Even though it's completely unnecessary, and usually not the right approach if only supporting modern versions of Python, new users still need to be taught it, because it can have unexpected side effects on the way their generators work. This redundancy increases the cognitive complexity of generator functions: new users need to be taught *two* ways of doing something, and then told "don't use the second way, we're only teaching it to you because someone else might use it in code you end up reading or maintaining, or because you might hit it accidentally and be struggling to figure out why your generator based iteration is unexpectedly finishing early". *That's* the problem PEP 479 solves: it's takes those 3 bullet points regarding the behaviour of StopIteration inside __next__ method implementations and makes them *no longer apply* to writing generator functions. If folks do run into them, they'll get a RuntimError and have the opportunity to debug it, and figure out where that came from. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 12/14/2014 10:02 AM, Nick Coghlan wrote:
I had forgotten this apparently important point. After re-reading the somewhat confusing 'Yield expressions' doc, I came up with this example. class subc: def __iter__(self): return self def __init__(self): self.i, self.it = -1, (0,1,2) def __next__(self): if self.i < 2: self.i += 1 return self.it[self.i] else: raise StopIteration('subc done') def subg(): for i in range(3): yield i return 'sub done' def sup(sub): print((yield from sub())) print(list(sup(subc))) print(list(sup(subg))) it = subg() result = [] try: while True: result.append(next(it)) except StopIteration as e: print(e.args[0]) print(result)
Pep 380 not only adds 'yield from', but also a mechanism for *any* iterator to optionally pass a terminal message back to *any* direct user that looks for the message. I don't know how I might use this feature, which is why I more or less ignored it at the time it was introduced, but it is part of asyncio's use of 'yield from'. For statements ignore the return message, so this is one possible reason to use while and explicit next inside try.
Or the return value can be accessed by catching the resulting StopIteration of next().
-- Terry Jan Reedy
On 14 December 2014 at 15:02, Nick Coghlan <ncoghlan@gmail.com> wrote:
Fair enough. I understand your point that the intention of the PEP is only to solve this for generators. I also understand that these threads are not working towards anything so this is my last post on the subject. I see nothing wrong with the rationale that "it's easier to to address this problem for generators" but that's very different from the "this problem is unique to generators" argument. Since the start of this thread the PEP is now modified so that it no longer explicitly makes the latter argument (although it still does not explicitly state the former).
I'm not sure why you single out 3.3 and 3.4 as it happens without yield from. In 2.7 (and anything since 2.2):
I will leave this subject now. Oscar
On Wed, Dec 10, 2014 at 8:35 AM, Chris Angelico <rosuav@gmail.com> wrote:
agreed that this is moistly semantics, so yes, "generator functions" is probably better. Also, IIUC, there is no such thing (or at least not one thing) as an "iterator" -- rather there are various things that conform to the iterator protocol. The most common of these is a class that implements __iter__ and __next__ methods (that behave the way specified) Should we call that an "iterator class"? Whereas there IS such a thing as a "generator" -- it is the type that is returned by a generator function or generator expression (comprehension?) And generators are one of the types that conform to the iterator protocol So, and here is where I am a bit confused, does this PEP change somethign about the generator type, or something about the syntax of for writing generator functions? (or both?) If it's really changing the behavior of the type, then the PEP should use "generator"
I think this confuses things, too: generator functions and iterators have always been completely different -- again, there is no such thing as an iterator at all, there are only things that conform to the protocol, and as I think Oscar pointed out, the behavior of StopIteration _inside_ such a type is not specified by the protocol. We could define an "iterator" as anything that conforms to the protocol -- probably how it is being used for the most part already. In which case. generators are a type of iterator custom classes with __iter__ and __next__ are another type of iterator some build-in objects are also iterators -- range objects, files, what have you. So how to write the above paragraph? How about something like: """ Under this proposal, generator functions and custom iterator classes would be more distinct than they currently are. They are two ways of writing an iterator, but with not only different syntax but different behavior with respect to the handling of StopIteration within the object itself. But generator functions have always been a compact way to write an iterator that "does some of the book keeping for you" -- with this PEP, they will do a bit more of the book keeping. """ """ Like the mixing of text and bytes in Python 2, the mixing of generators and iterators has resulted in certain perceived conveniences, """ I don't know that generators and iterators have ever been "mixed" in the way that text and bytes were. So I think this analogy just confused things. """ The distinction is simple: A generator function returns generator object. The latter is an iterator, having proper __iter__ and __next__ methods, while the former has neither and does not follow iterator protocol. """ I'm not sure there is confusion about that either -- at least not the confusion that is being addressed here. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
Chris Angelico wrote:
No, that's still too confused -- all of that was true before as well. Generator functions have never been iterators themselves. I think Nick has it nailed -- the difference is that code inside a generator function implementing a __next__ method will behave differently from that inside an ordinary function implementing a __next__ method. -- Greg
On Wed, Dec 10, 2014 at 1:34 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
That's hardly news though. *Except* for the possibility of raising StopIteration to end the iteration, the inside of a generator has always been completely different from the inside of a __next__ method implementation. -- --Guido van Rossum (python.org/~guido)
On 12/10/2014 4:37 PM, Guido van Rossum wrote:
Perhaps some Python equivalent of the C code will make it clearer what will and (mostly) will not change. Correct me if I am wrong, but I believe the function __call__ method must do something more or less like the following. class function: ... def __call__(self, *arg, **kwargs): "Setup execution frame and either run it or pass to generator." frame = execution_frame(self, args, kwargs) # several attributes of self are used to setup the frame if no_yield(self.code): run(frame) # until it stops return top(frame.stack) else: return generator(self.code, frame) The generator must look something like the following. class generator: def __iter__(self): return self def __init__(self, code, frame): self.gi_code = code self.gi_frame = frame self.gi_running = False def __next__(self): "Adapt gen-func yield, return to iterator return, raise S.I." self.gi_running = True run(self.gi_frame) if yielded(): return top(self.gi_frame.stack) else: # returned raise StopIteration() self.gi_running = False ... (close, send, and throw methods, not relevant here) (I probably have some details of gi_running wrong, and ignored how exhausted generator calls are handled.) Ignoring gi_code, which I believe is only there for introspection (such as gi_code.__name__, which I use for error-reporting), the link between the generator function and the generator's pre-written __next__ method is the suspended generator-function frame, stored as gi_frame. This is what make one generator instance different from another. Currently, a StopIteration raised during 'run(self.gi_frame)' is passed on like any other exception, but treated specially by the generator caller. The change is to wrap the frame execution or do the equivalent thereof. try: run(self.gi_frame) except StopIteration() raise RuntimeError All else remains the same. Do I have this right? The rationale for the change is the bulk of the PEP. However, the PEP rationale and its acceptance are based on the existing relationship between the execution frame of the gf body and the generator.__next__ method that translates the 'gf protocol' to the iterator protocol. Guido, Nick, and Greg have all pointed to this, but for me, thinking in terms of Python equivalents makes it clearer. -- Terry Jan Reedy
On Thu, Dec 11, 2014 at 8:49 PM, Terry Reedy <tjreedy@udel.edu> wrote:
That's pretty much how the proof-of-concept patch works, yes. (Though the parentheses after StopIteration seem wrong I think?) In theory, it should chain the exceptions, though that's not always currently working. Improvements to the patch welcomed :) ChrisA
On Wed, Dec 10, 2014 at 4:34 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
I think the problem is with the term "generator function" because what we call "generator function" is neither a generator nor a function. It is a callable that returns a generator. To our excuse, this abuse of language is not uncommon. People often say "integer function" when they mean a function with integer domain and range (or even just range).
On Thu, Dec 11, 2014 at 9:11 AM, Alexander Belopolsky <alexander.belopolsky@gmail.com> wrote:
Interesting. Are you saying the 'def' statement may produce a thing that isn't a function, even though it calls itself a function?
It's a function that returns an iterator, same as this is:
A generator function is a function that returns a generator object. Is this problematic? ChrisA
On Dec 10, 2014, at 14:22, Chris Angelico <rosuav@gmail.com> wrote:
def whatisthis(): return gen() That's not a generator function, but it is a function that returns a generator. (That's a bit of a silly case, but there's plenty of real-life code like this. In fact, IIRC, a couple of the functions in itertools are generator functions in the pure Python implementation but regular functions that return iterators in the C implementation.)
On Wed, Dec 10, 2014 at 05:11:43PM -0500, Alexander Belopolsky wrote:
I think the problem is with the term "generator function" because what we call "generator function" is neither a generator nor a function.
I'm afraid that is just wrong. "Generator functions" are functions. py> def gen(): ... yield 1 ... py> type(gen) <class 'function'> They are distinguishable from other functions by the presence of a flag on the __code__ object. The ``isgeneratorfunction`` function in the inspect module is short enough to reproduce here: def isgeneratorfunction(object): """Return true if the object is a user-defined generator function. Generator function objects provides same attributes as functions. See help(isfunction) for attributes listing.""" return bool((isfunction(object) or ismethod(object)) and object.__code__.co_flags & CO_GENERATOR) Generator functions are essentially syntactic sugar for building iterators from coroutines. The specific type of iterator they return is a generator, which is a built-in type (but not a built-in name). -- Steven
On Wed, Dec 10, 2014, at 07:36, Oscar Benjamin wrote:
The problem isn't what they do when the child iterator raises StopIteration. That's always been well-defined as "treat that particular child iterator as having ended" and behave however the function wants to treat that case (e.g. chain moves on to the next argument). The question is what they should do when *one of their other inputs* (such as a map or filter function) raises StopIteration.
participants (16)
-
Alexander Belopolsky -
Andrew Barnert -
Chris Angelico -
Chris Barker -
Ethan Furman -
Greg Ewing -
Guido van Rossum -
Juancarlo Añez -
Nathaniel Smith -
Nick Coghlan -
Oscar Benjamin -
random832＠fastmail.us -
Ron Adam -
Stephen J. Turnbull -
Steven D'Aprano -
Terry Reedy