From tomirendo at gmail.com Sat Nov 1 15:56:12 2014 From: tomirendo at gmail.com (yotam vaknin) Date: Sat, 1 Nov 2014 16:56:12 +0200 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration Message-ID: Hi, I would like to purpose that generator expressions will not catch StopIteration exception, if this exception did not come from the iterated object's __next__ function specifically. So generator expressions will be able to raise StopIteration by calculating the current value of the Generator. Here is an example of a use-case : def izip(*args): iters = [iter(obj) for obj in args] while True: yield tuple(next(it) for it in iters) a = izip([1,2],[3,4]) print(next(a),next(a),next(a)) #Currently prints : (1, 3) (2, 4) () list(izip([1,2],[3,4])) #Currently never returns Even thought this is the PEP described behaviour, I think this is an unwanted behaviour. I think Generator Expressions should work like List Comprehension in that sense: def iizip(*args): iters = [iter(obj) for obj in args] while True: yield tuple([next(it) for it in iters]) tuple(iizip([1,2],[3,4])) #Returns [(1, 3), (2, 4)] -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sat Nov 1 17:50:05 2014 From: guido at python.org (Guido van Rossum) Date: Sat, 1 Nov 2014 09:50:05 -0700 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: Message-ID: I think you're on to something. But I think both your examples have a problem, even though your second one "works". If we weren't forced by backward compatibility I would have made it much harder for StopIteration to "leak out". Currently a generator can either return or raise StopIteration to signal it is done, but I think it would have been better if StopIteration was treated as some kind of error in this case. Basically I think any time a StopIteration isn't caught by a for-loop or an explicit try/except StopIteraton, I feel there is a bug in the program, or at least it is hard to debug. I'm afraid that ship has sailed, though... On Sat, Nov 1, 2014 at 7:56 AM, yotam vaknin wrote: > Hi, > > I would like to purpose that generator expressions will not catch > StopIteration exception, if this exception did not come from the iterated > object's __next__ function specifically. So generator expressions will be > able to raise StopIteration by calculating the current value of the > Generator. > > Here is an example of a use-case : > > def izip(*args): > iters = [iter(obj) for obj in args] > while True: > yield tuple(next(it) for it in iters) > > a = izip([1,2],[3,4]) > print(next(a),next(a),next(a)) #Currently prints : (1, 3) (2, 4) () > list(izip([1,2],[3,4])) #Currently never returns > > Even thought this is the PEP described behaviour, I think this is an > unwanted behaviour. > > I think Generator Expressions should work like List Comprehension in that > sense: > > def iizip(*args): > iters = [iter(obj) for obj in args] > while True: > yield tuple([next(it) for it in iters]) > > tuple(iizip([1,2],[3,4])) #Returns [(1, 3), (2, 4)] > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sat Nov 1 21:02:19 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 01 Nov 2014 13:02:19 -0700 Subject: [Python-ideas] Deprecate dunder functions from operator module In-Reply-To: <54519687.6030708@stoneleaf.us> References: <20141030005321.GC26866@ando.pearwood.info> <54519687.6030708@stoneleaf.us> Message-ID: <54553C4B.1010404@stoneleaf.us> On 10/29/2014 06:38 PM, Ethan Furman wrote: > On 10/29/2014 05:53 PM, Steven D'Aprano wrote: >> >> I propose a few things: >> >> * institute a policy that, in the event of a new function being added >> to the operator module, only the dunderless version will be added; >> >> * change the documentation to make it clear that the dunderless >> versions should be used, rather than merely being "convenience" >> functions; >> >> * add a prominent note that the dunder versions exist for backwards >> compatibility only and should not be used in new code. > > +1 Actually, make that -1. I'm just crafting some tests to explore how NotImplemented impacts various classes, and the dunder versions make the whole thing much nicer. -- ~Ethan~ From tjreedy at udel.edu Sun Nov 2 00:07:08 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 01 Nov 2014 19:07:08 -0400 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: Message-ID: On 11/1/2014 12:50 PM, Guido van Rossum wrote: > I think you're on to something. But I think both your examples have a > problem, even though your second one "works". Both versions are buggy in that iizip() yields () infinitely, while zip() yields nothing. Fixes below. > If we weren't forced by backward compatibility I would have made it much > harder for StopIteration to "leak out". Currently a generator can either > return or raise StopIteration to signal it is done, but I think it would > have been better if StopIteration was treated as some kind of error in > this case. This would require some sort of additional special casing of StopIteration that we do not have now. Currently, it is limited to 'for' loops expecting and catching StopIteration as a signal to stop iterating. That is rather easy to understand. > Basically I think any time a StopIteration isn't caught by a > for-loop or an explicit try/except StopIteraton, I feel there is a bug > in the program, Outside of generator functions (and expressions), I agree as I cannot think of an exception when it is not. This has come up on Python list. > or at least it is hard to debug. Code within generator functions is different. Writing "raise StopIteration" instead of "return" is mostly a waste of keystrokes. As for next(it), StopIteration should usually propagate, as with an explicit raise and not be caught. The code below that 'works' (when it does work), works because the StopIteration from next(it) (when there is at least one) propagates to the list comp, which lets it pass to the generator, which lets it pass to the generator user. > I'm afraid that ship has sailed, though... > > On Sat, Nov 1, 2014 at 7:56 AM, yotam vaknin > > wrote: > I would like to purpose that generator expressions will not catch > StopIteration exception, if this exception did not come from the > iterated object's __next__ function specifically. For the purpose of your example, all instances of StopIteration are the same and might as well be the same instance. Since to my understanding generators and g.e.s already do not catch the StopIterations you say you want not caught, and since you need for it to not be caught in the code below, I do not understand exactly what you propose. > So generator > expressions will be able to raise StopIteration by calculating the > current value of the Generator. I cannot understand this. > def izip(*args): > iters = [iter(obj) for obj in args] > while True: > yield tuple(next(it) for it in iters) > > a = izip([1,2],[3,4]) > print(next(a),next(a),next(a)) #Currently prints : (1, 3) (2, 4) () > list(izip([1,2],[3,4])) #Currently never returns Better test code that avoid infinite looping: a = izip([1,2],[3,4]) for i in range(3): print(next(a)) One the third loop, the above prints (), while the below prints a traceback. With a = izip(), both print () 3 times. The problem is that when next(it) raises, you want the StopIteration instance propagated (not immediately caught), so that the generator-using code knows that the generator is exhausted. But the tuple call catches it first, so that, in combination with 'while True', the user never sees StopIteration A partial solution is to provoke StopIteration before calling tuple, so that it does propagate. That is what the list comp below does. But if args is empty, so is iters, and there is no next(it) to ever raise. For a complete solution that imitates zip and does not require an otherwise useless temporary list, replace the loop with this: while True: t = tuple(next(it) for it in iters) if not t: return yield t > Even thought this is the PEP described behaviour, I think this is an > unwanted behaviour. Not if you think carefully about what you want to happen when next(it) raises. I think generators and generators expressions should be left alone. > I think Generator Expressions should work like List Comprehension in > that sense: > > def iizip(*args): > iters = [iter(obj) for obj in args] > while True: > yield tuple([next(it) for it in iters]) This could be fixed with 'if not iters: return' as the second line. Replacing [genexp] with list(genexp) does not work because the latter, unlike the former, catches StopIteration. This is proof that the two are not exactly equivalent, and the such behavior difference I know of (excluding introspection, such as with trace). -- Terry Jan Reedy From ncoghlan at gmail.com Sun Nov 2 10:53:06 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 2 Nov 2014 19:53:06 +1000 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: Message-ID: On 2 November 2014 02:50, Guido van Rossum wrote: > I think you're on to something. But I think both your examples have a > problem, even though your second one "works". > > If we weren't forced by backward compatibility I would have made it much > harder for StopIteration to "leak out". Currently a generator can either > return or raise StopIteration to signal it is done, but I think it would > have been better if StopIteration was treated as some kind of error in this > case. Basically I think any time a StopIteration isn't caught by a for-loop > or an explicit try/except StopIteraton, I feel there is a bug in the > program, or at least it is hard to debug. > > I'm afraid that ship has sailed, though... The closest existing example of this kind of generator instance specific StopIteration handling that I can think of is the special case handling of StopIteration in contexlib._GeneratorContextManager.__exit__() (https://hg.python.org/cpython/file/3.4/Lib/contextlib.py#l63). There, the exception handling differentiates between "a specific StopIteration instance that we just threw into the subgenerator" (which it will allow to propagate) and "any other StopIteration instance, which indicates that the wrapped generator iterator terminated as expected" (which it will suppress). We had that wrong initially - if I recall correctly, it was PJE that noticed the problem before 2.5 was released. However, the only reason we were able to make it work is that we knew the exact identity of the exception we were throwing in, rather than just its type - we don't have that luxury in the general case. Getting back to the behaviour that prompted the thread, like a lot of exception handling quirks, it gets back to being very careful about the scope of exception handlers. In this case, the "next(it)" call is inside a generator expression, and hence inside the scope of the expression's StopIteration handling. By contrast, the comprehension version doesn't *have* any implicit exception handling, so the StopIteration escapes to terminate the containing generator. In terms of changing the behaviour of generator expressions to allow other StopIteration instances to propagate, I believe I do see one possible way to do it that limits the degree of backwards incompatibility. Firstly, you'd need to add a general purpose capability to generator iterators: def _set_default_exception(exc): """Supply a specific StopIteration instance to raise when the generator frame returns None""" ... Normally, when the generator execution falls off the end of the frame by returning, the interpreter raises StopIteration if the result is None, or StopIteration(result) if the result is not None. With the new method, you could set a specific instance to be raised when the underlying result of the frame is None. (Side note: "return" and "raise StopIteration" in a generator function aren't *exactly* the same, as only the former relies on the return->raise conversion supplied by the surrounding generator iterator object) That part would be entirely backwards compatible, and would allow you to distinguish whether calling "next", "send" or "throw" on any generator threw StopIteration because the underlying frame returned None (by checking if the StopIteration instance was the one you configured to be raised on a None result), or because it either returned a non-None value or else something running inside that frame threw StopIteration. The backwards incompatible part would be to then also change generator expressions to set a specific StopIteration instance to be raised when the underlying frame returned, and allow all other StopIteration instances to escape, just as contextlib._GeneratorContextManager.__exit__ allows StopIteration instances thrown from the body of the with statement to escape. I think the end result of such a change would definitely be less surprising, as it would make generator expressions behave more like the corresponding comprehensions, and eliminate a hidden infinite loop bug. However, I'm not sure if it's *sufficiently* less surprising to be worth changing - especially since it would mean incurring a small amount of additional runtime overhead for each generator expression. Regards, Nick. P.S. As additional background on the current difference in behaviour between list comprehensions and generator expressions, that has its roots in the same idiosyncrasy where putting a yield expression inside a comprehension actually *turns it into a generator expression*. Comprehensions are full closures, but they don't contain a yield expression, so you get a normal function, which the interpreter then calls. The interpreter doesn't actually do anything particularly special to make a generator expression instead - it just implicitly inserts a yield expression into the closure, which then automatically makes it a generator function instead of a normal one. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sun Nov 2 11:15:06 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 2 Nov 2014 20:15:06 +1000 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: Message-ID: On 2 November 2014 19:53, Nick Coghlan wrote: > On 2 November 2014 02:50, Guido van Rossum wrote: >> I think you're on to something. But I think both your examples have a >> problem, even though your second one "works". >> >> If we weren't forced by backward compatibility I would have made it much >> harder for StopIteration to "leak out". Currently a generator can either >> return or raise StopIteration to signal it is done, but I think it would >> have been better if StopIteration was treated as some kind of error in this >> case. Basically I think any time a StopIteration isn't caught by a for-loop >> or an explicit try/except StopIteraton, I feel there is a bug in the >> program, or at least it is hard to debug. >> >> I'm afraid that ship has sailed, though... > > The closest existing example of this kind of generator instance > specific StopIteration handling that I can think of is the special > case handling of StopIteration in > contexlib._GeneratorContextManager.__exit__() > (https://hg.python.org/cpython/file/3.4/Lib/contextlib.py#l63). There, > the exception handling differentiates between "a specific > StopIteration instance that we just threw into the subgenerator" > (which it will allow to propagate) and "any other StopIteration > instance, which indicates that the wrapped generator iterator > terminated as expected" (which it will suppress). We had that wrong > initially - if I recall correctly, it was PJE that noticed the problem > before 2.5 was released. However, the only reason we were able to make > it work is that we knew the exact identity of the exception we were > throwing in, rather than just its type - we don't have that luxury in > the general case. > > Getting back to the behaviour that prompted the thread, like a lot of > exception handling quirks, it gets back to being very careful about > the scope of exception handlers. In this case, the "next(it)" call is > inside a generator expression, and hence inside the scope of the > expression's StopIteration handling. By contrast, the comprehension > version doesn't *have* any implicit exception handling, so the > StopIteration escapes to terminate the containing generator. Bah, I should have fully read Terry's reply before responding. He's right, it's the tuple call that's suppressing the exception, not the generator expression itself. That changes the possible solution, by tweaking it to be an optional extension to the iterator protocol, allowing iterators to make the terminating exception configurable. def __setiterexc__(exc): """Specify the exception instance to raise when the iterator is exhausted""" ... Iterator consumers (like tuple) could then check for that method and use it to set a specific StopIteration instance, allowing all others to escape. I believe actually doing this would be adding too much complexity for too little gain, but it *is* possible. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From abarnert at yahoo.com Sun Nov 2 20:50:01 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 2 Nov 2014 11:50:01 -0800 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: Message-ID: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> This is related to the fact that, although the docs imply otherwise, [COMP] isn't exactly equivalent to list(COMP), because of cases like: def ensure_positive(x): if x<=0: raise StopIteration return True eggs = list(x for x in spam if raise_on_negative(x)) def ensure_positive(x); if x<=0: raise StopIteration return x eggs = list(ensure_positive(x) for x in spam) In both cases, this acts like a "takewhile": eggs ends up as a list of the initial positive values, and the first non-positive value is consumed and discarded. But if you do the same thing with a list comprehension, the comprehension is aborted by the StopIteration, and eggs never gets set (although the same values are consumed from spam, of course). IIRC, you asked me what the performance costs would be of changing listcomps to match, and for a trivial comp it worked out to be about 40% for the naive solution (build a genexpr, call it, call list) and about 20% with specially-optimized bytecode. So everyone agreed that even if this is a bug, that would be too much of a cost for too small of a fix. Of course here the issue is almost the opposite. But they're clearly related; how different comprehensions "leak" exceptions differs. Sent from a random iPhone On Nov 2, 2014, at 2:15, Nick Coghlan wrote: > On 2 November 2014 19:53, Nick Coghlan wrote: >> On 2 November 2014 02:50, Guido van Rossum wrote: >>> I think you're on to something. But I think both your examples have a >>> problem, even though your second one "works". >>> >>> If we weren't forced by backward compatibility I would have made it much >>> harder for StopIteration to "leak out". Currently a generator can either >>> return or raise StopIteration to signal it is done, but I think it would >>> have been better if StopIteration was treated as some kind of error in this >>> case. Basically I think any time a StopIteration isn't caught by a for-loop >>> or an explicit try/except StopIteraton, I feel there is a bug in the >>> program, or at least it is hard to debug. >>> >>> I'm afraid that ship has sailed, though... >> >> The closest existing example of this kind of generator instance >> specific StopIteration handling that I can think of is the special >> case handling of StopIteration in >> contexlib._GeneratorContextManager.__exit__() >> (https://hg.python.org/cpython/file/3.4/Lib/contextlib.py#l63). There, >> the exception handling differentiates between "a specific >> StopIteration instance that we just threw into the subgenerator" >> (which it will allow to propagate) and "any other StopIteration >> instance, which indicates that the wrapped generator iterator >> terminated as expected" (which it will suppress). We had that wrong >> initially - if I recall correctly, it was PJE that noticed the problem >> before 2.5 was released. However, the only reason we were able to make >> it work is that we knew the exact identity of the exception we were >> throwing in, rather than just its type - we don't have that luxury in >> the general case. >> >> Getting back to the behaviour that prompted the thread, like a lot of >> exception handling quirks, it gets back to being very careful about >> the scope of exception handlers. In this case, the "next(it)" call is >> inside a generator expression, and hence inside the scope of the >> expression's StopIteration handling. By contrast, the comprehension >> version doesn't *have* any implicit exception handling, so the >> StopIteration escapes to terminate the containing generator. > > Bah, I should have fully read Terry's reply before responding. He's > right, it's the tuple call that's suppressing the exception, not the > generator expression itself. > > That changes the possible solution, by tweaking it to be an optional > extension to the iterator protocol, allowing iterators to make the > terminating exception configurable. > > def __setiterexc__(exc): > """Specify the exception instance to raise when the iterator > is exhausted""" > ... > > Iterator consumers (like tuple) could then check for that method and > use it to set a specific StopIteration instance, allowing all others > to escape. > > I believe actually doing this would be adding too much complexity for > too little gain, but it *is* possible. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From tjreedy at udel.edu Sun Nov 2 22:00:46 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 02 Nov 2014 16:00:46 -0500 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> Message-ID: On 11/2/2014 2:50 PM, Andrew Barnert wrote: > This is related to the fact that, although the docs imply otherwise, > [COMP] isn't exactly equivalent to list(COMP), That purported equivalence is a common meme, which I may have helped spread. If it is implied in the doc, it should be changed. Before I answered on this thread yesterday, I looked for such an implication in the Language Reference (though not the Tutorial), and only found this carefully written description in the expressions chapter. "In this case, the elements of the new container are those that would be produced by considering each of the for or if clauses a block, nesting from left to right, and evaluating the expression to produce an element each time the innermost block is reached. Note that the comprehension is executed in a separate scope," IE, a comprehension is equivalent to the result of for and if statements in a separate Python function that first initializes a collection object (list, set, or dict) and augments the collection in the innermost scope with the element produced (which is a key,value pair for dicts). Such an equivalent function would not catch any exception raised by the innermost element expression. On the other hand, collection initializers, such as list or tuple, specifically catch StopIteration when fed an iterable. Hence the two cannot be equivalent when the element expression raises StopIteration. -- Terry Jan Reedy From abarnert at yahoo.com Sun Nov 2 23:37:57 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 2 Nov 2014 14:37:57 -0800 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> Message-ID: <3196CD31-F077-4A0F-990D-606F0592EB9A@yahoo.com> On Nov 2, 2014, at 13:00, Terry Reedy wrote: > On 11/2/2014 2:50 PM, Andrew Barnert wrote: >> This is related to the fact that, although the docs imply otherwise, >> [COMP] isn't exactly equivalent to list(COMP), > > That purported equivalence is a common meme, which I may have helped spread. If it is implied in the doc, it should be changed. Before I answered on this thread yesterday, I looked for such an implication in the Language Reference (though not the Tutorial), and only found this carefully written description in the expressions chapter. We looked through this last year and decided nothing needed to be changed in the docs, so I doubt it's worth repeating that effort. IIRC, the tutorial may have been confusing in the past but wasn't as of 3.3, and the only place that might confuse anyone was the what's new in 3.0, which is generally only changed to add notes about things which were un-/re-changed (like u"" strings). But again, even if I'm remembering wrong, I don't think it matters. All that being said, I don't really love the reference docs here. Neither 6.2.8 not anything else explicitly says what the semantics are. It's a pretty obvious guess that the syntax is interpreted the same as for comprehensions (as in 6.2.4), and that the values yielded are those that would be used to produce elements. But the docs don't actually say that. > "In this case, the elements of the new container are those that would be produced by considering each of the for or if clauses a block, nesting from left to right, and evaluating the expression to produce an element each time the innermost block is reached. Note that the comprehension is executed in a separate scope," > > IE, a comprehension is equivalent to the result of for and if statements in a separate Python function that first initializes a collection object (list, set, or dict) and augments the collection in the innermost scope with the element produced (which is a key,value pair for dicts). > > Such an equivalent function would not catch any exception raised by the innermost element expression. On the other hand, collection initializers, such as list or tuple, specifically catch StopIteration when fed an iterable. Hence the two cannot be equivalent when the element expression raises StopIteration. > > -- > Terry Jan Reedy > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From guido at python.org Mon Nov 3 04:01:05 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 2 Nov 2014 19:01:05 -0800 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> Message-ID: On Sun, Nov 2, 2014 at 1:00 PM, Terry Reedy wrote: > On 11/2/2014 2:50 PM, Andrew Barnert wrote: > >> This is related to the fact that, although the docs imply otherwise, >> [COMP] isn't exactly equivalent to list(COMP), >> > > That purported equivalence is a common meme, which I may have helped > spread. I may have started it. I was aware of the non-equivalence (only mostly-equivalence) in Python 2 and I wanted to make then identical in Python 3 -- having one construct being exactly equivalent to another reduce the amount of explaining needed. Unfortunately, people had started to depend on the (in my *current* opinion deplorable) behavior of generator expressions in the face of StopIteration thrown by arbitrary parts of the expression or condition, and the equivalence is still imperfect. At least the variable leakage has been fixed. I know that when we first introduced generators (not generator expressions) I was in favor of interpreting arbitrary things that raise StopIteration in a generator to cause the generator to terminate just as if it had decided to stop (i.e. 'return' or falling of the end), because I thought there were some useful patterns that could be written more compactly this way -- in particular, the pattern where a generator iterates over another iterator by calling next() on it, does some processing on the value thus produced, and then yielding the processed value (or not), and where the logical response to a StopIteration from the inner iterator is to exit the generator. For example: def only_positive(it): while True: x = next(it) if x > 0: yield x This *particular* example is much better written as: def only_positive(x): for x in it: if x > 0: yield x but the idea was that there might be variants where being constrained by a single for-loop would make the code less elegant if you had to catch the StopIteration and explicitly exit the generator. However, I don't think this idea has panned out. I haven't done a survey, but I have a feeling that in most cases where an explicit next() call is used (as opposed to a for-loop) there's a try/except Stopiteration around it, and a fair amount if time is wasted debugging situations where a StopIteration unexpectedly escapes and silently interrupts some loop over an unrelated generator (instead of loudly bubbling up to the top and causing a traceback, which would be more debuggable). And the use case of raising StopIteration from a condition used in a generator expression is iffy at best (it makes the condition function hard to use in other contexts, and it calls to attention the difference between generators and comprehensions). So I will go out on a limb here and say that this was a mistake and if we can think of easing the transitional pain it would be a good thing to fix this eventually. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Nov 3 05:02:25 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 3 Nov 2014 14:02:25 +1000 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> Message-ID: On 3 November 2014 13:01, Guido van Rossum wrote: > On Sun, Nov 2, 2014 at 1:00 PM, Terry Reedy wrote: >> >> On 11/2/2014 2:50 PM, Andrew Barnert wrote: >>> >>> This is related to the fact that, although the docs imply otherwise, >>> [COMP] isn't exactly equivalent to list(COMP), >> >> >> That purported equivalence is a common meme, which I may have helped >> spread. > > > I may have started it. I was aware of the non-equivalence (only > mostly-equivalence) in Python 2 and I wanted to make then identical in > Python 3 -- having one construct being exactly equivalent to another reduce > the amount of explaining needed. Unfortunately, people had started to depend > on the (in my *current* opinion deplorable) behavior of generator > expressions in the face of StopIteration thrown by arbitrary parts of the > expression or condition, and the equivalence is still imperfect. At least > the variable leakage has been fixed. I think I'm guilty as well - when I was working on the Python 3 changes, getting the *scoping* behaviour to be identical between comprehensions and generator expressions was one of the key objectives, so I regularly described it as making "[x for x in seq]" equivalent to "list(x for x in seq)". I unfortunately didn't notice the remaining exception handling differences at the time, or we might have been able to do something about it for 3.0 :( > However, I don't think this idea has panned out. I haven't done a survey, > but I have a feeling that in most cases where an explicit next() call is > used (as opposed to a for-loop) there's a try/except Stopiteration around > it, and a fair amount if time is wasted debugging situations where a > StopIteration unexpectedly escapes and silently interrupts some loop over an > unrelated generator (instead of loudly bubbling up to the top and causing a > traceback, which would be more debuggable). And the use case of raising > StopIteration from a condition used in a generator expression is iffy at > best (it makes the condition function hard to use in other contexts, and it > calls to attention the difference between generators and comprehensions). > > So I will go out on a limb here and say that this was a mistake and if we > can think of easing the transitional pain it would be a good thing to fix > this eventually. Having had to do the dance to work around the current behaviour in contextlib, I'm inclined to agree - there's a significant semantic difference between the "this iterable just terminated" StopIteration, and the "something threw StopIteration and nobody caught it", and the current model hides it. However, I also see potentially significant backwards compatibility problems when it comes to helper functions that throw StopIteration to terminate the calling generator - there would likely need to be some kind of thread local state and a helper "iterexit()" builtin and "PyIter_Exit()" C API to call instead of raising StopIteration directly. Making such a change would involve a lot of code churn just to phase out a relatively obscure issue that mostly just makes certain bugs harder to diagnose (as was the case with the generator expression based izip implementation in this thread), rather than directly causing bugs in its own right. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From guido at python.org Mon Nov 3 06:59:18 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 2 Nov 2014 21:59:18 -0800 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> Message-ID: On Sun, Nov 2, 2014 at 8:02 PM, Nick Coghlan wrote: > On 3 November 2014 13:01, Guido van Rossum wrote: > > On Sun, Nov 2, 2014 at 1:00 PM, Terry Reedy wrote: > >> > >> On 11/2/2014 2:50 PM, Andrew Barnert wrote: > >>> > >>> This is related to the fact that, although the docs imply otherwise, > >>> [COMP] isn't exactly equivalent to list(COMP), > >> > >> > >> That purported equivalence is a common meme, which I may have helped > >> spread. > > > > > > I may have started it. I was aware of the non-equivalence (only > > mostly-equivalence) in Python 2 and I wanted to make then identical in > > Python 3 -- having one construct being exactly equivalent to another > reduce > > the amount of explaining needed. Unfortunately, people had started to > depend > > on the (in my *current* opinion deplorable) behavior of generator > > expressions in the face of StopIteration thrown by arbitrary parts of the > > expression or condition, and the equivalence is still imperfect. At least > > the variable leakage has been fixed. > > I think I'm guilty as well - when I was working on the Python 3 > changes, getting the *scoping* behaviour to be identical between > comprehensions and generator expressions was one of the key > objectives, so I regularly described it as making "[x for x in seq]" > equivalent to "list(x for x in seq)". > > I unfortunately didn't notice the remaining exception handling > differences at the time, or we might have been able to do something > about it for 3.0 :( > > > However, I don't think this idea has panned out. I haven't done a survey, > > but I have a feeling that in most cases where an explicit next() call is > > used (as opposed to a for-loop) there's a try/except Stopiteration around > > it, and a fair amount if time is wasted debugging situations where a > > StopIteration unexpectedly escapes and silently interrupts some loop > over an > > unrelated generator (instead of loudly bubbling up to the top and > causing a > > traceback, which would be more debuggable). And the use case of raising > > StopIteration from a condition used in a generator expression is iffy at > > best (it makes the condition function hard to use in other contexts, and > it > > calls to attention the difference between generators and comprehensions). > > > > So I will go out on a limb here and say that this was a mistake and if we > > can think of easing the transitional pain it would be a good thing to fix > > this eventually. > > Having had to do the dance to work around the current behaviour in > contextlib, I'm inclined to agree - there's a significant semantic > difference between the "this iterable just terminated" StopIteration, > and the "something threw StopIteration and nobody caught it", and the > current model hides it. > > However, I also see potentially significant backwards compatibility > problems when it comes to helper functions that throw StopIteration to > terminate the calling generator - there would likely need to be some > kind of thread local state and a helper "iterexit()" builtin and > "PyIter_Exit()" C API to call instead of raising StopIteration > directly. > That's what I was afraid of. Can you point me to an example of code that depends on this that isn't trivial like Andrew Barnert's ensure_positive() example? I think that particular example, and the category it represents, are excessive cleverness that abuse the feature under discussion -- but you sound like you have helpers for context managers that couldn't be easily dismissed like that. > Making such a change would involve a lot of code churn just to phase > out a relatively obscure issue that mostly just makes certain bugs > harder to diagnose (as was the case with the generator expression > based izip implementation in this thread), rather than directly > causing bugs in its own right. > Maybe. But the real-life version of that bug can be *really* hard to find, and that's usually the kind of thing we'd like to fix. FWIW the implementation of my proposal is easy to describe (which the Zen approves of): when a StopIteration leaves a frame, replace it with some other exception (either a new, custom one or maybe just RuntimeError), chaining the original StopIteration. It's the consequences that are hard to survey and describe in this case (as they affect subtle code depending on the current semantics). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Nov 3 08:24:01 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 3 Nov 2014 17:24:01 +1000 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> Message-ID: On 3 November 2014 15:59, Guido van Rossum wrote: > On Sun, Nov 2, 2014 at 8:02 PM, Nick Coghlan wrote: >> However, I also see potentially significant backwards compatibility >> problems when it comes to helper functions that throw StopIteration to >> terminate the calling generator - there would likely need to be some >> kind of thread local state and a helper "iterexit()" builtin and >> "PyIter_Exit()" C API to call instead of raising StopIteration >> directly. > > > That's what I was afraid of. Can you point me to an example of code that > depends on this that isn't trivial like Andrew Barnert's ensure_positive() > example? I think that particular example, and the category it represents, > are excessive cleverness that abuse the feature under discussion -- but you > sound like you have helpers for context managers that couldn't be easily > dismissed like that. Sorry, I didn't mean to give that impression. I'm in a similar situation to you in that regard - any specific examples I can think of trip my "that's too obscure to be maintainable" alarm (if there is a reasonable use case, I'd expect it to involve full generators, rather than generator expressions). The code in contextlib relies on the way *generators* handle StopIteration, and if understand your proposal correctly, that would remain unchanged - only ordinary function calls would convert StopIteration to a different exception type, preserving the functional equivalence of a generator returning, raising StopIteration, or having StopIteration thrown into a yield point (it's that last one that contextlib relies on). >> Making such a change would involve a lot of code churn just to phase >> out a relatively obscure issue that mostly just makes certain bugs >> harder to diagnose (as was the case with the generator expression >> based izip implementation in this thread), rather than directly >> causing bugs in its own right. > > > Maybe. But the real-life version of that bug can be *really* hard to find, > and that's usually the kind of thing we'd like to fix. > > FWIW the implementation of my proposal is easy to describe (which the Zen > approves of): when a StopIteration leaves a frame, replace it with some > other exception (either a new, custom one or maybe just RuntimeError), > chaining the original StopIteration. That's far more elegant than the stateful possibilities I was considering. So generators would continue to leave StopIteration untouched (preserving the equivalence between returning from the frame and explicitly raising StopIteration from the generator body), and only ordinary function invocations would gain the StopIteration -> UnexpectedStopIteration translation (assuming we went with a new exception type)? > It's the consequences that are hard to survey and describe in this case (as > they affect subtle code depending on the current semantics). Aye. I'm reasonably OK with the notion of breaking overly clever (and hence hard to follow) generator expressions, but I'm a little more nervous about any change that means that factoring out "raise StopIteration" in a full generator function would stop working. That said, such a change would bring generator functions fully into line with the "flow control should be locally visible" principle that guided both the with statement design and asyncio - only a local return or raise statement could gracefully terminate the generator, StopIteration from a nested function call would always escape as the new exception type. If you wanted to factor out a helper function that terminated the generator you'd have to do "return yield from helper()" rather than just "helper()". Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From 4kir4.1i at gmail.com Mon Nov 3 11:41:30 2014 From: 4kir4.1i at gmail.com (Akira Li) Date: Mon, 03 Nov 2014 13:41:30 +0300 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> Message-ID: <87mw88wopx.fsf@gmail.com> Guido van Rossum writes: > On Sun, Nov 2, 2014 at 8:02 PM, Nick Coghlan wrote: > >> However, I also see potentially significant backwards compatibility >> problems when it comes to helper functions that throw StopIteration to >> terminate the calling generator - there would likely need to be some >> kind of thread local state and a helper "iterexit()" builtin and >> "PyIter_Exit()" C API to call instead of raising StopIteration >> directly. >> > > That's what I was afraid of. Can you point me to an example of code that > depends on this that isn't trivial like Andrew Barnert's ensure_positive() > example? I think that particular example, and the category it represents, > are excessive cleverness that abuse the feature under discussion -- but you > sound like you have helpers for context managers that couldn't be easily > dismissed like that. > > The pure Python implementation of itertools.groupby() provided in its docs [1] uses next() as the helper function that terminates the calling generator by raising StopIteration [1]: https://docs.python.org/3/library/itertools.html#itertools.groupby Here's a simplified example: from functools import partial def groupby(iterable): """ >>> ' '.join(k for k, g in groupby('AAAABBBCCDAABBB')) 'A B C D A B' >>> ' '.join(''.join(g) for k, g in groupby('AAAABBBCCDAABBB')) 'AAAA BBB CC D AA BBB' """ next_value = partial(next, iter(iterable)) def yield_same(group_value_): # generate group values nonlocal value while value == group_value_: yield value value = next_value() # exit on StopIteration group_value = value = object() while True: while value == group_value: # discard unconsumed values value = next_value() # exit on StopIteration group_value = value yield group_value, yield_same(group_value) The alternative is to return a sentinel from next(): def groupby_done(iterable): done = object() # sentinel next_value = partial(next, iter(iterable), done) def yield_same(group_value_): # generate group values nonlocal value while value == group_value_: yield value value = next_value() if value is done: return group_value = value = object() while value is not done: while value == group_value: # discard unconsumed values value = next_value() if value is done: return group_value = value yield group_value, yield_same(group_value) The first code example exploits the fact that `next(it)` continues to raise StopIteration on subsequent calls. The second code example has to propagate up the stack the termination condition manually (`while True` is replace with `while value is not done`). -- Akira From abarnert at yahoo.com Mon Nov 3 20:58:28 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 3 Nov 2014 11:58:28 -0800 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> Message-ID: On Nov 2, 2014, at 21:59, Guido van Rossum wrote: > On Sun, Nov 2, 2014 at 8:02 PM, Nick Coghlan wrote: >> On 3 November 2014 13:01, Guido van Rossum wrote: >> > On Sun, Nov 2, 2014 at 1:00 PM, Terry Reedy wrote: >> >> >> >> On 11/2/2014 2:50 PM, Andrew Barnert wrote: >> >>> >> >>> This is related to the fact that, although the docs imply otherwise, >> >>> [COMP] isn't exactly equivalent to list(COMP), >> >> >> >> >> >> That purported equivalence is a common meme, which I may have helped >> >> spread. >> > >> > >> > I may have started it. I was aware of the non-equivalence (only >> > mostly-equivalence) in Python 2 and I wanted to make then identical in >> > Python 3 -- having one construct being exactly equivalent to another reduce >> > the amount of explaining needed. Unfortunately, people had started to depend >> > on the (in my *current* opinion deplorable) behavior of generator >> > expressions in the face of StopIteration thrown by arbitrary parts of the >> > expression or condition, and the equivalence is still imperfect. At least >> > the variable leakage has been fixed. >> >> I think I'm guilty as well - when I was working on the Python 3 >> changes, getting the *scoping* behaviour to be identical between >> comprehensions and generator expressions was one of the key >> objectives, so I regularly described it as making "[x for x in seq]" >> equivalent to "list(x for x in seq)". >> >> I unfortunately didn't notice the remaining exception handling >> differences at the time, or we might have been able to do something >> about it for 3.0 :( >> >> > However, I don't think this idea has panned out. I haven't done a survey, >> > but I have a feeling that in most cases where an explicit next() call is >> > used (as opposed to a for-loop) there's a try/except Stopiteration around >> > it, and a fair amount if time is wasted debugging situations where a >> > StopIteration unexpectedly escapes and silently interrupts some loop over an >> > unrelated generator (instead of loudly bubbling up to the top and causing a >> > traceback, which would be more debuggable). And the use case of raising >> > StopIteration from a condition used in a generator expression is iffy at >> > best (it makes the condition function hard to use in other contexts, and it >> > calls to attention the difference between generators and comprehensions). >> > >> > So I will go out on a limb here and say that this was a mistake and if we >> > can think of easing the transitional pain it would be a good thing to fix >> > this eventually. >> >> Having had to do the dance to work around the current behaviour in >> contextlib, I'm inclined to agree - there's a significant semantic >> difference between the "this iterable just terminated" StopIteration, >> and the "something threw StopIteration and nobody caught it", and the >> current model hides it. >> >> However, I also see potentially significant backwards compatibility >> problems when it comes to helper functions that throw StopIteration to >> terminate the calling generator - there would likely need to be some >> kind of thread local state and a helper "iterexit()" builtin and >> "PyIter_Exit()" C API to call instead of raising StopIteration >> directly. > > That's what I was afraid of. Can you point me to an example of code that depends on this that isn't trivial like Andrew Barnert's ensure_positive() example? I think that particular example, and the category it represents, are excessive cleverness that abuse the feature under discussion The category is usually represented by the even more trivial and even more abusive example: (prime if prime<100 else throw(StopIteration) for prime in primes) I do see these, mostly on places like StackOverflow, where someone was shown this "cool trick" by someone else, used it without understanding it, and now has no idea how to debug his code. (However, a few people on this list suggested it as an alternative to adding some kind of "syntactic takewhile" to the language, so it's possible not everyone sees it as abuse, even though I think you and others called it abuse back then as well.) Anyway, I agree that explicitly disallowing it would make the language simpler, eliminate more bugs than useful idioms, and possibly open the door to other improvements. But if you can't justify this abuse as being actually illegal by a reading of the docs in 3.0-3.4, and people are potentially using it in real code, wouldn't that require a period of deprecation before it can be broken? > -- but you sound like you have helpers for context managers that couldn't be easily dismissed like that. > >> Making such a change would involve a lot of code churn just to phase >> out a relatively obscure issue that mostly just makes certain bugs >> harder to diagnose (as was the case with the generator expression >> based izip implementation in this thread), rather than directly >> causing bugs in its own right. > > Maybe. But the real-life version of that bug can be *really* hard to find, and that's usually the kind of thing we'd like to fix. > > FWIW the implementation of my proposal is easy to describe (which the Zen approves of): when a StopIteration leaves a frame, replace it with some other exception (either a new, custom one or maybe just RuntimeError), chaining the original StopIteration. > > It's the consequences that are hard to survey and describe in this case (as they affect subtle code depending on the current semantics). > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From antony.lee at berkeley.edu Mon Nov 3 22:38:43 2014 From: antony.lee at berkeley.edu (Antony Lee) Date: Mon, 3 Nov 2014 13:38:43 -0800 Subject: [Python-ideas] "or raise" syntax In-Reply-To: References: <5452534D.5060002@gmail.com> <545255A6.6000600@stoneleaf.us> Message-ID: If anything, a better idea would be to make "raise [... [from ...]]" an expression (possibly requiring parentheses, ? la yield), thus allowing it to be put in constructs such as the one suggested, but also in lambda expressions. Of course, the value of the expression doesn't have to be specified (the assignment target stays unbound, causing a NameError if we somehow catch the exception and later refer to the assignment target). 2014-10-30 11:45 GMT-07:00 Terry Reedy : > On 10/30/2014 11:13 AM, Ethan Furman wrote: > >> On 10/30/2014 08:03 AM, Javier Dehesa wrote: >> >>> >>> This happens to me with some frequency: >>> >>> result = f(args) >>> if not result: # of "if result is None:" >>> raise Exception(...) >>> >>> What if I could just say? >>> >>> result = f(args) or raise Exception(...) >>> >> >> Seems like a decent idea, but you can already have most of that: >> >> result = f(args) or raise_exc(ValueError, 'args must be ...') >> >> and then have 'raise_exc' do the exception raising work. >> > > No need to pass exception class and message separately instead of an > exception instance. > > def raiser(exc): > raise exc > > print(1 or raiser(ValueError('null value'))) > print(0 or raiser(ValueError('null value'))) > > >>> > 1 > Traceback (most recent call last): > File "c:\programs\python34\tem.py", line 5, in > print(0 or raiser(ValueError('null value'))) > File "c:\programs\python34\tem.py", line 2, in raiser > raise exc > ValueError: null value > > It does add another line to the trackback, but this is pretty minor. > > -- > Terry Jan Reedy > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Mon Nov 3 22:38:49 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 04 Nov 2014 10:38:49 +1300 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> Message-ID: <5457F5E9.8060205@canterbury.ac.nz> Guido van Rossum wrote: > FWIW the implementation of my proposal is easy to describe (which the > Zen approves of): when a StopIteration leaves a frame, replace it with > some other exception (either a new, custom one or maybe just > RuntimeError), chaining the original StopIteration. Wouldn't that break all existing iterators whose next() method is implemented in Python? Since the code that catches the StopIteration is always in a different frame then. Or are you only talking about generator frames here, not frames in general? -- Greg From guido at python.org Tue Nov 4 01:29:31 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 3 Nov 2014 16:29:31 -0800 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: <5457F5E9.8060205@canterbury.ac.nz> References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <5457F5E9.8060205@canterbury.ac.nz> Message-ID: On Mon, Nov 3, 2014 at 1:38 PM, Greg Ewing wrote: > Guido van Rossum wrote: > > FWIW the implementation of my proposal is easy to describe (which the Zen >> approves of): when a StopIteration leaves a frame, replace it with some >> other exception (either a new, custom one or maybe just RuntimeError), >> chaining the original StopIteration. >> > > Wouldn't that break all existing iterators whose next() method > is implemented in Python? Since the code that catches the > StopIteration is always in a different frame then. > > Or are you only talking about generator frames here, not > frames in general? > I have to apologize, I pretty much had this backwards. What I should have said is that a generator should always be terminated by a return or falling off the end, and if StopIteration is raised in the generator (either by an explicit raise or raised by something it calls) and not caught by an except clause in the same generator, it should be turned into something else, so it bubbles out as something that keeps getting bubbled up rather than silently ending a for-loop. OTOH StopIteration raised by a non-generator function should not be mutated. I'm sorry if this invalidates Nick's endorsement of the proposal. I definitely see this as a serious backward incompatibility: no matter how often it leads to buggy or obfuscated code, it's how things have always worked. Regarding Akira Li's examples of groupby(), unfortunately I find both versions inscrutable -- on a casual inspection I have no idea what goes on. I would have to think about how I would write groupby() myself (but I'm pretty sure I wouldn't us functools.partial(). :-) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron3200 at gmail.com Tue Nov 4 04:17:44 2014 From: ron3200 at gmail.com (Ron Adam) Date: Mon, 03 Nov 2014 21:17:44 -0600 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <5457F5E9.8060205@canterbury.ac.nz> Message-ID: On 11/03/2014 06:29 PM, Guido van Rossum wrote: > > Regarding Akira Li's examples of groupby(), unfortunately I find both > versions inscrutable -- on a casual inspection I have no idea what goes on. > I would have to think about how I would write groupby() myself (but I'm > pretty sure I wouldn't us functools.partial(). :-) I agree, and just happened to try. After several tries, it's tricky, I come to the conclusion it fits the pattern of peeking at the first value of the iterater best. It seems to be a fairly common pattern and hard to get right with iterators. For what it's worth here is what I came up with. :-) class PeekIter: """ Create an iterator which allows you to peek() at the next item to be yielded. It also stores the exception and re-raises it on the next next() call. """ def __init__(self, iterable): self._exn = None self._it = iter(iterable) self._peek = next(iter(iterable)) def peek(self): return self._peek def ok(self): #True if no exception occured when setting self._peek return self._exn == None def __iter__(self): return iter(self) def __next__(self): if self._exn != None: raise self._exn t = self._peek try: self._peek = next(self._it) except Exception as exn: self._exn = exn self._peek = None return t def group_by(iterable): """ Yield (key, (group))'s of like values from an iterator. """ itr = PeekIter(iterable) def group(it): # Yield only like values. k = it.peek() while it.peek() == k: yield next(it) while itr.ok(): yield itr.peek(), [x for x in group(itr)] print(' '.join(''.join(k) for k, v in group_by('AAAABBBCCDAABBB'))) # 'A B C D A B' print(' '.join(''.join(g) for k, g in group_by('AAAABBBCCDAABBB'))) # 'AAAA BBB CC D AA BBB' From ron3200 at gmail.com Tue Nov 4 04:50:38 2014 From: ron3200 at gmail.com (Ron Adam) Date: Mon, 03 Nov 2014 21:50:38 -0600 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <5457F5E9.8060205@canterbury.ac.nz> Message-ID: Just a very minor correction, list(group(itr) instead of list comp. ... On 11/03/2014 09:17 PM, Ron Adam wrote: > def group_by(iterable): > """ Yield (key, (group))'s of like values > from an iterator. > """ > itr = PeekIter(iterable) > > def group(it): > # Yield only like values. > k = it.peek() > while it.peek() == k: > yield next(it) > > while itr.ok(): > yield itr.peek(), [x for x in group(itr)] def group_by(iterable): """ Yield (key, (group))'s of like values from an iterator. """ def group(it): # Yield only like values. k = it.peek() while it.peek() == k: yield next(it) itr = PeekIter(iterable) while itr.ok(): yield itr.peek(), list(group(itr)) Cheers, Ron From steve at pearwood.info Tue Nov 4 04:57:04 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 4 Nov 2014 14:57:04 +1100 Subject: [Python-ideas] Deprecate dunder functions from operator module In-Reply-To: <54553C4B.1010404@stoneleaf.us> References: <20141030005321.GC26866@ando.pearwood.info> <54519687.6030708@stoneleaf.us> <54553C4B.1010404@stoneleaf.us> Message-ID: <20141104035704.GD17635@ando.pearwood.info> On Sat, Nov 01, 2014 at 01:02:19PM -0700, Ethan Furman wrote: > On 10/29/2014 06:38 PM, Ethan Furman wrote: > >On 10/29/2014 05:53 PM, Steven D'Aprano wrote: > >> > >>I propose a few things: > >> > >>* institute a policy that, in the event of a new function being added > >> to the operator module, only the dunderless version will be added; > >> > >>* change the documentation to make it clear that the dunderless > >> versions should be used, rather than merely being "convenience" > >> functions; > >> > >>* add a prominent note that the dunder versions exist for backwards > >> compatibility only and should not be used in new code. > > > >+1 > > Actually, make that -1. > > I'm just crafting some tests to explore how NotImplemented impacts various > classes, and the dunder versions make the whole thing much nicer. I'm surprised. I can't imagine dunder names making anything look nicer. It sounds like you are writing "operator.__add__" instead of "operator.add", which doesn't look nicer to me. Can you show me your code? Note that the dunder versions won't be going away soon, if at all, so if your code is relying on them, they will still be there. They just won't be prominently advertised. -- Steven From ethan at stoneleaf.us Tue Nov 4 05:04:10 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 03 Nov 2014 20:04:10 -0800 Subject: [Python-ideas] Deprecate dunder functions from operator module In-Reply-To: <20141104035704.GD17635@ando.pearwood.info> References: <20141030005321.GC26866@ando.pearwood.info> <54519687.6030708@stoneleaf.us> <54553C4B.1010404@stoneleaf.us> <20141104035704.GD17635@ando.pearwood.info> Message-ID: <5458503A.70500@stoneleaf.us> On 11/03/2014 07:57 PM, Steven D'Aprano wrote: > On Sat, Nov 01, 2014 at 01:02:19PM -0700, Ethan Furman wrote: >> On 10/29/2014 06:38 PM, Ethan Furman wrote: >>> On 10/29/2014 05:53 PM, Steven D'Aprano wrote: >>>> >>>> I propose a few things: >>>> >>>> * institute a policy that, in the event of a new function being added >>>> to the operator module, only the dunderless version will be added; >>>> >>>> * change the documentation to make it clear that the dunderless >>>> versions should be used, rather than merely being "convenience" >>>> functions; >>>> >>>> * add a prominent note that the dunder versions exist for backwards >>>> compatibility only and should not be used in new code. >>> >>> +1 >> >> Actually, make that -1. >> >> I'm just crafting some tests to explore how NotImplemented impacts various >> classes, and the dunder versions make the whole thing much nicer. > > I'm surprised. I can't imagine dunder names making anything look > nicer. It sounds like you are writing "operator.__add__" instead > of "operator.add", which doesn't look nicer to me. The dunder versions are being used to check if a particular method is in a type's dictionary, to see how the subclass should be tailored for a test. Without the dunder versions I would have had to create my own mapping between dunder and dunderless names. > Can you show me your code? Check out the "Role of NotImplemented" over on pydev. The script is the last half of the first post. -- ~Ethan~ From joshua at landau.ws Tue Nov 4 05:10:54 2014 From: joshua at landau.ws (Joshua Landau) Date: Tue, 4 Nov 2014 04:10:54 +0000 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <5457F5E9.8060205@canterbury.ac.nz> Message-ID: On 4 November 2014 00:29, Guido van Rossum wrote: > > Regarding Akira Li's examples of groupby(), unfortunately I find both > versions inscrutable -- on a casual inspection I have no idea what goes on. > I would have to think about how I would write groupby() myself (but I'm > pretty sure I wouldn't us functools.partial(). :-) It's worth noting that the functools.partial doesn't really do anything. Just changing next_value = partial(next, iter(iterable)) to iterator = iter(iterable) and "next_value()" to "next(iterator)" gets rid of it. I would have tackled it by letting the inner iterator set a "finished" flag and I would have exhausted the iterable by iterating over it: def groupby(iterable): # Make sure this is one-pass iterator = iter(iterable) finished = False # Yields a group def yield_group(): nonlocal finished, group_key # This was taken off the iterator # by the previous group yield group_key for item in iterator: if item != group_key: # Set up next group group_key = item return yield item # No more items in iterator finished = True # This is always the head of the next # or current group group_key = next(iterator) while not finished: group = yield_group() yield group_key, group # Make sure the iterator is exhausted for _ in group: pass # group_key will now be the head of the next group This does have a key difference. Whereas with groupby you have the confusing property that from itertools import groupby grps = groupby("|||---|||") a = next(grps) b = next(grps) c = next(grps) list(a[1]) #>>> ['|', '|', '|'] with mine this does not happen: grps = groupby("|||---|||") grps = groupby("|||---|||") a = next(grps) b = next(grps) c = next(grps) list(a[1]) #>>> [] Was this an oversight in the original design or is this actually desired? I would guess it's an oversight. From guido at python.org Tue Nov 4 05:49:45 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 3 Nov 2014 20:49:45 -0800 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <5457F5E9.8060205@canterbury.ac.nz> Message-ID: I guess implementing groupby() would be a good interview question. :-) Can we get back to the issue at hand, which is whether and how we can change the behavior of StopIteraton to be less error-prone? On Mon, Nov 3, 2014 at 8:10 PM, Joshua Landau wrote: > On 4 November 2014 00:29, Guido van Rossum wrote: > > > > Regarding Akira Li's examples of groupby(), unfortunately I find both > > versions inscrutable -- on a casual inspection I have no idea what goes > on. > > I would have to think about how I would write groupby() myself (but I'm > > pretty sure I wouldn't us functools.partial(). :-) > > It's worth noting that the functools.partial doesn't really do > anything. Just changing > > next_value = partial(next, iter(iterable)) > > to > > iterator = iter(iterable) > > and "next_value()" to "next(iterator)" gets rid of it. > > I would have tackled it by letting the inner iterator set a "finished" > flag and I would have exhausted the iterable by iterating over it: > > def groupby(iterable): > # Make sure this is one-pass > iterator = iter(iterable) > finished = False > > # Yields a group > def yield_group(): > nonlocal finished, group_key > > # This was taken off the iterator > # by the previous group > yield group_key > > for item in iterator: > if item != group_key: > # Set up next group > group_key = item > return > > yield item > > # No more items in iterator > finished = True > > # This is always the head of the next > # or current group > group_key = next(iterator) > > while not finished: > group = yield_group() > yield group_key, group > > # Make sure the iterator is exhausted > for _ in group: > pass > > # group_key will now be the head of the next group > > This does have a key difference. Whereas with groupby you have the > confusing property that > > from itertools import groupby > > grps = groupby("|||---|||") > a = next(grps) > b = next(grps) > c = next(grps) > list(a[1]) > #>>> ['|', '|', '|'] > > with mine this does not happen: > > grps = groupby("|||---|||") > > grps = groupby("|||---|||") > a = next(grps) > b = next(grps) > c = next(grps) > list(a[1]) > #>>> [] > > Was this an oversight in the original design or is this actually > desired? I would guess it's an oversight. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue Nov 4 06:00:51 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 4 Nov 2014 16:00:51 +1100 Subject: [Python-ideas] Deprecate dunder functions from operator module In-Reply-To: <5458503A.70500@stoneleaf.us> References: <20141030005321.GC26866@ando.pearwood.info> <54519687.6030708@stoneleaf.us> <54553C4B.1010404@stoneleaf.us> <20141104035704.GD17635@ando.pearwood.info> <5458503A.70500@stoneleaf.us> Message-ID: <20141104050051.GE17635@ando.pearwood.info> On Mon, Nov 03, 2014 at 08:04:10PM -0800, Ethan Furman wrote: > On 11/03/2014 07:57 PM, Steven D'Aprano wrote: > >>I'm just crafting some tests to explore how NotImplemented impacts various > >>classes, and the dunder versions make the whole thing much nicer. > > > >I'm surprised. I can't imagine dunder names making anything look > >nicer. It sounds like you are writing "operator.__add__" instead > >of "operator.add", which doesn't look nicer to me. > > The dunder versions are being used to check if a particular method is in a > type's dictionary, to see how the subclass should be tailored for a test. > Without the dunder versions I would have had to create my own mapping > between dunder and dunderless names. The dunder *methods* aren't going away. The dunder *functions* in operator won't be going away either. But even if they did, it is trivial to change: getattr(operator, op) # op looks like '__add__', etc. to getattr(operator, op.strip('_')) except of course for full generality you might need a mapping to cover cases where the operator function isn't the same as the dunder method. E.g. operator.truth, operator.contains. (I don't think they are relevant to your case though.) In any case, unless I'm missing something, I don't think that even full-blown removal of the dunder functions would affect your code in any significant way, and I'm not arguing for removal. > >Can you show me your code? > > Check out the "Role of NotImplemented" over on pydev. The script is the > last half of the first post. https://mail.python.org/pipermail/python-dev/2014-November/136875.html -- Steven From ron3200 at gmail.com Tue Nov 4 08:07:29 2014 From: ron3200 at gmail.com (Ron Adam) Date: Tue, 04 Nov 2014 01:07:29 -0600 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <5457F5E9.8060205@canterbury.ac.nz> Message-ID: A bad edit got me... Don't mean to side track the Stopiteration discussion, but also don't want anyone having to debug my mistakes. In the PeekIter __init__ class... On 11/03/2014 09:17 PM, Ron Adam wrote: > self._peek = next(iter(iterable)) to... self._peek = next(self._it) It was yielding an extra value on the first group. As for the handling of the StopIteration, Maybe adding a StopGenerator exception could ease the way. It could be guaranteed to not to escape the generator it's raised in, while a StopIteration could propagate out until it caught by a loop. Also StopGenerator(value) could be equivalent to return value. The two for now at least overlap. StopIterator(value) would be caught and converted. While a bare StopIterator would propogate out. The new exception would make for clearor and easier to understand code in coroutines I think. Another aspect of StopGenerator is it would escape inner loops of the generator it's raised in. Like return does. And so you throw a StopGenerator exception into generator who's yield is nested in multiple loops to stop it. Of course all bets are off until it's actually tried I think. Just because it sound good here, (or to me at the moment), doesn't mean it will work. Does it. :-) Cheers, Ron From tomirendo at gmail.com Wed Nov 5 00:12:05 2014 From: tomirendo at gmail.com (Yotam Vaknin) Date: Wed, 5 Nov 2014 01:12:05 +0200 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <5457F5E9.8060205@canterbury.ac.nz> Message-ID: Firstly, sorry for my buggy code that began all this mess. But what about adding a named parameter to the next function, specifying an exception to be raised on StopIteration, allowing it to propagate? ?-4 ???? 2014, ???? 06:49, Guido van Rossum ???/?: > I guess implementing groupby() would be a good interview question. :-) > > Can we get back to the issue at hand, which is whether and how we can change the behavior of StopIteraton to be less error-prone? > > > On Mon, Nov 3, 2014 at 8:10 PM, Joshua Landau wrote: >> On 4 November 2014 00:29, Guido van Rossum wrote: >> > >> > Regarding Akira Li's examples of groupby(), unfortunately I find both >> > versions inscrutable -- on a casual inspection I have no idea what goes on. >> > I would have to think about how I would write groupby() myself (but I'm >> > pretty sure I wouldn't us functools.partial(). :-) >> >> It's worth noting that the functools.partial doesn't really do >> anything. Just changing >> >> next_value = partial(next, iter(iterable)) >> >> to >> >> iterator = iter(iterable) >> >> and "next_value()" to "next(iterator)" gets rid of it. >> >> I would have tackled it by letting the inner iterator set a "finished" >> flag and I would have exhausted the iterable by iterating over it: >> >> def groupby(iterable): >> # Make sure this is one-pass >> iterator = iter(iterable) >> finished = False >> >> # Yields a group >> def yield_group(): >> nonlocal finished, group_key >> >> # This was taken off the iterator >> # by the previous group >> yield group_key >> >> for item in iterator: >> if item != group_key: >> # Set up next group >> group_key = item >> return >> >> yield item >> >> # No more items in iterator >> finished = True >> >> # This is always the head of the next >> # or current group >> group_key = next(iterator) >> >> while not finished: >> group = yield_group() >> yield group_key, group >> >> # Make sure the iterator is exhausted >> for _ in group: >> pass >> >> # group_key will now be the head of the next group >> >> This does have a key difference. Whereas with groupby you have the >> confusing property that >> >> from itertools import groupby >> >> grps = groupby("|||---|||") >> a = next(grps) >> b = next(grps) >> c = next(grps) >> list(a[1]) >> #>>> ['|', '|', '|'] >> >> with mine this does not happen: >> >> grps = groupby("|||---|||") >> >> grps = groupby("|||---|||") >> a = next(grps) >> b = next(grps) >> c = next(grps) >> list(a[1]) >> #>>> [] >> >> Was this an oversight in the original design or is this actually >> desired? I would guess it's an oversight. >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Wed Nov 5 01:55:51 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 04 Nov 2014 19:55:51 -0500 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <5457F5E9.8060205@canterbury.ac.nz> Message-ID: On 11/4/2014 6:12 PM, Yotam Vaknin wrote: > Firstly, sorry for my buggy code that began all this mess. > > But what about adding a named parameter to the next function, specifying > an exception to be raised on StopIteration, allowing it to propagate? There is already a 'default' option, so the two would have to be mutually exclusive. "next(iterator[, default]) Retrieve the next item from the iterator by calling its __next__() method. If default is given, it is returned if the iterator is exhausted, otherwise StopIteration is raised." I believe either adding an alternate exception, or raising the default if it is an exception, has been discussed before on this list. In cannot remember details other than the obvious fact that nothing was changed. -- Terry Jan Reedy From greg.ewing at canterbury.ac.nz Wed Nov 5 02:18:39 2014 From: greg.ewing at canterbury.ac.nz (Greg) Date: Wed, 05 Nov 2014 14:18:39 +1300 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <5457F5E9.8060205@canterbury.ac.nz> Message-ID: <54597AEF.5090106@canterbury.ac.nz> On 5/11/2014 1:55 p.m., Terry Reedy wrote: > There is already a 'default' option, so the two would have to be > mutually exclusive. > > "next(iterator[, default]) If it were a keyword-only argument, there wouldn't be any conflict. But I'm not sure I like the idea of adding more complexities to the iterator protocol. Anything that changes the signature of __next__ or __iter__ would be a bad idea, since all existing iterables would need to be updated to take it into account. If this is is to be done, it might be better to add a new optional dunder method, so that existing iterables would continue to work. -- Greg From ncoghlan at gmail.com Wed Nov 5 15:09:19 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 6 Nov 2014 00:09:19 +1000 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <5457F5E9.8060205@canterbury.ac.nz> Message-ID: On 4 Nov 2014 10:31, "Guido van Rossum" wrote: > > On Mon, Nov 3, 2014 at 1:38 PM, Greg Ewing wrote: >> >> Guido van Rossum wrote: >> >>> FWIW the implementation of my proposal is easy to describe (which the Zen approves of): when a StopIteration leaves a frame, replace it with some other exception (either a new, custom one or maybe just RuntimeError), chaining the original StopIteration. >> >> >> Wouldn't that break all existing iterators whose next() method >> is implemented in Python? Since the code that catches the >> StopIteration is always in a different frame then. >> >> Or are you only talking about generator frames here, not >> frames in general? > > > I have to apologize, I pretty much had this backwards. > > What I should have said is that a generator should always be terminated by a return or falling off the end, and if StopIteration is raised in the generator (either by an explicit raise or raised by something it calls) and not caught by an except clause in the same generator, it should be turned into something else, so it bubbles out as something that keeps getting bubbled up rather than silently ending a for-loop. OTOH StopIteration raised by a non-generator function should not be mutated. > > I'm sorry if this invalidates Nick's endorsement of the proposal. I actually thought this was what you meant originally, and while it *would* require changes to contextlib, they're fairly minor: the current check for StopIteration would change to catch "UnexpectedStopIteration" instead, and the exception identity check would look at __cause__ rather than directly at the caught exception. > I definitely see this as a serious backward incompatibility: no matter how often it leads to buggy or obfuscated code, it's how things have always worked. Aye, breaking the equivalence between "return" and "raise StopIteration" is pretty major. I'm not even sure a plausible transition plan is possible, as at least contextlib would trigger any warning we might issue. Regards, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Nov 5 22:47:09 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 6 Nov 2014 07:47:09 +1000 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <5457F5E9.8060205@canterbury.ac.nz> Message-ID: On 6 Nov 2014 00:09, "Nick Coghlan" wrote: > On 4 Nov 2014 10:31, "Guido van Rossum" wrote: > > > > > > I have to apologize, I pretty much had this backwards. > > > > What I should have said is that a generator should always be terminated by a return or falling off the end, and if StopIteration is raised in the generator (either by an explicit raise or raised by something it calls) and not caught by an except clause in the same generator, it should be turned into something else, so it bubbles out as something that keeps getting bubbled up rather than silently ending a for-loop. OTOH StopIteration raised by a non-generator function should not be mutated. > > > > I'm sorry if this invalidates Nick's endorsement of the proposal. > > I actually thought this was what you meant originally, and while it *would* require changes to contextlib, they're fairly minor: the current check for StopIteration would change to catch "UnexpectedStopIteration" instead, and the exception identity check would look at __cause__ rather than directly at the caught exception. > > > I definitely see this as a serious backward incompatibility: no matter how often it leads to buggy or obfuscated code, it's how things have always worked. > > Aye, breaking the equivalence between "return" and "raise StopIteration" is pretty major. > > I'm not even sure a plausible transition plan is possible, as at least contextlib would trigger any warning we might issue. And having said that... what if we introduced UnexpectedStopIteration but initially made it a subclass of StopIteration? We could issue a deprecation warning whenever we triggered the StopIteration -> UnexpectedStopIteration conversion, pointing out that at some point in the future (3.6? 3.7?), UnexpectedStopIteration will no longer be a subclass of StopIteration (perhaps becoming a subclass of RuntimeError instead?). contextlib could avoid the warning by preconstructing a suitable UnexpectedStopIteration instance and throwing *that* into the generator, rather than throwing in a StopIteration raised from the with statement body. Regards, Nick. > > Regards, > Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From apalala at gmail.com Thu Nov 6 01:35:56 2014 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Wed, 5 Nov 2014 20:05:56 -0430 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <5457F5E9.8060205@canterbury.ac.nz> Message-ID: On Wed, Nov 5, 2014 at 5:17 PM, Nick Coghlan wrote: > And having said that... what if we introduced UnexpectedStopIteration but > initially made it a subclass of StopIteration? Exactly! -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron3200 at gmail.com Thu Nov 6 04:34:05 2014 From: ron3200 at gmail.com (Ron Adam) Date: Wed, 05 Nov 2014 21:34:05 -0600 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <5457F5E9.8060205@canterbury.ac.nz> Message-ID: On 11/05/2014 03:47 PM, Nick Coghlan wrote: > > > I'm sorry if this invalidates Nick's endorsement of the proposal. > > > > I actually thought this was what you meant originally, and while it > *would* require changes to contextlib, they're fairly minor: the current > check for StopIteration would change to catch "UnexpectedStopIteration" > instead, and the exception identity check would look at __cause__ rather > than directly at the caught exception. > > > > > I definitely see this as a serious backward incompatibility: no matter > how often it leads to buggy or obfuscated code, it's how things have always > worked. > > > > Aye, breaking the equivalence between "return" and "raise StopIteration" > is pretty major. > > > > I'm not even sure a plausible transition plan is possible, as at least > contextlib would trigger any warning we might issue. > > And having said that... what if we introduced UnexpectedStopIteration but > initially made it a subclass of StopIteration? > > We could issue a deprecation warning whenever we triggered the > StopIteration -> UnexpectedStopIteration conversion, pointing out that at > some point in the future (3.6? 3.7?), UnexpectedStopIteration will no > longer be a subclass of StopIteration (perhaps becoming a subclass of > RuntimeError instead?). > > contextlib could avoid the warning by preconstructing a suitable > UnexpectedStopIteration instance and throwing *that* into the generator, > rather than throwing in a StopIteration raised from the with statement body. I really don't like the name Unexpected anything. It makes me think of blue screens and page faults. :-/ I'm also not sure how it's supposed to work or where it should come from. As near as I can tell these two examples below are equivalent. I think the thing that needs to be avoided is the case of the endless loop. It would be better to let the Exception be noisy. How would the second example be changed in order to do that? Or is there some other thing that needs to be fixed? Cheers, Ron def izip(*args): iters = [iter(obj) for obj in args] while True: yield list(next(it) for it in iters) #StopIteration suppressed #by list comprehension. #resulting in empty lists. #While Loop never exits. print("never printed") a = izip([1,2],[3,4]) print(next(a),next(a),next(a)) # (1, 3) (2, 4) () #list(izip([1,2],[3,4])) #Currently never returns def izip2(*args): iters = [iter(obj) for obj in args] while True: L = [] for it in iters: try: obj = next(it) # StopIteration suppressed here. except StopIteration: break # exit for-loop # "return" instead of "break"? L.append(obj) yield L # While Loop never exits. print("never printed") a = izip2([5,6],[7,8]) print(next(a),next(a),next(a)) # (5, 7) (6, 8) () list(izip2([5,6],[7,8])) #Currently never returns # Not only doesn't it exit, but it's building # an endless list of empty lists! From steve at pearwood.info Thu Nov 6 11:15:26 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 6 Nov 2014 21:15:26 +1100 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <5457F5E9.8060205@canterbury.ac.nz> Message-ID: <20141106101525.GC3597@ando.pearwood.info> On Thu, Nov 06, 2014 at 07:47:09AM +1000, Nick Coghlan wrote: > And having said that... what if we introduced UnexpectedStopIteration but > initially made it a subclass of StopIteration? > > We could issue a deprecation warning whenever we triggered the > StopIteration -> UnexpectedStopIteration conversion, pointing out that at > some point in the future (3.6? 3.7?), UnexpectedStopIteration will no > longer be a subclass of StopIteration (perhaps becoming a subclass of > RuntimeError instead?). I'm sorry, I have been trying to follow this thread, but there have been too many wrong turns and side-tracks for me to keep it straight. What is the problem this is supposed to solve? Is it just that list (and set and dict) comprehensions treat StopIteration differently than generator expressions? That is, that [expr for x in iterator] list(expr for x in iterator) are not exactly equivalent, if expr raises StopIteration. If so, it seems to me that you're adding a lot of conceptual baggage and complication for very little benefit, and this will probably confuse people far more than the current situation does. The different treatment of StopIteration in generator expressions and list comprehensions does not seem to be a problem for people in practice, judging by the python-list and tutor mailing lists. The current situation is simple to learn and understand: (1) Generator expressions *emit* StopIteration when they are done: py> next(iter([])) Traceback (most recent call last): File "", line 1, in StopIteration (2) Functions such as tuple, list, set, dict *absorb* StopIteration: py> list(iter([])) [] py> it = iter([]) py> list(next(it) for y in range(1000)) [] For-loops do the same, if StopIteration is raised in the "for x in iterable" header. That's how it knows the loop is done. The "for" part of a comprehension is the same. (3) But raising StopIteration in the expression part (or if part) of a comprehension does not absord the exception, it is treated like any other exception: py> [next(iter([])) for y in range(1000)] Traceback (most recent call last): File "", line 1, in File "", line 1, in StopIteration If that is surprising to anyone, I suggest it is because they haven't considered what happens when you raise StopIteration in the body of a for-loop: py> for y in range(1000): ... next(iter([])) ... Traceback (most recent call last): File "", line 2, in StopIteration To me, the status quo is consistent, understandable, and predictable. In contrast, you have: - a solution to something which I'm not sure is even a problem that needs solving; - but if it does, the solution seems quite magical, complicated, and hard to understand; - it is unclear (to me) under what circumstances StopIteration will be automatically converted to UnexpectedStopIteration; - and it seems to me that it will lead to surprising behaviour when people deliberately raise StopIteration only to have it mysteriously turn into a different exception, but only sometimes. It seems to me that if the difference between comprehensions and generator expressions really is a problem that needs solving, that the best way to proceed is using the __future__ mechanism. 3.5 could introduce from __future__ comprehensions_absorb_stopiteration and then 3.6 or 3.7 could make it the default behaviour. We're still breaking backwards compatibility, but at least we're doing it cleanly, without magic (well, except the __future__ magic, but that's well-known and acceptible magic). There will be a transition period during which people can choose to keep the old behaviour or the new, and then we transition to the new behaviour. This automatic transformation of some StopIterations into something else seems like it will be worse than the problem it is trying to fix. For what it is worth, I'm a strong -1 on changing the behaviour of comprehensions at all, but if we must change it in a backwards incompatible way, +1 on __future__ and -1 on changing the exceptions to a different exception. -- Steven From ron3200 at gmail.com Thu Nov 6 15:24:13 2014 From: ron3200 at gmail.com (Ron Adam) Date: Thu, 06 Nov 2014 08:24:13 -0600 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: <20141106101525.GC3597@ando.pearwood.info> References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <5457F5E9.8060205@canterbury.ac.nz> <20141106101525.GC3597@ando.pearwood.info> Message-ID: On 11/06/2014 04:15 AM, Steven D'Aprano wrote: > The current situation is simple to learn and understand: > > (1) Generator expressions*emit* StopIteration when they are done: > > py> next(iter([])) > Traceback (most recent call last): > File "", line 1, in > StopIteration It's the "when they are done" part that's having the issue. In some cases, they are never completely done because the StopIteration Error is handled incorrectly. The reason this doesn't show up more is that most iterators only have one loop, which is exited and the generator ends causing a different StopIteration to be emitted from the one that ends the loop. > (2) Functions such as tuple, list, set, dict*absorb* StopIteration: > > py> list(iter([])) > [] > py> it = iter([]) > py> list(next(it) for y in range(1000)) > [] Right, sometimes this doesn't happen. > For-loops do the same, if StopIteration is raised in the "for x in > iterable" header. That's how it knows the loop is done. The "for" part > of a comprehension is the same. I think this part is working as it should. > (3) But raising StopIteration in the expression part (or if part) of a > comprehension does not absord the exception, it is treated like any > other exception: > > py> [next(iter([])) for y in range(1000)] > Traceback (most recent call last): > File "", line 1, in > File "", line 1, in > StopIteration This is not always working. When a StopIteration is raised from the next call in the example that started the thread, it's getting replaced with the equivalent of a break, so it's never exiting the generator completely. I think it needs to be replaced with the equivalent of return, which will end the generator, and cause a StopIteration to be emitted. (As you describe here.) It looks like it's a bug in the C code for generator expressions. Since it only effects generator expressions that have iterators with nested loops, it may be fixible. Cheers, Ron From tjreedy at udel.edu Thu Nov 6 18:02:57 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 06 Nov 2014 12:02:57 -0500 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: <20141106101525.GC3597@ando.pearwood.info> References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <5457F5E9.8060205@canterbury.ac.nz> <20141106101525.GC3597@ando.pearwood.info> Message-ID: On 11/6/2014 5:15 AM, Steven D'Aprano wrote: > On Thu, Nov 06, 2014 at 07:47:09AM +1000, Nick Coghlan wrote: > >> And having said that... what if we introduced UnexpectedStopIteration but >> initially made it a subclass of StopIteration? >> >> We could issue a deprecation warning whenever we triggered the >> StopIteration -> UnexpectedStopIteration conversion, pointing out that at >> some point in the future (3.6? 3.7?), UnexpectedStopIteration will no >> longer be a subclass of StopIteration (perhaps becoming a subclass of >> RuntimeError instead?). > > I'm sorry, I have been trying to follow this thread, but there have > been too many wrong turns and side-tracks for me to keep it straight. > What is the problem this is supposed to solve? > > Is it just that list (and set and dict) comprehensions treat > StopIteration differently than generator expressions? That is, that > > [expr for x in iterator] > > list(expr for x in iterator) > > are not exactly equivalent, if expr raises StopIteration. > > If so, it seems to me that you're adding a lot of conceptual baggage and > complication for very little benefit, and this will probably confuse > people far more than the current situation does. The different treatment > of StopIteration in generator expressions and list comprehensions does > not seem to be a problem for people in practice, judging by the > python-list and tutor mailing lists. > > The current situation is simple to learn and understand: > > (1) Generator expressions *emit* StopIteration when they are done: > > py> next(iter([])) > Traceback (most recent call last): > File "", line 1, in > StopIteration 'iter([])' is a list_iterator, not a generator expression. Here is the example I think you wanted. >>> next(i for i in ()) Traceback (most recent call last): File "", line 1, in next(i for i in ()) StopIteration > (2) Functions such as tuple, list, set, dict *absorb* StopIteration: > > py> list(iter([])) > [] > py> it = iter([]) > py> list(next(it) for y in range(1000)) > [] > > For-loops do the same, if StopIteration is raised in the "for x in > iterable" header. That's how it knows the loop is done. The "for" part > of a comprehension is the same. > > > (3) But raising StopIteration in the expression part (or if part) of a > comprehension does not absord the exception, it is treated like any > other exception: > > py> [next(iter([])) for y in range(1000)] > Traceback (most recent call last): > File "", line 1, in > File "", line 1, in > StopIteration > > If that is surprising to anyone, I suggest it is because they haven't > considered what happens when you raise StopIteration in the body of a > for-loop: Which is the Python translation of the comprehension. > py> for y in range(1000): > ... next(iter([])) > ... > Traceback (most recent call last): > File "", line 2, in > StopIteration > > > To me, the status quo is consistent, understandable, and predictable. ... I agree. -- Terry Jan Reedy From ethan at stoneleaf.us Thu Nov 6 18:12:00 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 06 Nov 2014 09:12:00 -0800 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <5457F5E9.8060205@canterbury.ac.nz> <20141106101525.GC3597@ando.pearwood.info> Message-ID: <545BABE0.5030809@stoneleaf.us> I believe the issue is that, in some situations, the generator is absorbing StopIteration when it shouldn't. -- ~Ethan~ From guido at python.org Thu Nov 6 19:54:51 2014 From: guido at python.org (Guido van Rossum) Date: Thu, 6 Nov 2014 10:54:51 -0800 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: <20141106101525.GC3597@ando.pearwood.info> References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <5457F5E9.8060205@canterbury.ac.nz> <20141106101525.GC3597@ando.pearwood.info> Message-ID: TL;DR :-( The crux of the issue is that sometimes the StopIteration raised by an explicit next() call is "just" an exception, but sometimes it terminates some controlling loop. Most precedents clearly establish it as "just" an exception, e.g. (these are all nonsense) it = iter([1, 2, 3, -1]) for x in it: if x < 0: next(it) it = iter([1, 2, 3]) [x for x in it if next(it) < 0] Both these raise StopIteration and report a traceback. But generators and generator expressions are different. In a generator, any StopIteration that isn't caught by an exception handler in the body of the generator implicitly terminates the iteration, just like "return" or falling off the end. This was an explicitly designed feature, but I don't think has worked out very well, given that more often than not, a StopIteration that "escapes" from some expression is a bug. And because generator expressions are implemented using generators (duh), the following returns [] instead of raising StopIteration: it = iter([1, 2, 3]) list(x for x in it if next(it) < 0) This is confusing because it breaks the (intended) equivalence between list() and [] (even though we refer to the latter as a comprehension, the syntax inside the [] is the same as a generator expression. If I had had the right foresight, I would have made it an error to terminate a generator with a StopIteration, probably by raising another exception chained to the StopIteration (so the traceback shows the place where the StopIteration escaped). The question at hand is if we can fix this post-hoc, using clever tricks and (of course) a deprecation period. --Guido On Thu, Nov 6, 2014 at 2:15 AM, Steven D'Aprano wrote: > On Thu, Nov 06, 2014 at 07:47:09AM +1000, Nick Coghlan wrote: > > > And having said that... what if we introduced UnexpectedStopIteration but > > initially made it a subclass of StopIteration? > > > > We could issue a deprecation warning whenever we triggered the > > StopIteration -> UnexpectedStopIteration conversion, pointing out that at > > some point in the future (3.6? 3.7?), UnexpectedStopIteration will no > > longer be a subclass of StopIteration (perhaps becoming a subclass of > > RuntimeError instead?). > > I'm sorry, I have been trying to follow this thread, but there have > been too many wrong turns and side-tracks for me to keep it straight. > What is the problem this is supposed to solve? > > Is it just that list (and set and dict) comprehensions treat > StopIteration differently than generator expressions? That is, that > > [expr for x in iterator] > > list(expr for x in iterator) > > are not exactly equivalent, if expr raises StopIteration. > > If so, it seems to me that you're adding a lot of conceptual baggage and > complication for very little benefit, and this will probably confuse > people far more than the current situation does. The different treatment > of StopIteration in generator expressions and list comprehensions does > not seem to be a problem for people in practice, judging by the > python-list and tutor mailing lists. > > The current situation is simple to learn and understand: > > (1) Generator expressions *emit* StopIteration when they are done: > > py> next(iter([])) > Traceback (most recent call last): > File "", line 1, in > StopIteration > > > (2) Functions such as tuple, list, set, dict *absorb* StopIteration: > > py> list(iter([])) > [] > py> it = iter([]) > py> list(next(it) for y in range(1000)) > [] > > For-loops do the same, if StopIteration is raised in the "for x in > iterable" header. That's how it knows the loop is done. The "for" part > of a comprehension is the same. > > > (3) But raising StopIteration in the expression part (or if part) of a > comprehension does not absord the exception, it is treated like any > other exception: > > py> [next(iter([])) for y in range(1000)] > Traceback (most recent call last): > File "", line 1, in > File "", line 1, in > StopIteration > > If that is surprising to anyone, I suggest it is because they haven't > considered what happens when you raise StopIteration in the body of a > for-loop: > > py> for y in range(1000): > ... next(iter([])) > ... > Traceback (most recent call last): > File "", line 2, in > StopIteration > > > To me, the status quo is consistent, understandable, and predictable. > > In contrast, you have: > > - a solution to something which I'm not sure is even a problem > that needs solving; > > - but if it does, the solution seems quite magical, complicated, > and hard to understand; > > - it is unclear (to me) under what circumstances StopIteration > will be automatically converted to UnexpectedStopIteration; > > - and it seems to me that it will lead to surprising behaviour > when people deliberately raise StopIteration only to have it > mysteriously turn into a different exception, but only > sometimes. > > > It seems to me that if the difference between comprehensions and > generator expressions really is a problem that needs solving, that the > best way to proceed is using the __future__ mechanism. 3.5 could > introduce > > from __future__ comprehensions_absorb_stopiteration > > and then 3.6 or 3.7 could make it the default behaviour. > > We're still breaking backwards compatibility, but at least we're doing > it cleanly, without magic (well, except the __future__ magic, but that's > well-known and acceptible magic). There will be a transition period > during which people can choose to keep the old behaviour or the new, and > then we transition to the new behaviour. This automatic transformation > of some StopIterations into something else seems like it will be worse > than the problem it is trying to fix. > > For what it is worth, I'm a strong -1 on changing the behaviour of > comprehensions at all, but if we must change it in a backwards > incompatible way, +1 on __future__ and -1 on changing the exceptions to > a different exception. > > > > -- > Steven > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.rodola at gmail.com Thu Nov 6 19:59:31 2014 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Thu, 6 Nov 2014 19:59:31 +0100 Subject: [Python-ideas] including psutil in the standard library? In-Reply-To: <59DEA399-5915-4562-8042-1372CF9D4353@yahoo.com> References: <542319B1.2090606@ferrara.linux.it> <59DEA399-5915-4562-8042-1372CF9D4353@yahoo.com> Message-ID: On Tue, Oct 14, 2014 at 12:49 AM, Andrew Barnert < abarnert at yahoo.com.dmarc.invalid> wrote: > On Oct 13, 2014, at 14:40, Victor Stinner > wrote: > > > 2014-09-24 21:21 GMT+02:00 Stefano Borini < > stefano.borini at ferrara.linux.it>: > >> I am wondering if it would be possible to include psutil > >> (https://pypi.python.org/pypi/psutil ) in the standard library, and if > not, > >> what would be needed. > > > > The problem is that Python only releases a new major version every 18 > > months (or something like that), so adding a single new function will > > take so much time. > > > > Supporting a new platform, architecture, major version of an OS, etc. > > may also need to wait a new Python major release. > > > > Operating systems are evolving quite fast. For example, there is the > > new architecture AArch64, containers are more and more popular, > > systemd can also be used to provide features similar to psutil and it > > supports containers (running systemd, it even supports recursive > > containers....). Network becomes more virtual with Software Defined > > Networks (SDN, NFV, etc.). The Linux kernel is also extended at each > > release. Example of "recent" addition: /proc/pid/fdinfo/ directory. > > Does psutil provide NUMA information? NUMA also becomes more > > information nowadays for performances. > > > > The psutil also still evolves. For example, I see that the version 2.1 > > released at April, 2014 adds "netstat-like functionalities". The > > version 2.0 was only released one month before (March, 2014). > > > > The API of psutil changed a lot between 1.x and 2.0. The version 2.0 > > was only released a few months ago. Is it enough to consider that the > > API is now stable enough (was enough tested)? Giampaolo (author of the > > module) doesn't look to be confident in its own API ;-) He wrote "I > > still want to work on a couple of new features first (...) and be 100% > > sure that I'm happy with the API as it is right now". > > > > Maybe I'm wrong and the psutil is stable and can be "frozen" in the > > standard library. Maybe it's possible to extract a subpart of the > > psutil to keep the most stable part? > > > > I'm not strongly opposed to the integration of the psutil module into > > the stdlib. > > > > Is it hard to install the psutil module today? I see wheel packages > > and .exe installers for Windows. They are Linux packages (Debian, > > Ubuntu, Fedora, ArchLinux, etc.) > > This might be a good idea. > > It seems to me that there are two common uses for psutil. > > Some people want just a little bit more than Python makes available, and > they want it in a cross-platform (at least gnu vs. bsd vs. sysv, if not > unix vs. windows) way. That doesn't change that much over time, and adding > that much from psutil to the stdlib might make sense. > > Other people want all kinds of detailed process information, in a > platform-specific way, without having to grub through low-level APIs. That > changes frequently, and for those people, pip install psutil should > probably remain the right answer.+ > > (Of course there are also people who use psutil to do the exact same > things subprocess and os can already do better; for those people, the > answer is to stop doing that...) > > I guess the real question is, what are the use cases for the "stable > core"? Once we know that, we can identify whether it's stable enough and > simple enough to extract. I think this might make sense. Skimming through http://pythonhosted.org/psutil/ should give an idea of what's more "necessary" and what not. What's probably "less" necessary is: - psutil.disk_partitions() - psutil.net_connections() - psutil.users() - psutil.Process.open_files() (it turned out this is not reliable on Windows and FreeBSD) - psutil.Process.connections() - psutil.Process.memory_percent() ('cause it's a very simple utility method) APIs which are currently duplicated in Python stdlib: - psutil.disk_usage() (shutil.disk_usage, see http://bugs.python.org/issue12442) - psutil.Process.nice() (os.get/setpriority(), see: http://bugs.python.org/issue10784) - psutil.Process.cpu_affinity() (os.sched_get/setaffinity, see https://docs.python.org/3.5/library/os.html#os.sched_setaffinity) - psutil.cpu_count() (available as multiprocessing.cpu_count() but psutil can also distinguish between logical and physical cores) The rest is probably worth it for inclusion. As for my "doubts" regarding the stability of the API they basically comes down to one issue only: https://github.com/giampaolo/psutil/issues/428. In order to properly fix that issue I'll probably have to introduce a new "ZombieProcess" exception class other than "NoSuchProcess". That means users will have to change their code from this: for p in psutil.process_iter(): try: p.something() except psutil.NoSuchProcess: pass ...to this: for p in psutil.process_iter(): try: p.something() except (psutil.NoSuchProcess, psutil.ZombieProcess): pass I would consider the rest of the API quite stable. Please note that psutil currently supports: - Linux >= 2.6 (perhaps also 2.4) - Windows => XP - OSX => 10.4 (not sure about older versions) - FreeBSD => 9.0, perhaps also 8.0 - OpenSolaris (tested on OpenSolaris 10, not sure about other versions) With that said I expect that the moment psutil is included in the buildbot system a lot of currently undetected issues will be discovered as psutil has only been actively developed and tested on the platforms listed above (which are the ones I have access to). > Is it hard to install the psutil module today? I see wheel packages > and .exe installers for Windows. They are Linux packages (Debian, > Ubuntu, Fedora, ArchLinux, etc.) I would say it is reasonably easy, especially now that I'm providing wheels for Windows. -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Thu Nov 6 22:45:37 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 6 Nov 2014 22:45:37 +0100 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <5457F5E9.8060205@canterbury.ac.nz> <20141106101525.GC3597@ando.pearwood.info> Message-ID: <20141106224537.4fc55146@fsol> On Thu, 6 Nov 2014 10:54:51 -0800 Guido van Rossum wrote: > > If I had had the right foresight, I would have made it an error to > terminate a generator with a StopIteration, probably by raising another > exception chained to the StopIteration (so the traceback shows the place > where the StopIteration escaped). > > The question at hand is if we can fix this post-hoc, using clever tricks > and (of course) a deprecation period. Is there any point in fixing it? Who relies on such borderline cases? Regards Antoine. From apalala at gmail.com Fri Nov 7 01:31:32 2014 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Thu, 6 Nov 2014 20:01:32 -0430 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: <20141106224537.4fc55146@fsol> References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <5457F5E9.8060205@canterbury.ac.nz> <20141106101525.GC3597@ando.pearwood.info> <20141106224537.4fc55146@fsol> Message-ID: On Thu, Nov 6, 2014 at 5:15 PM, Antoine Pitrou wrote: > Is there any point in fixing it? Who relies on such borderline cases? import this? -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri Nov 7 10:43:12 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 7 Nov 2014 20:43:12 +1100 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <5457F5E9.8060205@canterbury.ac.nz> <20141106101525.GC3597@ando.pearwood.info> <20141106224537.4fc55146@fsol> Message-ID: <20141107094311.GD3597@ando.pearwood.info> On Thu, Nov 06, 2014 at 08:01:32PM -0430, Juancarlo A?ez wrote: > On Thu, Nov 6, 2014 at 5:15 PM, Antoine Pitrou wrote: > > > Is there any point in fixing it? Who relies on such borderline cases? > > > import this? What about it? There are 19 lines in the Zen, and I don't think any of them are particularly relevent here. Which line, or lines, were you thinking of? -- Steven From steve at pearwood.info Fri Nov 7 12:20:30 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 7 Nov 2014 22:20:30 +1100 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <5457F5E9.8060205@canterbury.ac.nz> <20141106101525.GC3597@ando.pearwood.info> Message-ID: <20141107112030.GA7768@ando.pearwood.info> On Thu, Nov 06, 2014 at 10:54:51AM -0800, Guido van Rossum wrote: > TL;DR :-( That's how I feel about this whole thread ;-) [...] > If I had had the right foresight, I would have made it an error to > terminate a generator with a StopIteration, probably by raising another > exception chained to the StopIteration (so the traceback shows the place > where the StopIteration escaped). > > The question at hand is if we can fix this post-hoc, using clever tricks > and (of course) a deprecation period. Do we need "clever tricks"? In my earlier email, I suggested that if this needs to be fixed, the best way to introduce a change in behaviour is with the __future__ mechanism. 3.5 could introduce from __future__ stopiteration_is_an_error (in my earlier post, I managed to get the suggested behaviour completely backwards) and then 3.6 could raise a warning and 3.7 could make it the default behaviour. We're still breaking backwards compatibility, but at least we're doing it cleanly, without clever and/or ugly hacks. There will be a transition period during which people can choose to keep the old behaviour or the new, and then we transition to the new behaviour. -- Steven From apalala at gmail.com Fri Nov 7 13:57:59 2014 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Fri, 7 Nov 2014 08:27:59 -0430 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: <20141107094311.GD3597@ando.pearwood.info> References: <5457F5E9.8060205@canterbury.ac.nz> <20141106101525.GC3597@ando.pearwood.info> <20141106224537.4fc55146@fsol> <20141107094311.GD3597@ando.pearwood.info> Message-ID: On Fri, Nov 7, 2014 at 5:13 AM, Steven D'Aprano wrote: > What about it? There are 19 lines in the Zen, and I don't think any of > them are particularly relevent here. Which line, or lines, were you > thinking of? > You asked what was the point in fixing the current behavior regarding StopIteration: - - Beautiful is better than ugly. - Explicit is better than implicit. - Simple is better than complex. - Flat is better than nested. - Readability counts. - Special cases aren't special enough to break the rules. - Errors should never pass silently. - In the face of ambiguity, refuse the temptation to guess. - There should be one-- and preferably only one --obvious way to do it. - Now is better than never. - If the implementation is hard to explain, it's a bad idea. Basically, that some of the behaviours Guido mentioned are unexpected consequences of a lack of foresight in the implementation of generator expressions. It is not in the Zen of Python to leave it as is just because some existing code may be relying on the odd behavior. Just to be clear, that StopIteration will cancel more than one iterator is an unintended behavior that is something difficult to explain, is of questionable usefulness, and is the source of difficult to catch bugs. Cheers, -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Nov 7 14:04:04 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 8 Nov 2014 00:04:04 +1100 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <5457F5E9.8060205@canterbury.ac.nz> <20141106101525.GC3597@ando.pearwood.info> <20141106224537.4fc55146@fsol> <20141107094311.GD3597@ando.pearwood.info> Message-ID: On Fri, Nov 7, 2014 at 11:57 PM, Juancarlo A?ez wrote: > It is not in the Zen of Python to leave it as is just because some existing > code may be relying on the odd behavior. Actually, it is. Practicality beats purity. :) ChrisA From solipsis at pitrou.net Fri Nov 7 14:15:37 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 7 Nov 2014 14:15:37 +0100 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <5457F5E9.8060205@canterbury.ac.nz> <20141106101525.GC3597@ando.pearwood.info> <20141106224537.4fc55146@fsol> Message-ID: <20141107141537.32310a1d@fsol> On Thu, 6 Nov 2014 20:01:32 -0430 Juancarlo A?ez wrote: > On Thu, Nov 6, 2014 at 5:15 PM, Antoine Pitrou wrote: > > > Is there any point in fixing it? Who relies on such borderline cases? > > > import this? I was more interested in an answer to my second question, actually ;-) i.e. "Who relies on such borderline cases?" Regards Antoine. From abarnert at yahoo.com Fri Nov 7 16:59:17 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 7 Nov 2014 07:59:17 -0800 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <5457F5E9.8060205@canterbury.ac.nz> <20141106101525.GC3597@ando.pearwood.info> Message-ID: <91F449C7-C6CB-4F1A-A9A4-D361B8B3A9AC@yahoo.com> On Nov 6, 2014, at 10:54, Guido van Rossum wrote: > This is confusing because it breaks the (intended) equivalence between list() and [] (even though we refer to the latter as a comprehension, the syntax inside the [] is the same as a generator expression. If this change (I mean the proposed clever workaround, not the "no terminating generators with StopIteration" change that's too radical) would be sufficient to make that equivalence actually true instead of just pretty close, I think that's reason enough to fix it on its own. Especially since that would make it easy to fix the genexpr docs. (Read 6.2.8 and tell me what it says the semantics of a genexpr are, and what values it yields. Now try to think of a way to fix that without repeating most of the text from 6.2.4, which nobody wants to do. If the docs could just define the semantics of genexprs, then define listcomps by saying that [] is equivalent to list(), that would be a lot simpler and clearer.) From guido at python.org Fri Nov 7 18:10:42 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 7 Nov 2014 09:10:42 -0800 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: <20141107112030.GA7768@ando.pearwood.info> References: <5457F5E9.8060205@canterbury.ac.nz> <20141106101525.GC3597@ando.pearwood.info> <20141107112030.GA7768@ando.pearwood.info> Message-ID: Trying to keep the thread on focus... On Fri, Nov 7, 2014 at 3:20 AM, Steven D'Aprano wrote: > On Thu, Nov 06, 2014 at 10:54:51AM -0800, Guido van Rossum wrote: > > If I had had the right foresight, I would have made it an error to > > terminate a generator with a StopIteration, probably by raising another > > exception chained to the StopIteration (so the traceback shows the place > > where the StopIteration escaped). > > > > The question at hand is if we can fix this post-hoc, using clever tricks > > and (of course) a deprecation period. > > Do we need "clever tricks"? In my earlier email, I suggested that if > this needs to be fixed, the best way to introduce a change in > behaviour is with the __future__ mechanism. 3.5 could introduce > > from __future__ stopiteration_is_an_error > > (in my earlier post, I managed to get the suggested behaviour completely > backwards) and then 3.6 could raise a warning and 3.7 could make it > the default behaviour. > Sure. (Though IMO __future__ itself is a clever hack.) > We're still breaking backwards compatibility, but at least we're doing > it cleanly, without clever and/or ugly hacks. There will be a transition > period during which people can choose to keep the old behaviour or the > new, and then we transition to the new behaviour. > We'd still need to specify the eventual behavior. I propose the following as a strawman (after the deprecation period is finished and the __future__ import is no longer needed): - If a StopIteration is about to bubble out of a generator frame, it is replaced with some other exception (maybe RuntimeError, maybe a new custom Exception subclass, but *not* deriving from StopIteration) which causes the next() call (which invoked the generator) to fail, passing that exception out. From then on it's just like any old exception. During the transition, we check if the generator was defined in the scope of the __future__, and if so, we do the the same thing; otherwise, we issue a warning and let the StopIteration bubble out, eventually terminating some loop or generator expression. It would be nice if, when the warning is made fatal (e.g. through the -W flag), the exception raised was the same one mentioned above (i.e. RuntimeError or a custom subclass -- I don't care much about this detail). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sat Nov 8 01:53:04 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 07 Nov 2014 19:53:04 -0500 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <5457F5E9.8060205@canterbury.ac.nz> <20141106101525.GC3597@ando.pearwood.info> <20141107112030.GA7768@ando.pearwood.info> Message-ID: On 11/7/2014 12:10 PM, Guido van Rossum wrote: > We'd still need to specify the eventual behavior. I propose the > following as a strawman (after the deprecation period is finished and > the __future__ import is no longer needed): Double meanings are a constant problem. The ambiguity of StopIteration (unintended bug indicator? intended stop signal?) is not needed within generator functions*. An explicit 'raise StopIteration' is more easily written 'return'. The rare. clever, 'yield expression' that means 'either yield the value of expression or raise StopIteration' can be re-written try: tem = expression except StopIteration: return yield tem An anonymous generator expression with a 'yield or raise' expresssion can be re-written as a named generator function. In this context, allowing 'yield or raise' is too clever in that it breaks the desired (intended?) equivalence between 'list(genexp)' and '[genexp]'. I agree that this is a bad tradeoff. Such alternative could be documented. * StopIteration raised within a __next__ method could be a bug, but a) this should be rare, b) the possibility cannot be eliminated, and c) it would be too expensive to not take StopIteration at face value. > - If a StopIteration is about to bubble out of a generator frame, it is > replaced with some other exception (maybe RuntimeError, I support this particular replacement. To me: 1. The proposal amounts to defining StopIteration escaping a running generator frame as a runtime error. > maybe a new custom Exception subclass, > but *not* deriving from StopIteration) 2. There are already more Exception classes than I can remember. 3. This error should be very rare after the transition. 4. A good error message will be at least as important as the class name. 5. A new, specific exception class is an invitation for people to write code that raises StopIteration so that its replacement can be caught in an except clause in the calling code. If the existing builtin exceptions other than StopIteration are not enough to choose from, one can define a custom class. > which causes the next() call (which invoked the generator) to fail, > passing that exception out. > From then on it's just like any old exception. Any replacement will be ordinary as soon as it is made, before it reaches next(), so I would move this last line up a couple of lines. -- Terry Jan Reedy From masklinn at masklinn.net Sat Nov 8 14:05:07 2014 From: masklinn at masklinn.net (Masklinn) Date: Sat, 8 Nov 2014 14:05:07 +0100 Subject: [Python-ideas] Store shared/locked state inside of the lock object Message-ID: <884DB226-A3EC-4A89-8668-8C2B835FBC4A@masklinn.net> Since the introduction of context managers, using the lock object itself has become easier and safer (in that there's very little chance of forgetting to release a lock anymore). A big annoyance remains though: relation between lock and lockee remains informal and there is no structured way to indicate: 1. whether a state-set is protected by a lock 2. which lock protects the state-set 3. which state-set is protected by a lock Some languages (e.g. Java) have tried to solve 2 and 3 using intrinsic locks, however * that does not solve 1 (and it becomes impossible to informally look for lock objects lying around and try to find corresponding state-sets) * it does not help much when state isn't coalesced in a single object, and for state hierarchies there is no way to express whether the whole hierarchy should be protected under the same lock (the root's) or each leaf should be locked individually. AFAIK intrinsic locks are not hierarchical themselves * things get very awkward when using alternate concurrency-management strategies such as explicit locks for security reason[0], the non-use of intrinsic locks has to be again documented informally A fairly small technical change I've been considering to improve this situation is to store the state-set inside the lock object, and only yield it through the context manager: that the state-set is protected by a lock is made obvious, and so is the relation between lock and state-set. I was delighted to discover that Rust's sync::Mutex[1] and sync::RWLock[2] follow a very similar strategy of owning the state-set It's not a panacea, it doesn't fix issues of lock acquisition ordering for instance, but I think it would go a fairly long way towards making correct use of locks easier in Python. The basic changes would be: * threading.Lock and threading.RLock would now take an optional `data` parameter * the parameter would be stored internally and not directly accessible by users of the lock * that parameter would be returned by __enter__ and provided to the current "owner" of the lock These should cause no forward-compatibility issues, Lock() currently takes no arguments, and its __enter__ returns no value. Possible improvements/questions/issues I can see: * with Lock, the locked state would not be available unless using as a context manager. RLock could allow getting the protected state only while locked by the current thread * as-is, the scheme requires mutable state as it's not possible to swap the internal state entirely. RLock could allow state-replacement when locked * because Python has no ownership concept, it would be possible for a consumer to keep a reference to the locked state and manipulate it without locking I don't consider the third issue to be huge, it could be mitigated by yielding a proxy to the internal state only valid for the current lock span. However I do not know if it's possible to create completely transparent proxies in Python. The first two issues are slightly more troubling and could be mitigated by yielding not the state-set alone but a proxy object living only for the current lock span (or both the state-set and a proxy object) that proxy would allow getting and setting the state-set, and would error-out after unlocking. Lock.acquire() could be altered to return the same proxy (or (state-set, proxy) pair) however it's currently defined as returning either True or False so that'd be a backwards- incompatible change. An alternative would be to add a new acquisition method or a new flag parameter changing the return value from True to these on acquisition. A drawback of this additional change is that it would require the lock object to keep track of the current live proxy(/proxies for rlock?), and invalidate it(/them) on unlocking, increasing its complexity much more than just adding a new attribute. Thoughts? [0] https://www.securecoding.cert.org/confluence/display/java/LCK00-J.+Use+private+final+lock+objects+to+synchronize+classes+that+may+interact+with+untrusted+code [1] http://doc.rust-lang.org/sync/struct.Mutex.html [2] http://doc.rust-lang.org/sync/struct.RWLock.html From phd at phdru.name Sat Nov 8 14:38:56 2014 From: phd at phdru.name (Oleg Broytman) Date: Sat, 8 Nov 2014 14:38:56 +0100 Subject: [Python-ideas] Store shared/locked state inside of the lock object In-Reply-To: <884DB226-A3EC-4A89-8668-8C2B835FBC4A@masklinn.net> References: <884DB226-A3EC-4A89-8668-8C2B835FBC4A@masklinn.net> Message-ID: <20141108133856.GA1706@phdru.name> Hi! On Sat, Nov 08, 2014 at 02:05:07PM +0100, Masklinn wrote: > ... a proxy to the internal state only valid for the current lock > span. However I do not know if it's possible to create completely > transparent proxies in Python. I think it's possible -- in a C extension. See http://www.egenix.com/products/python/mxBase/mxProxy/ for an example. (I don't have any opinion on the proposal.) Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From solipsis at pitrou.net Sat Nov 8 16:04:25 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 8 Nov 2014 16:04:25 +0100 Subject: [Python-ideas] Store shared/locked state inside of the lock object References: <884DB226-A3EC-4A89-8668-8C2B835FBC4A@masklinn.net> Message-ID: <20141108160425.1ffbd1f2@fsol> On Sat, 8 Nov 2014 14:05:07 +0100 Masklinn wrote: > > A fairly small technical change I've been considering to improve this > situation is to store the state-set inside the lock object, and only > yield it through the context manager: that the state-set is protected > by a lock is made obvious, and so is the relation between lock and > state-set. I was delighted to discover that Rust's sync::Mutex[1] and > sync::RWLock[2] follow a very similar strategy of owning the state-set > > It's not a panacea, it doesn't fix issues of lock acquisition ordering > for instance, but I think it would go a fairly long way towards making > correct use of locks easier in Python. > > The basic changes would be: > * threading.Lock and threading.RLock would now take an optional > `data` parameter > * the parameter would be stored internally and not directly accessible > by users of the lock > * that parameter would be returned by __enter__ and provided to the > current "owner" of the lock For clarity it should probably be a separate class (or set of classes), e.g. DataLock. Regards Antoine. From masklinn at masklinn.net Sat Nov 8 19:42:50 2014 From: masklinn at masklinn.net (Masklinn) Date: Sat, 8 Nov 2014 19:42:50 +0100 Subject: [Python-ideas] Store shared/locked state inside of the lock object In-Reply-To: <20141108160425.1ffbd1f2@fsol> References: <884DB226-A3EC-4A89-8668-8C2B835FBC4A@masklinn.net> <20141108160425.1ffbd1f2@fsol> Message-ID: On 2014-11-08, at 16:04 , Antoine Pitrou wrote: > On Sat, 8 Nov 2014 14:05:07 +0100 > Masklinn wrote: >> >> A fairly small technical change I've been considering to improve this >> situation is to store the state-set inside the lock object, and only >> yield it through the context manager: that the state-set is protected >> by a lock is made obvious, and so is the relation between lock and >> state-set. I was delighted to discover that Rust's sync::Mutex[1] and >> sync::RWLock[2] follow a very similar strategy of owning the state-set >> >> It's not a panacea, it doesn't fix issues of lock acquisition ordering >> for instance, but I think it would go a fairly long way towards making >> correct use of locks easier in Python. >> >> The basic changes would be: >> * threading.Lock and threading.RLock would now take an optional >> `data` parameter >> * the parameter would be stored internally and not directly accessible >> by users of the lock >> * that parameter would be returned by __enter__ and provided to the >> current "owner" of the lock > > For clarity it should probably be a separate class (or set of classes), > e.g. DataLock. On the one hand this'd allow completely ignoring backwards-compatibility issues wrt acquire() which is nice, on the other hand it would double the number of lock types and introduce redundancy as DataLock would be pretty much a strict superset of Lock, which is why I thought extending Lock made sense. From ethan at stoneleaf.us Sat Nov 8 21:01:43 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 08 Nov 2014 12:01:43 -0800 Subject: [Python-ideas] Store shared/locked state inside of the lock object In-Reply-To: References: <884DB226-A3EC-4A89-8668-8C2B835FBC4A@masklinn.net> <20141108160425.1ffbd1f2@fsol> Message-ID: <545E76A7.8060506@stoneleaf.us> On 11/08/2014 10:42 AM, Masklinn wrote: > On 2014-11-08, at 16:04 , Antoine Pitrou wrote: >> On Sat, 8 Nov 2014 14:05:07 +0100 Masklinn wrote: >>> >>> A fairly small technical change I've been considering to improve this >>> situation is to store the state-set inside the lock object, and only >>> yield it through the context manager: that the state-set is protected >>> by a lock is made obvious, and so is the relation between lock and >>> state-set. I was delighted to discover that Rust's sync::Mutex and >>> sync::RWLock follow a very similar strategy of owning the state-set >> >> For clarity it should probably be a separate class (or set of classes), >> e.g. DataLock. > > On the one hand this'd allow completely ignoring backwards-compatibility > issues wrt acquire() which is nice, on the other hand it would double > the number of lock types and introduce redundancy as DataLock would be > pretty much a strict superset of Lock, which is why I thought extending > Lock made sense. How does transforming existing locks into this kind of lock benefit existing code? If existing code has to change to take advantage of the new features, said code could just as easily change the name of the lock it was using. -- ~Ethan~ From masklinn at masklinn.net Sat Nov 8 23:09:57 2014 From: masklinn at masklinn.net (Masklinn) Date: Sat, 8 Nov 2014 23:09:57 +0100 Subject: [Python-ideas] Store shared/locked state inside of the lock object In-Reply-To: <545E76A7.8060506@stoneleaf.us> References: <884DB226-A3EC-4A89-8668-8C2B835FBC4A@masklinn.net> <20141108160425.1ffbd1f2@fsol> <545E76A7.8060506@stoneleaf.us> Message-ID: <34263F5C-A623-418F-A24C-322EBA957470@masklinn.net> On 2014-11-08, at 21:01 , Ethan Furman wrote: > On 11/08/2014 10:42 AM, Masklinn wrote: >> >> On the one hand this'd allow completely ignoring backwards-compatibility >> issues wrt acquire() which is nice, on the other hand it would double >> the number of lock types and introduce redundancy as DataLock would be >> pretty much a strict superset of Lock, which is why I thought extending >> Lock made sense. > > How does transforming existing locks into this kind of lock benefit existing code? I don't think I claimed that anywhere, as far as I think it makes absolutely no difference to existing code. > If existing code has to change to take advantage of the new features, said code could just as easily change the name of the lock it was using. Yes? From ncoghlan at gmail.com Tue Nov 11 13:21:12 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 11 Nov 2014 22:21:12 +1000 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: <20141106224537.4fc55146@fsol> References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <5457F5E9.8060205@canterbury.ac.nz> <20141106101525.GC3597@ando.pearwood.info> <20141106224537.4fc55146@fsol> Message-ID: On 7 November 2014 07:45, Antoine Pitrou wrote: > On Thu, 6 Nov 2014 10:54:51 -0800 > Guido van Rossum wrote: > > > > If I had had the right foresight, I would have made it an error to > > terminate a generator with a StopIteration, probably by raising another > > exception chained to the StopIteration (so the traceback shows the place > > where the StopIteration escaped). > > > > The question at hand is if we can fix this post-hoc, using clever tricks > > and (of course) a deprecation period. > > Is there any point in fixing it? Who relies on such borderline cases? > It's not about people relying on the current behaviour (it can't be, since we're talking about *changing* that behaviour), it's about "Errors should never pass silently". That is, the problematic cases that (at least arguably) may be worth fixing are those where: 1. StopIteration escapes from an expression (Error!) 2. Instead of causing a traceback, it terminates a containing generator (Passing silently!) As asyncio coroutines become more popular, I predict some serious head scratching from StopIteration escaping an asynchronous operation and getting thrown into a coroutine, which then terminates with a "return None" rather than propagating the exception as you might otherwise expect. The problem with this particular style of bug is that the only trace it leaves is a generator iterator that terminates earlier than expected - there's no traceback, log message, or any other indication of where something strange may be happening. Consider the following, from the original post in the thread: def izip(*args): iters = [iter(obj) for obj in args] while True: yield tuple([next(it) for it in iters]) The current behaviour of that construct is that, as soon as one of the iterators is empty: 1. next(it) throws StopIteration 2. the list comprehension unwinds the frame, and allows the exception to propagate 3. the generator iterator unwinds the frame, and allows the exception to propagate 4. the code invoking the iterator sees StopIteration and assumes iteration is complete If you switch to the generator expression version instead, the flow control becomes: 1. next(it) throws StopIteration 2. the generator expression unwinds the frame, and allows the exception to propagate 3. the iteration inside the tuple constructor sees StopIteration and halts 4. the generator iterator never terminates In that code, "next(it)" is a flow control operation akin to break (it terminates the nearest enclosing generator iterator, just as break terminates the nearest enclosing loop), but it's incredibly unclear that this is the case - there's no local indication that it may raise StopIteration, you need to "just know" that raising StopIteration is a possibility. Guido's suggestion is to consider looking for a viable way to break the equivalence between "return" and "raise StopIteration" in generator iterators - that way, the only way for the above code to work would be through a more explicit version that clearly tracks the flow control. Option 1 would be to assume we use a new exception, and are OK with folks catching it explicitly from __future__ import explicit_generator_return def izip(*args): iters = [iter(obj) for obj in args] while True: try: t = tuple(next(it) for it in iters) except UncaughtStopIteration: return # One of the iterators has been exhausted yield t Option 2 would be to assume the new exception is something generic like RuntimeError, requiring the inner loop to be converted to statement form: def izip(*args): iters = [iter(obj) for obj in args] while True: entry = [] for it in iters: try: item = next(it) except StopIteration: return # One of the iterators has been exhausted entry.append(item) yield tuple(entry) With option 2, you can also still rely on the fact that list comprehensions don't create a generator frame: def izip(*args): iters = [iter(obj) for obj in args] while True: try: entry = [next(it) for it in iters] except StopIteration: return # One of the iterators has been exhausted yield tuple(entry) The upside of the option 2 spellings is they'll work on all currently supported versions of Python, while the downside is the extra object construction they have to do if you want to yield something other than a list. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Nov 11 13:32:29 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 11 Nov 2014 22:32:29 +1000 Subject: [Python-ideas] Store shared/locked state inside of the lock object In-Reply-To: References: <884DB226-A3EC-4A89-8668-8C2B835FBC4A@masklinn.net> <20141108160425.1ffbd1f2@fsol> Message-ID: On 9 November 2014 04:42, Masklinn wrote: > On 2014-11-08, at 16:04 , Antoine Pitrou wrote: > > On Sat, 8 Nov 2014 14:05:07 +0100 > > Masklinn wrote: > >> > >> A fairly small technical change I've been considering to improve this > >> situation is to store the state-set inside the lock object, and only > >> yield it through the context manager: that the state-set is protected > >> by a lock is made obvious, and so is the relation between lock and > >> state-set. I was delighted to discover that Rust's sync::Mutex[1] and > >> sync::RWLock[2] follow a very similar strategy of owning the state-set > >> > >> It's not a panacea, it doesn't fix issues of lock acquisition ordering > >> for instance, but I think it would go a fairly long way towards making > >> correct use of locks easier in Python. > >> > >> The basic changes would be: > >> * threading.Lock and threading.RLock would now take an optional > >> `data` parameter > >> * the parameter would be stored internally and not directly accessible > >> by users of the lock > >> * that parameter would be returned by __enter__ and provided to the > >> current "owner" of the lock > > > > For clarity it should probably be a separate class (or set of classes), > > e.g. DataLock. > > On the one hand this'd allow completely ignoring backwards-compatibility > issues wrt acquire() which is nice, on the other hand it would double > the number of lock types and introduce redundancy as DataLock would be > pretty much a strict superset of Lock, which is why I thought extending > Lock made sense. > Merging it into Lock would make Lock itself harder to learn and use, so the separate DataLock notion sounds better to me - it keeps the documentation separate, so folks that just want a basic Lock or RLock don't need to care that DataLock exists. It's also worth considering just always making DataLock recursive, and not worrying about the non-recursive variant. If you'd like to experiment with this as a 3rd party module, Graham Dumpleton's wrapt library makes it possible to write almost completely transparent proxies in pure Python: http://wrapt.readthedocs.org/en/latest/wrappers.html Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron3200 at gmail.com Tue Nov 11 17:38:24 2014 From: ron3200 at gmail.com (Ron Adam) Date: Tue, 11 Nov 2014 10:38:24 -0600 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <5457F5E9.8060205@canterbury.ac.nz> <20141106101525.GC3597@ando.pearwood.info> <20141106224537.4fc55146@fsol> Message-ID: On 11/11/2014 06:21 AM, Nick Coghlan wrote: > > Option 2 would be to assume the new exception is something generic like > RuntimeError, requiring the inner loop to be converted to statement form: > > > def izip(*args): > iters = [iter(obj) for obj in args] > while True: > entry = [] > for it in iters: > try: > item = next(it) > except StopIteration: > return # One of the iterators has been exhausted > entry.append(item) > yield tuple(entry) When I was experimenting with this earlier, I needed the try-except to catch the StopIteration exception in order to do a "break". Which gave me the current behaviour of the generator expression being discussed. Replacing break with return, as above, gave the expected behaviour, but also just removing the try-except and letting the StopIteration propagate out, worked as well. That is, StopIteration(None) is equivalent to "return None" in the context above. Can you point me to the source file that implements generator expression byte code or C code? I wanted to look at that to see what was actually going on, but it looks like it may be a combination of a regular generator with a condition at some point to handle it slightly different. Cheers, Ron From greg.ewing at canterbury.ac.nz Tue Nov 11 21:49:00 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 12 Nov 2014 09:49:00 +1300 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <5457F5E9.8060205@canterbury.ac.nz> <20141106101525.GC3597@ando.pearwood.info> <20141106224537.4fc55146@fsol> Message-ID: <5462763C.9030706@canterbury.ac.nz> Ron Adam wrote: > Can you point me to the source file that implements generator expression > byte code or C code? I wanted to look at that to see what was actually > going on, but it looks like it may be a combination of a regular > generator with a condition at some point to handle it slightly different. There's not much difference between a generator expression and an ordinary generator. A generator expression gets compiled into a call of an anonymous generator function. >>> from dis import dis >>> def f(): ... return list(x for x in (1, 2, 3)) ... 2 0 LOAD_GLOBAL 0 (list) 3 LOAD_CONST 1 ( at 0x2a1d58, file "", line 2>) 6 MAKE_FUNCTION 0 9 LOAD_CONST 5 ((1, 2, 3)) 12 GET_ITER 13 CALL_FUNCTION 1 16 CALL_FUNCTION 1 19 RETURN_VALUE >>> dis(f.__code__.co_consts[1]) 2 0 LOAD_FAST 0 (.0) >> 3 FOR_ITER 11 (to 17) 6 STORE_FAST 1 (x) 9 LOAD_FAST 1 (x) 12 YIELD_VALUE 13 POP_TOP 14 JUMP_ABSOLUTE 3 >> 17 LOAD_CONST 0 (None) 20 RETURN_VALUE -- Greg From stefano.borini at ferrara.linux.it Tue Nov 11 22:51:24 2014 From: stefano.borini at ferrara.linux.it (Stefano Borini) Date: Tue, 11 Nov 2014 22:51:24 +0100 Subject: [Python-ideas] hardware stdlib package (Was: Re: including psutil in the standard library?) In-Reply-To: References: <542319B1.2090606@ferrara.linux.it> <59DEA399-5915-4562-8042-1372CF9D4353@yahoo.com> Message-ID: <546284DC.1050104@ferrara.linux.it> On 11/6/14 7:59 PM, Giampaolo Rodola' wrote: > - psutil.disk_partitions() > - psutil.net_connections() Actually, I was thinking that some of these should be part of a standard library "hardware" module, where you can inquire about the underlying hardware in a platform independent way. I checked around and I was unable to find a definitive package for this, which is a pity, because I consider it an important functionality that should definitely be a battery included. What I have found is, however, plenty of sparse code and packages trying to address this very need. To throw an idea, maybe it would be a good idea to aggregate all this code in a "hardware" package, stabilize it, and then include it as part of the standard library. I created a tentative github https://github.com/stefanoborini/hardware I'll scout around to see how to attack the problem, but maybe the ML has some important feedback I am definitely interested in. Stefano From ncoghlan at gmail.com Wed Nov 12 02:53:29 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 12 Nov 2014 11:53:29 +1000 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <5457F5E9.8060205@canterbury.ac.nz> <20141106101525.GC3597@ando.pearwood.info> <20141106224537.4fc55146@fsol> Message-ID: On 12 November 2014 02:38, Ron Adam wrote: > > > Can you point me to the source file that implements generator expression > byte code or C code? I wanted to look at that to see what was actually > going on, but it looks like it may be a combination of a regular generator > with a condition at some point to handle it slightly different. > When a StopIteration escapes from a generator frame, it's currently just propagated without modification. When the generator frame *returns*, the generator object converts that to raising StopIteration: https://hg.python.org/cpython/file/30a6c74ad87f/Objects/genobject.c#l117 An implementation of Guido's __future__ import idea would likely involve setting a flag on generator iterators when they're instantiated to say whether or not to intercept and convert StopIteration instances that escape from the generator frame. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Thu Nov 13 19:18:42 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 13 Nov 2014 10:18:42 -0800 Subject: [Python-ideas] A way to tell the gc it's OK to delete an object with a __del__ Message-ID: I've been poking into reference counting, circular references, etc, trying to understand how the garbage collector works. In practice, I'm trying to figure out a way to get a couple modules I work with not to leak memory: one is an extension to wxPython that I wrote, that creates a circular reference involving wx objects -- these, as is often the case with wrappers, have a custom __del__ method. The other is the python netcdf4 lib -- which has a circular reference built in, so that Datasets (which map to a file on disk) know about Variables (which map to data arrays within the file) and vice versa. We tried using weak references here, but it turns out that some users like to work with Variables without a reference to the Dataset, so that didn't work. Anyway, in my poking around, I think I understand that the gc won't delete "unreachable" objects if the have a custom __del__ method, because __del__ methods often handle resources that may depend on other objects, and bad (really bad) things can happen if they are deleted out of order, and the gc has no idea what order to delete. However -- some __del__ methods are perfectly safe regardless of delete order. So it would be nice to be able to somehow let the gc know that a particular object is safe to delete at any time. Is there a technical reason this can't be done? It seems something as simple as a __delete_in_any_order__ attribute would do it. Yes, this is dangerous, but authors of such extension objects need to know what they are doing anyway. Is there a technical complication I'm not thinking of? It seems if we could pull this off, we could eliminate one lasting ugly wart in python memory management. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From amauryfa at gmail.com Thu Nov 13 19:48:15 2014 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Thu, 13 Nov 2014 19:48:15 +0100 Subject: [Python-ideas] A way to tell the gc it's OK to delete an object with a __del__ In-Reply-To: References: Message-ID: 2014-11-13 19:18 GMT+01:00 Chris Barker : > I've been poking into reference counting, circular references, etc, trying > to understand how the garbage collector works. > > In practice, I'm trying to figure out a way to get a couple modules I work > with not to leak memory: one is an extension to wxPython that I wrote, that > creates a circular reference involving wx objects -- these, as is often the > case with wrappers, have a custom __del__ method. > > The other is the python netcdf4 lib -- which has a circular reference > built in, so that Datasets (which map to a file on disk) know about > Variables (which map to data arrays within the file) and vice versa. We > tried using weak references here, but it turns out that some users like to > work with Variables without a reference to the Dataset, so that didn't work. > > Anyway, in my poking around, I think I understand that the gc won't delete > "unreachable" objects if the have a custom __del__ method, because __del__ > methods often handle resources that may depend on other objects, and bad > (really bad) things can happen if they are deleted out of order, and the gc > has no idea what order to delete. > > However -- some __del__ methods are perfectly safe regardless of delete > order. > > So it would be nice to be able to somehow let the gc know that a > particular object is safe to delete at any time. > > Is there a technical reason this can't be done? It seems something as > simple as a __delete_in_any_order__ attribute would do it. Yes, this is > dangerous, but authors of such extension objects need to know what they are > doing anyway. > > Is there a technical complication I'm not thinking of? > > It seems if we could pull this off, we could eliminate one lasting ugly > wart in python memory management. > Did you try Python 3.4? https://docs.python.org/3/whatsnew/3.4.html#pep-442-safe-object-finalization -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Thu Nov 13 19:50:13 2014 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 13 Nov 2014 18:50:13 +0000 Subject: [Python-ideas] A way to tell the gc it's OK to delete an object with a __del__ In-Reply-To: References: Message-ID: <5464FD65.9070507@mrabarnett.plus.com> On 2014-11-13 18:18, Chris Barker wrote: > I've been poking into reference counting, circular references, etc, > trying to understand how the garbage collector works. > > In practice, I'm trying to figure out a way to get a couple modules I > work with not to leak memory: one is an extension to wxPython that I > wrote, that creates a circular reference involving wx objects -- these, > as is often the case with wrappers, have a custom __del__ method. > > The other is the python netcdf4 lib -- which has a circular reference > built in, so that Datasets (which map to a file on disk) know about > Variables (which map to data arrays within the file) and vice versa. We > tried using weak references here, but it turns out that some users like > to work with Variables without a reference to the Dataset, so that > didn't work. > > Anyway, in my poking around, I think I understand that the gc won't > delete "unreachable" objects if the have a custom __del__ method, > because __del__ methods often handle resources that may depend on other > objects, and bad (really bad) things can happen if they are deleted out > of order, and the gc has no idea what order to delete. > > However -- some __del__ methods are perfectly safe regardless of delete > order. > > So it would be nice to be able to somehow let the gc know that a > particular object is safe to delete at any time. > > Is there a technical reason this can't be done? It seems something as > simple as a __delete_in_any_order__ attribute would do it. Yes, this is > dangerous, but authors of such extension objects need to know what they > are doing anyway. > > Is there a technical complication I'm not thinking of? > > It seems if we could pull this off, we could eliminate one lasting ugly > wart in python memory management. > There's some code in this that you might find useful: http://permalink.gmane.org/gmane.comp.python.ideas/20334 From guido at python.org Thu Nov 13 21:04:52 2014 From: guido at python.org (Guido van Rossum) Date: Thu, 13 Nov 2014 12:04:52 -0800 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <5457F5E9.8060205@canterbury.ac.nz> <20141106101525.GC3597@ando.pearwood.info> <20141106224537.4fc55146@fsol> Message-ID: Sorry for my radio silence in this thread -- it was a combination of a conference and getting sick afterwards. IIUC Nick prefers reusing an existing exception over inventing a new one (to mean that a StopIteration was about to escape from a generator frame). The motivation is that this encourages catching the StopIteration at the source rather than relying on the new exception, thereby encouraging code that also works with previous versions. I like this. Nobody has suggested anything other than RuntimeError so let's go with that. I also like the idea of a __future__ import to request the new behavior in Python 3.5, and a warning if the error condition is detected while the new behavior isn't requested. I assume the __future__ import applies to generator function definitions, not calls. (But someone should reason this through before we commit.) Do we need a PEP for this or can we just go ahead? I haven't heard any pushback on the basic premise that this is a problem we'd like to fix. PS. If we decide not to go ahead with this, there's a small change to the semantics of "return" in a generator that might allow asyncio to distinguish between an intended return statement in the generator and an accidentally escaping StopIteration -- the return case should use a newly defined subclass of StopIteration. asyncio's _step() function can then tell the two situations apart easily. On Tue, Nov 11, 2014 at 4:21 AM, Nick Coghlan wrote: > On 7 November 2014 07:45, Antoine Pitrou wrote: > >> On Thu, 6 Nov 2014 10:54:51 -0800 >> Guido van Rossum wrote: >> > >> > If I had had the right foresight, I would have made it an error to >> > terminate a generator with a StopIteration, probably by raising another >> > exception chained to the StopIteration (so the traceback shows the place >> > where the StopIteration escaped). >> > >> > The question at hand is if we can fix this post-hoc, using clever tricks >> > and (of course) a deprecation period. >> >> Is there any point in fixing it? Who relies on such borderline cases? >> > > It's not about people relying on the current behaviour (it can't be, since > we're talking about *changing* that behaviour), it's about "Errors should > never pass silently". That is, the problematic cases that (at least > arguably) may be worth fixing are those where: > > 1. StopIteration escapes from an expression (Error!) > 2. Instead of causing a traceback, it terminates a containing generator > (Passing silently!) > > As asyncio coroutines become more popular, I predict some serious head > scratching from StopIteration escaping an asynchronous operation and > getting thrown into a coroutine, which then terminates with a "return None" > rather than propagating the exception as you might otherwise expect. > > The problem with this particular style of bug is that the only trace it > leaves is a generator iterator that terminates earlier than expected - > there's no traceback, log message, or any other indication of where > something strange may be happening. > > Consider the following, from the original post in the thread: > > def izip(*args): > iters = [iter(obj) for obj in args] > while True: > yield tuple([next(it) for it in iters]) > > The current behaviour of that construct is that, as soon as one of the > iterators is empty: > > 1. next(it) throws StopIteration > 2. the list comprehension unwinds the frame, and allows the exception to > propagate > 3. the generator iterator unwinds the frame, and allows the exception to > propagate > 4. the code invoking the iterator sees StopIteration and assumes iteration > is complete > > If you switch to the generator expression version instead, the flow > control becomes: > > 1. next(it) throws StopIteration > 2. the generator expression unwinds the frame, and allows the exception to > propagate > 3. the iteration inside the tuple constructor sees StopIteration and halts > 4. the generator iterator never terminates > > In that code, "next(it)" is a flow control operation akin to break (it > terminates the nearest enclosing generator iterator, just as break > terminates the nearest enclosing loop), but it's incredibly unclear that > this is the case - there's no local indication that it may raise > StopIteration, you need to "just know" that raising StopIteration is a > possibility. > > Guido's suggestion is to consider looking for a viable way to break the > equivalence between "return" and "raise StopIteration" in generator > iterators - that way, the only way for the above code to work would be > through a more explicit version that clearly tracks the flow control. > > Option 1 would be to assume we use a new exception, and are OK with folks > catching it explicitly > > from __future__ import explicit_generator_return > def izip(*args): > iters = [iter(obj) for obj in args] > while True: > try: > t = tuple(next(it) for it in iters) > except UncaughtStopIteration: > return # One of the iterators has been exhausted > yield t > > Option 2 would be to assume the new exception is something generic like > RuntimeError, requiring the inner loop to be converted to statement form: > > > def izip(*args): > iters = [iter(obj) for obj in args] > while True: > entry = [] > for it in iters: > try: > item = next(it) > except StopIteration: > return # One of the iterators has been exhausted > entry.append(item) > yield tuple(entry) > > With option 2, you can also still rely on the fact that list > comprehensions don't create a generator frame: > > def izip(*args): > iters = [iter(obj) for obj in args] > while True: > try: > entry = [next(it) for it in iters] > except StopIteration: > return # One of the iterators has been exhausted > yield tuple(entry) > > The upside of the option 2 spellings is they'll work on all currently > supported versions of Python, while the downside is the extra object > construction they have to do if you want to yield something other than a > list. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothy.c.delaney at gmail.com Thu Nov 13 22:31:04 2014 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Fri, 14 Nov 2014 08:31:04 +1100 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <5457F5E9.8060205@canterbury.ac.nz> <20141106101525.GC3597@ando.pearwood.info> <20141106224537.4fc55146@fsol> Message-ID: On 14 November 2014 07:04, Guido van Rossum wrote: > > PS. If we decide not to go ahead with this, there's a small change to the > semantics of "return" in a generator that might allow asyncio to > distinguish between an intended return statement in the generator and an > accidentally escaping StopIteration -- the return case should use a newly > defined subclass of StopIteration. asyncio's _step() function can then tell > the two situations apart easily. > I've only been peripherally following this, but that sounds like a good idea whether or not the rest is implemented. Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Thu Nov 13 23:20:16 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 13 Nov 2014 14:20:16 -0800 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <5457F5E9.8060205@canterbury.ac.nz> <20141106101525.GC3597@ando.pearwood.info> <20141106224537.4fc55146@fsol> Message-ID: <54652EA0.20204@stoneleaf.us> On 11/13/2014 12:04 PM, Guido van Rossum wrote: > > Do we need a PEP for this or can we just go ahead? I haven't heard any pushback on the basic premise that this is a > problem we'd like to fix. Given the confusion about what the problem is, and the possible fixes, I think a short PEP would be in order just so everyone is on the same page. I don't fully understand everything discussed so far (not a big user of generators), so I'll only throw my hat in as a backup volunteer to write the PEP if no one one else is able to take the time. -- ~Ethan~ From chris.barker at noaa.gov Fri Nov 14 00:18:57 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 13 Nov 2014 15:18:57 -0800 Subject: [Python-ideas] A way to tell the gc it's OK to delete an object with a __del__ In-Reply-To: References: Message-ID: On Thu, Nov 13, 2014 at 10:48 AM, Amaury Forgeot d'Arc wrote: > So it would be nice to be able to somehow let the gc know that a >> particular object is safe to delete at any time. >> >> Did you try Python 3.4? > > https://docs.python.org/3/whatsnew/3.4.html#pep-442-safe-object-finalization > > > no, I haven't -- very cool! This may be the final push to get me to py3! (though not for wxPython quite yet, sadly...) I hadn't noticed that in any of the py3.4 announcements (nor had I looked hard), but defiantly didn't find in in any of my googling about this issue. Thanks, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Nov 14 00:52:21 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 14 Nov 2014 10:52:21 +1100 Subject: [Python-ideas] A way to tell the gc it's OK to delete an object with a __del__ In-Reply-To: References: Message-ID: On Fri, Nov 14, 2014 at 10:18 AM, Chris Barker wrote: > This may be the final push to get me to py3! (though not for wxPython quite > yet, sadly...) Well, if you're coming here with language improvement suggestions, you'll definitely want to be on Py3. There won't be any features added to Py2. :) ChrisA From chris.barker at noaa.gov Fri Nov 14 01:58:52 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 13 Nov 2014 16:58:52 -0800 Subject: [Python-ideas] A way to tell the gc it's OK to delete an object with a __del__ In-Reply-To: References: Message-ID: On Thu, Nov 13, 2014 at 3:52 PM, Chris Angelico wrote: > On Fri, Nov 14, 2014 at 10:18 AM, Chris Barker > wrote: > > This may be the final push to get me to py3! (though not for wxPython > quite > > yet, sadly...) > > Well, if you're coming here with language improvement suggestions, > you'll definitely want to be on Py3. There won't be any features added > to Py2. :) > yes, I'm fully aware of that -- still the the real kick to get me there -- maybe this is it. I figure a language improvement will, at best get into the next minor version -- then it will be years? before you can count on most people using that version -- so proposing ideas is a whole different ball of wax than what I do operationally. -CHB > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Fri Nov 14 00:20:46 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 13 Nov 2014 15:20:46 -0800 Subject: [Python-ideas] A way to tell the gc it's OK to delete an object with a __del__ In-Reply-To: <5464FD65.9070507@mrabarnett.plus.com> References: <5464FD65.9070507@mrabarnett.plus.com> Message-ID: On Thu, Nov 13, 2014 at 10:50 AM, MRAB wrote: > There's some code in this that you might find useful: > > http://permalink.gmane.org/gmane.comp.python.ideas/20334 thanks -- that does look kind of like what I was thinking I'd need to do -- but in the long run what the gc to do it for me (which maybe it does in py3.4) But maybe I'll patch the netCDF lib with something like that -- folks will be using py < 3.4 for a long time yet :-( -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Nov 14 02:25:36 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 14 Nov 2014 12:25:36 +1100 Subject: [Python-ideas] A way to tell the gc it's OK to delete an object with a __del__ In-Reply-To: References: Message-ID: On Fri, Nov 14, 2014 at 11:58 AM, Chris Barker wrote: > I figure a language improvement will, at best get into the next minor > version -- then it will be years? before you can count on most people using > that version -- so proposing ideas is a whole different ball of wax than > what I do operationally. The next minor version of Py3 is 3.5, which is currently scheduled for a first alpha in February. If the feature misses that, the next minor release will be I think about 18 months later, give or take. The next minor version of Py2 is 2.8 and will not be happening. http://legacy.python.org/dev/peps/pep-0404/ ChrisA From tjreedy at udel.edu Fri Nov 14 06:23:07 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 14 Nov 2014 00:23:07 -0500 Subject: [Python-ideas] A way to tell the gc it's OK to delete an object with a __del__ In-Reply-To: References: Message-ID: On 11/13/2014 8:25 PM, Chris Angelico wrote: > On Fri, Nov 14, 2014 at 11:58 AM, Chris Barker wrote: >> I figure a language improvement will, at best get into the next minor >> version -- then it will be years? before you can count on most people using >> that version -- so proposing ideas is a whole different ball of wax than >> what I do operationally. > > The next minor version of Py3 is 3.5, which is currently scheduled for > a first alpha in February. If the feature misses that, the next minor > release will be I think about 18 months later, give or take. Beta1, scheduled for late May, is the new feature deadline, though sooner is better. -- Terry Jan Reedy From ncoghlan at gmail.com Fri Nov 14 08:41:32 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 14 Nov 2014 17:41:32 +1000 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: <54652EA0.20204@stoneleaf.us> References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <5457F5E9.8060205@canterbury.ac.nz> <20141106101525.GC3597@ando.pearwood.info> <20141106224537.4fc55146@fsol> <54652EA0.20204@stoneleaf.us> Message-ID: On 14 November 2014 08:20, Ethan Furman wrote: > On 11/13/2014 12:04 PM, Guido van Rossum wrote: > > > > Do we need a PEP for this or can we just go ahead? I haven't heard any > pushback on the basic premise that this is a > > problem we'd like to fix. > > Given the confusion about what the problem is, and the possible fixes, I > think a short PEP would be in order just so > everyone is on the same page. > Agreed, I think it's worth having an explanatory PEP at least for documentation purposes. It also makes it easier to reference from the 3.5 What's New, as there may be some code that's relying on the current behaviour that may need adjusting to use a custom exception type rather than StopIteration (or else refactoring to avoid crossing a generator frame boundary). Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Nov 14 08:45:55 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 14 Nov 2014 17:45:55 +1000 Subject: [Python-ideas] A way to tell the gc it's OK to delete an object with a __del__ In-Reply-To: References: <5464FD65.9070507@mrabarnett.plus.com> Message-ID: On 14 November 2014 09:20, Chris Barker wrote: > On Thu, Nov 13, 2014 at 10:50 AM, MRAB wrote: > >> There's some code in this that you might find useful: >> >> http://permalink.gmane.org/gmane.comp.python.ideas/20334 > > > thanks -- that does look kind of like what I was thinking I'd need to do > -- but in the long run what the gc to do it for me (which maybe it does in > py3.4) > > But maybe I'll patch the netCDF lib with something like that -- folks will > be using py < 3.4 for a long time yet :-( > PyPy's GC design is (very) different, and allows finalisation of reference cycles involving __del__ even in Python 2. That path naturally comes with its own C extension compatibility challenges, though. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Nov 14 18:30:54 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 14 Nov 2014 09:30:54 -0800 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <5457F5E9.8060205@canterbury.ac.nz> <20141106101525.GC3597@ando.pearwood.info> <20141106224537.4fc55146@fsol> <54652EA0.20204@stoneleaf.us> Message-ID: Someone please volunteer to write this PEP. I can review, but I need to save my Python time to work on the type hinting PEP. On Thu, Nov 13, 2014 at 11:41 PM, Nick Coghlan wrote: > On 14 November 2014 08:20, Ethan Furman wrote: > >> On 11/13/2014 12:04 PM, Guido van Rossum wrote: >> > >> > Do we need a PEP for this or can we just go ahead? I haven't heard any >> pushback on the basic premise that this is a >> > problem we'd like to fix. >> >> Given the confusion about what the problem is, and the possible fixes, I >> think a short PEP would be in order just so >> everyone is on the same page. >> > > Agreed, I think it's worth having an explanatory PEP at least for > documentation purposes. It also makes it easier to reference from the 3.5 > What's New, as there may be some code that's relying on the current > behaviour that may need adjusting to use a custom exception type rather > than StopIteration (or else refactoring to avoid crossing a generator frame > boundary). > > Regards, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Fri Nov 14 19:01:58 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 14 Nov 2014 10:01:58 -0800 Subject: [Python-ideas] A way to tell the gc it's OK to delete an object with a __del__ In-Reply-To: References: <5464FD65.9070507@mrabarnett.plus.com> Message-ID: On Thu, Nov 13, 2014 at 11:45 PM, Nick Coghlan wrote: > PyPy's GC design is (very) different, and allows finalisation of reference > cycles involving __del__ even in Python 2. That path naturally comes with > its own C extension compatibility challenges, though. > yup -- as does IronPython and Jython. But particularly for the scientific stack, switching to PyPy is a LOT harder than switching to 3.4 ! I certainly wouldn't do it for the garbage collection ;-) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From danilopeixoto5 at hotmail.com Fri Nov 14 23:54:38 2014 From: danilopeixoto5 at hotmail.com (Danilo Peixoto) Date: Fri, 14 Nov 2014 22:54:38 +0000 Subject: [Python-ideas] Python IDE (New IDE for Python scripting) Message-ID: An alternative IDE with a new dark and modern interface. http://s27.postimg.org/r4cjxf00z/Python_IDE.png (Concept) Concept shows the Python IDE functionality to organize projects and files in own software and quick information on the status bar. Some additional features: AutocompleteFind and replaceIncrease and decrease line indentand more... -------------- next part -------------- An HTML attachment was scrubbed... URL: From rajshorya at gmail.com Fri Nov 14 23:59:14 2014 From: rajshorya at gmail.com (Shorya Raj) Date: Sat, 15 Nov 2014 11:59:14 +1300 Subject: [Python-ideas] Python IDE (New IDE for Python scripting) In-Reply-To: References: Message-ID: Seems like it would "borrow" things from the PyCharm IDE. Looks quite nice though. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Sat Nov 15 00:43:41 2014 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Fri, 14 Nov 2014 17:43:41 -0600 Subject: [Python-ideas] Python IDE (New IDE for Python scripting) In-Reply-To: References: Message-ID: Wow...that looks awesome! On Fri, Nov 14, 2014 at 4:54 PM, Danilo Peixoto wrote: > An alternative IDE with a new dark and modern interface. > > http://s27.postimg.org/r4cjxf00z/Python_IDE.png (Concept) > > Concept shows the Python IDE functionality to organize projects and files > in own software and quick information on the status bar. > Some additional features: > > > - Autocomplete > - Find and replace > - Increase and decrease line indent > - and more... > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." Personal reality distortion fields are immune to contradictory evidence. - srean Check out my website: http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat Nov 15 02:17:12 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 15 Nov 2014 12:17:12 +1100 Subject: [Python-ideas] Python IDE (New IDE for Python scripting) In-Reply-To: References: Message-ID: <20141115011712.GJ2748@ando.pearwood.info> On Fri, Nov 14, 2014 at 10:54:38PM +0000, Danilo Peixoto wrote: > An alternative IDE with a new dark and modern interface. What about it? Do you have a proposal for it? This list is for ideas for the future development of Python the programming language and the standard library. How does your idea relate to the language or stdlib? Instead of posting a screen shot, do you have a repo where we can look at the code? -- Steven From rosuav at gmail.com Sat Nov 15 02:11:39 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 15 Nov 2014 12:11:39 +1100 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <5457F5E9.8060205@canterbury.ac.nz> <20141106101525.GC3597@ando.pearwood.info> <20141106224537.4fc55146@fsol> <54652EA0.20204@stoneleaf.us> Message-ID: On Sat, Nov 15, 2014 at 4:30 AM, Guido van Rossum wrote: > Someone please volunteer to write this PEP. I can review, but I need to save > my Python time to work on the type hinting PEP. After the stunning success :) of my last such endeavour (exception-catching expressions), I guess I could put my hand up for this one, if there's no better volunteer forthcoming. ChrisA From guido at python.org Sat Nov 15 03:18:01 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 14 Nov 2014 18:18:01 -0800 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <5457F5E9.8060205@canterbury.ac.nz> <20141106101525.GC3597@ando.pearwood.info> <20141106224537.4fc55146@fsol> <54652EA0.20204@stoneleaf.us> Message-ID: Sold! Thank you. On Fri, Nov 14, 2014 at 5:11 PM, Chris Angelico wrote: > On Sat, Nov 15, 2014 at 4:30 AM, Guido van Rossum > wrote: > > Someone please volunteer to write this PEP. I can review, but I need to > save > > my Python time to work on the type hinting PEP. > > After the stunning success :) of my last such endeavour > (exception-catching expressions), I guess I could put my hand up for > this one, if there's no better volunteer forthcoming. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From Steve.Dower at microsoft.com Sat Nov 15 06:29:40 2014 From: Steve.Dower at microsoft.com (Steve Dower) Date: Sat, 15 Nov 2014 05:29:40 +0000 Subject: [Python-ideas] Python IDE (New IDE for Python scripting) In-Reply-To: References: Message-ID: <7689848e2b034a4c9bcd01f05b78831b@DM2PR0301MB0734.namprd03.prod.outlook.com> There are already many alternatives to Idle, which I assume you're wanting to replace, and most are freely available and/or cross platform. This is a more crowded market than you may think, and there's no reason why the standard Python distribution is the right place for it (even Idle feels like unnecessary these days, but it wins because of history). Cheers, Steve Top-posted from my Windows Phone ________________________________ From: Danilo Peixoto Sent: ?11/?14/?2014 14:55 To: python-ideas at python.org Subject: [Python-ideas] Python IDE (New IDE for Python scripting) An alternative IDE with a new dark and modern interface. http://s27.postimg.org/r4cjxf00z/Python_IDE.png (Concept) Concept shows the Python IDE functionality to organize projects and files in own software and quick information on the status bar. Some additional features: * Autocomplete * Find and replace * Increase and decrease line indent * and more... -------------- next part -------------- An HTML attachment was scrubbed... URL: From gokoproject at gmail.com Sat Nov 15 07:12:58 2014 From: gokoproject at gmail.com (John Wong) Date: Sat, 15 Nov 2014 01:12:58 -0500 Subject: [Python-ideas] Python IDE (New IDE for Python scripting) In-Reply-To: <20141115011712.GJ2748@ando.pearwood.info> References: <20141115011712.GJ2748@ando.pearwood.info> Message-ID: On Fri, Nov 14, 2014 at 8:17 PM, Steven D'Aprano wrote: > > This list is for ideas for the future development of Python the > programming language and the standard library. How does your idea relate > to the language or stdlib? > https://docs.python.org/2/library/idle.html My guess is he wants to replace IDLE with a potentially shinier IDE out of the box, but obviously you made a good point proposal/code would be really nice. I suggest Danilo have a look at http://legacy.python.org/dev/peps/pep-0434/ and http://lwn.net/Articles/546188/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sat Nov 15 07:12:42 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 15 Nov 2014 01:12:42 -0500 Subject: [Python-ideas] Python IDE (New IDE for Python scripting) In-Reply-To: References: Message-ID: On 11/14/2014 5:54 PM, Danilo Peixoto wrote: > An alternative IDE with a new dark and modern interface. Q. When does a recycled decades-old design become new?' A. When an industry becomes part of the fashion industry. Except for the frame provided by the OS via tkinter, an Idle user can switch to white and colors on black, though it would be a bit tedious switching all 13 foreground/background pairs. Should a built-in retro-black theme be added? There is actually an issue (7949) about Idle not working well with dark GTK/KDE color schemes, and I believe such a color scheme would help. > http://s27.postimg.org/r4cjxf00z/Python_IDE.png (Concept) > > Concept shows the Python IDE functionality to organize projects and > files in own software and quick information on the status bar. Displaying the manual entry for a selected function is an interesting idea, though that space is too small. A popup would be better, I think. A function signature, however, would usually fit nicely. So Idle could put signatures on the currently mostly unused status line in addition to an ephemeral calltip popup box (which also includes part of docstrings). It could also be less fussy about cursor positioning before allowing calltip to enable access to the information. Thank you for provoking 4 concrete ideas for Idle improvement, even though perhaps not your intention. > Some additional features: > > * Autocomplete > * Find and replace > * Increase and decrease line indent > * and more... Already in Idle, though improvements are needed. -- Terry Jan Reedy From rosuav at gmail.com Sat Nov 15 07:23:21 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 15 Nov 2014 17:23:21 +1100 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <2A2674BF-394F-406B-8012-0FC7928F578E@yahoo.com> <5457F5E9.8060205@canterbury.ac.nz> <20141106101525.GC3597@ando.pearwood.info> <20141106224537.4fc55146@fsol> <54652EA0.20204@stoneleaf.us> Message-ID: On Fri, Nov 14, 2014 at 6:41 PM, Nick Coghlan wrote: > Agreed, I think it's worth having an explanatory PEP at least for > documentation purposes. Draft has been sent to the PEP editors, but if anyone wants to preview it, it's currently here: https://raw.githubusercontent.com/Rosuav/GenStopIter/master/pep-xxx.txt ChrisA From steve at pearwood.info Sat Nov 15 07:24:53 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 15 Nov 2014 17:24:53 +1100 Subject: [Python-ideas] Python IDE (New IDE for Python scripting) In-Reply-To: References: <20141115011712.GJ2748@ando.pearwood.info> Message-ID: <20141115062453.GK2748@ando.pearwood.info> On Sat, Nov 15, 2014 at 01:12:58AM -0500, John Wong wrote: > On Fri, Nov 14, 2014 at 8:17 PM, Steven D'Aprano > wrote: > > > > > This list is for ideas for the future development of Python the > > programming language and the standard library. How does your idea relate > > to the language or stdlib? > > > > https://docs.python.org/2/library/idle.html > My guess is he wants to replace IDLE with a potentially shinier IDE .....^^^^^ Yes, and that was my second guess too. (I am aware of IDLE :-) My first guess was that the OP was spamming us trying to get hits for his pet project. In either case, if there is only a mock-up of what this new and improved IDE will look like, but no actual code yet, then we're just wasting time discussing it. -- Steven From steve at pearwood.info Sat Nov 15 07:48:48 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 15 Nov 2014 17:48:48 +1100 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <5457F5E9.8060205@canterbury.ac.nz> <20141106101525.GC3597@ando.pearwood.info> <20141106224537.4fc55146@fsol> Message-ID: <20141115064848.GL2748@ando.pearwood.info> On Thu, Nov 13, 2014 at 12:04:52PM -0800, Guido van Rossum wrote: > Do we need a PEP for this or can we just go ahead? I haven't heard any > pushback on the basic premise that this is a problem we'd like to fix. *puts hand up* I'm not convinced that this is *a problem to be fixed*. At worst it is a "gotcha" to be aware of. The current behaviour is simple to understand: raising StopIteration halts the generator, end of story. I'm still not sure that I understand what the proposed fix is (a PEP will be good to explain that), but if I have understood it correctly, it turns a simple concept like "StopIteration halts the generator" into something more complicated: some StopIterations will halt the generator, others will be chained to a new, unrelated exception. Judging by the complete lack of questions about this on the tutor and python-list mailing lists, the slight difference in behaviour between generator expressions and comprehensions is not an issue in practice. I've seen people ask about the leakage of variables from comprehensions; I've never seen people ask about the different treatment of StopIteration. I have, however, seen people *rely* on that different treatment. E.g. to implement a short-circuiting generator expression that exits when a condition is reached: (expr for x in sequence if cond or stop()) where stop() raises StopIteration and halts the generator. If this change goes ahead, it will break code that does this. Having generator expressions and comprehensions behave exactly the same leads to the question, why do we have comprehensions? I guess the answer is "historical reasons", but from a pragmatic point of view being able to choose between [expr for x in sequence] list(expr for x in sequence) depending on how you want StopIteration to be treated may be useful. -- Steven From rosuav at gmail.com Sat Nov 15 08:06:27 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 15 Nov 2014 18:06:27 +1100 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: <20141115064848.GL2748@ando.pearwood.info> References: <5457F5E9.8060205@canterbury.ac.nz> <20141106101525.GC3597@ando.pearwood.info> <20141106224537.4fc55146@fsol> <20141115064848.GL2748@ando.pearwood.info> Message-ID: On Sat, Nov 15, 2014 at 5:48 PM, Steven D'Aprano wrote: > I'm still not sure that I understand > what the proposed fix is (a PEP will be good to explain that) Draft PEP exists, and now has your concerns incorporated. https://raw.githubusercontent.com/Rosuav/GenStopIter/master/pep-xxx.txt ChrisA From rosuav at gmail.com Sat Nov 15 10:29:47 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 15 Nov 2014 20:29:47 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators Message-ID: PEP: 479 Title: Change StopIteration handling inside generators Version: $Revision$ Last-Modified: $Date$ Author: Chris Angelico Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 15-Nov-2014 Python-Version: 3.5 Post-History: 15-Nov-2014 Abstract ======== This PEP proposes a semantic change to ``StopIteration`` when raised inside a generator, unifying the behaviour of list comprehensions and generator expressions somewhat. Rationale ========= The interaction of generators and ``StopIteration`` is currently somewhat surprising, and can conceal obscure bugs. An unexpected exception should not result in subtly altered behaviour, but should cause a noisy and easily-debugged traceback. Currently, ``StopIteration`` can be absorbed by the generator construct. Proposal ======== If a ``StopIteration`` is about to bubble out of a generator frame, it is replaced with some other exception (maybe ``RuntimeError``, maybe a new custom ``Exception`` subclass, but *not* deriving from ``StopIteration``) which causes the ``next()`` call (which invoked the generator) to fail, passing that exception out. From then on it's just like any old exception. [3]_ Consequences to existing code ============================= This change will affect existing code that depends on ``StopIteration`` bubbling up. The pure Python reference implementation of ``groupby`` [1]_ currently has comments "Exit on ``StopIteration``" where it is expected that the exception will propagate and then be handled. This will be unusual, but not unknown, and such constructs will fail. (Nick Coghlan comments: """If you wanted to factor out a helper function that terminated the generator you'd have to do "return yield from helper()" rather than just "helper()".""") As this can break code, it is proposed to utilize the ``__future__`` mechanism to introduce this, finally making it standard in Python 3.6 or 3.7. Alternate proposals =================== Supplying a specific exception to raise on return ------------------------------------------------- Nick Coghlan suggested a means of providing a specific ``StopIteration`` instance to the generator; if any other instance of ``StopIteration`` is raised, it is an error, but if that particular one is raised, the generator has properly completed. Making return-triggered StopIterations obvious ---------------------------------------------- For certain situations, a simpler and fully backward-compatible solution may be sufficient: when a generator returns, instead of raising ``StopIteration``, it raises a specific subclass of ``StopIteration`` which can then be detected. If it is not that subclass, it is an escaping exception rather than a return statement. Criticism ========= Unofficial and apocryphal statistics suggest that this is seldom, if ever, a problem. [4]_ Code does exist which relies on the current behaviour, and there is the concern that this would be unnecessary code churn to achieve little or no gain. References ========== .. [1] Initial mailing list comment (https://mail.python.org/pipermail/python-ideas/2014-November/029906.html) .. [2] Pure Python implementation of groupby (https://docs.python.org/3/library/itertools.html#itertools.groupby) .. [3] Proposal by GvR (https://mail.python.org/pipermail/python-ideas/2014-November/029953.html) .. [4] Response by Steven D'Aprano (https://mail.python.org/pipermail/python-ideas/2014-November/029994.html) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: From rosuav at gmail.com Sat Nov 15 13:57:34 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 15 Nov 2014 23:57:34 +1100 Subject: [Python-ideas] Change how Generator Expressions handle StopIteration In-Reply-To: References: <5457F5E9.8060205@canterbury.ac.nz> <20141106101525.GC3597@ando.pearwood.info> <20141106224537.4fc55146@fsol> <20141115064848.GL2748@ando.pearwood.info> Message-ID: On Sat, Nov 15, 2014 at 11:36 PM, Skip Montanaro wrote: > On Sat, Nov 15, 2014 at 1:06 AM, Chris Angelico wrote: >> >> Draft PEP exists, and now has your concerns incorporated. >> >> https://raw.githubusercontent.com/Rosuav/GenStopIter/master/pep-xxx.txt > > > Chris, > > 404 for me... (replying on-list as this will affect everyone now) It's now been allocated an actual number: https://raw.githubusercontent.com/Rosuav/GenStopIter/master/pep-0479.txt ChrisA From ncoghlan at gmail.com Sat Nov 15 15:13:38 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 16 Nov 2014 00:13:38 +1000 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: On 15 November 2014 19:29, Chris Angelico wrote: > PEP: 479 > Title: Change StopIteration handling inside generators > Version: $Revision$ > Last-Modified: $Date$ > Author: Chris Angelico > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 15-Nov-2014 > Python-Version: 3.5 > Post-History: 15-Nov-2014 > > > Abstract > ======== > > This PEP proposes a semantic change to ``StopIteration`` when raised > inside a generator, unifying the behaviour of list comprehensions and > generator expressions somewhat. > > > Rationale > ========= > > The interaction of generators and ``StopIteration`` is currently > somewhat surprising, and can conceal obscure bugs. An unexpected > exception should not result in subtly altered behaviour, but should > cause a noisy and easily-debugged traceback. Currently, > ``StopIteration`` can be absorbed by the generator construct. > Thanks for the write-up! Proposal > ======== > > If a ``StopIteration`` is about to bubble out of a generator frame, it > is replaced with some other exception (maybe ``RuntimeError``, maybe a > new custom ``Exception`` subclass, but *not* deriving from > ``StopIteration``) which causes the ``next()`` call (which invoked the > generator) to fail, passing that exception out. From then on it's > just like any old exception. [3]_ > [snip] > Alternate proposals > =================== > > Supplying a specific exception to raise on return > ------------------------------------------------- > > Nick Coghlan suggested a means of providing a specific > ``StopIteration`` instance to the generator; if any other instance of > ``StopIteration`` is raised, it is an error, but if that particular > one is raised, the generator has properly completed. > I think you can skip mentioning this particular idea in the PEP - I didn't like it even when I posted it, and both of Guido's ideas are much better :) > Making return-triggered StopIterations obvious > ---------------------------------------------- > > For certain situations, a simpler and fully backward-compatible > solution may be sufficient: when a generator returns, instead of > raising ``StopIteration``, it raises a specific subclass of > ``StopIteration`` which can then be detected. If it is not that > subclass, it is an escaping exception rather than a return statement. > There's an additional subtlety with this idea: if we add a new GeneratorReturn exception as a subclass of StopIteration, then generator iterators would likely also have to change to replace GeneratorReturn with a regular StopIteration (chaining appropriately via __cause__, and copying the return value across). >From the point of view of calling "next()" directly (rather than implicitly) this particular change makes it straightforward to distinguish between "the generator I called just finished" and "something inside the generator threw StopIteration". Due to the subclassing, implict next() invocations (e.g. in for loops, comprehensions, and container constructors) won't notice any difference. With such a change, we would actually likely modify the following code in contextlib._GeneratorContextManager.__exit__: try: self.gen.throw(exc_type, value, traceback) raise RuntimeError("generator didn't stop after throw()") except StopIteration as exc: # Generator suppressed the exception # unless it's a StopIteration instance we threw in return exc is not value except: if sys.exc_info()[1] is not value: raise To be the slightly more self-explanatory: try: self.gen.throw(type, value, traceback) raise RuntimeError("generator didn't stop after throw()") except GeneratorReturn: # Generator suppressed the exception return True except: if sys.exc_info()[1] is not value: raise The current proposal in the PEP actually doesn't let us simplify this contextlib code, but rather means we would have to make it more complicated to impedance match generator semantics with the context management protocol. To handle that change, we'd have to make the code something like the following (for clarity, I've assumed a new RuntimeError subclass, rather than RuntimeError itself): try: self.gen.throw(exc_type, value, traceback) raise RuntimeError("generator didn't stop after throw()") except StopIteration as exc: # Could becomes "return True" once the __future__ becomes the default return exc is not value except UnexpectedStopIteration as exc: if exc.__cause__ is not value: raise except: if sys.exc_info()[1] is not value: raise I definitely see value in adding a GeneratorReturn subclass to be able to tell the "returned" vs "raised StopIteration" cases apart from outside the generator (the current dance in contextlib only works because we have existing knowledge of the exact exception that was thrown in). I'm substantially less convinced of the benefit of changing generators to no longer suppress StopIteration. Yes, it's currently a rather odd corner case, but changing it *will* break code (at the very least, anyone using an old version of contextlib2, or who are otherwise relying on their own copy of contextlib rather than standard library one). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Nov 15 15:37:57 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 16 Nov 2014 01:37:57 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: On Sun, Nov 16, 2014 at 1:13 AM, Nick Coghlan wrote: > On 15 November 2014 19:29, Chris Angelico wrote: >> Nick Coghlan suggested a means of providing a specific >> ``StopIteration`` instance to the generator; if any other instance of >> ``StopIteration`` is raised, it is an error, but if that particular >> one is raised, the generator has properly completed. > > > I think you can skip mentioning this particular idea in the PEP - I didn't > like it even when I posted it, and both of Guido's ideas are much better :) Doesn't hurt to have some rejected alternates there :) >> For certain situations, a simpler and fully backward-compatible >> solution may be sufficient: when a generator returns, instead of >> raising ``StopIteration``, it raises a specific subclass of >> ``StopIteration`` which can then be detected. If it is not that >> subclass, it is an escaping exception rather than a return statement. > > There's an additional subtlety with this idea: if we add a new > GeneratorReturn exception as a subclass of StopIteration, then generator > iterators would likely also have to change to replace GeneratorReturn with a > regular StopIteration (chaining appropriately via __cause__, and copying the > return value across). Would have to do so automatically, meaning this is no simpler than the current proposal? Or would have to be always explicitly written to handle it? > I definitely see value in adding a GeneratorReturn subclass to be able to > tell the "returned" vs "raised StopIteration" cases apart from outside the > generator (the current dance in contextlib only works because we have > existing knowledge of the exact exception that was thrown in). I'm > substantially less convinced of the benefit of changing generators to no > longer suppress StopIteration. Yes, it's currently a rather odd corner case, > but changing it *will* break code (at the very least, anyone using an old > version of contextlib2, or who are otherwise relying on their own copy of > contextlib rather than standard library one). This is why it's proposed to use __future__ to protect it. If anyone's still using an old version of contextlib2 once 3.7 comes along, it'll break; but is there any reason to use Python 3.7 with a contextlib from elsewhere than its standard library? (I'm not familiar with contextlib2 or what it offers.) ChrisA From ncoghlan at gmail.com Sat Nov 15 16:21:36 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 16 Nov 2014 01:21:36 +1000 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: On 16 November 2014 00:37, Chris Angelico wrote: > On Sun, Nov 16, 2014 at 1:13 AM, Nick Coghlan wrote: > >> For certain situations, a simpler and fully backward-compatible > >> solution may be sufficient: when a generator returns, instead of > >> raising ``StopIteration``, it raises a specific subclass of > >> ``StopIteration`` which can then be detected. If it is not that > >> subclass, it is an escaping exception rather than a return statement. > > > > There's an additional subtlety with this idea: if we add a new > > GeneratorReturn exception as a subclass of StopIteration, then generator > > iterators would likely also have to change to replace GeneratorReturn > with a > > regular StopIteration (chaining appropriately via __cause__, and copying > the > > return value across). > > Would have to do so automatically, meaning this is no simpler than the > current proposal? Or would have to be always explicitly written to > handle it? > When GeneratorReturn escaped a generator frame, the interpreter would automatically convert it into an ordinary StopIteration instance. It's still simpler because it won't need the __future__ dance (as it doesn't involve any backwards incompatible changes). > > I definitely see value in adding a GeneratorReturn subclass to be able > to > > tell the "returned" vs "raised StopIteration" cases apart from outside > the > > generator (the current dance in contextlib only works because we have > > existing knowledge of the exact exception that was thrown in). I'm > > substantially less convinced of the benefit of changing generators to no > > longer suppress StopIteration. Yes, it's currently a rather odd corner > case, > > but changing it *will* break code (at the very least, anyone using an old > > version of contextlib2, or who are otherwise relying on their own copy of > > contextlib rather than standard library one). > > This is why it's proposed to use __future__ to protect it. Using __future__ still imposes a large cost on the community - docs need updating, code that relies on the existing behaviour has to be changed, developers need to adjust their mental models of how the language works. There needs to be a practical payoff for those costs - and at the moment, it's looking like we can actually get a reasonably large fraction of the gain without most of the pain by instead pursuing Guido's idea of a separate StopIteration subclass to distinguish returning from the outermost generator frame from raising StopIteration elsewhere in the generator. > If anyone's > still using an old version of contextlib2 once 3.7 comes along, it'll > break; but is there any reason to use Python 3.7 with a contextlib > from elsewhere than its standard library? Same reason folks use it now: consistent behaviour and features across a range of Python versions. However, that's not the key point - the key point is that working through the exact changes that would need to be made in contextlib persuaded me that I was wrong when I concluded that contextlib wouldn't be negatively affected. It's not much more complicated, but if we can find a fully supported example like that in the standard library, what other things might folks be doing with generators that *don't* fall into the category of "overly clever code that we don't mind breaking"? > > (I'm not familiar with > contextlib2 or what it offers.) > contexlib2 ~= 3.3 era contextlib that runs as far back as 2.6 (I initially created it as a proving ground for the idea that eventually become contextlib.ExitStack). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Nov 15 16:56:43 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 16 Nov 2014 02:56:43 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: On Sun, Nov 16, 2014 at 2:21 AM, Nick Coghlan wrote: > On 16 November 2014 00:37, Chris Angelico wrote: >> >> On Sun, Nov 16, 2014 at 1:13 AM, Nick Coghlan wrote: >> >> For certain situations, a simpler and fully backward-compatible >> >> solution may be sufficient: when a generator returns, instead of >> >> raising ``StopIteration``, it raises a specific subclass of >> >> ``StopIteration`` which can then be detected. If it is not that >> >> subclass, it is an escaping exception rather than a return statement. >> > >> > There's an additional subtlety with this idea: if we add a new >> > GeneratorReturn exception as a subclass of StopIteration, then generator >> > iterators would likely also have to change to replace GeneratorReturn >> > with a >> > regular StopIteration (chaining appropriately via __cause__, and copying >> > the >> > return value across). >> >> Would have to do so automatically, meaning this is no simpler than the >> current proposal? Or would have to be always explicitly written to >> handle it? > > When GeneratorReturn escaped a generator frame, the interpreter would > automatically convert it into an ordinary StopIteration instance. Okay, let me see if I have this straight. When a 'return' statement (including an implicit one at end-of-function) is encountered in any function which contains a 'yield' statement, it is implemented as "raise GeneratorReturn(value)" rather than as "raise StopIteration(value)" which is the current behaviour. However, if any GeneratorReturn would be raised in any way other than the 'return' statement, it would magically become a StopIteration instead. Is that correct? This does sound simpler. All the magic is in the boundary of the generator itself, nothing more. If a __next__ method raises either StopIteration or GeneratorReturn, or if any other function raises them, there's no special handling. Question: How does it "become" StopIteration? Is a new instance of StopIteration formed which copies in the other's ``value``? Is the type of this exception magically altered? Or is it a brand new exception with the __cause__ or __context__ set to carry the original? >> If anyone's >> still using an old version of contextlib2 once 3.7 comes along, it'll >> break; but is there any reason to use Python 3.7 with a contextlib >> from elsewhere than its standard library? > > > Same reason folks use it now: consistent behaviour and features across a > range of Python versions. > > However, that's not the key point - the key point is that working through > the exact changes that would need to be made in contextlib persuaded me that > I was wrong when I concluded that contextlib wouldn't be negatively > affected. > > It's not much more complicated, but if we can find a fully supported example > like that in the standard library, what other things might folks be doing > with generators that *don't* fall into the category of "overly clever code > that we don't mind breaking"? Fair enough. The breakage is a known problem, though; whatever's done is likely to cause at least some issues. If the alternate you describe above will break less (or almost none), then it'll be the best option. >> (I'm not familiar with >> contextlib2 or what it offers.) > > contexlib2 ~= 3.3 era contextlib that runs as far back as 2.6 (I initially > created it as a proving ground for the idea that eventually become > contextlib.ExitStack). Thanks, I figured it'd be like that. Since contextlib exists in 2.7, is contextlib2 meant to be legacy support only? ChrisA From ncoghlan at gmail.com Sat Nov 15 17:51:33 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 16 Nov 2014 02:51:33 +1000 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: On 16 November 2014 01:56, Chris Angelico wrote: > On Sun, Nov 16, 2014 at 2:21 AM, Nick Coghlan wrote: > > On 16 November 2014 00:37, Chris Angelico wrote: > >> > >> On Sun, Nov 16, 2014 at 1:13 AM, Nick Coghlan > wrote: > >> >> For certain situations, a simpler and fully backward-compatible > >> >> solution may be sufficient: when a generator returns, instead of > >> >> raising ``StopIteration``, it raises a specific subclass of > >> >> ``StopIteration`` which can then be detected. If it is not that > >> >> subclass, it is an escaping exception rather than a return statement. > >> > > >> > There's an additional subtlety with this idea: if we add a new > >> > GeneratorReturn exception as a subclass of StopIteration, then > generator > >> > iterators would likely also have to change to replace GeneratorReturn > >> > with a > >> > regular StopIteration (chaining appropriately via __cause__, and > copying > >> > the > >> > return value across). > >> > >> Would have to do so automatically, meaning this is no simpler than the > >> current proposal? Or would have to be always explicitly written to > >> handle it? > > > > When GeneratorReturn escaped a generator frame, the interpreter would > > automatically convert it into an ordinary StopIteration instance. > > Okay, let me see if I have this straight. When a 'return' statement > (including an implicit one at end-of-function) is encountered in any > function which contains a 'yield' statement, it is implemented as > "raise GeneratorReturn(value)" rather than as "raise > StopIteration(value)" which is the current behaviour. However, if any > GeneratorReturn would be raised in any way other than the 'return' > statement, it would magically become a StopIteration instead. Is that > correct? > That's not quite how generators work. While the "returning from a generator is equivalent to raise StopIteration" model is close enough that it's functionally equivalent to the actual behaviour in most cases (with the main difference being in how try/except blocks and context managers inside the generator react), this particular PEP is a situation where it's important to have a clear picture of the underlying details. When you have a generator iterator (the thing you get back when calling a generator function), there are two key components: * the generator iterator object itself * the generator frame where the code is running When you call next(gi), you're invoking the __next__ method on the *generator iterator*. It's that method which restarts evaluation of the generator frame at the point where it last left off, and interprets any results. Now, there are three things that can happen as a result of that frame evaluation: 1. It hits a yield point. In that case, gi.__next__ returns the yielded value. 2. It can return from the frame. In that case. gi.__next__ creates a *new* StopIteration instance (with an appropriate return value set) and raises it 3. It can throw an exception. In that case, gi.__next__ just allows it to propagate out (including if it's StopIteration) The following example illustrates the difference between cases 2 and 3 (in both cases, there's a StopIteration that terminates the hidden loop inside the list() call, the difference is in where that StopIteration is raised): >>> def genreturn(): ... yield ... try: ... return ... except: ... print("No exception") ... raise ... >>> list(genreturn()) [None] >>> def genraise(): ... yield ... try: ... raise StopIteration ... except: ... print("Exception!") ... raise ... >>> list(genraise()) Exception! [None] (The possible outcomes of gi.send() and gi.throw() are the same as those of next(gi). gi.throw() has the novel variant where the exception thrown in may propagate back out) The two change proposals being discussed are as follows: Current PEP (backwards incompatible): Change outcome 3 to convert StopIteration to RuntimeError (or a new exception type). Nothing else changes. Alternative (backwards compatible): Change outcome 2 to raise GeneratorReturn instead of StopIteration and outcome 3 to convert GeneratorReturn to StopIteration. The alternative *doesn't* do anything about the odd discrepancy between comprehensions and generator expressions that started the previous thread. It just adds a new capability where code that knows it's specifically dealing with a generator (like contextlib or asyncio) can more easily tell the difference between outcomes 2 and 3. > This does sound simpler. All the magic is in the boundary of the > generator itself, nothing more. If a __next__ method raises either > StopIteration or GeneratorReturn, or if any other function raises > them, there's no special handling. > All the magic is actually at the generator boundary regardless. The key differences between the two proposals are the decision to keep StopIteration as a common parent exception, and allow it to continue propagating out of generator frames unmodified. > Question: How does it "become" StopIteration? Is a new instance of > StopIteration formed which copies in the other's ``value``? Is the > type of this exception magically altered? Or is it a brand new > exception with the __cause__ or __context__ set to carry the original? > I'd suggest used the exception chaining machinery and creating a new exception with __cause__ and the generator return value set appropriately. >> (I'm not familiar with > >> contextlib2 or what it offers.) > > > > contexlib2 ~= 3.3 era contextlib that runs as far back as 2.6 (I > initially > > created it as a proving ground for the idea that eventually become > > contextlib.ExitStack). > > Thanks, I figured it'd be like that. Since contextlib exists in 2.7, > is contextlib2 meant to be legacy support only? > contextlib has actually been around since 2.5, but some features (most notably ExitStack) weren't added until much later. Like unittest2, contextlib2 allows access to newer stdlib features on older versions (I haven't used it as a testing ground for new ideas since ExitStack). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From Steve.Dower at microsoft.com Sat Nov 15 19:20:06 2014 From: Steve.Dower at microsoft.com (Steve Dower) Date: Sat, 15 Nov 2014 18:20:06 +0000 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: <7ecbb0e7ad7c4aa58314d63c454ed386@DM2PR0301MB0734.namprd03.prod.outlook.com> Since this changes the behavior of an object instance, how can __future__ help? If the generator definition is in a library but the code that raises StopIteration to terminate it is passed in from the users code, how is the user supposed to select the behavior they want? (This sounds to me like a similar problem to adding 'from __future__ import py3_string' to Py2, which we discussed a while ago. Happy to be shown that it isn't.) Cheers, Steve Top-posted from my Windows Phone ________________________________ From: Chris Angelico Sent: ?11/?15/?2014 1:30 To: python-ideas Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators PEP: 479 Title: Change StopIteration handling inside generators Version: $Revision$ Last-Modified: $Date$ Author: Chris Angelico Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 15-Nov-2014 Python-Version: 3.5 Post-History: 15-Nov-2014 Abstract ======== This PEP proposes a semantic change to ``StopIteration`` when raised inside a generator, unifying the behaviour of list comprehensions and generator expressions somewhat. Rationale ========= The interaction of generators and ``StopIteration`` is currently somewhat surprising, and can conceal obscure bugs. An unexpected exception should not result in subtly altered behaviour, but should cause a noisy and easily-debugged traceback. Currently, ``StopIteration`` can be absorbed by the generator construct. Proposal ======== If a ``StopIteration`` is about to bubble out of a generator frame, it is replaced with some other exception (maybe ``RuntimeError``, maybe a new custom ``Exception`` subclass, but *not* deriving from ``StopIteration``) which causes the ``next()`` call (which invoked the generator) to fail, passing that exception out. From then on it's just like any old exception. [3]_ Consequences to existing code ============================= This change will affect existing code that depends on ``StopIteration`` bubbling up. The pure Python reference implementation of ``groupby`` [1]_ currently has comments "Exit on ``StopIteration``" where it is expected that the exception will propagate and then be handled. This will be unusual, but not unknown, and such constructs will fail. (Nick Coghlan comments: """If you wanted to factor out a helper function that terminated the generator you'd have to do "return yield from helper()" rather than just "helper()".""") As this can break code, it is proposed to utilize the ``__future__`` mechanism to introduce this, finally making it standard in Python 3.6 or 3.7. Alternate proposals =================== Supplying a specific exception to raise on return ------------------------------------------------- Nick Coghlan suggested a means of providing a specific ``StopIteration`` instance to the generator; if any other instance of ``StopIteration`` is raised, it is an error, but if that particular one is raised, the generator has properly completed. Making return-triggered StopIterations obvious ---------------------------------------------- For certain situations, a simpler and fully backward-compatible solution may be sufficient: when a generator returns, instead of raising ``StopIteration``, it raises a specific subclass of ``StopIteration`` which can then be detected. If it is not that subclass, it is an escaping exception rather than a return statement. Criticism ========= Unofficial and apocryphal statistics suggest that this is seldom, if ever, a problem. [4]_ Code does exist which relies on the current behaviour, and there is the concern that this would be unnecessary code churn to achieve little or no gain. References ========== .. [1] Initial mailing list comment (https://mail.python.org/pipermail/python-ideas/2014-November/029906.html) .. [2] Pure Python implementation of groupby (https://docs.python.org/3/library/itertools.html#itertools.groupby) .. [3] Proposal by GvR (https://mail.python.org/pipermail/python-ideas/2014-November/029953.html) .. [4] Response by Steven D'Aprano (https://mail.python.org/pipermail/python-ideas/2014-November/029994.html) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From apalala at gmail.com Sat Nov 15 22:00:28 2014 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Sat, 15 Nov 2014 21:00:28 +0000 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators References: <7ecbb0e7ad7c4aa58314d63c454ed386@DM2PR0301MB0734.namprd03.prod.outlook.com> Message-ID: Exactly! The current behavior is not only likely undesirable, but it is also undocumented. Even if parts of stdlib rely on the current behavior, there's no need for a deprecation (read __future__) period. Undocumented features may change any time, because are mostly about implementation quirks (Isn't that rule documented somewhere in the Python docs?). In short: -1 deprecation (__future__); no need, because nothing documented gets broken +1 fix it now (3.5); the fix may be a change in the docs to validate the current behavior, and deprecate it (Yuk!) +1 Nick's design, which kind of leaves it the same and kind of fixes it p.s. What about 2.7? This fix is *not* a new feature. Cheers, -- Juanca On Sat Nov 15 2014 at 1:50:06 PM Steve Dower wrote: > Since this changes the behavior of an object instance, how can > __future__ help? If the generator definition is in a library but the code > that raises StopIteration to terminate it is passed in from the users code, > how is the user supposed to select the behavior they want? (This sounds to > me like a similar problem to adding 'from __future__ import py3_string' to > Py2, which we discussed a while ago. Happy to be shown that it isn't.) > > Cheers, > Steve > > Top-posted from my Windows Phone > ------------------------------ > From: Chris Angelico > Sent: ?11/?15/?2014 1:30 > To: python-ideas > Subject: [Python-ideas] PEP 479: Change StopIteration handling inside > generators > > PEP: 479 > Title: Change StopIteration handling inside generators > Version: $Revision$ > Last-Modified: $Date$ > Author: Chris Angelico > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 15-Nov-2014 > Python-Version: 3.5 > Post-History: 15-Nov-2014 > > > Abstract > ======== > > This PEP proposes a semantic change to ``StopIteration`` when raised > inside a generator, unifying the behaviour of list comprehensions and > generator expressions somewhat. > > > Rationale > ========= > > The interaction of generators and ``StopIteration`` is currently > somewhat surprising, and can conceal obscure bugs. An unexpected > exception should not result in subtly altered behaviour, but should > cause a noisy and easily-debugged traceback. Currently, > ``StopIteration`` can be absorbed by the generator construct. > > > Proposal > ======== > > If a ``StopIteration`` is about to bubble out of a generator frame, it > is replaced with some other exception (maybe ``RuntimeError``, maybe a > new custom ``Exception`` subclass, but *not* deriving from > ``StopIteration``) which causes the ``next()`` call (which invoked the > generator) to fail, passing that exception out. From then on it's > just like any old exception. [3]_ > > > Consequences to existing code > ============================= > > This change will affect existing code that depends on > ``StopIteration`` bubbling up. The pure Python reference > implementation of ``groupby`` [1]_ currently has comments "Exit on > ``StopIteration``" where it is expected that the exception will > propagate and then be handled. This will be unusual, but not unknown, > and such constructs will fail. > > (Nick Coghlan comments: """If you wanted to factor out a helper > function that terminated the generator you'd have to do "return > yield from helper()" rather than just "helper()".""") > > As this can break code, it is proposed to utilize the ``__future__`` > mechanism to introduce this, finally making it standard in Python 3.6 > or 3.7. > > > Alternate proposals > =================== > > Supplying a specific exception to raise on return > ------------------------------------------------- > > Nick Coghlan suggested a means of providing a specific > ``StopIteration`` instance to the generator; if any other instance of > ``StopIteration`` is raised, it is an error, but if that particular > one is raised, the generator has properly completed. > > > Making return-triggered StopIterations obvious > ---------------------------------------------- > > For certain situations, a simpler and fully backward-compatible > solution may be sufficient: when a generator returns, instead of > raising ``StopIteration``, it raises a specific subclass of > ``StopIteration`` which can then be detected. If it is not that > subclass, it is an escaping exception rather than a return statement. > > > Criticism > ========= > > Unofficial and apocryphal statistics suggest that this is seldom, if > ever, a problem. [4]_ Code does exist which relies on the current > behaviour, and there is the concern that this would be unnecessary > code churn to achieve little or no gain. > > > References > ========== > > .. [1] Initial mailing list comment > ( > https://mail.python.org/pipermail/python-ideas/2014-November/029906.html) > > .. [2] Pure Python implementation of groupby > (https://docs.python.org/3/library/itertools.html#itertools.groupby) > > .. [3] Proposal by GvR > ( > https://mail.python.org/pipermail/python-ideas/2014-November/029953.html) > > .. [4] Response by Steven D'Aprano > ( > https://mail.python.org/pipermail/python-ideas/2014-November/029994.html) > > > Copyright > ========= > > This document has been placed in the public domain. > > > > .. > Local Variables: > mode: indented-text > indent-tabs-mode: nil > sentence-end-double-space: t > fill-column: 70 > coding: utf-8 > End: > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Nov 15 23:40:02 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 16 Nov 2014 09:40:02 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: On Sun, Nov 16, 2014 at 3:51 AM, Nick Coghlan wrote: > On 16 November 2014 01:56, Chris Angelico wrote: >> Okay, let me see if I have this straight. When a 'return' statement >> (including an implicit one at end-of-function) is encountered in any >> function which contains a 'yield' statement, it is implemented as >> "raise GeneratorReturn(value)" rather than as "raise >> StopIteration(value)" which is the current behaviour. However, if any >> GeneratorReturn would be raised in any way other than the 'return' >> statement, it would magically become a StopIteration instead. Is that >> correct? > > When you call next(gi), you're invoking the __next__ method on the > *generator iterator*. It's that method which restarts evaluation of the > generator frame at the point where it last left off, and interprets any > results. > > Now, there are three things that can happen as a result of that frame > evaluation: > > 1. It hits a yield point. In that case, gi.__next__ returns the yielded > value. > > 2. It can return from the frame. In that case. gi.__next__ creates a > *new* StopIteration instance (with an appropriate return value set) and > raises it > > 3. It can throw an exception. In that case, gi.__next__ just allows it > to propagate out (including if it's StopIteration) Thank you for explaining. -- Cameron In case others were also oversimplifying in their heads, I've summarized the above into the PEP. > (The possible outcomes of gi.send() and gi.throw() are the same as those of > next(gi). gi.throw() has the novel variant where the exception thrown in may > propagate back out) Should that variant affect this proposal? What should happen if you throw StopIteration or GeneratorReturn into a generator? > The two change proposals being discussed are as follows: > > Current PEP (backwards incompatible): Change outcome 3 to convert > StopIteration to RuntimeError (or a new exception type). Nothing else > changes. > > Alternative (backwards compatible): Change outcome 2 to raise > GeneratorReturn instead of StopIteration and outcome 3 to convert > GeneratorReturn to StopIteration. > > The alternative *doesn't* do anything about the odd discrepancy between > comprehensions and generator expressions that started the previous thread. > It just adds a new capability where code that knows it's specifically > dealing with a generator (like contextlib or asyncio) can more easily tell > the difference between outcomes 2 and 3. Text along these lines added to PEP, thanks! >> Question: How does it "become" StopIteration? Is a new instance of >> StopIteration formed which copies in the other's ``value``? Is the >> type of this exception magically altered? Or is it a brand new >> exception with the __cause__ or __context__ set to carry the original? > > I'd suggest used the exception chaining machinery and creating a new > exception with __cause__ and the generator return value set appropriately. Makes sense. If the __cause__ is noticed at all (ie this doesn't just quietly stop a loop), it wants to be very noisy. >> Thanks, I figured it'd be like that. Since contextlib exists in 2.7, >> is contextlib2 meant to be legacy support only? > > contextlib has actually been around since 2.5, but some features (most > notably ExitStack) weren't added until much later. Like unittest2, > contextlib2 allows access to newer stdlib features on older versions (I > haven't used it as a testing ground for new ideas since ExitStack). If there is breakage from this, it would simply mean "older versions of contextlib2 are not compatible with Python 3.7, please upgrade your contextlib2" - several of the variants make it perfectly possible to write cross-version-compatible code. I would hope that this remains the case. ChrisA From rosuav at gmail.com Sun Nov 16 00:06:18 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 16 Nov 2014 10:06:18 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: On Sun, Nov 16, 2014 at 3:51 AM, Nick Coghlan wrote: > On 16 November 2014 01:56, Chris Angelico wrote: >> Okay, let me see if I have this straight. When a 'return' statement >> (including an implicit one at end-of-function) is encountered in any >> function which contains a 'yield' statement, it is implemented as >> "raise GeneratorReturn(value)" rather than as "raise >> StopIteration(value)" which is the current behaviour. However, if any >> GeneratorReturn would be raised in any way other than the 'return' >> statement, it would magically become a StopIteration instead. Is that >> correct? > > When you call next(gi), you're invoking the __next__ method on the > *generator iterator*. It's that method which restarts evaluation of the > generator frame at the point where it last left off, and interprets any > results. > > Now, there are three things that can happen as a result of that frame > evaluation: > > 1. It hits a yield point. In that case, gi.__next__ returns the yielded > value. > > 2. It can return from the frame. In that case. gi.__next__ creates a > *new* StopIteration instance (with an appropriate return value set) and > raises it > > 3. It can throw an exception. In that case, gi.__next__ just allows it > to propagate out (including if it's StopIteration) Thank you for explaining. -- Cameron In case others were also oversimplifying in their heads, I've summarized the above into the PEP. > (The possible outcomes of gi.send() and gi.throw() are the same as those of > next(gi). gi.throw() has the novel variant where the exception thrown in may > propagate back out) Should that variant affect this proposal? What should happen if you throw StopIteration or GeneratorReturn into a generator? > The two change proposals being discussed are as follows: > > Current PEP (backwards incompatible): Change outcome 3 to convert > StopIteration to RuntimeError (or a new exception type). Nothing else > changes. > > Alternative (backwards compatible): Change outcome 2 to raise > GeneratorReturn instead of StopIteration and outcome 3 to convert > GeneratorReturn to StopIteration. > > The alternative *doesn't* do anything about the odd discrepancy between > comprehensions and generator expressions that started the previous thread. > It just adds a new capability where code that knows it's specifically > dealing with a generator (like contextlib or asyncio) can more easily tell > the difference between outcomes 2 and 3. Text along these lines added to PEP, thanks! >> Question: How does it "become" StopIteration? Is a new instance of >> StopIteration formed which copies in the other's ``value``? Is the >> type of this exception magically altered? Or is it a brand new >> exception with the __cause__ or __context__ set to carry the original? > > I'd suggest used the exception chaining machinery and creating a new > exception with __cause__ and the generator return value set appropriately. Makes sense. If the __cause__ is noticed at all (ie this doesn't just quietly stop a loop), it wants to be very noisy. >> Thanks, I figured it'd be like that. Since contextlib exists in 2.7, >> is contextlib2 meant to be legacy support only? > > contextlib has actually been around since 2.5, but some features (most > notably ExitStack) weren't added until much later. Like unittest2, > contextlib2 allows access to newer stdlib features on older versions (I > haven't used it as a testing ground for new ideas since ExitStack). If there is breakage from this, it would simply mean "older versions of contextlib2 are not compatible with Python 3.7, please upgrade your contextlib2" - several of the variants make it perfectly possible to write cross-version-compatible code. I would hope that this remains the case. Latest version of PEP text incorporating the above changes: https://raw.githubusercontent.com/Rosuav/GenStopIter/master/pep-0479.txt (My apologies if this email has gone through more than once. I'm having major issues with my internet connection at the moment, and delivery is failing and being retried. Hopefully it really *is* failing, and not just saying so.) ChrisA From rosuav at gmail.com Sun Nov 16 00:06:29 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 16 Nov 2014 10:06:29 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: <7ecbb0e7ad7c4aa58314d63c454ed386@DM2PR0301MB0734.namprd03.prod.outlook.com> References: <7ecbb0e7ad7c4aa58314d63c454ed386@DM2PR0301MB0734.namprd03.prod.outlook.com> Message-ID: On Sun, Nov 16, 2014 at 5:20 AM, Steve Dower wrote: > Since this changes the behavior of an object instance, how can __future__ > help? If the generator definition is in a library but the code that raises > StopIteration to terminate it is passed in from the users code, how is the > user supposed to select the behavior they want? (This sounds to me like a > similar problem to adding 'from __future__ import py3_string' to Py2, which > we discussed a while ago. Happy to be shown that it isn't.) The behaviour selection would have to be based on the generator's definition. This proposal, in all its variants, is about what happens as the generator terminates; if you call on someone else's generator, and that someone hasn't applied the __future__ directive, you'll be in the current situation of not being able to distinguish 'return' from 'raise StopIteration'. But for your own generators, you can guarantee that they're distinct. ChrisA From rosuav at gmail.com Sun Nov 16 00:18:22 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 16 Nov 2014 10:18:22 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <7ecbb0e7ad7c4aa58314d63c454ed386@DM2PR0301MB0734.namprd03.prod.outlook.com> Message-ID: On Sun, Nov 16, 2014 at 8:00 AM, Juancarlo A?ez wrote: > Exactly! > > The current behavior is not only likely undesirable, but it is also > undocumented. I'm not sure about that. As Steven said, the current behaviour is simple: 1) When 'yield' is reached, a value is yielded. 2) When 'return' is reached, StopIteration is raised. 3) When an exception is raised, it is permitted to bubble up. Whether that is *correct* or not is the point of this PEP, but it is at least simple, and while it may not be documented per se, changing it is likely to break code. > Even if parts of stdlib rely on the current behavior, there's no need for a > deprecation (read __future__) period. > > Undocumented features may change any time, because are mostly about > implementation quirks (Isn't that rule documented somewhere in the Python > docs?). Maybe not, but let's get the proposal settled before figuring out how much deprecation period is needed. > In short: > > +1 fix it now (3.5); the fix may be a change in the docs to validate the > current behavior, and deprecate it (Yuk!) That would be pretty much what happens if the PEP is rejected: the current behaviour will be effectively validated (at least to the extent of "it's not worth the breakage"). > p.s. What about 2.7? This fix is *not* a new feature. That ultimately depends on the release manager, but I would not aim this at 2.7. Nick's proposal introduces a new exception type, which I think cuts this out of 2.7 consideration right there; both active proposals involve distinct changes to behaviour. I believe both of them require *at a minimum* a feature release, and quite probably a deprecation period (although that part may be arguable, as mentioned above). ChrisA From rob.cliffe at btinternet.com Sun Nov 16 01:49:01 2014 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Sun, 16 Nov 2014 00:49:01 +0000 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: <5467F47D.2080307@btinternet.com> On 15/11/2014 09:29, Chris Angelico wrote: > PEP: 479 > Title: Change StopIteration handling inside generators Thanks Chris for volunteering to do the donkey work again. I think the final draft of the PEP would be much more comprehensible and useful if it contained one or more examples illustrating how the old and new behaviour differ. I realise that this may not be appropriate until a consensus has been reached on what exactly the new behaviour *is*, so apologies if my grandmother already intends to suck this particular egg. Rob Cliffe From rosuav at gmail.com Sun Nov 16 02:21:48 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 16 Nov 2014 12:21:48 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: <5467F47D.2080307@btinternet.com> References: <5467F47D.2080307@btinternet.com> Message-ID: On Sun, Nov 16, 2014 at 11:49 AM, Rob Cliffe wrote: > On 15/11/2014 09:29, Chris Angelico wrote: >> >> PEP: 479 >> Title: Change StopIteration handling inside generators > > Thanks Chris for volunteering to do the donkey work again. > I think the final draft of the PEP would be much more comprehensible and > useful if it contained one or more examples illustrating how the old and new > behaviour differ. I realise that this may not be appropriate until a > consensus has been reached on what exactly the new behaviour *is*, so > apologies if my grandmother already intends to suck this particular egg. Agreed. And agreed on the analysis; I can't add examples till I know for sure what I'm adding examples _of_. The latest edit expanded on the details of the proposals, so it now may be possible to consider examples, but possibly we're still bikeshedding the nature of the proposals themselves. Correction. We're DEFINITELY still bikeshedding etc etc, but possibly we're still doing so to the extent that it's not worth adding examples yet. :) ChrisA From greg.ewing at canterbury.ac.nz Sun Nov 16 23:05:01 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 17 Nov 2014 11:05:01 +1300 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <7ecbb0e7ad7c4aa58314d63c454ed386@DM2PR0301MB0734.namprd03.prod.outlook.com> Message-ID: <54691F8D.1080703@canterbury.ac.nz> Chris Angelico wrote: > if you call on someone else's generator, > and that someone hasn't applied the __future__ directive, you'll be in > the current situation of not being able to distinguish 'return' from > 'raise StopIteration'. But for your own generators, you can guarantee > that they're distinct. This suggests that the use of a __future__ directive is not really appropriate, since it can affect code outside of the module with the __future__ directive in it. -- Greg From steve at pearwood.info Mon Nov 17 01:03:02 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 17 Nov 2014 11:03:02 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: <54691F8D.1080703@canterbury.ac.nz> References: <7ecbb0e7ad7c4aa58314d63c454ed386@DM2PR0301MB0734.namprd03.prod.outlook.com> <54691F8D.1080703@canterbury.ac.nz> Message-ID: <20141117000302.GS2748@ando.pearwood.info> On Mon, Nov 17, 2014 at 11:05:01AM +1300, Greg Ewing wrote: > Chris Angelico wrote: > >if you call on someone else's generator, > >and that someone hasn't applied the __future__ directive, you'll be in > >the current situation of not being able to distinguish 'return' from > >'raise StopIteration'. But for your own generators, you can guarantee > >that they're distinct. > > This suggests that the use of a __future__ directive is not > really appropriate, since it can affect code outside of the > module with the __future__ directive in it. I don't see how that is different from any other __future__ directive. They are all per-module, and if you gain access to an object from another module, it will behave as specified in the module that created it, not the module that imported it. How is this different? -- Steven From rosuav at gmail.com Mon Nov 17 02:29:05 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 17 Nov 2014 12:29:05 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: <20141117000302.GS2748@ando.pearwood.info> References: <7ecbb0e7ad7c4aa58314d63c454ed386@DM2PR0301MB0734.namprd03.prod.outlook.com> <54691F8D.1080703@canterbury.ac.nz> <20141117000302.GS2748@ando.pearwood.info> Message-ID: On Mon, Nov 17, 2014 at 11:03 AM, Steven D'Aprano wrote: > On Mon, Nov 17, 2014 at 11:05:01AM +1300, Greg Ewing wrote: >> Chris Angelico wrote: >> >if you call on someone else's generator, >> >and that someone hasn't applied the __future__ directive, you'll be in >> >the current situation of not being able to distinguish 'return' from >> >'raise StopIteration'. But for your own generators, you can guarantee >> >that they're distinct. >> >> This suggests that the use of a __future__ directive is not >> really appropriate, since it can affect code outside of the >> module with the __future__ directive in it. > > I don't see how that is different from any other __future__ directive. > They are all per-module, and if you gain access to an object from > another module, it will behave as specified in the module that created > it, not the module that imported it. How is this different? Well, let's see. For feature in sorted(__future__.all_feature_names): absolute_import: Affects implementation of a keyword barry_as_FLUFL: Not entirely sure what this one actually accomplishes. :) division: Changes the meaning of one operator. generators: Introduces a keyword nested_scopes: Alters the compilation of source to byte-code(?) print_function: Removes a keyword unicode_literals: Alters the type used for literals with_statement: Introduces a keyword Apart from the joke, it seems that every __future__ directive is there to affect the compilation, not execution, of its module: that is, once a module has been compiled to .pyc, it shouldn't matter whether it used __future__ or not. Regardless of unicode_literals, you can create bytes literals with b'asdf' and unicode literals with u'asdf'. I'm not entirely sure about division (can you call on true-division without the future directive?), but in any case, it's all done at compilation time, as can be seen interactively: Python 2.7.3 (default, Mar 13 2014, 11:03:55) [GCC 4.7.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> def division1(x,y): ... return x/y, x//y, x%y ... >>> from __future__ import division >>> def division2(x,y): ... return x/y, x//y, x%y ... >>> import dis >>> dis.dis(division1) 2 0 LOAD_FAST 0 (x) 3 LOAD_FAST 1 (y) 6 BINARY_DIVIDE 7 LOAD_FAST 0 (x) 10 LOAD_FAST 1 (y) 13 BINARY_FLOOR_DIVIDE 14 LOAD_FAST 0 (x) 17 LOAD_FAST 1 (y) 20 BINARY_MODULO 21 BUILD_TUPLE 3 24 RETURN_VALUE >>> dis.dis(division2) 2 0 LOAD_FAST 0 (x) 3 LOAD_FAST 1 (y) 6 BINARY_TRUE_DIVIDE 7 LOAD_FAST 0 (x) 10 LOAD_FAST 1 (y) 13 BINARY_FLOOR_DIVIDE 14 LOAD_FAST 0 (x) 17 LOAD_FAST 1 (y) 20 BINARY_MODULO 21 BUILD_TUPLE 3 24 RETURN_VALUE So to make this consistent with all other __future__ directives, there would need to be some kind of safe way to define this: perhaps an attribute on the generator object. Something like this: >>> def gen(x): ... yield x ... raise StopIteration(42) ... >>> g=gen(123) >>> list(g) [123] >>> gen.__distinguish_returns__ = True >>> g=gen(123) >>> list(g) Traceback (most recent call last): File "", line 1, in File "", line 3, in gen StopIteration: 42 The attribute on the function would be what affects behaviour; the __future__ directive applies that attribute to all generator functions in its module (including genexprs). Once the __future__ directive becomes automatic, the attribute can and will be dropped - any code which interrogates it MUST be prepared to stop interrogating it once the feature applies to all modules. Does that sound reasonable? Should it be added to the PEP? ChrisA From guido at python.org Mon Nov 17 02:58:34 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 16 Nov 2014 17:58:34 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <7ecbb0e7ad7c4aa58314d63c454ed386@DM2PR0301MB0734.namprd03.prod.outlook.com> <54691F8D.1080703@canterbury.ac.nz> <20141117000302.GS2748@ando.pearwood.info> Message-ID: (I'm catching up on this thread from the end.) On Sun, Nov 16, 2014 at 5:29 PM, Chris Angelico wrote: > On Mon, Nov 17, 2014 at 11:03 AM, Steven D'Aprano > wrote: > > On Mon, Nov 17, 2014 at 11:05:01AM +1300, Greg Ewing wrote: > >> Chris Angelico wrote: > >> >if you call on someone else's generator, > >> >and that someone hasn't applied the __future__ directive, you'll be in > >> >the current situation of not being able to distinguish 'return' from > >> >'raise StopIteration'. But for your own generators, you can guarantee > >> >that they're distinct. > >> > >> This suggests that the use of a __future__ directive is not > >> really appropriate, since it can affect code outside of the > >> module with the __future__ directive in it. > > > > I don't see how that is different from any other __future__ directive. > > They are all per-module, and if you gain access to an object from > > another module, it will behave as specified in the module that created > > it, not the module that imported it. How is this different? > > Well, let's see. For feature in sorted(__future__.all_feature_names): > > absolute_import: Affects implementation of a keyword > barry_as_FLUFL: Not entirely sure what this one actually accomplishes. :) > division: Changes the meaning of one operator. > generators: Introduces a keyword > nested_scopes: Alters the compilation of source to byte-code(?) > print_function: Removes a keyword > unicode_literals: Alters the type used for literals > with_statement: Introduces a keyword > > Apart from the joke, it seems that every __future__ directive is there > to affect the compilation, not execution, of its module: that is, once > a module has been compiled to .pyc, it shouldn't matter whether it > used __future__ or not. Regardless of unicode_literals, you can create > bytes literals with b'asdf' and unicode literals with u'asdf'. I'm not > entirely sure about division (can you call on true-division without > the future directive?), but in any case, it's all done at compilation > time, as can be seen interactively: > > Python 2.7.3 (default, Mar 13 2014, 11:03:55) > [GCC 4.7.2] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> def division1(x,y): > ... return x/y, x//y, x%y > ... > >>> from __future__ import division > >>> def division2(x,y): > ... return x/y, x//y, x%y > ... > >>> import dis > >>> dis.dis(division1) > 2 0 LOAD_FAST 0 (x) > 3 LOAD_FAST 1 (y) > 6 BINARY_DIVIDE > 7 LOAD_FAST 0 (x) > 10 LOAD_FAST 1 (y) > 13 BINARY_FLOOR_DIVIDE > 14 LOAD_FAST 0 (x) > 17 LOAD_FAST 1 (y) > 20 BINARY_MODULO > 21 BUILD_TUPLE 3 > 24 RETURN_VALUE > >>> dis.dis(division2) > 2 0 LOAD_FAST 0 (x) > 3 LOAD_FAST 1 (y) > 6 BINARY_TRUE_DIVIDE > 7 LOAD_FAST 0 (x) > 10 LOAD_FAST 1 (y) > 13 BINARY_FLOOR_DIVIDE > 14 LOAD_FAST 0 (x) > 17 LOAD_FAST 1 (y) > 20 BINARY_MODULO > 21 BUILD_TUPLE 3 > 24 RETURN_VALUE > > So to make this consistent with all other __future__ directives, there > would need to be some kind of safe way to define this: perhaps an > attribute on the generator object. Something like this: > > >>> def gen(x): > ... yield x > ... raise StopIteration(42) > ... > >>> g=gen(123) > >>> list(g) > [123] > >>> gen.__distinguish_returns__ = True > >>> g=gen(123) > >>> list(g) > Traceback (most recent call last): > File "", line 1, in > File "", line 3, in gen > StopIteration: 42 > > The attribute on the function would be what affects behaviour; the > __future__ directive applies that attribute to all generator functions > in its module (including genexprs). Once the __future__ directive > becomes automatic, the attribute can and will be dropped - any code > which interrogates it MUST be prepared to stop interrogating it once > the feature applies to all modules. > > Does that sound reasonable? Should it be added to the PEP? > I agree with you and Steven that this is a fine use of __future__. What a generator does with a StopIteration that is about to bubble out of its frame is up to that generator. I don't think it needs to be a flag on the *function* though -- IMO it should be a flag on the code object. (And the flag should somehow be transferred to the stack frame when the function is executed, so the right action can be taken when an exception is about to bubble out of that frame.) One other small point: let's change the PEP to just propose RuntimeError, and move the "some other exception" to the "rejected ideas" section. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Nov 17 03:11:15 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 17 Nov 2014 13:11:15 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <7ecbb0e7ad7c4aa58314d63c454ed386@DM2PR0301MB0734.namprd03.prod.outlook.com> <54691F8D.1080703@canterbury.ac.nz> <20141117000302.GS2748@ando.pearwood.info> Message-ID: On Mon, Nov 17, 2014 at 12:58 PM, Guido van Rossum wrote: > I agree with you and Steven that this is a fine use of __future__. What a > generator does with a StopIteration that is about to bubble out of its frame > is up to that generator. I don't think it needs to be a flag on the > *function* though -- IMO it should be a flag on the code object. (And the > flag should somehow be transferred to the stack frame when the function is > executed, so the right action can be taken when an exception is about to > bubble out of that frame.) > > One other small point: let's change the PEP to just propose RuntimeError, > and move the "some other exception" to the "rejected ideas" section. Changes incorporated, thanks! I'm not familiar with the details of stack frame handling, so I've taken the cop-out approach and just quoted you directly into the PEP. PEP draft: https://raw.githubusercontent.com/Rosuav/GenStopIter/master/pep-0479.txt GitHub hosted repo, if you want to follow changes etc: https://github.com/Rosuav/GenStopIter ChrisA From greg.ewing at canterbury.ac.nz Mon Nov 17 04:22:42 2014 From: greg.ewing at canterbury.ac.nz (Greg) Date: Mon, 17 Nov 2014 16:22:42 +1300 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <7ecbb0e7ad7c4aa58314d63c454ed386@DM2PR0301MB0734.namprd03.prod.outlook.com> <54691F8D.1080703@canterbury.ac.nz> <20141117000302.GS2748@ando.pearwood.info> Message-ID: <54696A02.6050602@canterbury.ac.nz> On 17/11/2014 2:29 p.m., Chris Angelico wrote: > barry_as_FLUFL: Not entirely sure what this one actually accomplishes. :) It determines whether "not equal" is spelled "!=" or "<>", so it fits the pattern of being compile-time-only. -- Greg From rosuav at gmail.com Mon Nov 17 04:34:52 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 17 Nov 2014 14:34:52 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: <54696A02.6050602@canterbury.ac.nz> References: <7ecbb0e7ad7c4aa58314d63c454ed386@DM2PR0301MB0734.namprd03.prod.outlook.com> <54691F8D.1080703@canterbury.ac.nz> <20141117000302.GS2748@ando.pearwood.info> <54696A02.6050602@canterbury.ac.nz> Message-ID: On Mon, Nov 17, 2014 at 2:22 PM, Greg wrote: > On 17/11/2014 2:29 p.m., Chris Angelico wrote: > >> barry_as_FLUFL: Not entirely sure what this one actually accomplishes. :) > > > It determines whether "not equal" is spelled "!=" or "<>", so it > fits the pattern of being compile-time-only. Right. So, are there any __future__ directives that have any effect on byte-code? I'm not seeing any, though that may just mean I didn't recognize one. ChrisA From g.brandl at gmx.net Mon Nov 17 10:26:13 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 17 Nov 2014 10:26:13 +0100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <7ecbb0e7ad7c4aa58314d63c454ed386@DM2PR0301MB0734.namprd03.prod.outlook.com> <54691F8D.1080703@canterbury.ac.nz> <20141117000302.GS2748@ando.pearwood.info> Message-ID: On 11/17/2014 03:11 AM, Chris Angelico wrote: > On Mon, Nov 17, 2014 at 12:58 PM, Guido van Rossum wrote: >> I agree with you and Steven that this is a fine use of __future__. What a >> generator does with a StopIteration that is about to bubble out of its frame >> is up to that generator. I don't think it needs to be a flag on the >> *function* though -- IMO it should be a flag on the code object. (And the >> flag should somehow be transferred to the stack frame when the function is >> executed, so the right action can be taken when an exception is about to >> bubble out of that frame.) >> >> One other small point: let's change the PEP to just propose RuntimeError, >> and move the "some other exception" to the "rejected ideas" section. > > Changes incorporated, thanks! I'm not familiar with the details of > stack frame handling, so I've taken the cop-out approach and just > quoted you directly into the PEP. > > PEP draft: > https://raw.githubusercontent.com/Rosuav/GenStopIter/master/pep-0479.txt I fixed a typo and syntax and made a content clarification ("generator constructed" -> "generator function constructed"; the generator object itself is constructed when the function is called) and committed the changes. Georg From rosuav at gmail.com Mon Nov 17 11:27:22 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 17 Nov 2014 21:27:22 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <7ecbb0e7ad7c4aa58314d63c454ed386@DM2PR0301MB0734.namprd03.prod.outlook.com> <54691F8D.1080703@canterbury.ac.nz> <20141117000302.GS2748@ando.pearwood.info> Message-ID: On Mon, Nov 17, 2014 at 8:26 PM, Georg Brandl wrote: > I fixed a typo and syntax and made a content clarification ("generator > constructed" -> "generator function constructed"; the generator object > itself is constructed when the function is called) and committed the changes. Thanks Georg! This means today's version is now visible here: http://legacy.python.org/dev/peps/pep-0479/ ChrisA From ncoghlan at gmail.com Mon Nov 17 13:56:32 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 17 Nov 2014 22:56:32 +1000 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <7ecbb0e7ad7c4aa58314d63c454ed386@DM2PR0301MB0734.namprd03.prod.outlook.com> <54691F8D.1080703@canterbury.ac.nz> <20141117000302.GS2748@ando.pearwood.info> <54696A02.6050602@canterbury.ac.nz> Message-ID: On 17 November 2014 13:34, Chris Angelico wrote: > On Mon, Nov 17, 2014 at 2:22 PM, Greg wrote: > > On 17/11/2014 2:29 p.m., Chris Angelico wrote: > > > >> barry_as_FLUFL: Not entirely sure what this one actually accomplishes. > :) > > > > > > It determines whether "not equal" is spelled "!=" or "<>", so it > > fits the pattern of being compile-time-only. > > Right. So, are there any __future__ directives that have any effect on > byte-code? I'm not seeing any, though that may just mean I didn't > recognize one. > True division (in Python 2) is a nice simple one to look at, since it just swaps one bytecode for another (BINARY_DIVIDE -> BINARY_TRUE_DIVIDE) >>> def floor_div(x, y): ... return x / y ... >>> from __future__ import division >>> def true_div(x, y): ... return x / y ... >>> import dis >>> dis.dis(floor_div) 2 0 LOAD_FAST 0 (x) 3 LOAD_FAST 1 (y) 6 BINARY_DIVIDE 7 RETURN_VALUE >>> dis.dis(true_div) 2 0 LOAD_FAST 0 (x) 3 LOAD_FAST 1 (y) 6 BINARY_TRUE_DIVIDE 7 RETURN_VALUE The compiler actually stores a whole pile of useful info on code objects that doesn't show up in the disassembly output (switching to Python 3 for more up to date dis module goodness): >>> import dis >>> def true_div(x, y): ... return x / y ... >>> dis.show_code(true_div) Name: true_div Filename: Argument count: 2 Kw-only arguments: 0 Number of locals: 2 Stack size: 2 Flags: OPTIMIZED, NEWLOCALS, NOFREE Constants: 0: None Variable names: 0: x 1: y So conveying to the generator iterator whether or not "from __future__ import generator_return" was in effect would just be a matter of the compiler setting a new flag on the generator code object. For *affected generators* (i.e. those defined in a module where the new future statement was in effect), StopIteration escaping would be considered a RuntimeError. For almost all code, such RuntimeErrors would look like any other RuntimError raised by a broken generator implementation. The only code which would *have* to change immediately as a "Porting to Python 3.5" requirement is code like that in contextlib, which throws StopIteration into generators, and currently expects to get it back out unmodified. Such code will need to be updated to also handle RuntimError instances where the direct cause is the StopIteration exception that was thrown in. Other affected code (such as the "next() bubbling up" groupby example) would keep working unless the __future__ statement was in effect. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeanpierreda at gmail.com Mon Nov 17 14:50:36 2014 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Mon, 17 Nov 2014 05:50:36 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: On Sat, Nov 15, 2014 at 1:29 AM, Chris Angelico wrote: > The interaction of generators and ``StopIteration`` is currently > somewhat surprising, and can conceal obscure bugs. An unexpected > exception should not result in subtly altered behaviour, but should > cause a noisy and easily-debugged traceback. Currently, > ``StopIteration`` can be absorbed by the generator construct. Specifically it's absorbed by the caller of the generator, because the caller doesn't know the difference between next(x) raising StopIteration because the iterator specifically wants to stop, vs because of accident. As another alternative, how about a new iterator protocol that is defined without this ambiguity? Code at the bottom of my post to help explain: define a new method __nextx__ which doesn't use StopIteration for any signalling, instead, it returns None if there are no values to return, and returns a special value Some(v) if it wants to return a value v. Both next(it) and nextx(it) are made to work for any iterator that is defined using either protocol, but for loops and Python builtins all use nextx internally. Generators define __next__ unless you from __future__ import iterators, in which case they define __nextx__ instead. In this way, old code can't tell the difference between accidental StopIteration and deliberate StopIteration, but new code (using nextx instead of next, and using __future__ import'd generators) can. No backwards incompatibility is introduced, and you can still insert StopIteration into a generator and get it back out -- using both next() where it is ambiguous and nextx() where it is not. Yes, it's ugly to have two different iterator protocols, but not that ugly. In fact, this would be Python's third (I have omitted that third protocol in the below example, for the sake of clarity). I find the proposed solution more scary, in that it's sort of a "hack" to get around an old mistake, rather than a correction to that mistake, and it introduces complexity that can't be removed in principle. (Also, it's very unusual.) class Some: def __init__(self, value): self.value = value def next(it): v = nextx(it) if v is None: raise StopIteration return v.value def nextx(it): if hasattr(it, '__nextx__'): v = it.__nextx__() if v is None or isinstance(v, Some): return v raise TypeError("__nextx__ must return Some(...) or None, not %r" % (v,)) if hasattr(it, '__next__'): try: return Some(it.__next__()) except StopIteration: return None raise TypeError -- Devin From rosuav at gmail.com Mon Nov 17 15:04:01 2014 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 18 Nov 2014 01:04:01 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: On Tue, Nov 18, 2014 at 12:50 AM, Devin Jeanpierre wrote: > As another alternative, how about a new iterator protocol that is > defined without this ambiguity? Code at the bottom of my post to help > explain: define a new method __nextx__ which doesn't use > StopIteration for any signalling, instead, it returns None if there > are no values to return, and returns a special value Some(v) if it > wants to return a value v. Both next(it) and nextx(it) are made to > work for any iterator that is defined using either protocol, but for > loops and Python builtins all use nextx internally. Generators define > __next__ unless you from __future__ import iterators, in which case > they define __nextx__ instead. I had actually contemplated adding a "what if __next__ returned a sentinel instead of raising an exception" possibility to the PEP, if only for completeness. Since someone else has suggested it too now, it may be worth doing. Rather than a wrapper around every returned value, what I'd be inclined toward is a special sentinel that otherwise cannot be returned. This could be a dedicated, arbitrary object, or something globally unique, or something locally unique. One option that comes to mind is to have the generator return _itself_ to signal that it's returned. I don't think this option will be better than the current front runners, but would you like me to add it for completeness? The biggest downside is that it might give a false positive; you can't, for instance, have an iterator "all_objects()" which returns, like the name says, every object currently known to Python. (I don't know that CPython is capable of implementing that, but there's no reason another Python couldn't, and it might be useful.) I expect that's why the exception system was used instead; can anyone confirm that? ChrisA From ram at rachum.com Mon Nov 17 16:50:27 2014 From: ram at rachum.com (Ram Rachum) Date: Mon, 17 Nov 2014 17:50:27 +0200 Subject: [Python-ideas] Implement `itertools.permutations.__getitem__` and `itertools.permutations.index` In-Reply-To: References: <20140505171538.GR4273@ando> <20140506023902.GV4273@ando> <18aa148c-f6cb-414c-9f6c-de5011568204@googlegroups.com> Message-ID: Hi guys, I just wanted to give an update on this: I just released my own code that does this to PyPI: https://pypi.python.org/pypi/combi Thanks, Ram. On Tue, Jun 10, 2014 at 9:24 AM, Ram Rachum wrote: > I'll email you if/when it's released :) > > > On Tue, Jun 10, 2014 at 9:04 AM, Neil Girdhar > wrote: > >> I really like this and hope that it eventually makes it into the stdlib. >> It's also a good argument for your other suggestion whereby some of the >> itertools to return Iterables rather than Iterators like range does. >> >> Best, >> >> Neil >> >> On Wednesday, May 7, 2014 1:43:20 PM UTC-4, Ram Rachum wrote: >> >>> I'm probably going to implement it in my python_toolbox package. I >>> already implemented 30% and it's really cool. It's at the point where I >>> doubt that I want it in the stdlib because I've gotten so much awesome >>> functionality into it and I'd hate to (a) have 80% of it stripped and (b) >>> have the class names changed to be non-Pythonic :) >>> >>> >>> On Wed, May 7, 2014 at 8:40 PM, Tal Einat wrote: >>> >>> On Wed, May 7, 2014 at 8:21 PM, Ram Rachum wrote: >>>> > Hi Tal, >>>> > >>>> > I'm using it for a project of my own (optimizing keyboard layout) but >>>> I >>>> > can't make the case that it's useful for the stdlib. I'd understand >>>> if it >>>> > would be omitted for not being enough of a common need. >>>> >>>> At the least, this (a function for getting a specific permutation by >>>> lexicographical-order index) could make a nice cookbook recipe. >>>> >>>> - Tal >>>> >>> >>> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> -- >> >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "python-ideas" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/python-ideas/in2gUQMFUzA/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> python-ideas+unsubscribe at googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Nov 17 18:28:41 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 17 Nov 2014 09:28:41 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <7ecbb0e7ad7c4aa58314d63c454ed386@DM2PR0301MB0734.namprd03.prod.outlook.com> <54691F8D.1080703@canterbury.ac.nz> <20141117000302.GS2748@ando.pearwood.info> Message-ID: On Mon, Nov 17, 2014 at 2:27 AM, Chris Angelico wrote: > On Mon, Nov 17, 2014 at 8:26 PM, Georg Brandl wrote: > > I fixed a typo and syntax and made a content clarification ("generator > > constructed" -> "generator function constructed"; the generator object > > itself is constructed when the function is called) and committed the > changes. > > Thanks Georg! This means today's version is now visible here: > > http://legacy.python.org/dev/peps/pep-0479/ > Off-topic: the new python.org site now supports PEPs, so please switch to URLs like this: https://www.python.org/dev/peps/pep-0479/ (if you don't like the formatting send a pull request to https://github.com/python/pythondotorg). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Nov 17 18:40:20 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 17 Nov 2014 09:40:20 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: I don't want to contemplate a new __next__ protocol. The existing protocol was carefully designed and tuned to have minimal memory overhead (even to the point where the exception instance returned may be reused). Wapping each result would just result in an extra allocation + deallocation per iteration, unless you can play games with reference counts or do something else to complicate the semantics). Introducing __nextx__ would require thousands of libraries implementing this to incur churn as they feel the pressure to switch to the new protocol, and the compatibility issue would be felt everywhere. The problem we're trying to fix is unique to generators (thereby also implicating generator expressions). On Mon, Nov 17, 2014 at 6:04 AM, Chris Angelico wrote: > On Tue, Nov 18, 2014 at 12:50 AM, Devin Jeanpierre > wrote: > > As another alternative, how about a new iterator protocol that is > > defined without this ambiguity? Code at the bottom of my post to help > > explain: define a new method __nextx__ which doesn't use > > StopIteration for any signalling, instead, it returns None if there > > are no values to return, and returns a special value Some(v) if it > > wants to return a value v. Both next(it) and nextx(it) are made to > > work for any iterator that is defined using either protocol, but for > > loops and Python builtins all use nextx internally. Generators define > > __next__ unless you from __future__ import iterators, in which case > > they define __nextx__ instead. > > I had actually contemplated adding a "what if __next__ returned a > sentinel instead of raising an exception" possibility to the PEP, if > only for completeness. Since someone else has suggested it too now, it > may be worth doing. > > Rather than a wrapper around every returned value, what I'd be > inclined toward is a special sentinel that otherwise cannot be > returned. This could be a dedicated, arbitrary object, or something > globally unique, or something locally unique. One option that comes to > mind is to have the generator return _itself_ to signal that it's > returned. > > I don't think this option will be better than the current front > runners, but would you like me to add it for completeness? The biggest > downside is that it might give a false positive; you can't, for > instance, have an iterator "all_objects()" which returns, like the > name says, every object currently known to Python. (I don't know that > CPython is capable of implementing that, but there's no reason another > Python couldn't, and it might be useful.) I expect that's why the > exception system was used instead; can anyone confirm that? > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Mon Nov 17 18:33:56 2014 From: mistersheik at gmail.com (Neil Girdhar) Date: Mon, 17 Nov 2014 12:33:56 -0500 Subject: [Python-ideas] Implement `itertools.permutations.__getitem__` and `itertools.permutations.index` In-Reply-To: References: <20140505171538.GR4273@ando> <20140506023902.GV4273@ando> <18aa148c-f6cb-414c-9f6c-de5011568204@googlegroups.com> Message-ID: Looks great! Why did you go with "get_rapplied", "unrapplied", etc. instead of having a copy() method and using settable properties? On Mon, Nov 17, 2014 at 10:50 AM, Ram Rachum wrote: > Hi guys, > > I just wanted to give an update on this: I just released my own code that > does this to PyPI: https://pypi.python.org/pypi/combi > > > Thanks, > Ram. > > On Tue, Jun 10, 2014 at 9:24 AM, Ram Rachum wrote: > >> I'll email you if/when it's released :) >> >> >> On Tue, Jun 10, 2014 at 9:04 AM, Neil Girdhar >> wrote: >> >>> I really like this and hope that it eventually makes it into the >>> stdlib. It's also a good argument for your other suggestion whereby some >>> of the itertools to return Iterables rather than Iterators like range does. >>> >>> Best, >>> >>> Neil >>> >>> On Wednesday, May 7, 2014 1:43:20 PM UTC-4, Ram Rachum wrote: >>> >>>> I'm probably going to implement it in my python_toolbox package. I >>>> already implemented 30% and it's really cool. It's at the point where I >>>> doubt that I want it in the stdlib because I've gotten so much awesome >>>> functionality into it and I'd hate to (a) have 80% of it stripped and (b) >>>> have the class names changed to be non-Pythonic :) >>>> >>>> >>>> On Wed, May 7, 2014 at 8:40 PM, Tal Einat wrote: >>>> >>>> On Wed, May 7, 2014 at 8:21 PM, Ram Rachum wrote: >>>>> > Hi Tal, >>>>> > >>>>> > I'm using it for a project of my own (optimizing keyboard layout) >>>>> but I >>>>> > can't make the case that it's useful for the stdlib. I'd understand >>>>> if it >>>>> > would be omitted for not being enough of a common need. >>>>> >>>>> At the least, this (a function for getting a specific permutation by >>>>> lexicographical-order index) could make a nice cookbook recipe. >>>>> >>>>> - Tal >>>>> >>>> >>>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> -- >>> >>> --- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "python-ideas" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/python-ideas/in2gUQMFUzA/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> python-ideas+unsubscribe at googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ram at rachum.com Mon Nov 17 18:44:46 2014 From: ram at rachum.com (Ram Rachum) Date: Mon, 17 Nov 2014 19:44:46 +0200 Subject: [Python-ideas] Implement `itertools.permutations.__getitem__` and `itertools.permutations.index` In-Reply-To: References: <20140505171538.GR4273@ando> <20140506023902.GV4273@ando> <18aa148c-f6cb-414c-9f6c-de5011568204@googlegroups.com> Message-ID: Thanks :) Settable properties are a no-go because I wanted permutation spaces to be immutable. Since the cost of creating a new space is nil (permutations are created on-demand, not on space creation) there isn't a reason to mutate an existing space. I don't think using a copy() method to would be very nice. But I guess it's a matter of taste. On Mon, Nov 17, 2014 at 7:33 PM, Neil Girdhar wrote: > Looks great! > > Why did you go with "get_rapplied", "unrapplied", etc. instead of having a > copy() method and using settable properties? > > On Mon, Nov 17, 2014 at 10:50 AM, Ram Rachum wrote: > >> Hi guys, >> >> I just wanted to give an update on this: I just released my own code that >> does this to PyPI: https://pypi.python.org/pypi/combi >> >> >> Thanks, >> Ram. >> >> On Tue, Jun 10, 2014 at 9:24 AM, Ram Rachum wrote: >> >>> I'll email you if/when it's released :) >>> >>> >>> On Tue, Jun 10, 2014 at 9:04 AM, Neil Girdhar >>> wrote: >>> >>>> I really like this and hope that it eventually makes it into the >>>> stdlib. It's also a good argument for your other suggestion whereby some >>>> of the itertools to return Iterables rather than Iterators like range does. >>>> >>>> Best, >>>> >>>> Neil >>>> >>>> On Wednesday, May 7, 2014 1:43:20 PM UTC-4, Ram Rachum wrote: >>>> >>>>> I'm probably going to implement it in my python_toolbox package. I >>>>> already implemented 30% and it's really cool. It's at the point where I >>>>> doubt that I want it in the stdlib because I've gotten so much awesome >>>>> functionality into it and I'd hate to (a) have 80% of it stripped and (b) >>>>> have the class names changed to be non-Pythonic :) >>>>> >>>>> >>>>> On Wed, May 7, 2014 at 8:40 PM, Tal Einat wrote: >>>>> >>>>> On Wed, May 7, 2014 at 8:21 PM, Ram Rachum wrote: >>>>>> > Hi Tal, >>>>>> > >>>>>> > I'm using it for a project of my own (optimizing keyboard layout) >>>>>> but I >>>>>> > can't make the case that it's useful for the stdlib. I'd understand >>>>>> if it >>>>>> > would be omitted for not being enough of a common need. >>>>>> >>>>>> At the least, this (a function for getting a specific permutation by >>>>>> lexicographical-order index) could make a nice cookbook recipe. >>>>>> >>>>>> - Tal >>>>>> >>>>> >>>>> >>>> _______________________________________________ >>>> Python-ideas mailing list >>>> Python-ideas at python.org >>>> https://mail.python.org/mailman/listinfo/python-ideas >>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>>> -- >>>> >>>> --- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "python-ideas" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/python-ideas/in2gUQMFUzA/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to >>>> python-ideas+unsubscribe at googlegroups.com. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeanpierreda at gmail.com Mon Nov 17 18:53:17 2014 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Mon, 17 Nov 2014 09:53:17 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: On Mon, Nov 17, 2014 at 9:40 AM, Guido van Rossum wrote: > I don't want to contemplate a new __next__ protocol. The existing protocol > was carefully designed and tuned to have minimal memory overhead (even to > the point where the exception instance returned may be reused). Wapping each > result would just result in an extra allocation + deallocation per > iteration, unless you can play games with reference counts or do something > else to complicate the semantics). Introducing __nextx__ would require > thousands of libraries implementing this to incur churn as they feel the > pressure to switch to the new protocol, and the compatibility issue would be > felt everywhere. This sounds totally reasonable to me. > The problem we're trying to fix is unique to generators (thereby also > implicating generator expressions). I suppose since you're only fixing generators, then that is literally the only problem you are trying to fix, but it is more general than that. I have encountered this sort of problem writing __next__ by hand in Python -- that is, that bugs inside code I call result in silent control flow changes rather than a visible exception. -- Devin From mistersheik at gmail.com Mon Nov 17 19:01:32 2014 From: mistersheik at gmail.com (Neil Girdhar) Date: Mon, 17 Nov 2014 13:01:32 -0500 Subject: [Python-ideas] Implement `itertools.permutations.__getitem__` and `itertools.permutations.index` In-Reply-To: References: <20140505171538.GR4273@ando> <20140506023902.GV4273@ando> <18aa148c-f6cb-414c-9f6c-de5011568204@googlegroups.com> Message-ID: Sure, but even list has copy(). I meant settable property on the generator not on the generated permutation. On Mon, Nov 17, 2014 at 12:44 PM, Ram Rachum wrote: > Thanks :) > > Settable properties are a no-go because I wanted permutation spaces to be > immutable. Since the cost of creating a new space is nil (permutations are > created on-demand, not on space creation) there isn't a reason to mutate an > existing space. > > I don't think using a copy() method to would be very nice. But I guess > it's a matter of taste. > > On Mon, Nov 17, 2014 at 7:33 PM, Neil Girdhar > wrote: > >> Looks great! >> >> Why did you go with "get_rapplied", "unrapplied", etc. instead of having >> a copy() method and using settable properties? >> >> On Mon, Nov 17, 2014 at 10:50 AM, Ram Rachum wrote: >> >>> Hi guys, >>> >>> I just wanted to give an update on this: I just released my own code >>> that does this to PyPI: https://pypi.python.org/pypi/combi >>> >>> >>> Thanks, >>> Ram. >>> >>> On Tue, Jun 10, 2014 at 9:24 AM, Ram Rachum wrote: >>> >>>> I'll email you if/when it's released :) >>>> >>>> >>>> On Tue, Jun 10, 2014 at 9:04 AM, Neil Girdhar >>>> wrote: >>>> >>>>> I really like this and hope that it eventually makes it into the >>>>> stdlib. It's also a good argument for your other suggestion whereby some >>>>> of the itertools to return Iterables rather than Iterators like range does. >>>>> >>>>> Best, >>>>> >>>>> Neil >>>>> >>>>> On Wednesday, May 7, 2014 1:43:20 PM UTC-4, Ram Rachum wrote: >>>>> >>>>>> I'm probably going to implement it in my python_toolbox package. I >>>>>> already implemented 30% and it's really cool. It's at the point where I >>>>>> doubt that I want it in the stdlib because I've gotten so much awesome >>>>>> functionality into it and I'd hate to (a) have 80% of it stripped and (b) >>>>>> have the class names changed to be non-Pythonic :) >>>>>> >>>>>> >>>>>> On Wed, May 7, 2014 at 8:40 PM, Tal Einat wrote: >>>>>> >>>>>> On Wed, May 7, 2014 at 8:21 PM, Ram Rachum >>>>>>> wrote: >>>>>>> > Hi Tal, >>>>>>> > >>>>>>> > I'm using it for a project of my own (optimizing keyboard layout) >>>>>>> but I >>>>>>> > can't make the case that it's useful for the stdlib. I'd >>>>>>> understand if it >>>>>>> > would be omitted for not being enough of a common need. >>>>>>> >>>>>>> At the least, this (a function for getting a specific permutation by >>>>>>> lexicographical-order index) could make a nice cookbook recipe. >>>>>>> >>>>>>> - Tal >>>>>>> >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> Python-ideas mailing list >>>>> Python-ideas at python.org >>>>> https://mail.python.org/mailman/listinfo/python-ideas >>>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>>>> -- >>>>> >>>>> --- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "python-ideas" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/python-ideas/in2gUQMFUzA/unsubscribe >>>>> . >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> python-ideas+unsubscribe at googlegroups.com. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Nov 17 19:01:46 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 17 Nov 2014 10:01:46 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <7ecbb0e7ad7c4aa58314d63c454ed386@DM2PR0301MB0734.namprd03.prod.outlook.com> <54691F8D.1080703@canterbury.ac.nz> <20141117000302.GS2748@ando.pearwood.info> <54696A02.6050602@canterbury.ac.nz> Message-ID: On Mon, Nov 17, 2014 at 4:56 AM, Nick Coghlan wrote: > On 17 November 2014 13:34, Chris Angelico wrote: > >> Right. So, are there any __future__ directives that have any effect on >> byte-code? I'm not seeing any, though that may just mean I didn't >> recognize one. >> > > True division (in Python 2) is a nice simple one to look at, since it just > swaps one bytecode for another (BINARY_DIVIDE -> BINARY_TRUE_DIVIDE) > > >>> def floor_div(x, y): > ... return x / y > ... > >>> from __future__ import division > >>> def true_div(x, y): > ... return x / y > ... > >>> import dis > >>> dis.dis(floor_div) > 2 0 LOAD_FAST 0 (x) > 3 LOAD_FAST 1 (y) > 6 BINARY_DIVIDE > 7 RETURN_VALUE > >>> dis.dis(true_div) > 2 0 LOAD_FAST 0 (x) > 3 LOAD_FAST 1 (y) > 6 BINARY_TRUE_DIVIDE > 7 RETURN_VALUE > > The compiler actually stores a whole pile of useful info on code objects > that doesn't show up in the disassembly output (switching to Python 3 for > more up to date dis module goodness): > > >>> import dis > >>> def true_div(x, y): > ... return x / y > ... > >>> dis.show_code(true_div) > Name: true_div > Filename: > Argument count: 2 > Kw-only arguments: 0 > Number of locals: 2 > Stack size: 2 > Flags: OPTIMIZED, NEWLOCALS, NOFREE > Constants: > 0: None > Variable names: > 0: x > 1: y > > So conveying to the generator iterator whether or not "from __future__ > import generator_return" was in effect would just be a matter of the > compiler setting a new flag on the generator code object. For *affected > generators* (i.e. those defined in a module where the new future statement > was in effect), StopIteration escaping would be considered a RuntimeError. > > For almost all code, such RuntimeErrors would look like any other > RuntimError raised by a broken generator implementation. > > The only code which would *have* to change immediately as a "Porting to > Python 3.5" requirement is code like that in contextlib, which throws > StopIteration into generators, and currently expects to get it back out > unmodified. Such code will need to be updated to also handle RuntimError > instances where the direct cause is the StopIteration exception that was > thrown in. > > Other affected code (such as the "next() bubbling up" groupby example) > would keep working unless the __future__ statement was in effect. > Thanks, this is exactly what I was thinking of. The new flag could be named REPLACE_STOPITERATION. Then the __future__ import could be named replace_stopiteration_in_generators (it needs more description than the flag name because the flag is already defined in the context of a generator, while the __future__ import must still establish that context). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Nov 17 19:02:36 2014 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 18 Nov 2014 05:02:36 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <7ecbb0e7ad7c4aa58314d63c454ed386@DM2PR0301MB0734.namprd03.prod.outlook.com> <54691F8D.1080703@canterbury.ac.nz> <20141117000302.GS2748@ando.pearwood.info> Message-ID: On Tue, Nov 18, 2014 at 4:28 AM, Guido van Rossum wrote: > Off-topic: the new python.org site now supports PEPs, so please switch to > URLs like this: https://www.python.org/dev/peps/pep-0479/ (if you don't like > the formatting send a pull request to > https://github.com/python/pythondotorg). Oh, nice. Google searches for PEPs still find legacy rather than that, so it may be worth setting legacy to redirect to www. Formatting looks fine, except that the subheadings look bolder than the main; I'll check for issues and post one, though probably not a PR. ChrisA From rosuav at gmail.com Mon Nov 17 19:09:49 2014 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 18 Nov 2014 05:09:49 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: On Tue, Nov 18, 2014 at 4:40 AM, Guido van Rossum wrote: > Wapping each result would just result in an extra allocation + deallocation > per iteration... Which is why I would be more inclined to use a sentinel of some sort... but that has its own problems. There's no perfect solution, so status quo wins unless a really compelling case can be made. I could toss something into the Alternate Proposals section, but I wouldn't be personally supporting it. ChrisA From guido at python.org Mon Nov 17 19:09:36 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 17 Nov 2014 10:09:36 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: On Mon, Nov 17, 2014 at 9:53 AM, Devin Jeanpierre wrote: [Guido] > > The problem we're trying to fix is unique to generators (thereby also > > implicating generator expressions). > > I suppose since you're only fixing generators, then that is literally > the only problem you are trying to fix, but it is more general than > that. I have encountered this sort of problem writing __next__ by hand > in Python -- that is, that bugs inside code I call result in silent > control flow changes rather than a visible exception. > I assume this is something where the __next__() method on your iterator class calls next() on some other iterator and accidentally doesn't catch the StopIteration coming out of it (or, more likely, this happens several calls deep, making it more interesting to debug). That particular problem is not unique to __next__ and StopIteration -- the same thing can (and does!) happen with __getitem__ and KeyError or IndexError, and with __getattr[ibute]__ and AttributeError. In all these cases I think there isn't much we can do apart from adding lint rules. If you are writing __next__ as a method on an iterator class, one way or another you are going to have to raise StopIteration when there isn't another element, and similarly __getitem__ has to raise KeyError or IndexError, etc. In the generator case, we have a better way to signal the end -- a return statement (or falling off the end). And that's why we can even contemplate doing something different when StopIteration is raised in the generator. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Nov 17 19:12:50 2014 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 18 Nov 2014 05:12:50 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: On Tue, Nov 18, 2014 at 4:53 AM, Devin Jeanpierre wrote: > I suppose since you're only fixing generators, then that is literally > the only problem you are trying to fix, but it is more general than > that. I have encountered this sort of problem writing __next__ by hand > in Python -- that is, that bugs inside code I call result in silent > control flow changes rather than a visible exception. If you're writing __next__ by hand, there's nothing anyone else can do about a bubbled-up StopIteration. What you can do is wrap your code in try/except: def __next__(self): try: # whatever except StopIteration: raise RuntimeError raise StopIteration If your "whatever" section returns a value, that's what the result will be. If it fails to return a value, StopIteration will be raised. And if StopIteration is raised, it'll become RuntimeError. But this has to be inside __next__. This can't be done externally. (Note, untested code. May have bugs.) ChrisA From guido at python.org Mon Nov 17 19:17:21 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 17 Nov 2014 10:17:21 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: On Mon, Nov 17, 2014 at 10:09 AM, Chris Angelico wrote: > On Tue, Nov 18, 2014 at 4:40 AM, Guido van Rossum > wrote: > > Wapping each result would just result in an extra allocation + > deallocation > > per iteration... > > Which is why I would be more inclined to use a sentinel of some > sort... but that has its own problems. There's no perfect solution, so > status quo wins unless a really compelling case can be made. I could > toss something into the Alternate Proposals section, but I wouldn't be > personally supporting it. > Trust me, we went down this rabbit hole when we designed the iterator protocol. The problem is that that sentinel object must have a name (otherwise how would you know when you had seen the sentinel), which means that there is at least one dict (the namespace defining that name, probably the builtins module) that has the sentinel as one of its values, which means that iterating over that particular dict's values would see the sentinel as a legitimate value (and terminate prematurely). You could fix this by allocating a unique sentinel for every iteration, but there would be some additional overhead for that too (the caller and the iterator both need to hold on to the sentinel object). In any case, as I tried to say before, redesigning the iterator protocol is most definitely out of scope here (you could write a separate PEP and I'd reject it instantly). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Nov 17 19:23:25 2014 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 18 Nov 2014 05:23:25 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: On Tue, Nov 18, 2014 at 5:17 AM, Guido van Rossum wrote: > Trust me, we went down this rabbit hole when we designed the iterator > protocol. The problem is that that sentinel object must have a name > (otherwise how would you know when you had seen the sentinel), which means > that there is at least one dict (the namespace defining that name, probably > the builtins module) that has the sentinel as one of its values, which means > that iterating over that particular dict's values would see the sentinel as > a legitimate value (and terminate prematurely). You could fix this by > allocating a unique sentinel for every iteration, but there would be some > additional overhead for that too (the caller and the iterator both need to > hold on to the sentinel object). That's a much more solid argument against it than I had. (Also amusing to try to contemplate.) The iterator protocol absolutely demands a completely out-of-band means of signalling "I have nothing to return now". > In any case, as I tried to say before, redesigning the iterator protocol is > most definitely out of scope here (you could write a separate PEP and I'd > reject it instantly). That's a PEP I'll leave for someone else to write. :) ChrisA From ethan at stoneleaf.us Mon Nov 17 19:26:54 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 17 Nov 2014 10:26:54 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: <546A3DEE.5040206@stoneleaf.us> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 11/17/2014 10:23 AM, Chris Angelico wrote: > On Tue, Nov 18, 2014 at 5:17 AM, Guido van Rossum wrote: >> >> In any case, as I tried to say before, redesigning the iterator protocol is most definitely out of scope here >> (you could write a separate PEP and I'd reject it instantly). > > That's a PEP I'll leave for someone else to write. :) On the other hand, if you did write it (now), and Guido rejected it (of course), then that would mean PEP 479 is sure to be accepted! Third time's the charm! ;) - -- ~Ethan~ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iQIcBAEBAgAGBQJUaj3uAAoJENZ7D1rrH75NzRYP/2NcwwNOQvD71RlhJG0k0sUY xLgjzLSCVQbR1jliF1Ej9BEZhFhd1I4lg/msjDPNBSHABPtsbrRTzZiac4x3OAEz 5L5qCB7hIaHg+gDKEbCiOl5NAZUbZL+Qs1tFdlSjeCti/YUAVE1DiD4WQ+kXHxZ4 OF9Nj5urHrGQYh//HR3RryQddihPBkM9Y40oucRwQE32jzcJacgmEvOM4aVvJHb8 2rBfl7VjpHwC2dV51R3cYpwIu2Qqk9i4kEnkB9xDtj9HQl3YT+qEyAuZWOs7v3cH cJOjBuIxJGYF/CbjRE5/mCFegvGv4Lf2C8QSeKlqR549IPhso+qcnLvTQNdBWer0 g9L8Gsk1whUR9GQ3HGaupAkAo8rbZKM9fjvEtnPvcWAPcGZDfeFG5MEl82SCOsLS aUl+v5u2sVozxIsY9fr8s30X2HZReaVjuJFDLCosZmi3RcqYL7O0x5+Mo4GPgsbi YxyCYU0c0LTTk7yljweYL/oWXyUS/hAxz2VNMyAjDNw1Uy1kWwS6qAW5FcMxfdlz I92CBx0PXPLz1hhHYsJ/YnHIJpL9+HRHHRRdJOnHOGCyb7AX6jN3ruMGYaTClrYa G2ez3JMeu0NhTuzpFF2mjO0PXTu/HuHq3bdv44jhZS0Zn67OQmNk845bZi5U3EaW oJTRKMM/PTZNpspQE0/n =ZYwx -----END PGP SIGNATURE----- From ram at rachum.com Mon Nov 17 20:37:05 2014 From: ram at rachum.com (Ram Rachum) Date: Mon, 17 Nov 2014 21:37:05 +0200 Subject: [Python-ideas] Implement `itertools.permutations.__getitem__` and `itertools.permutations.index` In-Reply-To: References: <20140505171538.GR4273@ando> <20140506023902.GV4273@ando> <18aa148c-f6cb-414c-9f6c-de5011568204@googlegroups.com> Message-ID: I'm not sure what you mean. Can you please give an example? On Mon, Nov 17, 2014 at 8:01 PM, Neil Girdhar wrote: > Sure, but even list has copy(). I meant settable property on the > generator not on the generated permutation. > > On Mon, Nov 17, 2014 at 12:44 PM, Ram Rachum wrote: > >> Thanks :) >> >> Settable properties are a no-go because I wanted permutation spaces to be >> immutable. Since the cost of creating a new space is nil (permutations are >> created on-demand, not on space creation) there isn't a reason to mutate an >> existing space. >> >> I don't think using a copy() method to would be very nice. But I guess >> it's a matter of taste. >> >> On Mon, Nov 17, 2014 at 7:33 PM, Neil Girdhar >> wrote: >> >>> Looks great! >>> >>> Why did you go with "get_rapplied", "unrapplied", etc. instead of having >>> a copy() method and using settable properties? >>> >>> On Mon, Nov 17, 2014 at 10:50 AM, Ram Rachum wrote: >>> >>>> Hi guys, >>>> >>>> I just wanted to give an update on this: I just released my own code >>>> that does this to PyPI: https://pypi.python.org/pypi/combi >>>> >>>> >>>> Thanks, >>>> Ram. >>>> >>>> On Tue, Jun 10, 2014 at 9:24 AM, Ram Rachum wrote: >>>> >>>>> I'll email you if/when it's released :) >>>>> >>>>> >>>>> On Tue, Jun 10, 2014 at 9:04 AM, Neil Girdhar >>>>> wrote: >>>>> >>>>>> I really like this and hope that it eventually makes it into the >>>>>> stdlib. It's also a good argument for your other suggestion whereby some >>>>>> of the itertools to return Iterables rather than Iterators like range does. >>>>>> >>>>>> Best, >>>>>> >>>>>> Neil >>>>>> >>>>>> On Wednesday, May 7, 2014 1:43:20 PM UTC-4, Ram Rachum wrote: >>>>>> >>>>>>> I'm probably going to implement it in my python_toolbox package. I >>>>>>> already implemented 30% and it's really cool. It's at the point where I >>>>>>> doubt that I want it in the stdlib because I've gotten so much awesome >>>>>>> functionality into it and I'd hate to (a) have 80% of it stripped and (b) >>>>>>> have the class names changed to be non-Pythonic :) >>>>>>> >>>>>>> >>>>>>> On Wed, May 7, 2014 at 8:40 PM, Tal Einat wrote: >>>>>>> >>>>>>> On Wed, May 7, 2014 at 8:21 PM, Ram Rachum >>>>>>>> wrote: >>>>>>>> > Hi Tal, >>>>>>>> > >>>>>>>> > I'm using it for a project of my own (optimizing keyboard layout) >>>>>>>> but I >>>>>>>> > can't make the case that it's useful for the stdlib. I'd >>>>>>>> understand if it >>>>>>>> > would be omitted for not being enough of a common need. >>>>>>>> >>>>>>>> At the least, this (a function for getting a specific permutation by >>>>>>>> lexicographical-order index) could make a nice cookbook recipe. >>>>>>>> >>>>>>>> - Tal >>>>>>>> >>>>>>> >>>>>>> >>>>>> _______________________________________________ >>>>>> Python-ideas mailing list >>>>>> Python-ideas at python.org >>>>>> https://mail.python.org/mailman/listinfo/python-ideas >>>>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>>>>> -- >>>>>> >>>>>> --- >>>>>> You received this message because you are subscribed to a topic in >>>>>> the Google Groups "python-ideas" group. >>>>>> To unsubscribe from this topic, visit >>>>>> https://groups.google.com/d/topic/python-ideas/in2gUQMFUzA/unsubscribe >>>>>> . >>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>> python-ideas+unsubscribe at googlegroups.com. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Nov 17 20:38:27 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 17 Nov 2014 11:38:27 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: <546A3DEE.5040206@stoneleaf.us> References: <546A3DEE.5040206@stoneleaf.us> Message-ID: FWIW, I spent some time this morning close-reading the PEP, and made a somewhat significant set of updates -- adding more specifics about the proposed new __futue__ statement and the new code object flag, tracking down a few more examples of code that would be affected, and some other minor edits. Here's the diff: https://hg.python.org/peps/rev/8de949863677 Hopefully the new version will soon be here: https://www.python.org/dev/peps/pep-0479 Note that I am definitely not yet deciding on this PEP. I would love it if people sent in examples of code using generator expressions that would be affected by this change (either by highlighting a bug in the code or by breaking what currently works). If this PEP gets rejected, we could resurrect the GeneratorExit proposal currently listed as an alternative -- although the more I think about that the less I think it's worth it, except for the very specific case of asyncio (thinking of which, I should add something to the PEP about that too). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Mon Nov 17 20:40:39 2014 From: mistersheik at gmail.com (Neil Girdhar) Date: Mon, 17 Nov 2014 14:40:39 -0500 Subject: [Python-ideas] Implement `itertools.permutations.__getitem__` and `itertools.permutations.index` In-Reply-To: References: <20140505171538.GR4273@ando> <20140506023902.GV4273@ando> <18aa148c-f6cb-414c-9f6c-de5011568204@googlegroups.com> Message-ID: perm_space = PermSpace(3) perm_space.degrees = 3 list(perm_space) perm_space.degrees = None list(perm_space) etc. On Mon, Nov 17, 2014 at 2:37 PM, Ram Rachum wrote: > I'm not sure what you mean. Can you please give an example? > > On Mon, Nov 17, 2014 at 8:01 PM, Neil Girdhar > wrote: > >> Sure, but even list has copy(). I meant settable property on the >> generator not on the generated permutation. >> >> On Mon, Nov 17, 2014 at 12:44 PM, Ram Rachum wrote: >> >>> Thanks :) >>> >>> Settable properties are a no-go because I wanted permutation spaces to >>> be immutable. Since the cost of creating a new space is nil (permutations >>> are created on-demand, not on space creation) there isn't a reason to >>> mutate an existing space. >>> >>> I don't think using a copy() method to would be very nice. But I guess >>> it's a matter of taste. >>> >>> On Mon, Nov 17, 2014 at 7:33 PM, Neil Girdhar >>> wrote: >>> >>>> Looks great! >>>> >>>> Why did you go with "get_rapplied", "unrapplied", etc. instead of >>>> having a copy() method and using settable properties? >>>> >>>> On Mon, Nov 17, 2014 at 10:50 AM, Ram Rachum wrote: >>>> >>>>> Hi guys, >>>>> >>>>> I just wanted to give an update on this: I just released my own code >>>>> that does this to PyPI: https://pypi.python.org/pypi/combi >>>>> >>>>> >>>>> Thanks, >>>>> Ram. >>>>> >>>>> On Tue, Jun 10, 2014 at 9:24 AM, Ram Rachum wrote: >>>>> >>>>>> I'll email you if/when it's released :) >>>>>> >>>>>> >>>>>> On Tue, Jun 10, 2014 at 9:04 AM, Neil Girdhar >>>>>> wrote: >>>>>> >>>>>>> I really like this and hope that it eventually makes it into the >>>>>>> stdlib. It's also a good argument for your other suggestion whereby some >>>>>>> of the itertools to return Iterables rather than Iterators like range does. >>>>>>> >>>>>>> Best, >>>>>>> >>>>>>> Neil >>>>>>> >>>>>>> On Wednesday, May 7, 2014 1:43:20 PM UTC-4, Ram Rachum wrote: >>>>>>> >>>>>>>> I'm probably going to implement it in my python_toolbox package. I >>>>>>>> already implemented 30% and it's really cool. It's at the point where I >>>>>>>> doubt that I want it in the stdlib because I've gotten so much awesome >>>>>>>> functionality into it and I'd hate to (a) have 80% of it stripped and (b) >>>>>>>> have the class names changed to be non-Pythonic :) >>>>>>>> >>>>>>>> >>>>>>>> On Wed, May 7, 2014 at 8:40 PM, Tal Einat >>>>>>>> wrote: >>>>>>>> >>>>>>>> On Wed, May 7, 2014 at 8:21 PM, Ram Rachum >>>>>>>>> wrote: >>>>>>>>> > Hi Tal, >>>>>>>>> > >>>>>>>>> > I'm using it for a project of my own (optimizing keyboard >>>>>>>>> layout) but I >>>>>>>>> > can't make the case that it's useful for the stdlib. I'd >>>>>>>>> understand if it >>>>>>>>> > would be omitted for not being enough of a common need. >>>>>>>>> >>>>>>>>> At the least, this (a function for getting a specific permutation >>>>>>>>> by >>>>>>>>> lexicographical-order index) could make a nice cookbook recipe. >>>>>>>>> >>>>>>>>> - Tal >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> Python-ideas mailing list >>>>>>> Python-ideas at python.org >>>>>>> https://mail.python.org/mailman/listinfo/python-ideas >>>>>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>>>>>> -- >>>>>>> >>>>>>> --- >>>>>>> You received this message because you are subscribed to a topic in >>>>>>> the Google Groups "python-ideas" group. >>>>>>> To unsubscribe from this topic, visit >>>>>>> https://groups.google.com/d/topic/python-ideas/in2gUQMFUzA/unsubscribe >>>>>>> . >>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>> python-ideas+unsubscribe at googlegroups.com. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ram at rachum.com Mon Nov 17 20:57:55 2014 From: ram at rachum.com (Ram Rachum) Date: Mon, 17 Nov 2014 21:57:55 +0200 Subject: [Python-ideas] Implement `itertools.permutations.__getitem__` and `itertools.permutations.index` In-Reply-To: References: <20140505171538.GR4273@ando> <20140506023902.GV4273@ando> <18aa148c-f6cb-414c-9f6c-de5011568204@googlegroups.com> Message-ID: Ah, I understand. I don't like it, though I'm not sure I can really express why. One reason is that this would mean that PermSpaces would be mutable, and thus not hashable and not usable as keys in dicts and sets. It would also mean I couldn't use the `CachedProperty` pattern all over the class as I do today. So these are a couple of reasons, there might be more that didn't come to me now. On Mon, Nov 17, 2014 at 9:40 PM, Neil Girdhar wrote: > perm_space = PermSpace(3) > > perm_space.degrees = 3 > > list(perm_space) > > perm_space.degrees = None > > list(perm_space) > > etc. > > On Mon, Nov 17, 2014 at 2:37 PM, Ram Rachum wrote: > >> I'm not sure what you mean. Can you please give an example? >> >> On Mon, Nov 17, 2014 at 8:01 PM, Neil Girdhar >> wrote: >> >>> Sure, but even list has copy(). I meant settable property on the >>> generator not on the generated permutation. >>> >>> On Mon, Nov 17, 2014 at 12:44 PM, Ram Rachum wrote: >>> >>>> Thanks :) >>>> >>>> Settable properties are a no-go because I wanted permutation spaces to >>>> be immutable. Since the cost of creating a new space is nil (permutations >>>> are created on-demand, not on space creation) there isn't a reason to >>>> mutate an existing space. >>>> >>>> I don't think using a copy() method to would be very nice. But I guess >>>> it's a matter of taste. >>>> >>>> On Mon, Nov 17, 2014 at 7:33 PM, Neil Girdhar >>>> wrote: >>>> >>>>> Looks great! >>>>> >>>>> Why did you go with "get_rapplied", "unrapplied", etc. instead of >>>>> having a copy() method and using settable properties? >>>>> >>>>> On Mon, Nov 17, 2014 at 10:50 AM, Ram Rachum wrote: >>>>> >>>>>> Hi guys, >>>>>> >>>>>> I just wanted to give an update on this: I just released my own code >>>>>> that does this to PyPI: https://pypi.python.org/pypi/combi >>>>>> >>>>>> >>>>>> Thanks, >>>>>> Ram. >>>>>> >>>>>> On Tue, Jun 10, 2014 at 9:24 AM, Ram Rachum wrote: >>>>>> >>>>>>> I'll email you if/when it's released :) >>>>>>> >>>>>>> >>>>>>> On Tue, Jun 10, 2014 at 9:04 AM, Neil Girdhar >>>>>> > wrote: >>>>>>> >>>>>>>> I really like this and hope that it eventually makes it into the >>>>>>>> stdlib. It's also a good argument for your other suggestion whereby some >>>>>>>> of the itertools to return Iterables rather than Iterators like range does. >>>>>>>> >>>>>>>> Best, >>>>>>>> >>>>>>>> Neil >>>>>>>> >>>>>>>> On Wednesday, May 7, 2014 1:43:20 PM UTC-4, Ram Rachum wrote: >>>>>>>> >>>>>>>>> I'm probably going to implement it in my python_toolbox package. I >>>>>>>>> already implemented 30% and it's really cool. It's at the point where I >>>>>>>>> doubt that I want it in the stdlib because I've gotten so much awesome >>>>>>>>> functionality into it and I'd hate to (a) have 80% of it stripped and (b) >>>>>>>>> have the class names changed to be non-Pythonic :) >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, May 7, 2014 at 8:40 PM, Tal Einat >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> On Wed, May 7, 2014 at 8:21 PM, Ram Rachum >>>>>>>>>> wrote: >>>>>>>>>> > Hi Tal, >>>>>>>>>> > >>>>>>>>>> > I'm using it for a project of my own (optimizing keyboard >>>>>>>>>> layout) but I >>>>>>>>>> > can't make the case that it's useful for the stdlib. I'd >>>>>>>>>> understand if it >>>>>>>>>> > would be omitted for not being enough of a common need. >>>>>>>>>> >>>>>>>>>> At the least, this (a function for getting a specific permutation >>>>>>>>>> by >>>>>>>>>> lexicographical-order index) could make a nice cookbook recipe. >>>>>>>>>> >>>>>>>>>> - Tal >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Python-ideas mailing list >>>>>>>> Python-ideas at python.org >>>>>>>> https://mail.python.org/mailman/listinfo/python-ideas >>>>>>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>>>>>>> -- >>>>>>>> >>>>>>>> --- >>>>>>>> You received this message because you are subscribed to a topic in >>>>>>>> the Google Groups "python-ideas" group. >>>>>>>> To unsubscribe from this topic, visit >>>>>>>> https://groups.google.com/d/topic/python-ideas/in2gUQMFUzA/unsubscribe >>>>>>>> . >>>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>>> python-ideas+unsubscribe at googlegroups.com. >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Mon Nov 17 20:59:50 2014 From: mistersheik at gmail.com (Neil Girdhar) Date: Mon, 17 Nov 2014 14:59:50 -0500 Subject: [Python-ideas] Implement `itertools.permutations.__getitem__` and `itertools.permutations.index` In-Reply-To: References: <20140505171538.GR4273@ando> <20140506023902.GV4273@ando> <18aa148c-f6cb-414c-9f6c-de5011568204@googlegroups.com> Message-ID: Those are good reasons, thanks. :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue Nov 18 00:05:17 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 18 Nov 2014 10:05:17 +1100 Subject: [Python-ideas] Implement `itertools.permutations.__getitem__` and `itertools.permutations.index` In-Reply-To: References: <18aa148c-f6cb-414c-9f6c-de5011568204@googlegroups.com> Message-ID: <20141117230517.GU2748@ando.pearwood.info> On Mon, Nov 17, 2014 at 09:57:55PM +0200, Ram Rachum wrote: > Ah, I understand. I don't like it, though I'm not sure I can really express > why. One reason is that this would mean that PermSpaces would be mutable, > and thus not hashable and not usable as keys in dicts and sets. I'm having trouble thinking of any reason to use a PermSpace as a dict key or set element. Do you have a concrete example of doing so? -- Steven From ram at rachum.com Tue Nov 18 00:18:14 2014 From: ram at rachum.com (Ram Rachum) Date: Tue, 18 Nov 2014 01:18:14 +0200 Subject: [Python-ideas] Implement `itertools.permutations.__getitem__` and `itertools.permutations.index` In-Reply-To: <20141117230517.GU2748@ando.pearwood.info> References: <18aa148c-f6cb-414c-9f6c-de5011568204@googlegroups.com> <20141117230517.GU2748@ando.pearwood.info> Message-ID: I actually did that in the project I made Combi for. I had a bunch of perm spaces that signified certain configurations (can't get more specific, sorry) and I wanted to find the best configuration in each space. So I had a dict from each space to the best configuration in that space. On Tue, Nov 18, 2014 at 1:05 AM, Steven D'Aprano wrote: > On Mon, Nov 17, 2014 at 09:57:55PM +0200, Ram Rachum wrote: > > Ah, I understand. I don't like it, though I'm not sure I can really > express > > why. One reason is that this would mean that PermSpaces would be mutable, > > and thus not hashable and not usable as keys in dicts and sets. > > I'm having trouble thinking of any reason to use a PermSpace as a dict > key or set element. Do you have a concrete example of doing so? > > > > -- > Steven > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue Nov 18 00:56:42 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 18 Nov 2014 10:56:42 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <546A3DEE.5040206@stoneleaf.us> Message-ID: <20141117235641.GW2748@ando.pearwood.info> On Mon, Nov 17, 2014 at 11:38:27AM -0800, Guido van Rossum wrote: > Hopefully the new version will soon be here: > > https://www.python.org/dev/peps/pep-0479 > > Note that I am definitely not yet deciding on this PEP. I would love it if > people sent in examples of code using generator expressions that would be > affected by this change (either by highlighting a bug in the code or by > breaking what currently works). Over a week ago I raised this issue on python-list mailing list. I expected a storm of bike-shedding, because that's the sort of place p-l is :-) but got just two people commenting. The thread, for anyone interested: https://mail.python.org/pipermail/python-list/2014-November/680757.html One response suggested that it is not generators which do the wrong thing, but comprehensions, and that comprehensions should be changed to behave like generators: https://mail.python.org/pipermail/python-list/2014-November/680758.html That should probably be put in the PEP, even if it is not an option being considered, it at least evidence that "some people" find the behaviour of generators more natural than that of comprehensions. -- Steve From guido at python.org Tue Nov 18 01:13:23 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 17 Nov 2014 16:13:23 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: <20141117235641.GW2748@ando.pearwood.info> References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> Message-ID: Thanks, I added the survey and a summary to the PEP. If you hear about any examples please do forward them here! On Mon, Nov 17, 2014 at 3:56 PM, Steven D'Aprano wrote: > On Mon, Nov 17, 2014 at 11:38:27AM -0800, Guido van Rossum wrote: > > > Hopefully the new version will soon be here: > > > > https://www.python.org/dev/peps/pep-0479 > > > > Note that I am definitely not yet deciding on this PEP. I would love it > if > > people sent in examples of code using generator expressions that would be > > affected by this change (either by highlighting a bug in the code or by > > breaking what currently works). > > Over a week ago I raised this issue on python-list mailing list. I > expected a storm of bike-shedding, because that's the sort of place p-l > is :-) but got just two people commenting. > > The thread, for anyone interested: > > https://mail.python.org/pipermail/python-list/2014-November/680757.html > > One response suggested that it is not generators which do the wrong > thing, but comprehensions, and that comprehensions should be changed to > behave like generators: > > https://mail.python.org/pipermail/python-list/2014-November/680758.html > > That should probably be put in the PEP, even if it is not an option > being considered, it at least evidence that "some people" find the > behaviour of generators more natural than that of comprehensions. > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Nov 18 05:50:14 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 17 Nov 2014 20:50:14 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> Message-ID: Nick, I think we've gone through enough clarifications of the PEP now to be clear on the proposal. I saw in one of your earliest replies (right after Chris posted his first draft) that you're hesitant to support the PEP because of what would have to change to contextlib. What I couldn't quite read is whether you think that the proposal by itself is not an improvement, or whether you're just worried about compatibility. Apparently you know of a large group of users who use an older 3rd party version of contextlib, and for whom that older, 3rd party contextlib should keep working with future versions of Python 3 without updating their version of contextlib -- did I get that right? What exactly is the constraint there that makes their version of contextlib immutable even though the version of Python they are using may move forward? Separate from this special case, I am also worried about backward compatibility, and I have yet to get a good idea for how widespread code is that depends on StopIteration bubbling out from generators. I also don't have a good idea how often this issue bites users, but I have a feeling it does bite. E.g. this quote from c.l.py ( https://mail.python.org/pipermail/python-list/2014-November/680775.html): """ I did find it annoying occasionally that raising StopIteration inside a generator expression conveys a different behavior than elsewhere. It did take me quite a while to understand why that is so, but after that it did not cause me much of a headache anymore. """ -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From wolfgang.maier at biologie.uni-freiburg.de Tue Nov 18 18:40:30 2014 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Tue, 18 Nov 2014 18:40:30 +0100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> Message-ID: <546B848E.6040406@biologie.uni-freiburg.de> On 11/18/2014 05:50 AM, Guido van Rossum wrote: > > Separate from this special case, I am also worried about backward > compatibility, and I have yet to get a good idea for how widespread code > is that depends on StopIteration bubbling out from generators. I also > don't have a good idea how often this issue bites users, but I have a > feeling it does bite. E.g. this quote from c.l.py > (https://mail.python.org/pipermail/python-list/2014-November/680775.html): > > """ > I did find it annoying occasionally that raising StopIteration inside a > generator expression conveys a different behavior than elsewhere. It did > take me quite a while to understand why that is so, but after that it > did not cause me much of a headache anymore. > """ > I just remembered one use of the current behavior. Two years ago or so, I was suggesting on this list a possibility for early termination of comprehensions when a particular value is encountered. In other words, an equivalent to: l = [] for x in seq: if x == y: break l.append(x) At the time, somebody suggested (roughly): def stop(): raise StopIteration l = list(x for x in seq if x!=y or stop()) which, for the very reasons discussed in this thread, works only as a generator expression and not in comprehension form. I used this solution in some not particularly important piece of code so I wouldn't despair if it wasn't compatible with the next release of the language. Also, I have a feeling that some of you may consider this sort of a hack in the first place. Just thought I'd mention it here for completeness. Wolfgang From ethan at stoneleaf.us Tue Nov 18 20:09:43 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 18 Nov 2014 11:09:43 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> Message-ID: <546B9977.9020903@stoneleaf.us> On 11/17/2014 08:50 PM, Guido van Rossum wrote: > > Separate from this special case, I am also worried about backward > compatibility, and I have yet to get a good idea for how widespread code is > that depends on StopIteration bubbling out from generators. I also don't > have a good idea how often this issue bites users, but I have a feeling it > does bite. One argument for making the change*: When we're writing __next__, or __getattr__, etc., it is obvious that we are playing with internals and have to be extra careful of what other exceptions might be raised in that code. Contrariwise, the only indication of something special about a generator is the presence of the yield keyword -- for ordinary use (such as in for loops) it doesn't matter whether the called function returns a list, tuple, iterator, generator, or whatever, as long as it can be iterated over, and so when writing a generator, or converting an iterable-returning function into a generator, there's nothing obvious saying, "Hey! Watch out for a StopIteration somewhere else in this block of code!" * I make no statement as to how strong this argument is, but there you have it. :) -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From guido at python.org Tue Nov 18 21:36:35 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 18 Nov 2014 12:36:35 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: <546B9977.9020903@stoneleaf.us> References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> <546B9977.9020903@stoneleaf.us> Message-ID: That's pretty much the reason I gave this my attention at all. :-) On Tue, Nov 18, 2014 at 11:09 AM, Ethan Furman wrote: > On 11/17/2014 08:50 PM, Guido van Rossum wrote: > > > > Separate from this special case, I am also worried about backward > > compatibility, and I have yet to get a good idea for how widespread code > is > > that depends on StopIteration bubbling out from generators. I also don't > > have a good idea how often this issue bites users, but I have a feeling > it > > does bite. > > One argument for making the change*: When we're writing __next__, or > __getattr__, etc., it is obvious that we are > playing with internals and have to be extra careful of what other > exceptions might be raised in that code. > Contrariwise, the only indication of something special about a generator > is the presence of the yield keyword -- for > ordinary use (such as in for loops) it doesn't matter whether the called > function returns a list, tuple, iterator, > generator, or whatever, as long as it can be iterated over, and so when > writing a generator, or converting an > iterable-returning function into a generator, there's nothing obvious > saying, "Hey! Watch out for a StopIteration > somewhere else in this block of code!" > > * I make no statement as to how strong this argument is, but there you > have it. :) > > -- > ~Ethan~ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Tue Nov 18 23:54:57 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 18 Nov 2014 17:54:57 -0500 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: <546B848E.6040406@biologie.uni-freiburg.de> References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> <546B848E.6040406@biologie.uni-freiburg.de> Message-ID: On 11/18/2014 12:40 PM, Wolfgang Maier wrote: > I just remembered one use of the current behavior. Two years ago or so, > I was suggesting on this list a possibility for early termination of > comprehensions when a particular value is encountered. In other words, > an equivalent to: > > l = [] > for x in seq: > if x == y: > break > l.append(x) I believe I thought then that one should write the explicit loop rather than overload the 'comprehension' concept. > At the time, somebody suggested (roughly): > > def stop(): > raise StopIteration > > l = list(x for x in seq if x!=y or stop()) If stop is defined in another file, such as 'utility', this is a bit nasty. A new maintainer comes along and changes that to a list comprehension, or perhaps decides a set rather than a list is needed, and changes it to a set comprehension instead of set() call and bingo!, a bug. Or someone imitates the pattern, but with [] instead of list. > which, for the very reasons discussed in this thread, works only as a > generator expression and not in comprehension form. With this example, where the StopIteration source could be much more obscure than next(), I now understand Guido's concern about hard-to-understand bugs. From a maintainability view, it should not matter if one calls a function on a naked comprehension (making it a genexp) or uses the > I used this solution in some not particularly important piece of code so > I wouldn't despair if it wasn't compatible with the next release of the > language. > Also, I have a feeling that some of you may consider this sort of a hack > in the first place. -- Terry Jan Reedy From rosuav at gmail.com Wed Nov 19 00:08:29 2014 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 19 Nov 2014 10:08:29 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> <546B848E.6040406@biologie.uni-freiburg.de> Message-ID: On Wed, Nov 19, 2014 at 9:54 AM, Terry Reedy wrote: > On 11/18/2014 12:40 PM, Wolfgang Maier wrote: > >> I just remembered one use of the current behavior. Two years ago or so, >> I was suggesting on this list a possibility for early termination of >> comprehensions when a particular value is encountered. In other words, >> an equivalent to: >> >> l = [] >> for x in seq: >> if x == y: >> break >> l.append(x) > > > I believe I thought then that one should write the explicit loop rather than > overload the 'comprehension' concept. I'm not sure about that. Comprehensions can already be filtered; is it such a jump from there to a "filter" that aborts on a certain condition? It may not be language-supported, but I don't see that it's illogical; and any use of a loop that appends to a list is rightly considered code smell. ChrisA From steve at pearwood.info Wed Nov 19 02:15:48 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 19 Nov 2014 12:15:48 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> <546B848E.6040406@biologie.uni-freiburg.de> Message-ID: <20141119011548.GZ2748@ando.pearwood.info> On Wed, Nov 19, 2014 at 10:08:29AM +1100, Chris Angelico wrote: > On Wed, Nov 19, 2014 at 9:54 AM, Terry Reedy wrote: > > On 11/18/2014 12:40 PM, Wolfgang Maier wrote: > > > >> I just remembered one use of the current behavior. Two years ago or so, > >> I was suggesting on this list a possibility for early termination of > >> comprehensions when a particular value is encountered. In other words, > >> an equivalent to: > >> > >> l = [] > >> for x in seq: > >> if x == y: > >> break > >> l.append(x) > > > > > > I believe I thought then that one should write the explicit loop rather than > > overload the 'comprehension' concept. > > I'm not sure about that. Comprehensions can already be filtered; is it > such a jump from there to a "filter" that aborts on a certain > condition? It may not be language-supported, but I don't see that it's > illogical; It certainly isn't. It's an obvious extension to the concept: terminate the loop rather than filter it. At least two languages support early termination: http://clojuredocs.org/clojure_core/clojure.core/for http://docs.racket-lang.org/guide/for.html and it keeps getting asked for: http://www.reddit.com/r/Python/comments/ciec3/is_there_anything_like_a_list_comprehension/ http://stackoverflow.com/questions/5505891/using-while-in-list-comprehension-or-generator-expressions http://stackoverflow.com/questions/16931214/short-circuiting-list-comprehensions https://www.daniweb.com/software-development/python/threads/293381/break-a-list-comprehension https://mail.python.org/pipermail/python-ideas/2014-February/026036.html https://mail.python.org/pipermail/python-ideas/2013-January/018969.html There's a rejected PEP: https://www.python.org/dev/peps/pep-3142/ and alternative solutions (write an explicit generator function, use itertools.takewhile). So there's obviously a need for this sort of thing, and (expr for x in iterable if cond() or stop()) seems to be a common solution. I'm not sure if that's a neat trick or a deplorable hack :-) but either way this PEP will break code using it. > and any use of a loop that appends to a list is rightly > considered code smell. I'm afraid I don't understand that comment. Why is appending to a list inside a loop a code smell? That's exactly what list comps do. -- Steven From rosuav at gmail.com Wed Nov 19 03:01:17 2014 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 19 Nov 2014 13:01:17 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: <20141119011548.GZ2748@ando.pearwood.info> References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> <546B848E.6040406@biologie.uni-freiburg.de> <20141119011548.GZ2748@ando.pearwood.info> Message-ID: On Wed, Nov 19, 2014 at 12:15 PM, Steven D'Aprano wrote: >> and any use of a loop that appends to a list is rightly >> considered code smell. > > I'm afraid I don't understand that comment. Why is appending to a list > inside a loop a code smell? That's exactly what list comps do. That's precisely why. If I write code like this: l = [] for i in something: l.append(func(i)) then I should rework it into a comprehension. Having a filter doesn't change that: l = [] for i in something: if i: l.append(func(i)) That's still possible with a list comp, and should be rewritten as one. But having a break in there *does* change it, because there's no way in the language to do that. The question is: Is it better to abuse StopIteration or to turn the list comp back into an explicit loop? And if anyone chose the former, their code will break. ChrisA From guido at python.org Wed Nov 19 03:56:42 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 18 Nov 2014 18:56:42 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> <546B848E.6040406@biologie.uni-freiburg.de> <20141119011548.GZ2748@ando.pearwood.info> Message-ID: On Tue, Nov 18, 2014 at 6:01 PM, Chris Angelico wrote: > On Wed, Nov 19, 2014 at 12:15 PM, Steven D'Aprano > wrote: > >> and any use of a loop that appends to a list is rightly > >> considered code smell. > > > > I'm afraid I don't understand that comment. Why is appending to a list > > inside a loop a code smell? That's exactly what list comps do. > > That's precisely why. If I write code like this: > > l = [] > for i in something: > l.append(func(i)) > > then I should rework it into a comprehension. Having a filter doesn't > change that: > > l = [] > for i in something: > if i: > l.append(func(i)) > > That's still possible with a list comp, and should be rewritten as > one. But having a break in there *does* change it, because there's no > way in the language to do that. The question is: Is it better to abuse > StopIteration or to turn the list comp back into an explicit loop? And > if anyone chose the former, their code will break. > Not everything you do with an explicit loop can be done with a comprehension, and that's by design. Comprehensions should be easier to reason about than code using for-loops. And generator expressions should work the same way, except for producing results in a lazy fashion. The StopIteration hack breaks this equivalence and hampers the ability to reason, since you can't tell whether a predicate might raise StopIteration. It was never my intention that generator expressions behaved this way -- it was an accidental feature that surprised me when it was first shown to me, and I've never gotten used to it. (And I don't care whether you say it is "obvious", call it "stop()", and only use it in an "idiomatic" fashion -- it's still a surprise for anyone who has to debug code involving it.) The only thing standing in the way of fixing this is the recognition that there may be a fair amount of code out there that depends on this hack, and which will have to be rewritten. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Wed Nov 19 04:37:28 2014 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 19 Nov 2014 14:37:28 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> <546B848E.6040406@biologie.uni-freiburg.de> <20141119011548.GZ2748@ando.pearwood.info> Message-ID: On Wed, Nov 19, 2014 at 1:56 PM, Guido van Rossum wrote: > The only thing standing in the way of fixing this is the recognition that > there may be a fair amount of code out there that depends on this hack, and > which will have to be rewritten. Has anyone come across any more non-trivial examples? We have contextlib (in the standard library) and contextlib2 (third-party), plus a number of StackOverflow posts and such. Are there any other known pieces of code that would be seriously hampered by this change? ChrisA From guido at python.org Wed Nov 19 05:22:41 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 18 Nov 2014 20:22:41 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> <546B848E.6040406@biologie.uni-freiburg.de> <20141119011548.GZ2748@ando.pearwood.info> Message-ID: On Tue, Nov 18, 2014 at 7:37 PM, Chris Angelico wrote: > On Wed, Nov 19, 2014 at 1:56 PM, Guido van Rossum > wrote: > > The only thing standing in the way of fixing this is the recognition that > > there may be a fair amount of code out there that depends on this hack, > and > > which will have to be rewritten. > > Has anyone come across any more non-trivial examples? We have > contextlib (in the standard library) and contextlib2 (third-party), > plus a number of StackOverflow posts and such. Are there any other > known pieces of code that would be seriously hampered by this change? > One possible way to find out would be to write a simple version of a patch (maybe one that doesn't use __future__ but just always converts StopIteration to RuntimeError when it bubbling out of a generator frame) and run the stdlib tests, then see how many tests this breaks. (I understand if you don't want to write it. But maybe someone does. :-) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Wed Nov 19 17:01:29 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 19 Nov 2014 18:01:29 +0200 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> Message-ID: On 18.11.14 06:50, Guido van Rossum wrote: > Separate from this special case, I am also worried about backward > compatibility, and I have yet to get a good idea for how widespread code > is that depends on StopIteration bubbling out from generators. I also > don't have a good idea how often this issue bites users, but I have a > feeling it does bite. E.g. current difflib implementation depends on this behavior. ... while True: ... while(found_diff is False): from_line, to_line, found_diff = next(line_pair_iterator) ... From ncoghlan at gmail.com Wed Nov 19 17:03:27 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 20 Nov 2014 02:03:27 +1000 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> Message-ID: On 18 November 2014 14:50, Guido van Rossum wrote: > Nick, > > I think we've gone through enough clarifications of the PEP now to be > clear on the proposal. I saw in one of your earliest replies (right after > Chris posted his first draft) that you're hesitant to support the PEP > because of what would have to change to contextlib. What I couldn't quite > read is whether you think that the proposal by itself is not an > improvement, or whether you're just worried about compatibility. > I think it's an improvement - I really like the fact that it brings generators into line with your reasoning in the with statement PEP that flow control constructs should be locally visible. At the moment, "raise StopIteration" in a generator context is effectively non-local flow control, as it means any function call (explicit or implicit) or yield point may gracefully stop generator execution, rather than only return statements. >From the point of view of a generator *consumer*, I'm now happy that the backwards incompatibility is limited to cases where you throw in StopIteration and want to check if you got the same StopIteration instance back - you'll now also need to check for RuntimeError with __cause__ set to the StopIteration instance you threw in. You can construct scenarios where such a check will give a false positive, but they're getting seriously contrived at that point. That's obscure enough that I think it's on par with other behavioural tweaks we've included in the "Porting to Python X.Y" guides in the past. > Apparently you know of a large group of users who use an older 3rd party > version of contextlib, and for whom that older, 3rd party contextlib should > keep working with future versions of Python 3 without updating their > version of contextlib -- did I get that right? What exactly is the > constraint there that makes their version of contextlib immutable even > though the version of Python they are using may move forward? > I don't even remember how that came up now, but it was entirely hypothetical, and I think it can be ignored as a concern. As you say, being able to update to Python 3.5 without also being able to update to a new version of contextlib2 would just be weird. Even if such a strange scenario did somehow come up, it would still be possible to switch to a conditional import where they used the stdlib version if available, and only fell back to contextlib2 on earlier versions of Python. > Separate from this special case, I am also worried about backward > compatibility, and I have yet to get a good idea for how widespread code is > that depends on StopIteration bubbling out from generators. I also don't > have a good idea how often this issue bites users, but I have a feeling it > does bite. E.g. this quote from c.l.py ( > https://mail.python.org/pipermail/python-list/2014-November/680775.html): > > """ > I did find it annoying occasionally that raising StopIteration inside a > generator expression conveys a different behavior than elsewhere. It did > take me quite a while to understand why that is so, but after that it > did not cause me much of a headache anymore. > """ > One advantage of needing a __future__ import on the generator author side is that you always have the option of changing your mind, and *never* making the new behaviour the default. That wouldn't be a *good* outcome, but I don't think it would be intolerable. OTOH, I'm also not sure the status quo is sufficiently problematic to be worth changing. Yes, it's a little weird, but is it *that* much weirder than the unavoidable issues with exceptions thrown in __next__, __getitem__, __getattr__ and other special methods where a particular type of exception is handled directly by the interpreter? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Wed Nov 19 17:24:07 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 20 Nov 2014 03:24:07 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> Message-ID: On Thu, Nov 20, 2014 at 3:03 AM, Nick Coghlan wrote: > OTOH, I'm also not sure the status quo is sufficiently problematic to be > worth changing. Yes, it's a little weird, but is it *that* much weirder than > the unavoidable issues with exceptions thrown in __next__, __getitem__, > __getattr__ and other special methods where a particular type of exception > is handled directly by the interpreter? If you write __next__, you write in a "raise StopIteration" when it's done. If you write __getattr__, you write in "raise AttributeError" if the attribute shouldn't exist. Those are sufficiently explicit that it should be reasonably clear that the exception is the key. But when you write a generator, you don't explicitly raise: def gen(): yield 1 yield 2 yield 3 return 4 The distinction in __next__ is between returning something and raising something. The distinction in a generator is between "yield" and "return". Why should a generator author have to be concerned about one particular exception having magical meaning? Imagine this scenario: def producer(): """Return user input, or raise KeyboardInterrupt""" return input("Enter the next string: ") def consumer(): """Process the user's input""" while True: try: command = producer() except KeyboardInterrupt: break dispatch(command) Okay, now let's make a mock producer: strings = ["do stuff","do more stuff","blah blah"] def mock_producer() if strings: return strings.pop(0) raise KeyboardInterrupt That's how __next__ works, only with a different exception, and I think people would agree that this is NOT a good use of KeyboardInterrupt. If you put a few extra layers in between the producer and consumer, you'd be extremely surprised that an unexpected KeyboardInterrupt just quietly terminated a loop. Yet this is exactly what the generator-and-for-loop model creates: a situation in which StopIteration, despite not being seen at either end of the code, now has magical properties. Without the generator, *only* __next__ has this effect, and that's exactly where it's documented to be. Does that make for more justification? Unexpected exceptions bubbling up is better than unexpected exceptions quietly terminating loops? ChrisA From ncoghlan at gmail.com Wed Nov 19 17:45:27 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 20 Nov 2014 02:45:27 +1000 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> Message-ID: On 20 November 2014 02:24, Chris Angelico wrote: > Does that make for more justification? Unexpected exceptions bubbling > up is better than unexpected exceptions quietly terminating loops? > The part I found most compelling was when you pointed out that in the special method implementations, the normal return path was always spelled with "return", while the "value missing" result was indicated with a special kind of exception (StopIteration, AttributeError, IndexError or KeyError), and then any other exception was consider unexpected. Generators add the third notion of being able to suspend execution via "yield", which then left them with two different ways of spelling termination inside the frame: "return" OR "raise StopIteration". The second spelling ("raise StopIteration") is then inherently surprising, as it's entirely redundant, *except* in that it allows you to effectively have a "hidden return" in a generator frame that can't be done anywhere else. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Wed Nov 19 23:46:39 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 19 Nov 2014 17:46:39 -0500 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> Message-ID: On 11/19/2014 11:24 AM, Chris Angelico wrote: > On Thu, Nov 20, 2014 at 3:03 AM, Nick Coghlan wrote: >> OTOH, I'm also not sure the status quo is sufficiently problematic to be >> worth changing. Yes, it's a little weird, but is it *that* much weirder than >> the unavoidable issues with exceptions thrown in __next__, __getitem__, >> __getattr__ and other special methods where a particular type of exception >> is handled directly by the interpreter? > > If you write __next__, you write in a "raise StopIteration" when it's > done. If you write __getattr__, you write in "raise AttributeError" if > the attribute shouldn't exist. Those are sufficiently explicit that it > should be reasonably clear that the exception is the key. But when you > write a generator, you don't explicitly raise: > > def gen(): > yield 1 > yield 2 > yield 3 > return 4 > > The distinction in __next__ is between returning something and raising > something. The distinction in a generator is between "yield" and > "return". Which, as I said a week ago, is why there is no need for "raise StopIteration" in a generator function. The doc clearly states the limited intended use of StopIteration. ''' exception StopIteration Raised by built-in function next() and an iterator?s __next__() method to signal that there are no further items produced by the iterator. ''' StopIteration is exposed so it can be raised in user coded __next__() and caught when using explicit next(). If it was only used for builtins and for loops, it would not need to be visible. > Why should a generator author have to be concerned about one > particular exception having magical meaning? I am not sure of your intent with this rhetorical (?) question. > Imagine this scenario: > > def producer(): > """Return user input, or raise KeyboardInterrupt""" > return input("Enter the next string: ") The prompt should be "Enter the next string or hit ^C to quit: ". > def consumer(): > """Process the user's input""" > while True: > try: > command = producer() > except KeyboardInterrupt: > break > dispatch(command) > Okay, now let's make a mock producer: > > strings = ["do stuff","do more stuff","blah blah"] > def mock_producer() > if strings: return strings.pop(0) > raise KeyboardInterrupt > That's how __next__ works, only with a different exception, and I > think people would agree that this is NOT a good use of > KeyboardInterrupt. It is avoidable because the return type of producer is limited to strings. Therefore, producer could (and perhaps should) itself catch KeyboardInterrupt and return None, which is intended for such use. Consumer would then be simplified by replacing 3 lines with "if command is None: break". > If you put a few extra layers in between the > producer and consumer, you'd be extremely surprised that an unexpected > KeyboardInterrupt just quietly terminated a loop. Yet this is exactly > what the generator-and-for-loop model creates: a situation in which > StopIteration, despite not being seen at either end of the code, now > has magical properties. Without the generator, *only* __next__ has > this effect, and that's exactly where it's documented to be. > > Does that make for more justification? Unexpected exceptions bubbling > up is better than unexpected exceptions quietly terminating loops? -- Terry Jan Reedy From rosuav at gmail.com Wed Nov 19 23:57:43 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 20 Nov 2014 09:57:43 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> Message-ID: On Thu, Nov 20, 2014 at 9:46 AM, Terry Reedy wrote: > On 11/19/2014 11:24 AM, Chris Angelico wrote: >> The distinction in __next__ is between returning something and raising >> something. The distinction in a generator is between "yield" and >> "return". > > > Which, as I said a week ago, is why there is no need for "raise > StopIteration" in a generator function. The doc clearly states the limited > intended use of StopIteration. > ''' > exception StopIteration > Raised by built-in function next() and an iterator?s __next__() method > to signal that there are no further items produced by the iterator. > ''' > StopIteration is exposed so it can be raised in user coded __next__() and > caught when using explicit next(). If it was only used for builtins and for > loops, it would not need to be visible. > >> Why should a generator author have to be concerned about one >> particular exception having magical meaning? > > I am not sure of your intent with this rhetorical (?) question. Yes, rhetorical. Basically saying the same as you are: that StopIteration is a part of __next__, not generators. >> Imagine this scenario: >> >> def producer(): >> """Return user input, or raise KeyboardInterrupt""" >> return input("Enter the next string: ") > > The prompt should be "Enter the next string or hit ^C to quit: ". Yeah, the point is about its interaction with the rest of the program, not the human. >> def consumer(): >> """Process the user's input""" >> while True: >> try: >> command = producer() >> except KeyboardInterrupt: >> break >> dispatch(command) > > >> Okay, now let's make a mock producer: >> >> strings = ["do stuff","do more stuff","blah blah"] >> def mock_producer() >> if strings: return strings.pop(0) >> raise KeyboardInterrupt > > >> That's how __next__ works, only with a different exception, and I >> think people would agree that this is NOT a good use of >> KeyboardInterrupt. > > > It is avoidable because the return type of producer is limited to strings. > Therefore, producer could (and perhaps should) itself catch > KeyboardInterrupt and return None, which is intended for such use. Consumer > would then be simplified by replacing 3 lines with "if command is None: > break". Sure it does. But suppose it does some parsing on the string first, and that parsing might return literally any object. The structure of the program is the same, but now it really does need to signal "no more stuff" in some way other than return value. Just trying to concoct a situation similar to generators/for loops, using a different exception. I'm fairly sure there's no way to make the above system seem truly plausible, because KeyboardInterrupt is a bad exception for the purpose; but it's still broadly similar, and I think the same applies: StopException should be *only* inside __next__() and next(). Since generators can distinguish yield from return, they don't need to distinguish return from raise. ChrisA From rosuav at gmail.com Wed Nov 19 23:58:43 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 20 Nov 2014 09:58:43 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> Message-ID: On Thu, Nov 20, 2014 at 9:57 AM, Chris Angelico wrote: > Since generators can distinguish yield from > return, they don't need to distinguish return from raise. Bad grammar, should edit before posting. Since generators can distinguish value from no value by using yield and return, they don't need to use yield and raise. ChrisA From steve at pearwood.info Thu Nov 20 01:25:47 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 20 Nov 2014 11:25:47 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> Message-ID: <20141120002546.GH2748@ando.pearwood.info> On Thu, Nov 20, 2014 at 02:45:27AM +1000, Nick Coghlan wrote: > The part I found most compelling was when you pointed out that in the > special method implementations, the normal return path was always spelled > with "return", while the "value missing" result was indicated with a > special kind of exception (StopIteration, AttributeError, IndexError or > KeyError), and then any other exception was consider unexpected. > > Generators add the third notion of being able to suspend execution via > "yield", which then left them with two different ways of spelling > termination inside the frame: "return" OR "raise StopIteration". The second > spelling ("raise StopIteration") is then inherently surprising, as it's > entirely redundant, *except* in that it allows you to effectively have a > "hidden return" in a generator frame that can't be done anywhere else. I'm not sure that many people outside of this and the python-dev mailing lists would find the use of "raise StopIteration" surprising. Rather, I expect that they will find the use of an explicit "return" inside a generator surprising. People are informally taught that generators use yield *instead of* return, so seeing both in the same function is a surprise. (Most generators quitely fall out the bottom with no explicit end.) I don't claim that doing so is Pythonic or even good practice, but I am sure that there are a lot of people who believe that raising StopIteration to exit a generator is (1) supported and (2) preferred. Examples of code in the wild using StopIteration to exit: http://code.openhub.net/file?fid=ezlejSoT2q7PWrhgNkpdU55MWOA&cid=jVcYOxnQhvU&s=raise%20StopIteration&fp=301369&mp&projSelected=true#L0 http://code.openhub.net/file?fid=M0gWWCpn-avqHO_jnsYcG2T81lg&cid=VKn_M0_GgKM&s=raise%20StopIteration&fp=301283&mp&projSelected=true#L0 http://code.openhub.net/file?fid=pDrrTI8lyh0LO_6rTCk9npC96SE&cid=Y8jg8v1AyqU&s=raise%20StopIteration&fp=41191&mp&projSelected=true#L0 http://code.openhub.net/file?fid=PTjGrE_5rOhyZhL1CUrPBtRk7n8&cid=tWtPpAs4E1g&s=raise%20StopIteration&fp=210789&mp&projSelected=true#L0 http://code.openhub.net/file?fid=WzkucGktJhjsP8cj4BO6Wcnbx-0&cid=fsj7E8vdVMA&s=raise%20StopIteration&fp=401086&mp&projSelected=true#L0 http://stackoverflow.com/questions/6784934/python-yield-and-stopiteration-in-one-loop http://stackoverflow.com/questions/14183803/in-pythons-generators-what-is-the-difference-between-raise-stopiteration-and That last example not only uses raise to exit the generator, but the author actually guesses that it is the more Pythonic way to do so. Here is a description of the generator protocol which could easily lead the reader to conclude that raising StopIteration is the correct way to exit a generator: To support this protocol, functions with yield statement are compiled specially as generators. They return a generator object when they are called. The returned object supports the iteration interface with an automatically created __next__() method to resume execution. Generator functions may have a return simply terminates the generation of values by raising a StopIteration exceptions after any normal function exit. http://www.bogotobogo.com/python/python_generators.php At this point, I'm convinced that there is a good argument for a __future__ import changing this behaviour. But I suspect that making this the default behaviour in the future will break a lot of code. -- Steven From rosuav at gmail.com Thu Nov 20 02:34:08 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 20 Nov 2014 12:34:08 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: <20141120002546.GH2748@ando.pearwood.info> References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> <20141120002546.GH2748@ando.pearwood.info> Message-ID: On Thu, Nov 20, 2014 at 11:25 AM, Steven D'Aprano wrote: > I'm not sure that many people outside of this and the python-dev mailing > lists would find the use of "raise StopIteration" surprising. Rather, I > expect that they will find the use of an explicit "return" inside a > generator surprising. People are informally taught that generators use > yield *instead of* return, so seeing both in the same function is a > surprise. (Most generators quitely fall out the bottom with no explicit > end.) Interesting. But "yield instead of return" doesn't automatically say "and then raise StopIteration to early-abort"; I'd say the informal description is fine, it just needs to be modified differently once people actually want an early abort. ("You can still use 'return' for its other purpose, terminating a function before reaching the end.") > Examples of code in the wild using StopIteration to exit: > > http://code.openhub.net/file?fid=PTjGrE_5rOhyZhL1CUrPBtRk7n8&cid=tWtPpAs4E1g&s=raise%20StopIteration&fp=210789&mp&projSelected=true#L0 Trivially unnecessary, and as soon as there's a bug report, the "What's New In 3.7" page will explain that it needs to be removed. > http://code.openhub.net/file?fid=WzkucGktJhjsP8cj4BO6Wcnbx-0&cid=fsj7E8vdVMA&s=raise%20StopIteration&fp=401086&mp&projSelected=true#L0 I have no idea what this one is doing, but it looks like it's half way to what's wanted here. Catch the exception and deal with it... this proposal just means the "deal with it" part needs to be reworded into a return statement. All it needs is for "What's New in 3.5" to recommend use of 'return' instead of 'raise StopIteration', and all these cases will be easily compatible with all [1] versions of Python. > http://stackoverflow.com/questions/6784934/python-yield-and-stopiteration-in-one-loop The accepted answer correctly advises the function be written to simply return. This will work fine. The other answer has a local "raise StopIteration", which can be translated into a simple "return". > http://stackoverflow.com/questions/14183803/in-pythons-generators-what-is-the-difference-between-raise-stopiteration-and > > That last example not only uses raise to exit the generator, but the > author actually guesses that it is the more Pythonic way to do so. The question's author does, but the accepted answer recommends "return". This may result in the odd question here or there, but it's not a major problem. Any time a generator has "raise StopIteration" in its own body, it can simply become "return". That's easy. The issue comes up when it's not raising that itself, but is letting it bubble up - maybe from a next() call. def takewhiletrue(iter): while True: # coincidental with the function name # try: val = next(iter) # except StopIteration: return if not val: break yield val This won't come up in a simple search for "raise StopIteration", and if you have something like this where the condition is almost always going to be reached eventually, you might not notice the problem for a long time. How would you know to add the commented-out lines? What kind of code search would you use to detect this? > Here is a description of the generator protocol which could easily lead > the reader to conclude that raising StopIteration is the correct way to > exit a generator: > > To support this protocol, functions with yield statement > are compiled specially as generators. They return a generator > object when they are called. The returned object supports the > iteration interface with an automatically created __next__() > method to resume execution. Generator functions may have a > return simply terminates the generation of values by raising > a StopIteration exceptions after any normal function exit. > > http://www.bogotobogo.com/python/python_generators.php Even that does recommend 'return'. If anyone reads that, writes "raise StopIteration", sees code bombing with RuntimeError, and then comes to python-list, we can explain that the recommended method is "return". I have no problem with this. There are plenty of much-worse practices that people pick up - mixing bytes and text, using backslashes in Windows path names without doubling them or using raw literals, etc, etc, etc. In lots of cases they'll seem to work ("C:\Program Files\New Stuff\Testing" will work, until you lower-case the name), but when they break, you just have to fix them. This wouldn't be the first Python minor version to tighten up requirements to remove bug magnets. > At this point, I'm convinced that there is a good argument for a > __future__ import changing this behaviour. But I suspect that making > this the default behaviour in the future will break a lot of code. I suspect that a huge percentage of the code so broken can be trivially fixed just by search/replacing "raise StopIteration" with "return". There'll be only a very few where the StopIteration is raised from some other function and needs to be caught and dealt with - and fixing those is just as likely to reveal bugs needing fixing. ChrisA From steve at pearwood.info Thu Nov 20 03:06:15 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 20 Nov 2014 13:06:15 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> Message-ID: <20141120020615.GI2748@ando.pearwood.info> On Thu, Nov 20, 2014 at 03:24:07AM +1100, Chris Angelico wrote: > If you write __next__, you write in a "raise StopIteration" when it's > done. If you write __getattr__, you write in "raise AttributeError" if > the attribute shouldn't exist. Those are sufficiently explicit that it > should be reasonably clear that the exception is the key. But when you > write a generator, you don't explicitly raise: That's not true in practice. See my reply to Nick, there is lots of code out there which uses StopIteration to exit generators. Some of that code isn't very good code -- I've seen "raise StopIteration" immediately before falling out the bottom of the generator -- but until now it has worked and the impression some people have gotten is that it is actually preferred. > def gen(): > yield 1 > yield 2 > yield 3 > return 4 Until 3.2, that was a syntax error. For the majority of people who are still using Python 2.7, it is *still* a syntax error. To write this in a backwards-compatible way, you have to exit the generator with: raise StopIteration(2) > The distinction in __next__ is between returning something and raising > something. The distinction in a generator is between "yield" and > "return". Why should a generator author have to be concerned about one > particular exception having magical meaning? I would put it another way: informally, the distinction between a generator and a function is that generators use yield where functions use return. Most people are happy with that informal definition, a full pedantic explanation of co-routines will just confuse them or bore them. The rule they will learn is: * use return in functions * use yield in generators That makes generators that use both surprising. Since most generators either run forever or fall out the bottom when they are done, I expect that seeing a generator with a return in it is likely to surprise a lot of people. I've known that return works for many years, and I still give a double-take whenever I see it in a generator. > Imagine this scenario: > > def producer(): > """Return user input, or raise KeyboardInterrupt""" > return input("Enter the next string: ") > > def consumer(): > """Process the user's input""" > while True: > try: > command = producer() > except KeyboardInterrupt: > break > dispatch(command) > > > Okay, now let's make a mock producer: > > strings = ["do stuff","do more stuff","blah blah"] > def mock_producer() > if strings: return strings.pop(0) > raise KeyboardInterrupt > > That's how __next__ works, only with a different exception, and I > think people would agree that this is NOT a good use of > KeyboardInterrupt. Why not? How else are you going to communicate something out of band to the consumer except via an exception? We can argue about whether KeyboardInterrupt is the right exception to use or not, but if you insist that this is a bad protocol then you're implicitly saying that the iterator protocol is also a bad protocol. > If you put a few extra layers in between the > producer and consumer, you'd be extremely surprised that an unexpected > KeyboardInterrupt just quietly terminated a loop. You might be, but since I've paid attention to the protocol rules, I won't be. Sorry to be harsh, but how clear do we have to be? StopIteration terminates iterators, and generators are iterators. That rule may or may not be inconvenient, it might be annoying (but sometimes useful), it might hide bugs, it might even be something that we can easily forget until reminded, but if it comes as a "surprise" that just means you don't know how the iterator protocol works. There are good reasons for changing this behaviour, but pandering to people who don't know how the iterator protocol works is not one of them. > Yet this is exactly > what the generator-and-for-loop model creates: a situation in which > StopIteration, despite not being seen at either end of the code, now > has magical properties. That's exactly how the protocol works. Even if you write "return" in your generator, it still raises StopIteration. > Without the generator, *only* __next__ has > this effect, and that's exactly where it's documented to be. The documentation states that __next__ raises StopIteration, it doesn't say that *only* __next__ should raise StopIteration. https://docs.python.org/3/library/stdtypes.html#iterator.__next__ I trust that we all expect to be able to factor out the raise into a helper function or method, yes? It truly would be surprising if this failed: class MyIterator: def __iter__(self): return self def __next__(self): return something() def something(): # Toy helper function. if random.random() < 0.5: return "Spam!" raise StopIteration Now let's write this as a generator: def gen(): while True: yield something() which is much nicer than: def gen(): while True: try: yield something() except StopIteration: return # converted by Python into raise StopIteration -- Steven From rosuav at gmail.com Thu Nov 20 03:24:02 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 20 Nov 2014 13:24:02 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: <20141120020615.GI2748@ando.pearwood.info> References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> <20141120020615.GI2748@ando.pearwood.info> Message-ID: On Thu, Nov 20, 2014 at 1:06 PM, Steven D'Aprano wrote: > On Thu, Nov 20, 2014 at 03:24:07AM +1100, Chris Angelico wrote: > >> If you write __next__, you write in a "raise StopIteration" when it's >> done. If you write __getattr__, you write in "raise AttributeError" if >> the attribute shouldn't exist. Those are sufficiently explicit that it >> should be reasonably clear that the exception is the key. But when you >> write a generator, you don't explicitly raise: > > That's not true in practice. See my reply to Nick, there is lots of code > out there which uses StopIteration to exit generators. Some of that code > isn't very good code -- I've seen "raise StopIteration" immediately > before falling out the bottom of the generator -- but until now it has > worked and the impression some people have gotten is that it is actually > preferred. Yes, I thought it was rare. I stand corrected. Reword that to "you don't *need to* explicitly raise", since you can simply return, and it becomes true again, though. >> def gen(): >> yield 1 >> yield 2 >> yield 3 >> return 4 > > Until 3.2, that was a syntax error. For the majority of people who are > still using Python 2.7, it is *still* a syntax error. To write this in a > backwards-compatible way, you have to exit the generator with: > > raise StopIteration(2) In most cases you won't need to put a value on it, so bare "return" will work just fine. I just put a return value onto it so it wouldn't look trivially useless. >> The distinction in __next__ is between returning something and raising >> something. The distinction in a generator is between "yield" and >> "return". Why should a generator author have to be concerned about one >> particular exception having magical meaning? > > I would put it another way: informally, the distinction between a > generator and a function is that generators use yield where functions > use return. Most people are happy with that informal definition, a full > pedantic explanation of co-routines will just confuse them or bore them. > The rule they will learn is: > > * use return in functions > * use yield in generators > > That makes generators that use both surprising. Since most generators > either run forever or fall out the bottom when they are done, I expect > that seeing a generator with a return in it is likely to surprise a lot > of people. I've known that return works for many years, and I still > give a double-take whenever I see it in a generator. But it's just as surprising to put "raise StopIteration" into it. It's normal to put that into __next__, it's not normal to need it in a generator. Either way, it's something unusual; so let's go with the unusual "return" rather than the unusual "raise". >> That's how __next__ works, only with a different exception, and I >> think people would agree that this is NOT a good use of >> KeyboardInterrupt. > > Why not? How else are you going to communicate something out of band to > the consumer except via an exception? > > We can argue about whether KeyboardInterrupt is the right exception to > use or not, but if you insist that this is a bad protocol then you're > implicitly saying that the iterator protocol is also a bad protocol. Well, that's exactly what I do mean. KeyboardInterrupt is not a good way for two parts of a program to communicate with each other, largely because it can be raised unexpectedly. Which is the point of this PEP: raising StopIteration unexpectedly should also result in a noisy traceback. > I trust that we all expect to be able to factor out the raise into a > helper function or method, yes? It truly would be surprising if this > failed: > > > class MyIterator: > def __iter__(self): > return self > def __next__(self): > return something() > > > def something(): > # Toy helper function. > if random.random() < 0.5: > return "Spam!" > raise StopIteration > > > > Now let's write this as a generator: > > def gen(): > while True: > yield something() > > > which is much nicer than: > > def gen(): > while True: > try: > yield something() > except StopIteration: > return # converted by Python into raise StopIteration Sure. There was a suggestion that "return yield from something()" would work, though, which - I can't confirm that this works, but assuming it does - would be a lot tidier. But there's still a difference. Your first helper function was specifically a __next__ helper. It was tied intrinsically to the iterator protocol. If you want to call a __next__ helper (or actually call next(iter) on something) inside a generator, you'll have to - if this change goes through - cope with the fact that generator protocol says "return" where __next__ protocol says "raise StopIteration". If you want a generator helper, it'd look like this: def something(): # Toy helper function. if random.random() < 0.5: yield "Spam!" def gen(): yield from something() Voila! Now it's a generator helper, following generator protocol. Every bit as tidy as the original. Let's write a __getitem__ helper: def something(x): # Toy helper function. if random.random() < 0.5: return "Spam!" raise KeyError(x) class X: def __getitem__(self, x): return something(x) Same thing. As soon as you get into raising these kinds of exceptions, you're tying your helper to a specific protocol. All that's happening with PEP 479 is that generator and iterator protocol are being distinguished slightly. ChrisA From rosuav at gmail.com Thu Nov 20 04:44:25 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 20 Nov 2014 14:44:25 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> <546B848E.6040406@biologie.uni-freiburg.de> <20141119011548.GZ2748@ando.pearwood.info> Message-ID: On Wed, Nov 19, 2014 at 3:22 PM, Guido van Rossum wrote: > One possible way to find out would be to write a simple version of a patch > (maybe one that doesn't use __future__ but just always converts > StopIteration to RuntimeError when it bubbling out of a generator frame) and > run the stdlib tests, then see how many tests this breaks. (I understand if > you don't want to write it. But maybe someone does. :-) I poked around a bit in the code and managed to come up with this. It doesn't chain the previous exception, so the traceback is a little scanty, but it does turn a StopIteration into a RuntimeError. (It might also leak the original StopIteration. I'm not sure.) Prior to this patch, I had 377 of 390 tests passing flawlessly and no failures (just skips and warnings); with this applied, six failures. diff -r 23ab1197df0b Objects/genobject.c --- a/Objects/genobject.c Wed Nov 19 13:21:40 2014 +0200 +++ b/Objects/genobject.c Thu Nov 20 13:43:44 2014 +1100 @@ -130,6 +130,14 @@ } Py_CLEAR(result); } + else if (!result) + { + if (PyErr_ExceptionMatches(PyExc_StopIteration)) + { + PyErr_SetString(PyExc_RuntimeError, + "generator raised StopIteration"); + } + } if (!result || f->f_stacktop == NULL) { /* generator can't be rerun, so release the frame */ However, I'm not sure about setting the context. In errors.c is a function _PyErr_ChainExceptions which appears to do a similar job, so I imitated its code. Here's the result: else if (!result) { if (PyErr_ExceptionMatches(PyExc_StopIteration)) { PyObject *exc, *val, *val2, *tb; PyErr_Fetch(&exc, &val, &tb); PyErr_NormalizeException(&exc, &val, &tb); Py_DECREF(exc); Py_XDECREF(tb); PyErr_SetString(PyExc_RuntimeError, "generator raised StopIteration"); PyErr_Fetch(&exc, &val2, &tb); PyErr_NormalizeException(&exc, &val2, &tb); PyException_SetContext(val2, val); PyErr_Restore(exc, val2, tb); } } The context is being set, but without a traceback. ############# def good_gen(): yield 1 return 2 def evil_gen(): yield 1 raise StopIteration(2) # In absence of PEP 479 changes, the above two should be virtually indistinguishable. print("Starting.") good = tuple(good_gen()) print("Good:", good, good == (1,)) evil = tuple(evil_gen()) print("Evil:", evil, evil == (1,)) ############# Starting. Good: (1,) True StopIteration: 2 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "../test_pep0479.py", line 14, in evil = tuple(evil_gen()) RuntimeError: generator raised StopIteration What am I missing here? Do I need to force something to construct a full traceback before it can show the line number that actually raised StopIteration? ChrisA From ethan at stoneleaf.us Thu Nov 20 05:19:41 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 19 Nov 2014 20:19:41 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: <20141120002546.GH2748@ando.pearwood.info> References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> <20141120002546.GH2748@ando.pearwood.info> Message-ID: <546D6BDD.4040007@stoneleaf.us> On 11/19/2014 04:25 PM, Steven D'Aprano wrote: > > Here is a description of the generator protocol which could easily lead > the reader to conclude that raising StopIteration is the correct way to > exit a generator: > > To support this protocol, functions with yield statement > are compiled specially as generators. They return a generator > object when they are called. The returned object supports the > iteration interface with an automatically created __next__() > method to resume execution. Generator functions may have a > return simply terminates the generation of values by raising > a StopIteration exceptions after any normal function exit. > > http://www.bogotobogo.com/python/python_generators.php We are not, however, responsible for third-party documentation. > At this point, I'm convinced that there is a good argument for a > __future__ import changing this behaviour. But I suspect that making > this the default behaviour in the future will break a lot of code. Isn't that the case with every __future__ directive that becomes the standard? Folks have an entire minor release to make the adjustment. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From steve at pearwood.info Thu Nov 20 06:30:51 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 20 Nov 2014 16:30:51 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> <20141120002546.GH2748@ando.pearwood.info> Message-ID: <20141120053050.GA20200@ando.pearwood.info> On Thu, Nov 20, 2014 at 12:34:08PM +1100, Chris Angelico wrote: > On Thu, Nov 20, 2014 at 11:25 AM, Steven D'Aprano wrote: > > Examples of code in the wild using StopIteration to exit: > > > > http://code.openhub.net/file?fid=PTjGrE_5rOhyZhL1CUrPBtRk7n8&cid=tWtPpAs4E1g&s=raise%20StopIteration&fp=210789&mp&projSelected=true#L0 > > Trivially unnecessary, and as soon as there's a bug report, the > "What's New In 3.7" page will explain that it needs to be removed. The point isn't that it is easy to fix. I'm sure that there will be cases of code that are not easy to fix. The point is that we're breaking working code and causing code churn. We're not fixing a bug. We're changing behaviour people rely on. That ought to make us more conservative about breaking their code. > This may result in the odd question here or there, but it's not a > major problem. And neither is the existing behaviour. We're weighing up whether the small benefit in fixing this wart is worth the pain. The PEP isn't approved yet, and right from the beginning Guido said that he feared that fixing this might be too disruptive. I'm trying to get a feel for how disruptive it will be. I did a quick and informal survey of the developers I work with. The dreaded lurgy has gone through our office, so most of them are away ill, but of those still here (all two of them) one of them couldn't remember whether you exit a generator with "yield nil" or "return nil" (he's a Ruby and Objective-C guy when we're not paying him to write Python) and the other one said that the whole problem is that generators exist in the first place, Python should get rid of them and allow people to define their own using macros (he likes to think of himself as a Lisp and Scheme guru :-) Make of that what you will. -- Steven From rosuav at gmail.com Thu Nov 20 06:38:01 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 20 Nov 2014 16:38:01 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: <20141120053050.GA20200@ando.pearwood.info> References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> <20141120002546.GH2748@ando.pearwood.info> <20141120053050.GA20200@ando.pearwood.info> Message-ID: On Thu, Nov 20, 2014 at 4:30 PM, Steven D'Aprano wrote: > the other one said that the whole problem is that generators exist in > the first place, Python should get rid of them and allow people to > define their own using macros (he likes to think of himself as a Lisp > and Scheme guru :-) There's a language that lets you define anything you like. It's called "file on disk". If you don't like how it runs, you just put a little shebang at the top and the whole rest of the file is interpreted differently... On one side of the balance is code breakage. On the other side is currently broken code where bugs will be found. Which is the stronger argument? I'm inclined toward the latter, but neither has a huge body of code to back it. ChrisA From steve at pearwood.info Thu Nov 20 06:50:03 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 20 Nov 2014 16:50:03 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: <546D6BDD.4040007@stoneleaf.us> References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> <20141120002546.GH2748@ando.pearwood.info> <546D6BDD.4040007@stoneleaf.us> Message-ID: <20141120055003.GB20200@ando.pearwood.info> On Wed, Nov 19, 2014 at 08:19:41PM -0800, Ethan Furman wrote: > We are not, however, responsible for third-party documentation. No, of course not, but we should be aware that: * some people believe that raising StopIteration is an acceptable way to exit a generator; and * doing so has worked fine since generators were introduced back in Python 2.2. I wonder whether people who learned about generators back in the 2.2 days will have stronger opinions about raising StopIteration than more recent users? I remember learning that an explicit raise was the way to exit a generator, and sure enough the 2.2 What's New says this: Inside a generator function, the return statement can only be used without a value, and signals the end of the procession of values; afterwards the generator cannot return any further values. return with a value, such as return 5, is a syntax error inside a generator function. The end of the generator?s results can also be indicated by raising StopIteration manually, or by just letting the flow of execution fall off the bottom of the function. https://docs.python.org/3/whatsnew/2.2.html#pep-255-simple-generators That's not to say that we can't change the behaviour, but neither can we say it is undocumented or blame third parties. > > At this point, I'm convinced that there is a good argument for a > > __future__ import changing this behaviour. But I suspect that making > > this the default behaviour in the future will break a lot of code. > > Isn't that the case with every __future__ directive that becomes the > standard? Folks have an entire minor release to make the adjustment. Huh, you say that like it's a long time :-) -- Steven From rosuav at gmail.com Thu Nov 20 06:54:20 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 20 Nov 2014 16:54:20 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> <546B848E.6040406@biologie.uni-freiburg.de> <20141119011548.GZ2748@ando.pearwood.info> Message-ID: On Thu, Nov 20, 2014 at 2:44 PM, Chris Angelico wrote: > > I poked around a bit in the code and managed to come up with this. It > doesn't chain the previous exception, so the traceback is a little > scanty, but it does turn a StopIteration into a RuntimeError. (It > might also leak the original StopIteration. I'm not sure.) Prior to > this patch, I had 377 of 390 tests passing flawlessly and no failures > (just skips and warnings); with this applied, six failures. > With the attached demo patch, all tests pass except test_generators, which explicitly tests stuff about the correlation between return and StopIteration. There's the contextlib changes, a couple of places that were raising StopIteration and should be returning, and a couple that were letting StopIteration bubble and now need to catch it and return. I've deliberately not followed PEP 8 here, in the interests of minimizing diff size; in several cases, blocks of code ought to be indented a level, but I cheated and created a half-indentation to show how little actually changes. If anyone would care to try this on their own codebases, that'd be helpful. ChrisA -------------- next part -------------- A non-text attachment was scrubbed... Name: pep0479.patch Type: text/x-patch Size: 4590 bytes Desc: not available URL: From ncoghlan at gmail.com Thu Nov 20 11:01:40 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 20 Nov 2014 20:01:40 +1000 Subject: [Python-ideas] Return expressions Message-ID: This idea occurred to me in context of PEP 479 and the "or stop()" hack for generator expressions. I'm not a big enough fan of the idea to pursue it myself, but if anyone is bothered by the prospect of PEP 479 taking the "or stop()" technique away, this may be worth pursuing further. As a reminder of how the hack works, the following generator: >>> def takewhile(iterable, pred): ... for x in iter(iterable): ... if not pred(x): ... return ... yield x ... >>> list(takewhile(range(10), (lambda x: x < 5))) [0, 1, 2, 3, 4] Can currently be converted to a generator expression with the aid of a helper function: >>> def stop(): ... raise StopIteration ... >>> list(x for x in range(10) if x < 5 or stop()) [0, 1, 2, 3, 4] Under PEP 479, that will raise RuntimeError instead. If return was an expression rather than a statement (ala the yield statement -> expression conversion in PEP 342), then the following would be functionally equivalent to the current "or stop()" trick: >>> list(x for x in range(10) if x < 5 or return) [0, 1, 2, 3, 4] This is emphatically *not* executable pseudocode - it only makes sense in terms of the translation to the corresponding full generator function definition. However, the translation remains exact in terms of the normal semantics of the "return" keyword, unlike various other proposals to use the "while" keyword to provide comparable functionality. For comprehensions, the meaning of a bare return would likely need to be tweaked slightly to be the same as reaching the end of the frame: returning the object being constructed by the comprehension, rather than None. >>> [x for x in range(10) if x < 5 or return] [0, 1, 2, 3, 4] >>> {x for x in range(10) if x < 5 or return} {0, 1, 2, 3, 4} >>> {x:x for x in range(10) if x < 5 or return} {0:0, 1:1, 2:2, 3:3, 4:4} Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Nov 20 11:13:52 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 20 Nov 2014 20:13:52 +1000 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> <546B848E.6040406@biologie.uni-freiburg.de> <20141119011548.GZ2748@ando.pearwood.info> Message-ID: On 20 November 2014 15:54, Chris Angelico wrote: > On Thu, Nov 20, 2014 at 2:44 PM, Chris Angelico wrote: > > > > I poked around a bit in the code and managed to come up with this. It > > doesn't chain the previous exception, so the traceback is a little > > scanty, but it does turn a StopIteration into a RuntimeError. (It > > might also leak the original StopIteration. I'm not sure.) Prior to > > this patch, I had 377 of 390 tests passing flawlessly and no failures > > (just skips and warnings); with this applied, six failures. > > > > With the attached demo patch, all tests pass except test_generators, > which explicitly tests stuff about the correlation between return and > StopIteration. There's the contextlib changes, a couple of places that > were raising StopIteration and should be returning, and a couple that > were letting StopIteration bubble and now need to catch it and return. > I've deliberately not followed PEP 8 here, in the interests of > minimizing diff size; in several cases, blocks of code ought to be > indented a level, but I cheated and created a half-indentation to show > how little actually changes. > Thanks Chris - could you make a tracker issue and attach this there? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Thu Nov 20 11:43:54 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 20 Nov 2014 21:43:54 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> <546B848E.6040406@biologie.uni-freiburg.de> <20141119011548.GZ2748@ando.pearwood.info> Message-ID: On Thu, Nov 20, 2014 at 9:13 PM, Nick Coghlan wrote: > On 20 November 2014 15:54, Chris Angelico wrote: >> >> On Thu, Nov 20, 2014 at 2:44 PM, Chris Angelico wrote: >> > >> > I poked around a bit in the code and managed to come up with this. It >> > doesn't chain the previous exception, so the traceback is a little >> > scanty, but it does turn a StopIteration into a RuntimeError. (It >> > might also leak the original StopIteration. I'm not sure.) Prior to >> > this patch, I had 377 of 390 tests passing flawlessly and no failures >> > (just skips and warnings); with this applied, six failures. >> > >> >> With the attached demo patch, all tests pass except test_generators, >> which explicitly tests stuff about the correlation between return and >> StopIteration. There's the contextlib changes, a couple of places that >> were raising StopIteration and should be returning, and a couple that >> were letting StopIteration bubble and now need to catch it and return. >> I've deliberately not followed PEP 8 here, in the interests of >> minimizing diff size; in several cases, blocks of code ought to be >> indented a level, but I cheated and created a half-indentation to show >> how little actually changes. > > > Thanks Chris - could you make a tracker issue and attach this there? http://bugs.python.org/issue22906 ChrisA From python at mrabarnett.plus.com Thu Nov 20 13:54:36 2014 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 20 Nov 2014 12:54:36 +0000 Subject: [Python-ideas] Return expressions In-Reply-To: References: Message-ID: <546DE48C.5000805@mrabarnett.plus.com> On 2014-11-20 10:01, Nick Coghlan wrote: > This idea occurred to me in context of PEP 479 and the "or stop()" hack > for generator expressions. I'm not a big enough fan of the idea to > pursue it myself, but if anyone is bothered by the prospect of PEP 479 > taking the "or stop()" technique away, this may be worth pursuing further. > > As a reminder of how the hack works, the following generator: > > >>> def takewhile(iterable, pred): > ... for x in iter(iterable): > ... if not pred(x): > ... return > ... yield x > ... > >>> list(takewhile(range(10), (lambda x: x < 5))) > [0, 1, 2, 3, 4] > > Can currently be converted to a generator expression with the aid of a > helper function: > > >>> def stop(): > ... raise StopIteration > ... > >>> list(x for x in range(10) if x < 5 or stop()) > [0, 1, 2, 3, 4] > > Under PEP 479, that will raise RuntimeError instead. If return was an > expression rather than a statement (ala the yield statement -> > expression conversion in PEP 342), then the following would be > functionally equivalent to the current "or stop()" trick: > > >>> list(x for x in range(10) if x < 5 or return) > [0, 1, 2, 3, 4] > > This is emphatically *not* executable pseudocode - it only makes sense > in terms of the translation to the corresponding full generator function > definition. However, the translation remains exact in terms of the > normal semantics of the "return" keyword, unlike various other proposals > to use the "while" keyword to provide comparable functionality. > > For comprehensions, the meaning of a bare return would likely need to be > tweaked slightly to be the same as reaching the end of the frame: > returning the object being constructed by the comprehension, rather than > None. > > >>> [x for x in range(10) if x < 5 or return] > [0, 1, 2, 3, 4] > >>> {x for x in range(10) if x < 5 or return} > {0, 1, 2, 3, 4} > >>> {x:x for x in range(10) if x < 5 or return} > {0:0, 1:1, 2:2, 3:3, 4:4} > I'd prefer 'while' instead: >>> [x for x in range(10) while x < 5] [0, 1, 2, 3, 4] >>> {x for x in range(10) while x < 5} {0, 1, 2, 3, 4} >>> {x:x for x in range(10) while x < 5} {0:0, 1:1, 2:2, 3:3, 4:4} From njs at pobox.com Thu Nov 20 14:02:23 2014 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 20 Nov 2014 13:02:23 +0000 Subject: [Python-ideas] Return expressions In-Reply-To: References: Message-ID: On Thu, Nov 20, 2014 at 10:01 AM, Nick Coghlan wrote: > >>> list(x for x in range(10) if x < 5 or return) > [0, 1, 2, 3, 4] I'd be concerned about the confusion engendered by 'return' having totally different semantics within very small regions, e.g. we'd have def takewhile(iterable, pred): return (x for x in iterable if pred(x) or return) Esp. because normally 'return' is very powerful -- right now any mention of that token anywhere in a function body jumps out of the entire function, so you really have to scan for them actively when trying to understand a function's flow control. Not a big fan of a flow control expression in general, really; yield expressions are kinda flow-controlly, but they just pause execution, they don't disrupt the logical through-line of a block of code. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From ncoghlan at gmail.com Thu Nov 20 14:36:26 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 20 Nov 2014 23:36:26 +1000 Subject: [Python-ideas] Return expressions In-Reply-To: <546DE48C.5000805@mrabarnett.plus.com> References: <546DE48C.5000805@mrabarnett.plus.com> Message-ID: On 20 November 2014 22:54, MRAB wrote: > I'd prefer 'while' instead: > > >>> [x for x in range(10) while x < 5] > [0, 1, 2, 3, 4] > >>> {x for x in range(10) while x < 5} > {0, 1, 2, 3, 4} > >>> {x:x for x in range(10) while x < 5} > {0:0, 1:1, 2:2, 3:3, 4:4} > That's been suggested many times, and always fails on the grounds of being completely incompatible with the expansions of comprehensions and generator expressions as nested compound statements. The return expression idea is one that reflects the actual deep structure of the nested scopes that underlie the syntactic sugar, so it's compatible with the expansion. This idea was mainly aimed at folks that might be inclined to worry about the possible loss of the "or stop()" generator expression trick in PEP 479, though. If that trick wasn't currently possible, and wasn't being proposed for removal, I never would have posted this idea. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From encukou at gmail.com Thu Nov 20 15:43:04 2014 From: encukou at gmail.com (Petr Viktorin) Date: Thu, 20 Nov 2014 15:43:04 +0100 Subject: [Python-ideas] Return expressions In-Reply-To: References: <546DE48C.5000805@mrabarnett.plus.com> Message-ID: On Thu, Nov 20, 2014 at 2:36 PM, Nick Coghlan wrote: > On 20 November 2014 22:54, MRAB wrote: >> >> I'd prefer 'while' instead: >> >> >>> [x for x in range(10) while x < 5] >> [0, 1, 2, 3, 4] >> >>> {x for x in range(10) while x < 5} >> {0, 1, 2, 3, 4} >> >>> {x:x for x in range(10) while x < 5} >> {0:0, 1:1, 2:2, 3:3, 4:4} > > > That's been suggested many times, and always fails on the grounds of being > completely incompatible with the expansions of comprehensions and generator > expressions as nested compound statements. > > The return expression idea is one that reflects the actual deep structure of > the nested scopes that underlie the syntactic sugar, so it's compatible with > the expansion. > > This idea was mainly aimed at folks that might be inclined to worry about > the possible loss of the "or stop()" generator expression trick in PEP 479, > though. If that trick wasn't currently possible, and wasn't being proposed > for removal, I never would have posted this idea. > The "while" expresses intent, "return" caters to low-level mechanics. The "nested compound statements" explanation is already not that good: why does the value come first, not last? For readability*. The difference between function/nested compound statements syntax and comprehension syntax is already so big, and the low-level "while x:"?"if not x: break" translation is so easy, that between the two the readability of "while" should win. Outside of comprehensions, an expression that never has a value seems quite backwards (and dangerous, in this case). * (or you might say, for similarity to math notation) From python at mrabarnett.plus.com Thu Nov 20 16:49:53 2014 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 20 Nov 2014 15:49:53 +0000 Subject: [Python-ideas] Return expressions In-Reply-To: References: <546DE48C.5000805@mrabarnett.plus.com> Message-ID: <546E0DA1.10609@mrabarnett.plus.com> On 2014-11-20 14:43, Petr Viktorin wrote: > On Thu, Nov 20, 2014 at 2:36 PM, Nick Coghlan wrote: > > On 20 November 2014 22:54, MRAB wrote: > >> > >> I'd prefer 'while' instead: > >> > >> >>> [x for x in range(10) while x < 5] > >> [0, 1, 2, 3, 4] > >> >>> {x for x in range(10) while x < 5} > >> {0, 1, 2, 3, 4} > >> >>> {x:x for x in range(10) while x < 5} > >> {0:0, 1:1, 2:2, 3:3, 4:4} > > > > > > That's been suggested many times, and always fails on the grounds of being > > completely incompatible with the expansions of comprehensions and generator > > expressions as nested compound statements. > > > > The return expression idea is one that reflects the actual deep structure of > > the nested scopes that underlie the syntactic sugar, so it's compatible with > > the expansion. > > > > This idea was mainly aimed at folks that might be inclined to worry about > > the possible loss of the "or stop()" generator expression trick in PEP 479, > > though. If that trick wasn't currently possible, and wasn't being proposed > > for removal, I never would have posted this idea. > > > > The "while" expresses intent, "return" caters to low-level mechanics. > The "nested compound statements" explanation is already not that good: > why does the value come first, not last? For readability*. The > difference between function/nested compound statements syntax and > comprehension syntax is already so big, and the low-level "while > x:"?"if not x: break" translation is so easy, that between the two the > readability of "while" should win. I can see the problem with 'while': if there are multiple 'for' parts, they are equivalent to nested 'for' loops, so you might assume that a 'while' part would also be equivalent to a nested 'while' loop, whereas it would, in fact, be controlling the preceding 'for' part. In other words, it would be: for x in range(10) while x < 5: .... but could be seen as: for x in range(10): while x < 5: .... > Outside of comprehensions, an expression that never has a value seems > quite backwards (and dangerous, in this case). > > * (or you might say, for similarity to math notation) > From ron3200 at gmail.com Thu Nov 20 17:11:30 2014 From: ron3200 at gmail.com (Ron Adam) Date: Thu, 20 Nov 2014 10:11:30 -0600 Subject: [Python-ideas] Return expressions In-Reply-To: References: Message-ID: On 11/20/2014 04:01 AM, Nick Coghlan wrote: > This idea occurred to me in context of PEP 479 and the "or stop()" hack for > generator expressions. I'm not a big enough fan of the idea to pursue it > myself, but if anyone is bothered by the prospect of PEP 479 taking the "or > stop()" technique away, this may be worth pursuing further. > > As a reminder of how the hack works, the following generator: > > >>> def takewhile(iterable, pred): > ... for x in iter(iterable): > ... if not pred(x): > ... return > ... yield x > ... > >>> list(takewhile(range(10), (lambda x: x < 5))) > [0, 1, 2, 3, 4] Just a note: If this generator was used on an iterator more than once it would drop items between groups. Takewhile fits the peek-ahead pattern I mentioned in an earlier thread that groupby matches. I think groupby manages to hide the looked ahead value in the yield path, but it still does look ahead, just not in an obvious way. It's my understanding looking ahead is problomatic for genrators because they may have side effects when next() is called on them. Like reading input or writing output, or accessing some other object outside the generator. It seams to me that most (if not all) partial iterations will either want to yield the "last" value tested, or stop "before" the last value tested. The generator expression version of the above has issues with both of those. Cheers, Ron From ncoghlan at gmail.com Thu Nov 20 17:21:53 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 21 Nov 2014 02:21:53 +1000 Subject: [Python-ideas] Return expressions In-Reply-To: <546E0DA1.10609@mrabarnett.plus.com> References: <546DE48C.5000805@mrabarnett.plus.com> <546E0DA1.10609@mrabarnett.plus.com> Message-ID: On 21 November 2014 01:49, MRAB wrote: > > > On 2014-11-20 14:43, Petr Viktorin wrote: >> >> The "while" expresses intent, "return" caters to low-level mechanics. >> The "nested compound statements" explanation is already not that good: >> why does the value come first, not last? For readability*. The >> difference between function/nested compound statements syntax and >> comprehension syntax is already so big, and the low-level "while >> x:"?"if not x: break" translation is so easy, that between the two the >> readability of "while" should win. > > I can see the problem with 'while': if there are multiple 'for' parts, they are equivalent to nested > 'for' loops, so you might assume that a 'while' part would also be equivalent to a nested 'while' > loop, whereas it would, in fact, be controlling the preceding 'for' part. > > In other words, it would be: > > for x in range(10) while x < 5: > .... > > but could be seen as: > > for x in range(10): > while x < 5: PEP 3142 was the last attempt at asking this question - Guido isn't keen on the idea, so he rejected it the last time we did a pass through the PEPs that weren't being actively worked on. So while the odds still aren't good, a proposal that kept the statement and expression forms consistent might still have some chance, by proposing that the statement example above: for x in range(10) while x < 5: .... Be equivalent to: for x in range(10): if not x < 5: break .... That is, there'd only be one loop, rather than two. At that point, a "while" clause in a comprehension or generator expression would be part of that enhanced syntax, rather than a standalone while loop. That does also suggest another possible alternative to the "or stop()" trick - allow *break* as an expression. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rosuav at gmail.com Thu Nov 20 17:36:16 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 21 Nov 2014 03:36:16 +1100 Subject: [Python-ideas] Return expressions In-Reply-To: References: <546DE48C.5000805@mrabarnett.plus.com> <546E0DA1.10609@mrabarnett.plus.com> Message-ID: On Fri, Nov 21, 2014 at 3:21 AM, Nick Coghlan wrote: > That does also suggest another possible alternative to the "or stop()" > trick - allow *break* as an expression. Or just as a keyword component of a comprehension, in the same way 'for' is. Since 'if' already has meaning, [x for x in range(10) if x >= 5 break] would be confusing, so probably it'd have to be the slightly-awkward-to-read [x for x in range(10) break x >= 5], adding a termination condition to the preceding loop. ChrisA From random832 at fastmail.us Thu Nov 20 18:35:47 2014 From: random832 at fastmail.us (random832 at fastmail.us) Date: Thu, 20 Nov 2014 12:35:47 -0500 Subject: [Python-ideas] Return expressions In-Reply-To: <546E0DA1.10609@mrabarnett.plus.com> References: <546DE48C.5000805@mrabarnett.plus.com> <546E0DA1.10609@mrabarnett.plus.com> Message-ID: <1416504947.266514.193437377.1DE3BBF5@webmail.messagingengine.com> On Thu, Nov 20, 2014, at 10:49, MRAB wrote: > I can see the problem with 'while': if there are multiple 'for' parts, > they are equivalent to nested > 'for' loops, so you might assume that a 'while' part would also be > equivalent to a nested 'while' > loop, whereas it would, in fact, be controlling the preceding 'for' part. > > In other words, it would be: > > for x in range(10) while x < 5: > .... > > but could be seen as: > > for x in range(10): > while x < 5: > .... I've always thought of comprehensions as a sort of "inside-out" loop, due to the value at the beginning: (f(a) for a in range) is roughly equivalent to for a in range: yield f(a). Due to this, I was actually somewhat surprised to find that multiple for clauses don't work as if they were nested inside-out loops >>> [(a, b) for a in range(3) for b in range(3)] [(0, 0), (0, 1), (0, 2), (1, 0), (1, 1), (1, 2), (2, 0), (2, 1), (2, 2)] >>> [[(a, b) for a in range(3)] for b in range(3)] [[(0, 0), (1, 0), (2, 0)], [(0, 1), (1, 1), (2, 1)], [(0, 2), (1, 2), (2, 2)]] I think people trying to use "while" for this are coming from an english-grammar point of view. From njs at pobox.com Thu Nov 20 19:02:31 2014 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 20 Nov 2014 18:02:31 +0000 Subject: [Python-ideas] Return expressions In-Reply-To: References: <546DE48C.5000805@mrabarnett.plus.com> <546E0DA1.10609@mrabarnett.plus.com> Message-ID: On Thu, Nov 20, 2014 at 4:36 PM, Chris Angelico wrote: > On Fri, Nov 21, 2014 at 3:21 AM, Nick Coghlan wrote: >> That does also suggest another possible alternative to the "or stop()" >> trick - allow *break* as an expression. > > Or just as a keyword component of a comprehension, in the same way > 'for' is. Since 'if' already has meaning, [x for x in range(10) if x >>= 5 break] would be confusing, so probably it'd have to be the > slightly-awkward-to-read [x for x in range(10) break x >= 5], adding a > termination condition to the preceding loop. 'break' is certainly better than 'return' -- currently 'break' means "look for the nearest enclosing loop", not "look for the enclosing function", so it'd preserve that at least. [x for x in range(10) if x >= 5 else break]? with 'else break' handled in the grammar as a non-compositional phrase like 'is not'? Not sure how this interacts with multiple mixed for and if clauses -- I've never wrapped my head around those semantics. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From antony.lee at berkeley.edu Thu Nov 20 19:24:12 2014 From: antony.lee at berkeley.edu (Antony Lee) Date: Thu, 20 Nov 2014 10:24:12 -0800 Subject: [Python-ideas] Return expressions In-Reply-To: References: <546DE48C.5000805@mrabarnett.plus.com> <546E0DA1.10609@mrabarnett.plus.com> Message-ID: This is fairly similar to a recent thread suggesting an "or raise" construct ("x = f() or raise ValueError" as a shortcut for "x = f(); if not x: raise ValueError"). I suggested that "raise" be made a value-less expression, so that "x = f() or raise ..." can be intepreted normally. As a side effect, this would make lambdas slightly more powerful too. Having "return" and "break" as value-less expressions would be nice too. The interaction with nested loops in generators would be the natural one, without any generator-specific syntax: ((x, y) if stay_in_inner(x, y) else break for x in X for y in Y) is the same as for x in X: for y in Y: if stay_in_inner(x, y): yield x, y else: break or, with break-expression everywhere, as a separate generator: for x in X: for y in Y: (yield x, y) if stay_in_inner(x, y) else break whereas ((x, y) if stay_in_gen(x, y) else return for x in X for y in Y) is the same as for x in X: for y in Y: if stay_in_gen(x, y): yield x, y else: return or for x in X: for y in Y: (yield x, y) if stay_in_gen(x, y) else return i.e. the intepretation of a genexp is still to take the nested "for"s and "if"s after the leading expression, add suitable colons, newlines and indentation, and insert the leading expression at the end. Antony 2014-11-20 10:02 GMT-08:00 Nathaniel Smith : > On Thu, Nov 20, 2014 at 4:36 PM, Chris Angelico wrote: > > On Fri, Nov 21, 2014 at 3:21 AM, Nick Coghlan > wrote: > >> That does also suggest another possible alternative to the "or stop()" > >> trick - allow *break* as an expression. > > > > Or just as a keyword component of a comprehension, in the same way > > 'for' is. Since 'if' already has meaning, [x for x in range(10) if x > >>= 5 break] would be confusing, so probably it'd have to be the > > slightly-awkward-to-read [x for x in range(10) break x >= 5], adding a > > termination condition to the preceding loop. > > 'break' is certainly better than 'return' -- currently 'break' means > "look for the nearest enclosing loop", not "look for the enclosing > function", so it'd preserve that at least. > > [x for x in range(10) if x >= 5 else break]? > > with 'else break' handled in the grammar as a non-compositional phrase > like 'is not'? Not sure how this interacts with multiple mixed for and > if clauses -- I've never wrapped my head around those semantics. > > -n > > -- > Nathaniel J. Smith > Postdoctoral researcher - Informatics - University of Edinburgh > http://vorpus.org > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Thu Nov 20 22:13:48 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 21 Nov 2014 10:13:48 +1300 Subject: [Python-ideas] Return expressions In-Reply-To: References: Message-ID: <546E598C.8000905@canterbury.ac.nz> Nick Coghlan wrote: > >>> list(x for x in range(10) if x < 5 or return) > [0, 1, 2, 3, 4] > > This is emphatically *not* executable pseudocode - it only makes sense > in terms of the translation to the corresponding full generator function -1 on messing with the semantics of 'return' (which would have implications beyond this use case) just so you can write something in one particular context that doesn't read intuitively. If early exit from comprehensions is really a desirable feature (and Guido doesn't seem to think so) it would be better addressed with an explicit language feature, e.g. list(x for x in range(10) while x < 5) [x for x in range(10) while x < 5] This makes the intent perfectly clear and works just as well in all kinds of comprehensions. It breaks the strict equivalence between comprehension syntax and the corresponding nested loop code, but you can't have everything. -- Greg From wolfgang.maier at biologie.uni-freiburg.de Thu Nov 20 23:39:58 2014 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Thu, 20 Nov 2014 23:39:58 +0100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> <20141120020615.GI2748@ando.pearwood.info> Message-ID: On 20.11.2014 03:24, Chris Angelico wrote: > On Thu, Nov 20, 2014 at 1:06 PM, Steven D'Aprano wrote: > >> I trust that we all expect to be able to factor out the raise into a >> helper function or method, yes? It truly would be surprising if this >> failed: >> >> >> class MyIterator: >> def __iter__(self): >> return self >> def __next__(self): >> return something() >> >> >> def something(): >> # Toy helper function. >> if random.random() < 0.5: >> return "Spam!" >> raise StopIteration >> >> >> >> Now let's write this as a generator: >> >> def gen(): >> while True: >> yield something() >> >> >> which is much nicer than: >> >> def gen(): >> while True: >> try: >> yield something() >> except StopIteration: >> return # converted by Python into raise StopIteration > > Sure. There was a suggestion that "return yield from something()" > would work, though, which - I can't confirm that this works, but > assuming it does - would be a lot tidier. But there's still a > difference. Your first helper function was specifically a __next__ > helper. It was tied intrinsically to the iterator protocol. If you > want to call a __next__ helper (or actually call next(iter) on > something) inside a generator, you'll have to - if this change goes > through - cope with the fact that generator protocol says "return" > where __next__ protocol says "raise StopIteration". If you want a > generator helper, it'd look like this: > > def something(): > # Toy helper function. > if random.random() < 0.5: > yield "Spam!" > > def gen(): > yield from something() > > Voila! Now it's a generator helper, following generator protocol. > Every bit as tidy as the original. Let's write a __getitem__ helper: > Hmm, I'm not convinced by these toy examples, but I did inspect some of my own code for incompatibility with the proposed change. I found that there really is only one recurring pattern I use that I'd have to change and that is how I've implemented several file parsers. I tend to write them like this: def parser (file_object): while True: title_line = next(file_object) # will terminate after the last record try: # read and process the rest of the record here except StopIteration: # this record is incomplete raise OSError('Invalid file format') yield processed_record So I'm catching StopIteration raised by the underlying IOWrapper only if it occurs in illegal places (with regard to the file format the parser expects), but not when it indicates the end of a correct file. I always thought of letting the Error bubble up as a way to keep the parser transparent. Now in this case, I think, I would have to change this to: def parser (io_object): while True: try: title_line = next(io_object) except StopIteration: return ... which I could certainly do without too much effort, but could this be one of the more widespread sources of incompatibility that Steve imagines ? Wolfgang From ron3200 at gmail.com Thu Nov 20 23:50:03 2014 From: ron3200 at gmail.com (Ron Adam) Date: Thu, 20 Nov 2014 16:50:03 -0600 Subject: [Python-ideas] Return expressions In-Reply-To: References: Message-ID: On 11/20/2014 04:01 AM, Nick Coghlan wrote: > Can currently be converted to a generator expression with the aid of a > helper function: > > >>> def stop(): > ... raise StopIteration > ... > >>> list(x for x in range(10) if x < 5 or stop()) > [0, 1, 2, 3, 4] How about making a stop_generator() function builtin, just for this feature? It could raise a specific StopIteration instance, possibly the same subclass/instance as return does in generators, so the generator would know to catch it instead of complaining. As a builting, all the mechanics are kept out of the way as implementation details. For those who already use stop() in many places, if there are any one who does that often, they could just do stop = stop_generator at the top of their programs. I think "stop" is too general a name in this case. It's fine when the function is defined right above where it is used, but as a builtin a more specific name would be better. Cheers, Ron From wolfgang.maier at biologie.uni-freiburg.de Thu Nov 20 23:58:51 2014 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Thu, 20 Nov 2014 23:58:51 +0100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: <20141120020615.GI2748@ando.pearwood.info> References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> <20141120020615.GI2748@ando.pearwood.info> Message-ID: On 20.11.2014 03:06, Steven D'Aprano wrote: > On Thu, Nov 20, 2014 at 03:24:07AM +1100, Chris Angelico wrote: > >> Without the generator, *only* __next__ has >> this effect, and that's exactly where it's documented to be. > > The documentation states that __next__ raises StopIteration, it doesn't > say that *only* __next__ should raise StopIteration. > > https://docs.python.org/3/library/stdtypes.html#iterator.__next__ > > I trust that we all expect to be able to factor out the raise into a > helper function or method, yes? It truly would be surprising if this > failed: > > > class MyIterator: > def __iter__(self): > return self > def __next__(self): > return something() > > > def something(): > # Toy helper function. > if random.random() < 0.5: > return "Spam!" > raise StopIteration > > > > Now let's write this as a generator: > > def gen(): > while True: > yield something() > > > which is much nicer than: > > def gen(): > while True: > try: > yield something() > except StopIteration: > return # converted by Python into raise StopIteration > I find this example a compelling argument against the PEP. Personally, I'm dealing a lot more often with refactoring a generator function into a iterator class than I'm rewriting generator expressions into comprehensions (at least the exotic kinds that would reveal their inequality). So for me at least, the burden of having to remember that I can let (and should let) StopIteration bubble up inside __next__, but not in generator functions weighs in heavier than the equality argument and the protection against hard-to-diagnose (but rarely occurring) bugs in nested generator functions. From rosuav at gmail.com Fri Nov 21 00:30:50 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 21 Nov 2014 10:30:50 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> <20141120020615.GI2748@ando.pearwood.info> Message-ID: On Fri, Nov 21, 2014 at 9:58 AM, Wolfgang Maier wrote: > I find this example a compelling argument against the PEP. Personally, I'm > dealing a lot more often with refactoring a generator function into a > iterator class than I'm rewriting generator expressions into comprehensions > (at least the exotic kinds that would reveal their inequality). > So for me at least, the burden of having to remember that I can let (and > should let) StopIteration bubble up inside __next__, but not in generator > functions weighs in heavier than the equality argument and the protection > against hard-to-diagnose (but rarely occurring) bugs in nested generator > functions. Compare my earlier response to Steven, though: it's not difficult to refactor a generator into a helper-generator, rather than refactor a generator into a helper-__next__. This proposal would force a decoupling of generator protocol from __next__ protocol. The ugliness in Steven's examples comes from trying to use a __next__ helper in a generator. It'd be just as ugly trying to refactor __getitem__ to make use of a __getattr__ helper - you'd have to catch AttributeError and turn it into KeyError at the boundary between the two protocols. ChrisA From greg.ewing at canterbury.ac.nz Thu Nov 20 22:28:32 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 21 Nov 2014 10:28:32 +1300 Subject: [Python-ideas] Return expressions In-Reply-To: References: Message-ID: <546E5D00.6040005@canterbury.ac.nz> Ron Adam wrote: > It's my understanding looking ahead is problomatic for genrators because > they may have side effects when next() is called on them. Not just generators, but any iterator. Any algorithm that requires lookahead has problems if you apply it more than once to the same iterator. Something has to store the lookahead value in between times, and the iterator protocol just doesn't allow for that. -- Greg From guido at python.org Fri Nov 21 00:51:07 2014 From: guido at python.org (Guido van Rossum) Date: Thu, 20 Nov 2014 15:51:07 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> <20141120020615.GI2748@ando.pearwood.info> Message-ID: On Thu, Nov 20, 2014 at 2:39 PM, Wolfgang Maier < wolfgang.maier at biologie.uni-freiburg.de> wrote: > [...] > Hmm, I'm not convinced by these toy examples, but I did inspect some of my > own code for incompatibility with the proposed change. I found that there > really is only one recurring pattern I use that I'd have to change and that > is how I've implemented several file parsers. I tend to write them like > this: > > def parser (file_object): > while True: > title_line = next(file_object) # will terminate after the last > record > > try: > # read and process the rest of the record here > except StopIteration: > # this record is incomplete > raise OSError('Invalid file format') > yield processed_record > > So I'm catching StopIteration raised by the underlying IOWrapper only if > it occurs in illegal places (with regard to the file format the parser > expects), but not when it indicates the end of a correct file. > I always thought of letting the Error bubble up as a way to keep the > parser transparent. > Now in this case, I think, I would have to change this to: > > def parser (io_object): > while True: > try: > title_line = next(io_object) > except StopIteration: > return > ... > > which I could certainly do without too much effort, but could this be one > of the more widespread sources of incompatibility that Steve imagines ? There's probably something important missing from your examples. The above while-loop is equivalent to for title_line in io_object: ... If you're okay with getting RuntimeError instead of OSError for an undesirable StopIteration, you can just drop the except clause altogether. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Fri Nov 21 06:00:31 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 21 Nov 2014 00:00:31 -0500 Subject: [Python-ideas] Return expressions In-Reply-To: <546E0DA1.10609@mrabarnett.plus.com> References: <546DE48C.5000805@mrabarnett.plus.com> <546E0DA1.10609@mrabarnett.plus.com> Message-ID: On 11/20/2014 10:49 AM, MRAB wrote: > I can see the problem with 'while': if there are multiple 'for' parts, > they are equivalent to nested > 'for' loops, x for x in A if x=B is also equivalent to nested compound statements. > so you might assume that a 'while' part would also be > equivalent to a nested 'while' > loop, whereas it would, in fact, be controlling the preceding 'for' part. -- Terry Jan Reedy From wolfgang.maier at biologie.uni-freiburg.de Fri Nov 21 11:19:40 2014 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Fri, 21 Nov 2014 11:19:40 +0100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> <20141120020615.GI2748@ando.pearwood.info> Message-ID: On 21.11.2014 00:51, Guido van Rossum wrote: > On Thu, Nov 20, 2014 at 2:39 PM, Wolfgang Maier > > > wrote: > > [...] > Hmm, I'm not convinced by these toy examples, but I did inspect some > of my own code for incompatibility with the proposed change. I found > that there really is only one recurring pattern I use that I'd have > to change and that is how I've implemented several file parsers. I > tend to write them like this: > > def parser (file_object): > while True: > title_line = next(file_object) # will terminate after the > last record > > try: > # read and process the rest of the record here > except StopIteration: > # this record is incomplete > raise OSError('Invalid file format') > yield processed_record > > There's probably something important missing from your examples. The > above while-loop is equivalent to > > for title_line in io_object: > ... > My reason for not using a for loop here is that I'm trying to read from a file where several lines form a record, so I'm reading the title line of a record (and if there is no record in the file any more I want the parser generator to terminate/return. If a title line is read successfully then I'm reading the record's body lines inside a try/except, i.e. where it says "# read and process the rest of the record here" in my shortened code I am actually calling next several times again to retrieve the body lines (and while reading these lines an unexpected StopIteration in the IOWrapper is considered a file format error). I realize that I could also use a for loop and still call next(file_object) inside it, but I find this a potentially confusing pattern that I'm trying to avoid by using the while loop and all explicit next(). Compare: for title_line in file_object: record_body = next(file_object) # in reality record_body is generated using several next calls # depending on the content found in the record body while it's read yield (title_line, record_body) vs while True: title_line = next(file_object) body = next(file_object) yield (title_line, body) To me, the for loop version suggests to me that the content of file_object is read in line by line by the loop (even though the name title_line tries to hint at this being not true). Only when I inspect the loop body I see that further items are retrieved with next() and, thus, skipped in the for iteration. The while loop, on the other hand, makes the number of iterations very clear by showing all of them in the loop body. Would you agree that this is justification enough for while instead of for or is it only me who thinks that a for loop makes the code read awkward ? > If you're okay with getting RuntimeError instead of OSError for an > undesirable StopIteration, you can just drop the except clause altogether. Right, I could do this if the PEP-described behavior was in effect today. From rosuav at gmail.com Fri Nov 21 11:26:24 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 21 Nov 2014 21:26:24 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> <20141120020615.GI2748@ando.pearwood.info> Message-ID: On Fri, Nov 21, 2014 at 9:19 PM, Wolfgang Maier wrote: > My reason for not using a for loop here is that I'm trying to read from a > file where several lines form a record, so I'm reading the title line of a > record (and if there is no record in the file any more I want the parser > generator to terminate/return. If a title line is read successfully then I'm > reading the record's body lines inside a try/except, i.e. where it says "# > read and process the rest of the record here" in my shortened code I am > actually calling next several times again to retrieve the body lines (and > while reading these lines an unexpected StopIteration in the IOWrapper is > considered a file format error). > I realize that I could also use a for loop and still call next(file_object) > inside it, but I find this a potentially confusing pattern that I'm trying > to avoid by using the while loop and all explicit next(). I agree. The last example in the PEP is a cut-down form of your parser, and I raise the exact same concern: https://www.python.org/dev/peps/pep-0479/#examples The use of the for loop strongly implies that the loop body will be executed once for each thing in the iterable, which isn't true if you next() it in the body. Legal? Sure. Confusing? Definitely. ChrisA From wolfgang.maier at biologie.uni-freiburg.de Fri Nov 21 11:38:40 2014 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Fri, 21 Nov 2014 11:38:40 +0100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> <20141120020615.GI2748@ando.pearwood.info> Message-ID: On 21.11.2014 11:26, Chris Angelico wrote: > > I agree. The last example in the PEP is a cut-down form of your > parser, and I raise the exact same concern: > > https://www.python.org/dev/peps/pep-0479/#examples > > The use of the for loop strongly implies that the loop body will be > executed once for each thing in the iterable, which isn't true if you > next() it in the body. Legal? Sure. Confusing? Definitely. > Yes, that example illustrates the exact same thing I tried to describe. Nice! From raymond.hettinger at gmail.com Fri Nov 21 12:24:40 2014 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Fri, 21 Nov 2014 03:24:40 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com> > On Nov 15, 2014, at 1:29 AM, Chris Angelico wrote: > > Abstract > ======== > > This PEP proposes a semantic change to ``StopIteration`` when raised > inside a generator, unifying the behaviour of list comprehensions and > generator expressions somewhat. Please let me know if I'm reading the PEP correctly. Does the proposal break all existing code in generators that uses next() to raise StopIteration or that raises StopIteration explicitly? For example, here is the pure python recipe for itertools.accumulate() show in the docs at https://docs.python.org/3/library/itertools.html#itertool-functions : def accumulate(iterable, func=operator.add): 'Return running totals' # accumulate([1,2,3,4,5]) --> 1 3 6 10 15 # accumulate([1,2,3,4,5], operator.mul) --> 1 2 6 24 120 it = iter(iterable) total = next(it) yield total for element in it: total = func(total, element) yield total Or would it break the traditional examples of how to write something like izip() using a generator? def izip(iterable1, iterable2): it1 = iter(iterable1) it2 = iter(iterable2) while True: v1 = next(it1) v2 = next(it2) yield v1, v2 assert list(izip('ab', 'cde')) == [('a', 'c'), ('b', 'd')] assert list(izip('abc', 'cd')) == [('a', 'c'), ('b', 'd')] My initial reading of the PEP was a bit unsettling because the listed examples (such as unwrap() and parser()) were a series of cases where code that was currently working just fine for the last decade would break and need be changed to less pleasant looking code. Also, the PEP motivation seemed somewhat weak. Instead of listing known bugs or real-world development difficulties, it seems to hinge almost entirely some "being surprised" that list comprehensions and generator expressions aren't the same in every regard (they aren't). AFAICT, that suggestion is that an incorrect expectation of perfect symmetry warrants a number of what the author calls "consequences for existing code". It seems that if the real problem is one of false expectations or surprises, the direct solution would be to provide clearer examples of how things actually work and to disabuse the idea that list comprehensions and generator expressions are more interchangeable than they actually are. Raymond P.S. On a more general note, I think that our biggest problem in the Python world is getting people to switch to Python 3. If we really want that to happen, we should develop a strong aversion to proposals that further increase the semantic difference between Python 2 and Python 3. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Nov 21 12:50:52 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 21 Nov 2014 22:50:52 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com> References: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com> Message-ID: On Fri, Nov 21, 2014 at 10:24 PM, Raymond Hettinger wrote: > > Please let me know if I'm reading the PEP correctly. > Does the proposal break all existing code in generators > that uses next() to raise StopIteration or that raises > StopIteration explicitly? > > For example, here is the pure python recipe for itertools.accumulate() > show in the docs at > https://docs.python.org/3/library/itertools.html#itertool-functions : > > > def accumulate(iterable, func=operator.add): > 'Return running totals' > # accumulate([1,2,3,4,5]) --> 1 3 6 10 15 > # accumulate([1,2,3,4,5], operator.mul) --> 1 2 6 24 120 > it = iter(iterable) > total = next(it) > yield total > for element in it: > total = func(total, element) > yield total The case where the iterable is empty would now raise, yes. > Or would it break the traditional examples of how to write something > like izip() using a generator? > > def izip(iterable1, iterable2): > it1 = iter(iterable1) > it2 = iter(iterable2) > while True: > v1 = next(it1) > v2 = next(it2) > yield v1, v2 > > assert list(izip('ab', 'cde')) == [('a', 'c'), ('b', 'd')] > assert list(izip('abc', 'cd')) == [('a', 'c'), ('b', 'd')] Yes, this would be affected. This proposal causes a separation of generators and iterators, so it's no longer possible to pretend that they're the same thing. > Also, the PEP motivation seemed somewhat weak. Instead of listing > known bugs or real-world development difficulties, it seems to hinge > almost entirely some "being surprised" that list comprehensions and > generator expressions aren't the same in every regard (they aren't). The main point is one of exceptions being silently suppressed. Iterator protocol involves the StopIteration exception; generator protocol doesn't, yet currently a generator that raises StopIteration will quietly terminate. It's as if every generator is wrapped inside "try: ..... except StopIteration: pass". Would you accept any function being written with that kind of implicit suppression of any other exception? > P.S. On a more general note, I think that our biggest problem > in the Python world is getting people to switch to Python 3. > If we really want that to happen, we should develop a strong > aversion to proposals that further increase the semantic > difference between Python 2 and Python 3. The recommended form of the code will work exactly the same way in both versions: explicitly catching StopIteration and using it as a signal that the function should terminate. The only difference is the behaviour of the non-recommended practice of allowing an exception to bubble part-way and then be implicitly caught. ChrisA From ncoghlan at gmail.com Fri Nov 21 13:55:34 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 21 Nov 2014 22:55:34 +1000 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com> Message-ID: On 21 November 2014 21:50, Chris Angelico wrote: > The recommended form of the code will work exactly the same way in > both versions: explicitly catching StopIteration and using it as a > signal that the function should terminate. The only difference is the > behaviour of the non-recommended practice of allowing an exception to > bubble part-way and then be implicitly caught. Raymond's point is that for a long time, the equivalence between "return" and "raise StopIteration" in a generator function has been explicit. The dissatisfaction with the "non-local flow control" aspects of the latter really only started to creep in around Python 2.5 (based on the explicit decision to avoid non-local flow control behaviour in the definition of the with statement in PEP 343), and this PEP is the first time this longstanding behaviour of generators has been seriously questioned at the python-dev level. Guido also didn't add himself as a co-author on the PEP, so it isn't clear on first reading that *he's* the one considering the change, rather than it being an independent suggestion on your part :) I suspect enough evidence of breakage is accumulating to tip the balance back to "not worth the hassle", but it would also be possible to just *add* the "from __future__ import generator_stop" feature, and postpone a decision on making that the only available behaviour. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From wolfgang.maier at biologie.uni-freiburg.de Fri Nov 21 14:31:33 2014 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Fri, 21 Nov 2014 14:31:33 +0100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com> References: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com> Message-ID: On 21.11.2014 12:24, Raymond Hettinger wrote: > >> On Nov 15, 2014, at 1:29 AM, Chris Angelico >> > > wrote: >> >> Abstract >> ======== >> >> This PEP proposes a semantic change to ``StopIteration`` when raised >> inside a generator, unifying the behaviour of list comprehensions and >> generator expressions somewhat. > > > Please let me know if I'm reading the PEP correctly. > Does the proposal break all existing code in generators > that uses next() to raise StopIteration or that raises > StopIteration explicitly? > > For example, here is the pure python recipe for itertools.accumulate() > show in the docs at > https://docs.python.org/3/library/itertools.html#itertool-functions : > > > def accumulate(iterable, func=operator.add): > 'Return running totals' > # accumulate([1,2,3,4,5]) --> 1 3 6 10 15 > # accumulate([1,2,3,4,5], operator.mul) --> 1 2 6 24 120 > it = iter(iterable) > total = next(it) > yield total > for element in it: > total = func(total, element) > yield total > > Or would it break the traditional examples of how to write something > like izip() using a generator? > > def izip(iterable1, iterable2): > it1 = iter(iterable1) > it2 = iter(iterable2) > while True: > v1 = next(it1) > v2 = next(it2) > yield v1, v2 > > assert list(izip('ab', 'cde')) == [('a', 'c'), ('b', 'd')] > assert list(izip('abc', 'cd')) == [('a', 'c'), ('b', 'd')] > Since I already learnt quite a lot from following this thread: I checked yesterday what the docs have to say about the pure-python equivalent of python3's zip() because I expected it to look like the above izip recipe (making it incompatible with the PEP behavior). However, I found that the given equivalent code is: def zip(*iterables): # zip('ABCD', 'xy') --> Ax By sentinel = object() iterators = [iter(it) for it in iterables] while iterators: result = [] for it in iterators: elem = next(it, sentinel) if elem is sentinel: return result.append(elem) yield tuple(result) i.e., there is no unprotected next call in this example. What surprised me though is that the protection here is done via the default argument of next() while more typically you'd use a try/except clause. So what's the difference between the two ? Specifically, with a default value given will next just catch StopIteration, which you could do less verbosely yourself and/or is there some speed gain from the fact that the Error has to be propagated up one level less ? Is there a guideline when to use try/except vs. next with a default value ? Thanks, Wolfgang From guido at python.org Fri Nov 21 16:39:40 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 21 Nov 2014 07:39:40 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <546A3DEE.5040206@stoneleaf.us> <20141117235641.GW2748@ando.pearwood.info> <20141120020615.GI2748@ando.pearwood.info> Message-ID: On Fri, Nov 21, 2014 at 2:19 AM, Wolfgang Maier < wolfgang.maier at biologie.uni-freiburg.de> wrote: > On 21.11.2014 00:51, Guido van Rossum wrote: > >> On Thu, Nov 20, 2014 at 2:39 PM, Wolfgang Maier >> > > >> wrote: >> >> [...] >> Hmm, I'm not convinced by these toy examples, but I did inspect some >> of my own code for incompatibility with the proposed change. I found >> that there really is only one recurring pattern I use that I'd have >> to change and that is how I've implemented several file parsers. I >> tend to write them like this: >> >> def parser (file_object): >> while True: >> title_line = next(file_object) # will terminate after the >> last record >> >> try: >> # read and process the rest of the record here >> except StopIteration: >> # this record is incomplete >> raise OSError('Invalid file format') >> yield processed_record >> >> There's probably something important missing from your examples. The >> above while-loop is equivalent to >> >> for title_line in io_object: >> ... >> >> > My reason for not using a for loop here is that I'm trying to read from a > file where several lines form a record, so I'm reading the title line of a > record (and if there is no record in the file any more I want the parser > generator to terminate/return. If a title line is read successfully then > I'm reading the record's body lines inside a try/except, i.e. where it says > "# read and process the rest of the record here" in my shortened code I am > actually calling next several times again to retrieve the body lines (and > while reading these lines an unexpected StopIteration in the IOWrapper is > considered a file format error). > I realize that I could also use a for loop and still call > next(file_object) inside it, but I find this a potentially confusing > pattern that I'm trying to avoid by using the while loop and all explicit > next(). Compare: > > for title_line in file_object: > record_body = next(file_object) > # in reality record_body is generated using several next calls > # depending on the content found in the record body while it's read > yield (title_line, record_body) > > vs > > while True: > title_line = next(file_object) > body = next(file_object) > yield (title_line, body) > > To me, the for loop version suggests to me that the content of file_object > is read in line by line by the loop (even though the name title_line tries > to hint at this being not true). Only when I inspect the loop body I see > that further items are retrieved with next() and, thus, skipped in the for > iteration. The while loop, on the other hand, makes the number of > iterations very clear by showing all of them in the loop body. > > Would you agree that this is justification enough for while instead of for > or is it only me who thinks that a for loop makes the code read awkward ? > Now that you have explained it I see your point. > If you're okay with getting RuntimeError instead of OSError for an >> undesirable StopIteration, you can just drop the except clause altogether. >> > > Right, I could do this if the PEP-described behavior was in effect today. > So shouldn't you be voting *for* the PEP? -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Fri Nov 21 16:53:15 2014 From: barry at python.org (Barry Warsaw) Date: Fri, 21 Nov 2014 10:53:15 -0500 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators References: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com> Message-ID: <20141121105315.2ef5545e@anarchist.wooz.org> On Nov 21, 2014, at 10:55 PM, Nick Coghlan wrote: >I suspect enough evidence of breakage is accumulating to tip the >balance back to "not worth the hassle", but it would also be possible >to just *add* the "from __future__ import generator_stop" feature, and >postpone a decision on making that the only available behaviour. I have no opinion on the actual PEP, but I'm not sure the above is a good resolution. future imports should be for things that have a clear path to default behavior in some future release. I don't think we should incur technical debt to future-ize a feature that won't eventually get adopted. Such a thing will just be another wart that will be difficult to remove for backward compatibility. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From abarnert at yahoo.com Fri Nov 21 17:22:55 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 21 Nov 2014 08:22:55 -0800 Subject: [Python-ideas] Return expressions In-Reply-To: References: <546DE48C.5000805@mrabarnett.plus.com> <546E0DA1.10609@mrabarnett.plus.com> Message-ID: Sent from a random iPhone On Nov 20, 2014, at 8:21, Nick Coghlan wrote: > On 21 November 2014 01:49, MRAB wrote: >> >> >> On 2014-11-20 14:43, Petr Viktorin wrote: >>> >>> The "while" expresses intent, "return" caters to low-level mechanics. >>> The "nested compound statements" explanation is already not that good: >>> why does the value come first, not last? For readability*. The >>> difference between function/nested compound statements syntax and >>> comprehension syntax is already so big, and the low-level "while >>> x:"?"if not x: break" translation is so easy, that between the two the >>> readability of "while" should win. >> >> I can see the problem with 'while': if there are multiple 'for' parts, they are equivalent to nested >> 'for' loops, so you might assume that a 'while' part would also be equivalent to a nested 'while' >> loop, whereas it would, in fact, be controlling the preceding 'for' part. >> >> In other words, it would be: >> >> for x in range(10) while x < 5: >> .... >> >> but could be seen as: >> >> for x in range(10): >> while x < 5: > > > PEP 3142 was the last attempt at asking this question - Guido isn't > keen on the idea, so he rejected it the last time we did a pass > through the PEPs that weren't being actively worked on. The fact few times this has came up on -ideas or -dev, people suggested a variety of alternative syntaxes for this idea, just as is happening this time, and so far, they've always been subsets of the same ideas. There are syntaxes that map nicely to the nested statement translation but don't read well, syntaxes that read well but make no sense as nested statements, syntaxes that sort of meet both those criteria but are so verbose nobody would use them, syntaxes that only make sense if everyone knows that comprehensions (including genexprs) are run inside their own hidden functions, syntaxes that only make sense if you pretend they aren't, syntaxes that require a change to normal for statements that wouldn't ever realistically be used there, and syntaxes that change the basic principle behind how comprehensions work. (And maybe my suggestion to make or stop() work in listcomps the same way fits in here, even though the goal was to unify the semantics of all genexprs and other comprehensions, rather than to provide this functionality.) So far, Guido hasn't liked any of them, and none of them have gotten the kind of widespread buy-in from the community that seem likely to convince him to take another look. So it seems like a waste of time to rehash them every year or so. Maybe what we need to do is update PEP 3142, gathering all of the alternative ideas, and expanding on the rationale, so Guido can reject that, and next time this comes up, everyone will have something to read before having the same arguments about the same proposals? From steve at pearwood.info Fri Nov 21 17:30:32 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 22 Nov 2014 03:30:32 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com> Message-ID: <20141121163032.GN2748@ando.pearwood.info> On Fri, Nov 21, 2014 at 10:50:52PM +1100, Chris Angelico wrote: > On Fri, Nov 21, 2014 at 10:24 PM, Raymond Hettinger > wrote: > > Or would it break the traditional examples of how to write something > > like izip() using a generator? > > > > def izip(iterable1, iterable2): > > it1 = iter(iterable1) > > it2 = iter(iterable2) > > while True: > > v1 = next(it1) > > v2 = next(it2) > > yield v1, v2 > > > > assert list(izip('ab', 'cde')) == [('a', 'c'), ('b', 'd')] > > assert list(izip('abc', 'cd')) == [('a', 'c'), ('b', 'd')] > > Yes, this would be affected. This proposal causes a separation of > generators and iterators, so it's no longer possible to pretend that > they're the same thing. But generators and iterators *are the same thing*. (Generator functions are not iterators, but generators themselves are.) Iterators don't have a specific type, but they obey the iterator protocol: py> def gen(): ... yield 1 ... yield 2 ... py> it = gen() py> iter(it) is it True py> hasattr(it, '__next__') True `it` is an iterator. > > Also, the PEP motivation seemed somewhat weak. Instead of listing > > known bugs or real-world development difficulties, it seems to hinge > > almost entirely some "being surprised" that list comprehensions and > > generator expressions aren't the same in every regard (they aren't). > > The main point is one of exceptions being silently suppressed. > Iterator protocol involves the StopIteration exception; generator > protocol doesn't, yet currently a generator that raises StopIteration > will quietly terminate. It's as if every generator is wrapped inside > "try: ..... except StopIteration: pass". Would you accept any function > being written with that kind of implicit suppression of any other > exception? Yes. That's how the classic pre-iterator iteration protocol works: py> class K: ... def __getitem__(self, i): ... if i == 5: raise IndexError ... return i ... py> x = K() py> list(x) [0, 1, 2, 3, 4] Context managers support suppressing any exception which occurs: If the suite was exited due to an exception, and the return value from the __exit__() method was false, the exception is reraised. If the return value was true, the exception is suppressed, and execution continues with the statement following the with statement. https://docs.python.org/3/reference/compound_stmts.html#the-with-statement So there's two examples, one of the oldest going back to Python 1 days, and one of the newest. There may be others. -- Steven From ethan at stoneleaf.us Fri Nov 21 17:36:04 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 21 Nov 2014 08:36:04 -0800 Subject: [Python-ideas] Return expressions In-Reply-To: References: <546DE48C.5000805@mrabarnett.plus.com> <546E0DA1.10609@mrabarnett.plus.com> Message-ID: <546F69F4.9000301@stoneleaf.us> On 11/21/2014 08:22 AM, Andrew Barnert wrote: > > Maybe what we need to do is update PEP 3142, gathering all of the alternative ideas, > and expanding on the rationale, so Guido can reject that, and next time this comes > up, everyone will have something to read before having the same arguments about the > same proposals? Are we accepting nominations? I have the perfect fellow in mind. :) (And in case Chris decides to nominate me in retaliation, I decline. ;) -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From rosuav at gmail.com Fri Nov 21 17:43:38 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 22 Nov 2014 03:43:38 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: <20141121163032.GN2748@ando.pearwood.info> References: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com> <20141121163032.GN2748@ando.pearwood.info> Message-ID: On Sat, Nov 22, 2014 at 3:30 AM, Steven D'Aprano wrote: > On Fri, Nov 21, 2014 at 10:50:52PM +1100, Chris Angelico wrote: >> Yes, this would be affected. This proposal causes a separation of >> generators and iterators, so it's no longer possible to pretend that >> they're the same thing. > > But generators and iterators *are the same thing*. (Generator functions > are not iterators, but generators themselves are.) Iterators don't have > a specific type, but they obey the iterator protocol: I can write many other factory functions which return iterators. They are not, themselves, iterators, and therefore should not be expected to follow iterator protocol. def gen(): return iter([1,2]) > py> it = gen() > py> iter(it) is it > True > py> hasattr(it, '__next__') > True > > `it` is an iterator. The above function works with those tests, too. Generator functions are functions that return iterators, and the __next__ method of the returned object is what follows iterator protocol. >> The main point is one of exceptions being silently suppressed. >> Iterator protocol involves the StopIteration exception; generator >> protocol doesn't, yet currently a generator that raises StopIteration >> will quietly terminate. It's as if every generator is wrapped inside >> "try: ..... except StopIteration: pass". Would you accept any function >> being written with that kind of implicit suppression of any other >> exception? > > Yes. > > That's how the classic pre-iterator iteration protocol works: > > py> class K: > ... def __getitem__(self, i): > ... if i == 5: raise IndexError > ... return i > ... > py> x = K() > py> list(x) > [0, 1, 2, 3, 4] That's following getitem protocol, and it's part of that protocol for the raising of IndexError to be the way of not returning any value. But what's more surprising is that raising StopIteration will also silently halt iteration, which I think is not good: >>> class K: def __getitem__(self, i): if i == 5: raise StopIteration return i >>> list(K()) [0, 1, 2, 3, 4] > Context managers support suppressing any exception which occurs: > > If the suite was exited due to an exception, and the return > value from the __exit__() method was false, the exception is > reraised. If the return value was true, the exception is > suppressed, and execution continues with the statement > following the with statement. > > https://docs.python.org/3/reference/compound_stmts.html#the-with-statement Context managers get a chance to function like a try/except block. If one silently and unexpectedly suppresses an exception, it's going to be surprising; but more likely, it's as clear and explicit as an actual try/except block. This isn't "as soon as you use a 'with' block, any XyzError will jump to the end of the block and keep going". ChrisA From rosuav at gmail.com Fri Nov 21 17:49:15 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 22 Nov 2014 03:49:15 +1100 Subject: [Python-ideas] Return expressions In-Reply-To: <546F69F4.9000301@stoneleaf.us> References: <546DE48C.5000805@mrabarnett.plus.com> <546E0DA1.10609@mrabarnett.plus.com> <546F69F4.9000301@stoneleaf.us> Message-ID: On Sat, Nov 22, 2014 at 3:36 AM, Ethan Furman wrote: > On 11/21/2014 08:22 AM, Andrew Barnert wrote: >> >> Maybe what we need to do is update PEP 3142, gathering all of the alternative ideas, >> and expanding on the rationale, so Guido can reject that, and next time this comes >> up, everyone will have something to read before having the same arguments about the >> same proposals? > > Are we accepting nominations? I have the perfect fellow in mind. :) > > (And in case Chris decides to nominate me in retaliation, I decline. ;) Heh! Great, I'm going to get a name for writing the PEPs that exist solely to be rejected... I was hoping PEP 479 would be accepted easily, given who was advocating it, but now it's starting to look like I doomed the proposal :) ChrisA From ethan at stoneleaf.us Fri Nov 21 17:49:47 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 21 Nov 2014 08:49:47 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com> References: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com> Message-ID: <546F6D2B.1040007@stoneleaf.us> On 11/21/2014 03:24 AM, Raymond Hettinger wrote: > > Also, the PEP motivation seemed somewhat weak. Instead of listing > known bugs or real-world development difficulties, it seems to hinge > almost entirely some "being surprised" that list comprehensions and > generator expressions aren't the same in every regard (they aren't). I believe the motivation is more along the lines of the difficulty and time wasted in debugging a malfunctioning program when a generator stops early because a StopIteration escaped instead of having some other exception raised. This would be along the same lines as not allowing sum to work with str -- a more valid case, IMO, because the sum restriction is performance based, while this change would actually prevent breakage... or more accurately, put the breakage at the cause and make it much easier to fix. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From ethan at stoneleaf.us Fri Nov 21 17:52:59 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 21 Nov 2014 08:52:59 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: <20141121163032.GN2748@ando.pearwood.info> References: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com> <20141121163032.GN2748@ando.pearwood.info> Message-ID: <546F6DEB.50906@stoneleaf.us> On 11/21/2014 08:30 AM, Steven D'Aprano wrote: > > But generators and iterators *are the same thing*. (Generator functions > are not iterators, but generators themselves are.) Iterators don't have > a specific type, but they obey the iterator protocol: Um, no, they aren't. One cannot 'send()' into any ol' iterator. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From steve at pearwood.info Fri Nov 21 18:16:23 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 22 Nov 2014 04:16:23 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: <546F6DEB.50906@stoneleaf.us> References: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com> <20141121163032.GN2748@ando.pearwood.info> <546F6DEB.50906@stoneleaf.us> Message-ID: <20141121171622.GP2748@ando.pearwood.info> On Fri, Nov 21, 2014 at 08:52:59AM -0800, Ethan Furman wrote: > On 11/21/2014 08:30 AM, Steven D'Aprano wrote: > > > > But generators and iterators *are the same thing*. (Generator functions > > are not iterators, but generators themselves are.) Iterators don't have > > a specific type, but they obey the iterator protocol: > > Um, no, they aren't. One cannot 'send()' into any ol' iterator. "Must not support send()" has never been part of the definition of iterators. The `Iterator` ABC also recognises generators as iterators: py> def gen(): ... yield 1 ... py> from collections import Iterator py> isinstance(gen(), Iterator) True and they are documented as iterators: Python?s generators provide a convenient way to implement the iterator protocol. If a container object?s __iter__() method is implemented as a generator, it will automatically return an iterator object (technically, a generator object) supplying the __iter__() and __next__() methods. https://docs.python.org/3/library/stdtypes.html#generator-types I don't understand where this idea that generators aren't iterators has come from, unless it is confusion between the generator *function* and the generator object itself. -- Steven From abarnert at yahoo.com Fri Nov 21 18:18:29 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 21 Nov 2014 09:18:29 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: <546F6DEB.50906@stoneleaf.us> References: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com> <20141121163032.GN2748@ando.pearwood.info> <546F6DEB.50906@stoneleaf.us> Message-ID: <592A1ABA-41FE-4643-B6B2-CFD733982EB6@yahoo.com> On Nov 21, 2014, at 8:52, Ethan Furman wrote: > On 11/21/2014 08:30 AM, Steven D'Aprano wrote: >> >> But generators and iterators *are the same thing*. (Generator functions >> are not iterators, but generators themselves are.) Iterators don't have >> a specific type, but they obey the iterator protocol: > > Um, no, they aren't. One cannot 'send()' into any ol' iterator. Generators are a subtype of iterators. They support the iterator protocol completely, and in the same way as any other iterator. They also support extensions to that protocol--e.g., send(). But they also have a relationship to a generator function or generator expression, which you could call a "protocol" but if so it's not one expressible at the level of the language. I think that leads to a bit of confusion when speaking loosely. When someone says "the generator protocol vs. the iterator protocol" the "obviously correct" meaning is send and throw, but it's not what people always mean. Then again, the word "generator" itself leads to confusion when speaking loosely. Maybe it would be clearer if "generator" had no meaning; generator functions return generator iterators. But I don't think this confusion has caused serious problems over the decades, so I doubt the more minor confusion at issue here is likely to be serious. From antony.lee at berkeley.edu Fri Nov 21 18:41:31 2014 From: antony.lee at berkeley.edu (Antony Lee) Date: Fri, 21 Nov 2014 09:41:31 -0800 Subject: [Python-ideas] Return expressions In-Reply-To: References: <546DE48C.5000805@mrabarnett.plus.com> <546E0DA1.10609@mrabarnett.plus.com> <546F69F4.9000301@stoneleaf.us> Message-ID: I would like to believe that "break"-as-a-expression solves most of these issues. It is reasonable, though, to drop the "return"-as-an-expression part of the proposal, because you need to know that the comprehension is run in a separate function to understand it (it's a bit ironic that I am dropping the original proposal to defend my own, now...). Consider ((x, y) for x in l1 if f1(x) or break for y in l2 if f2(y) or break) This maps directly to for x in l1: if f1(x) or break: for y in l2: if f2(y) or break: yield x, y which is literally a copy-paste of the second part followed by yielding the first part. I think this reads reasonably well but this is obviously a subjective issue. Antony 2014-11-21 8:49 GMT-08:00 Chris Angelico : > On Sat, Nov 22, 2014 at 3:36 AM, Ethan Furman wrote: > > On 11/21/2014 08:22 AM, Andrew Barnert wrote: > >> > >> Maybe what we need to do is update PEP 3142, gathering all of the > alternative ideas, > >> and expanding on the rationale, so Guido can reject that, and next time > this comes > >> up, everyone will have something to read before having the same > arguments about the > >> same proposals? > > > > Are we accepting nominations? I have the perfect fellow in mind. :) > > > > (And in case Chris decides to nominate me in retaliation, I decline. ;) > > Heh! Great, I'm going to get a name for writing the PEPs that exist > solely to be rejected... I was hoping PEP 479 would be accepted > easily, given who was advocating it, but now it's starting to look > like I doomed the proposal :) > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Fri Nov 21 19:06:46 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 21 Nov 2014 10:06:46 -0800 Subject: [Python-ideas] Return expressions In-Reply-To: References: <546DE48C.5000805@mrabarnett.plus.com> <546E0DA1.10609@mrabarnett.plus.com> <546F69F4.9000301@stoneleaf.us> Message-ID: On Nov 21, 2014, at 9:41, Antony Lee wrote: > I would like to believe that "break"-as-a-expression solves most of these issues. This one has been raised every time in the past. Since we don't have the PEP, I'll try to summarize the problems. But first: > It is reasonable, though, to drop the "return"-as-an-expression part of the proposal, because you need to know that the comprehension is run in a separate function to understand it (it's a bit ironic that I am dropping the original proposal to defend my own, now...). There's a more significant difference between the two proposals. A break expression allows you to break out of a single loop, but gives you no way to break out of the whole thing (as your example shows). A return expression allows you to break out of the whole thing, but gives you no way to break out of a single loop. That's why they're not interchangeable in explicit loop code. (Also, notice that every so often, someone proposed a numbered or labeled break, and the answer is always "Why would you need that? If your code has enough loops that you need to break out of some but not all, refactor those some into a separate function and then use return.") This also raises the question of why one of break/continue/return should be an expression but not the others. If your answer is "because you only need continue when there's code not controlled by the if, which is impossible in a comprehension" then you're pretty much admitting that the flow control expression thing is not really a general purpose thing, but a hack made for comprehensions. > Consider > > ((x, y) for x in l1 if f1(x) or break for y in l2 if f2(y) or break) > > This maps directly to > > for x in l1: > if f1(x) or break: > for y in l2: > if f2(y) or break: > yield x, y When you write it this way, it's pretty clear that you're abusing or as an else, and that it comes in the wrong place, and that you're cramming side effects into an expression with no value. Would you put anything else with side effects here, even a call to a logging function? You're also introducing unnecessary indentation, and making the code more verbose by comparison with the obvious alternative: for x in l1: if not f1(x): break for y in l2: if not f2(y): break yield x, y Yes, that alternative can't be written as a comprehension. But that doesn't mean we should come up with a less obvious, more verbose, and harder to reason about alternative just because it can be written as a comprehension, even though we'd never write it as an explicit loop. If you want to go that route, we might as well just enshrine the "or stop()" hack instead of trying to break it. Also, can you imagine using break as an expression in any other context besides as an or operand in an if statement directly under a for statement? From chris.barker at noaa.gov Fri Nov 21 18:51:41 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 21 Nov 2014 09:51:41 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: <592A1ABA-41FE-4643-B6B2-CFD733982EB6@yahoo.com> References: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com> <20141121163032.GN2748@ando.pearwood.info> <546F6DEB.50906@stoneleaf.us> <592A1ABA-41FE-4643-B6B2-CFD733982EB6@yahoo.com> Message-ID: On Fri, Nov 21, 2014 at 9:18 AM, Andrew Barnert < abarnert at yahoo.com.dmarc.invalid> wrote: > Generators are a subtype of iterators. They support the iterator protocol > completely, and in the same way as any other iterator. They also support > extensions to that protocol--e.g., send(). But they also have a > relationship to a generator function or generator expression, > interesting -- I've always called those "generator comprehensions" -- but anyway, -- do they have a special relationship? I can put any iterable in a generator expression: gen_exp = (i for i in [3,4,5,6]) the result is a generator: In [5]: type(gen_exp) Out[5]: generator so I guess you could call that a "special relationship" -- but it looks to me kind of like an alternate constructor. But in any case, you can use a generator created by a generator expression or a generator function the same way you can use a iterable or an iterator class. Then again, the word "generator" itself leads to confusion when speaking > loosely. Maybe it would be clearer if "generator" had no meaning; generator > functions return generator iterators. not sure how that would help -- a generator is a type, and it is created by either calling a generator function or a generator expression. if there is confusion, it's when folks call a generator function a "generator" Anyway, I just went back and read the PEP, and I'm still confused -- would the PEP make generators behave more like iterator classes, or less like them? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeanpierreda at gmail.com Fri Nov 21 19:25:14 2014 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Fri, 21 Nov 2014 10:25:14 -0800 Subject: [Python-ideas] Return expressions In-Reply-To: References: <546DE48C.5000805@mrabarnett.plus.com> <546E0DA1.10609@mrabarnett.plus.com> <546F69F4.9000301@stoneleaf.us> Message-ID: On Fri, Nov 21, 2014 at 9:41 AM, Antony Lee wrote: > I would like to believe that "break"-as-a-expression solves most of these > issues. It is reasonable, though, to drop the "return"-as-an-expression > part of the proposal, because you need to know that the comprehension is run > in a separate function to understand it (it's a bit ironic that I am > dropping the original proposal to defend my own, now...). > > Consider > > ((x, y) for x in l1 if f1(x) or break for y in l2 if f2(y) or break) > > This maps directly to > > for x in l1: > if f1(x) or break: > for y in l2: > if f2(y) or break: > yield x, y > > which is literally a copy-paste of the second part followed by yielding the > first part. I think this reads reasonably well but this is obviously a > subjective issue. I really hope "or break" doesn't become an idiom if break is turned into an expression. I find [x if x < N else break for x in ...] so much more readable than [x for x in ... if x < N or break] This is very near (but not directly equivalent) to the relatively idiomatic: for x in ...: if x < N: yield x else: break -- Devin From antony.lee at berkeley.edu Fri Nov 21 19:26:06 2014 From: antony.lee at berkeley.edu (Antony Lee) Date: Fri, 21 Nov 2014 10:26:06 -0800 Subject: [Python-ideas] Return expressions In-Reply-To: References: <546DE48C.5000805@mrabarnett.plus.com> <546E0DA1.10609@mrabarnett.plus.com> <546F69F4.9000301@stoneleaf.us> Message-ID: 2014-11-21 10:06 GMT-08:00 Andrew Barnert : > On Nov 21, 2014, at 9:41, Antony Lee wrote: > > > I would like to believe that "break"-as-a-expression solves most of > these issues. > > This one has been raised every time in the past. Since we don't have the > PEP, I'll try to summarize the problems. But first: > > > It is reasonable, though, to drop the "return"-as-an-expression part of > the proposal, because you need to know that the comprehension is run in a > separate function to understand it (it's a bit ironic that I am dropping > the original proposal to defend my own, now...). > > There's a more significant difference between the two proposals. A break > expression allows you to break out of a single loop, but gives you no way > to break out of the whole thing (as your example shows). A return > expression allows you to break out of the whole thing, but gives you no way > to break out of a single loop. That's why they're not interchangeable in > explicit loop code. (Also, notice that every so often, someone proposed a > numbered or labeled break, and the answer is always "Why would you need > that? If your code has enough loops that you need to break out of some but > not all, refactor those some into a separate function and then use return.") > > This also raises the question of why one of break/continue/return should > be an expression but not the others. If your answer is "because you only > need continue when there's code not controlled by the if, which is > impossible in a comprehension" then you're pretty much admitting that the > flow control expression thing is not really a general purpose thing, but a > hack made for comprehensions. > I guess you're going to make me flip-flop and go back to defend having all three of "break", "return" and "raise" as expressions (I can't see a single use of "continue" as an expression but would be happy to be proven wrong). The issue of "return" as an expression is obviously that it exposes the internals of generators (i.e. they are in their own function) but I can live with that. ("raise" as expressions is a separate issue, the main application being to stuff it into lambdas.) > > > Consider > > > > ((x, y) for x in l1 if f1(x) or break for y in l2 if f2(y) or break) > > > > This maps directly to > > > > for x in l1: > > if f1(x) or break: > > for y in l2: > > if f2(y) or break: > > yield x, y > > When you write it this way, it's pretty clear that you're abusing or as an > else, and that it comes in the wrong place, and that you're cramming side > effects into an expression with no value. Would you put anything else with > side effects here, even a call to a logging function? > > You're also introducing unnecessary indentation, and making the code more > verbose by comparison with the obvious alternative: > > for x in l1: > if not f1(x): break > for y in l2: > if not f2(y): break > yield x, y > > Yes, that alternative can't be written as a comprehension. But that > doesn't mean we should come up with a less obvious, more verbose, and > harder to reason about alternative just because it can be written as a > comprehension, even though we'd never write it as an explicit loop. If you > want to go that route, we might as well just enshrine the "or stop()" hack > instead of trying to break it. > The introduction of unnecessary indentation is honestly not there to make the non-generator example look bad, but simply to show how the proposal addresses the issue of translating nested loops. For the non-nested loop case, a simpler and arguably more readable way to write it would be (x if f(x) else break for x in X) Perhaps giving the nested case first was poor PR; yes, the nested case is slightly hackish but I am fairly sure anybody can figure out the meaning of the above. > > Also, can you imagine using break as an expression in any other context > besides as an or operand in an if statement directly under a for statement? > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Nov 21 19:28:58 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 22 Nov 2014 05:28:58 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com> <20141121163032.GN2748@ando.pearwood.info> <546F6DEB.50906@stoneleaf.us> <592A1ABA-41FE-4643-B6B2-CFD733982EB6@yahoo.com> Message-ID: On Sat, Nov 22, 2014 at 4:51 AM, Chris Barker wrote: > Anyway, I just went back and read the PEP, and I'm still confused -- would > the PEP make generators behave more like iterator classes, or less like > them? Neutral. A generator function, an iterator class, etc, etc, etc, exists solely to construct an iterator. That iterator has a __next__ method, which either returns a value or raises StopIteration, or raises some other exception (which bubbles up). There are two easy ways to write iterators. One is to construct a class: class Iter: def __init__(self): self.x = 0 def __iter__(self): return self def __next__(self): if self.x == 3: raise StopIteration self.x += 1 return self.x Another is to write a generator function: def gen(): yield 1 yield 2 yield 3 Both Iter and gen are callables which return iterators. Both of them will produce three integers and then raise StopIteration. Both will, as is good form for iterators, continue to raise StopIteration thereafter. And neither Iter nor gen is, itself, an iterator. One is a class which constructs iterators. The other is a generator function, which also constructs iterators. That's all. In Iter.__next__, I wrote code which chose between "return" and "raise StopIteration" to define its result; in gen(), I wrote code which chose between "yield" and "return" (in this case, the implicit return at the end of the function) to define its result. The only change made by this proposal is that StopIteration becomes, in a generator, like any other unexpected exception. It creates a separation between "iterator protocol" (which is implemented by __next__) and "generator protocol" (which is written in the body of a function with 'yield' in it). ChrisA From ethan at stoneleaf.us Fri Nov 21 18:27:03 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 21 Nov 2014 09:27:03 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: <20141121171622.GP2748@ando.pearwood.info> References: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com> <20141121163032.GN2748@ando.pearwood.info> <546F6DEB.50906@stoneleaf.us> <20141121171622.GP2748@ando.pearwood.info> Message-ID: <546F75E7.9040409@stoneleaf.us> On 11/21/2014 09:16 AM, Steven D'Aprano wrote: > On Fri, Nov 21, 2014 at 08:52:59AM -0800, Ethan Furman wrote: >> On 11/21/2014 08:30 AM, Steven D'Aprano wrote: >>> >>> But generators and iterators *are the same thing*. (Generator functions >>> are not iterators, but generators themselves are.) Iterators don't have >>> a specific type, but they obey the iterator protocol: >> >> Um, no, they aren't. One cannot 'send()' into any ol' iterator. > > "Must not support send()" has never been part of the definition of > iterators. I suspect a language use issue: *are the same thing* implies to me a type check, not an isinstance check. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From abarnert at yahoo.com Fri Nov 21 19:48:05 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 21 Nov 2014 10:48:05 -0800 Subject: [Python-ideas] Return expressions In-Reply-To: <546F69F4.9000301@stoneleaf.us> References: <546DE48C.5000805@mrabarnett.plus.com> <546E0DA1.10609@mrabarnett.plus.com> <546F69F4.9000301@stoneleaf.us> Message-ID: If someone _is_ going to do this, I have one more suggestion that I didn't raise last time around, because I don't want to argue for it, but for completeness: Briefly: add if and while clauses to the for statement; redefine comprehensions as nested for loops. The basic problem is that Python comprehensions map to arbitrarily nested for and if statements, and there's no way to add "while" semantics cleanly with another nested statement. If you look at the languages that have while clauses, this is not how their comprehensions work. Some of them, they have nested loops only, but each loop has optional modifiers that Python's don't: an if or when clause that filters, a while clause that terminates early, maybe unless and until clauses that do the same but with negated conditions (although until has a choice between doing the last iteration or not doing it). This allows for everything you can do in Python except multiple if clauses on the same for clause (which are usually better rewritten with and--or, when that's too unwieldy, you probably shouldn't have been writing it as a one-liner). In others, there's a single loop, but it takes nested iterable expressions, so you only get one set of modifiers for all of the iterables. Others just don't have any syntax for nested iteration at all; you have to turn it upside-down and use the outer loop as the iterable for the inner loop. This would have another minor arguable benefit. Every once in a while I see some code like this: for x in (x for x in xs if f(x)): ... because the writer didn't think long enough to realize he doesn't need the comprehension. This code is obviously a lot better as: for x in xs if f(x): But that example shows exactly why I don't think this is worth arguing for. The same code is even better as: for x in xs: if f(x): And that brings us back to the reason Clojure, Racket, etc. have clauses on their for loops: Because they don't have loop statements, they have a loop function. (Well, usually it's a macro that quotes the arguments so you don't have to lambda all your expressions into functions, and to allow you to avoid at least one excess repetition, but let's ignore that.) The when and while clauses are just keyword arguments to the function. More generally, they encourage or require you to write everything as a complex expression, which is the exact opposite of what Python encourages. Still, just because the idea comes from languages that are very different from Python doesn't necessarily mean it wouldn't fit; after all, that's how we got comprehensions in the first place. Sent from a random iPhone On Nov 21, 2014, at 8:36, Ethan Furman wrote: > On 11/21/2014 08:22 AM, Andrew Barnert wrote: >> >> Maybe what we need to do is update PEP 3142, gathering all of the alternative ideas, >> and expanding on the rationale, so Guido can reject that, and next time this comes >> up, everyone will have something to read before having the same arguments about the >> same proposals? > > Are we accepting nominations? I have the perfect fellow in mind. :) > > (And in case Chris decides to nominate me in retaliation, I decline. ;) > > -- > ~Ethan~ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From chris.barker at noaa.gov Fri Nov 21 20:33:03 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 21 Nov 2014 11:33:03 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators Message-ID: I htink I'm getting closer to clarity here, but: let's say I have this silly helper function (similar to the one in the PEP): ## (sorry -- python2) def helper(x): if x > 2: raise StopIteration else: return x # I call that helper from both an iterator class and a generator function. class Iter(object): def __init__(self): self.x = 0 def __iter__(self): return self def next(self): if self.x == 3: raise StopIteration self.x += 1 return helper(self.x) def gen(): yield helper(1) yield helper(2) yield helper(3) # first just loop through them: for i in Iter(): print i for i in gen(): print i # they act exactly the same # then put them inside generator expression and loop through that: for j in ( i for i in gen() ): print j for j in ( i for i in Iter() ): print j # they still act exactly the same. Would this PEP chance that? would that StopIteration bubble up through the iterator class, but not through the generator function? If so then I'm -1 on the PEP. The only change made by this proposal is that StopIteration becomes, > in a generator, like any other unexpected exception. but from the user's perspective, an iterator and a generator should look the same. And I think from the perspective of the author of a given generator function, they should look as much the same as possible-- i.e. if a StopIteration is raised internally, it will behave the same way as it would in a iterator class. It creates a > separation between "iterator protocol" (which is implemented by > __next__) and "generator protocol" (which is written in the body of a > function with 'yield' in it). > terminology messing me up here -- I understood the "iterator protocol" top mean the __iter__ and __next__ -- and a generator satisfies that protocol. Is there a "generator protocol" that means somethign different? Maybe that's what this whole sub-thread is about: I never thought there was such a thing as a "generator protocol", and I don't see why there should be. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Nov 21 22:55:53 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 22 Nov 2014 08:55:53 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: On Sat, Nov 22, 2014 at 6:33 AM, Chris Barker wrote: > And I think from the perspective of the author of a given generator > function, they should look as much the same as possible-- i.e. if a > StopIteration is raised internally, it will behave the same way as it would > in a iterator class. Ah, but why? When you write a function like this: def gen(): yield 1 yield 2 why should StopIteration mean any more to it than, say, KeyError does? >> It creates a >> separation between "iterator protocol" (which is implemented by >> __next__) and "generator protocol" (which is written in the body of a >> function with 'yield' in it). > > > terminology messing me up here -- I understood the "iterator protocol" top > mean the __iter__ and __next__ -- and a generator satisfies that protocol. > Is there a "generator protocol" that means somethign different? A generator *object* does. A generator *function* doesn't - it follows a protocol that consists of the "yield" and "return" statements. ChrisA From chris.barker at noaa.gov Fri Nov 21 17:53:19 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 21 Nov 2014 08:53:19 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: <20141121163032.GN2748@ando.pearwood.info> References: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com> <20141121163032.GN2748@ando.pearwood.info> Message-ID: As someone who has written maybe one generator expression in production code, I have little opinion on the PEP. But as someone that teaches Python, I have a comment on: On Fri, Nov 21, 2014 at 10:50:52PM +1100, Chris Angelico wrote: > > Yes, this would be affected. This proposal causes a separation of > > generators and iterators, so it's no longer possible to pretend that > > they're the same thing. > As pointed out by Steven, the _are_ the same thing. When I teach interators and generators, I get a bit tangled up explaining what the difference is, and why Python has both. This is what I say: Conceptually ( outside of language constructs): An "generator" is something that, well, generates value on the fly, as requested, until there are no more to generate, and then terminates. A "iterator" on the other had is something that produces the values in a pre-existing sequence of values, until there are no more. IN practice, python uses the exact same protocol (the iterator protocol -- __iter__, __next__) for both, so that you can write, e.g. a for loop, and not have to know whether the underlying object you are looping through is iterating or generating... As you can write a "generator" in the sense above in a class that supports the iterator protocol (and, can, in fact, write an "iterator" with a generator function), then I say that generator functions really are only syntactic sugar -- they are short and sweet and do much of the book keeping for you. But given all that keeping the protocols as similar as possible is a *good* thing, not a bad one -- they should behave as much as possible teh same. If StopIteration bubbles up from inside an iterator, wouldn't that silently terminate as well? Honestly, I'm a bit lost -- but my point is this -- generators and iterators should behave as much the same as possible. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Nov 21 23:09:07 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 22 Nov 2014 09:09:07 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com> <20141121163032.GN2748@ando.pearwood.info> Message-ID: On Sat, Nov 22, 2014 at 3:53 AM, Chris Barker wrote: > As you can write a "generator" in the sense above in a class that supports > the iterator protocol (and, can, in fact, write an "iterator" with a > generator function), then I say that generator functions really are only > syntactic sugar -- they are short and sweet and do much of the book keeping > for you. If you want to consider them that way, then sure - but part of the bookkeeping they do for you is the management of the StopIteration exception. That becomes purely an implementation detail. You don't actually use it when you write a generator function. ChrisA From guido at python.org Fri Nov 21 23:29:27 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 21 Nov 2014 14:29:27 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com> <20141121163032.GN2748@ando.pearwood.info> Message-ID: On Fri, Nov 21, 2014 at 8:53 AM, Chris Barker wrote: > As someone who has written maybe one generator expression in production > code, I have little opinion on the PEP. > > But as someone that teaches Python, I have a comment on: > > On Fri, Nov 21, 2014 at 10:50:52PM +1100, Chris Angelico wrote: >> > Yes, this would be affected. This proposal causes a separation of >> > generators and iterators, so it's no longer possible to pretend that >> > they're the same thing. >> > > As pointed out by Steven, the _are_ the same thing. When I teach > interators and generators, I get a bit tangled up explaining what the > difference is, and why Python has both. This is what I say: > > Conceptually ( outside of language constructs): > > An "generator" is something that, well, generates value on the fly, as > requested, until there are no more to generate, and then terminates. > > A "iterator" on the other had is something that produces the values in a > pre-existing sequence of values, until there are no more. > > IN practice, python uses the exact same protocol (the iterator protocol -- > __iter__, __next__) for both, so that you can write, e.g. a for loop, and > not have to know whether the underlying object you are looping through is > iterating or generating... > > As you can write a "generator" in the sense above in a class that supports > the iterator protocol (and, can, in fact, write an "iterator" with a > generator function), then I say that generator functions really are only > syntactic sugar -- they are short and sweet and do much of the book keeping > for you. > > But given all that keeping the protocols as similar as possible is a > *good* thing, not a bad one -- they should behave as much as possible teh > same. > > If StopIteration bubbles up from inside an iterator, wouldn't that > silently terminate as well? > > Honestly, I'm a bit lost -- but my point is this -- generators and > iterators should behave as much the same as possible. > I'm sorry you see it that way; we must have done a terrible job explaining this in the past. :-( The behavior for the *consumer* of the iteration is unchanged (call next() until it raises StopIteration -- or let a for-loop take care of the details for you). The interface for the *producer* has never been all that similar: In a generator you *yield* subsequent values until you are done; but if you are not using a generator, you must define a __next__() method (next() in Python 2) that *returns* a single value each time, until it's done, and then it has to raise StopIteration. There is no need to raise StopIteration from a generator, you just return when you are done. Insisting that raising StopIteration in a generator makes it more similar to a __next__() method ignores the fact that producing values is done in a completely different ways. So, again, the PEP does not change anything about iterators, and generators will continue to follow the iterator protocol. The change is only for generator authors (and, most importantly, for people using a certain hack in generator expressions). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Sat Nov 22 00:06:10 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 21 Nov 2014 15:06:10 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com> <20141121163032.GN2748@ando.pearwood.info> Message-ID: On Fri, Nov 21, 2014 at 2:29 PM, Guido van Rossum wrote: > Honestly, I'm a bit lost -- but my point is this -- generators and >> iterators should behave as much the same as possible. >> > > I'm sorry you see it that way; we must have done a terrible job explaining > this in the past. :-( > well, others have found examples in old docs that mingle StopIteration and generators...so I guess so, but I'm not sure I'm that misinformed. It still seems to me that there are two ways to write the same thing. The behavior for the *consumer* of the iteration is unchanged > got it -- the issue at hand is what happens to a StopIteration that is raised by something the generator calls. I think the point of this PEP is that the author og a generator function is thinking about using "yield" to provide the next value, and return (explicit or implicit) to stop the generation of objects. That return is raise a StopIteration, but the author isn't thinking about it. So why would they want to think about having to trap StopIteration when calling other functions. While the author of a iterator class is thinking about the __next__ method and raising a StopIteration to terminate. So s/he would naturally think about trapping StopIteration when calling functions? I suppose that makes some sense, but to me it seems like a generator function is a different syntax for creating what is essentially the same thing -- why shouldn't it have the same behavior? and of you are writing a generator, presumably you know how it's going to get use -- i.e. by somethign that expects a StopIteration -- it's not like you're ignorant of the whole idea. Consider this far fetched situation: Either a iterator class or a generator function could take a function object to call to do part of its work. If that function happened to raise a StopIteration -- now the user would have to know which type of object they were workign with, so they would know how to handle the termination of the iter/gener-artion OK -- far more far fetched than the proceeding example of confusion, but the point is this: AFAIU, the current distinction between generators and iterators is how they are written -- i.e. syntax, essentially. But this PEP would change the behavior of generators in some small way, creating a distinction that doesn't currently exist. So, again, the PEP does not change anything about iterators, and generators > will continue to follow the iterator protocol. The change is only for > generator authors > I guess this is where I'm not sure -- it seems to me that the behavior of generators is being change, not the syntax -- so while mostly of concern to generator authors, it is, in fact, a chance in behavior that can be seen by the consumer of (maybe only an oddly designed) generator. In practice, that difference may only matter to folks using that particular hack in generator expression, but it is indeed a change. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Nov 22 00:40:37 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 22 Nov 2014 10:40:37 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com> <20141121163032.GN2748@ando.pearwood.info> Message-ID: On Sat, Nov 22, 2014 at 10:06 AM, Chris Barker wrote: > I think the point of this PEP is that the author og a generator function is > thinking about using "yield" to provide the next value, and return (explicit > or implicit) to stop the generation of objects. That return is raise a > StopIteration, but the author isn't thinking about it. > > So why would they want to think about having to trap StopIteration when > calling other functions. > > While the author of a iterator class is thinking about the __next__ method > and raising a StopIteration to terminate. So s/he would naturally think > about trapping StopIteration when calling functions? > > I suppose that makes some sense, but to me it seems like a generator > function is a different syntax for creating what is essentially the same > thing -- why shouldn't it have the same behavior? Let's suppose you use a Python-like macro language to generate Python code. In the macro language, you can write stuff like this: class X: iterator: return 5 return 6 return 7 iterator_done And it will get compiled into something like this: class X: def __init__(self): self._iter_state = 0 def __iter__(self): return self def __next__(self): self._iter_state += 1 if self._iter_state == 1: return 5 if self._iter_state == 2: return 6 if self._iter_state == 3: return 7 raise StopIteration This is a reasonably plausible macro language, right? It's basically still a class definition, but it lets you leave out a whole bunch of boilerplate. Now, the question is: As you write the simplified version, should you ever need to concern yourself with StopIteration? I posit no, you should not; it's not a part of the macro language at all. Of course, if this *were* how things were done, it would probably be implemented as a very thin wrapper, exposing all its details to your code; but there's no reason that it *needs* to be so thin. The language you're writing in doesn't need to have any concept of a StopIteration exception, because it doesn't need to use an exception to signal "no more results". > and of you are writing a generator, presumably you know how it's going to > get use -- i.e. by somethign that expects a StopIteration -- it's not like > you're ignorant of the whole idea. Not necessarily. Can we get someone here who knows asyncio and coroutines, and can comment on the use of such generators? > Consider this far fetched situation: > > Either a iterator class or a generator function could take a function object > to call to do part of its work. If that function happened to raise a > StopIteration -- now the user would have to know which type of object they > were workign with, so they would know how to handle the termination of the > iter/gener-artion Either a __getattr__ or a __getitem__ could use a helper function to do part of its work, too, but either the helper needs to know which, or it needs some other way of signalling. They're different protocols, so they're handled differently. If Python wanted to conflate all of these, there could be a single "NoReturnValue" exception, used by every function which needs to be able to return absolutely any object and also to be able to signal "I don't have anything to return". But no, Python has separate exceptions for signalling "I don't have any such key", "I don't have any such attribute", and "I don't have any more things to iterate over". Generators don't need any of them, because - like my macro language above - they have two different keywords and two different byte-codes (yield vs return). In many cases, the helper function doesn't actually need the capability to return *absolutely any object*. In that case, the obvious solution would be to have it return None to say "nothing to return", and then the calling function can either translate that into the appropriate exception, or return rather than yielding, as appropriate. That would also make the helper more useful to other stand-alone functions. But even if your helper has to be able to return absolutely anything, you still have a few options: 1) Design the helper as part of __next__, and explicitly catch the exception. def nexthelper(): if condition: return value raise StopIteration def __next__(self): return nexthelper() def gen(): try: yield nexthelper() except StopIteration: pass 2) Write the helper as a generator, and explicitly next() it if you need that: def genhelper(): if condition: yield value def __next__(self): return next(genhelper()) def gen(): yield from genhelper() 3) Return status and value. I don't like this, but it does work. def tuplehelper(): if condition: return True, value return False, None def __next__(self): ok, val = tuplehelper() if ok: return val raise StopIteration def gen(): ok, val = tuplehelper() if ok: yield val All these methods work perfectly, because they have a clear boundary between protocols. If you want to write a __getitem__ that calls on the same helper, you can do that, and have __getitem__ itself raise appropriately if there's nothing to return. > AFAIU, the current distinction between generators and iterators is how they > are written -- i.e. syntax, essentially. But this PEP would change the > behavior of generators in some small way, creating a distinction that > doesn't currently exist. Generators are currently a leaky abstraction for iterator classes. This PEP plugs a leak that's capable of masking bugs. > I guess this is where I'm not sure -- it seems to me that the behavior of > generators is being change, not the syntax -- so while mostly of concern to > generator authors, it is, in fact, a chance in behavior that can be seen by > the consumer of (maybe only an oddly designed) generator. In practice, that > difference may only matter to folks using that particular hack in generator > expression, but it is indeed a change. The only way a consumer will see a change of behaviour is if the generator author used this specific hack (in which case, instead of the generator quietly terminating, a RuntimeError will bubble up - hopefully all the way up until a human sees it). In terms of this PEP, that's a bug in the generator. Bug-free generators will not appear any different to the consumer. ChrisA From antony.lee at berkeley.edu Sat Nov 22 00:55:30 2014 From: antony.lee at berkeley.edu (Antony Lee) Date: Fri, 21 Nov 2014 15:55:30 -0800 Subject: [Python-ideas] Counter comparison Message-ID: As Counter objects effectively behave like multi-sets, it seems reasonable to overload <, <=, >, >= as for set objects to check whether a Counter is a sub/super-set of another Counter: c < d <===> all(c[k] < d[k] for k in c) Thoughts? Antony -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Sat Nov 22 01:27:59 2014 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 22 Nov 2014 00:27:59 +0000 Subject: [Python-ideas] Counter comparison In-Reply-To: References: Message-ID: <546FD88F.6000709@mrabarnett.plus.com> On 2014-11-21 23:55, Antony Lee wrote: > As Counter objects effectively behave like multi-sets, it seems > reasonable to overload <, <=, >, >= as for set objects to check whether > a Counter is a sub/super-set of another Counter: > > c < d <===> all(c[k] < d[k] for k in c) > > Thoughts? > More correctly, it would be: all(c[k] < d[k] for k in set(c) | set(d)) From python at 2sn.net Sat Nov 22 01:56:50 2014 From: python at 2sn.net (Alexander Heger) Date: Sat, 22 Nov 2014 11:56:50 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com> <20141121163032.GN2748@ando.pearwood.info> Message-ID: > got it -- the issue at hand is what happens to a StopIteration that is > raised by something the generator calls. > > I think the point of this PEP is that the author og a generator function is > thinking about using "yield" to provide the next value, and return (explicit > or implicit) to stop the generation of objects. That return is raise a > StopIteration, but the author isn't thinking about it. > > So why would they want to think about having to trap StopIteration when > calling other functions. > > While the author of a iterator class is thinking about the __next__ method > and raising a StopIteration to terminate. So s/he would naturally think > about trapping StopIteration when calling functions? > > I suppose that makes some sense, but to me it seems like a generator > function is a different syntax for creating what is essentially the same > thing -- why shouldn't it have the same behavior? > > and of you are writing a generator, presumably you know how it's going to > get use -- i.e. by somethign that expects a StopIteration -- it's not like > you're ignorant of the whole idea. > > Consider this far fetched situation: > > Either a iterator class or a generator function could take a function object > to call to do part of its work. If that function happened to raise a > StopIteration -- now the user would have to know which type of object they > were workign with, so they would know how to handle the termination of the > iter/gener-artion I think this is a good point. Maybe a way to obtain equivalency to the generator functions in this case is to "break" this example for the iterator object as well, in that StopIteration has to be raised in the frame of the generator object; if it raised in a different context, e.g., a function called by __next__, that StopIteration should also be converted to a RuntimeError similar to what is proposed in the PEP for the generator functions. Maybe this is not what Chris intends to happen, but it would make things consistent. -Alexander From python at mrabarnett.plus.com Sat Nov 22 04:26:50 2014 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 22 Nov 2014 03:26:50 +0000 Subject: [Python-ideas] Counter comparison In-Reply-To: References: <546FD88F.6000709@mrabarnett.plus.com> Message-ID: <5470027A.7010504@mrabarnett.plus.com> On 2014-11-22 01:36, Antony Lee wrote: > I guess it depends on whether you think {"a": 1} < {"a": 2, "b": -1} > or not. I actually thought about it and initially decided that the > inequality should hold, but given that most_common works fine with > negative counts, perhaps it shouldn't, indeed. > The other point is that if the 'b' entry was in the first, then it would be checked, and the negative count would matter, but as it wasn't, it won't, and it won't. > 2014-11-21 16:27 GMT-08:00 MRAB >: > > On 2014-11-21 23:55, Antony Lee wrote: > > As Counter objects effectively behave like multi-sets, it seems > reasonable to overload <, <=, >, >= as for set objects to > check whether > a Counter is a sub/super-set of another Counter: > > c < d <===> all(c[k] < d[k] for k in c) > > Thoughts? > > More correctly, it would be: > > all(c[k] < d[k] for k in set(c) | set(d)) > From steve at pearwood.info Sat Nov 22 11:48:27 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 22 Nov 2014 21:48:27 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com> <20141121163032.GN2748@ando.pearwood.info> Message-ID: <20141122104827.GQ2748@ando.pearwood.info> On Sat, Nov 22, 2014 at 10:40:37AM +1100, Chris Angelico wrote: > Let's suppose you use a Python-like macro language to generate Python > code. In the macro language, you can write stuff like this: Is there really any point in hypothesing imaginary macro languages when we have a concrete and existing language (Python itself) to look at? [snip made-up example] > This is a reasonably plausible macro language, right? It's basically > still a class definition, but it lets you leave out a whole bunch of > boilerplate. Now, the question is: As you write the simplified > version, should you ever need to concern yourself with StopIteration? Sure, why not? It is part of the concrete protocol: iterators raise StopIteration to halt. That's not a secret, and it is not an implementation detail, it is a concrete, public part of the API. > I posit no, you should not; it's not a part of the macro language at > all. This is why talking about imaginary macro languages is pointless. You say it is not part of the macro language. I say it is. Since the language doesn't actually exist, who is to say which is right? In real Python code, "raise StopIteration" does exist, and does work in generators. Sometimes the fact that it works is a nuisance, when you have an unexpected StopIteration. Sometimes the fact that it works is exactly what you want, when you have an expected StopIteration. You seem to think that allowing a generator function to delegate the decision to halt to a helper function is a Bad Thing. I say it is a Good Thing, even if it occasionally makes buggy code a bit harder to debug. [...] > > and of you are writing a generator, presumably you know how it's going to > > get use -- i.e. by somethign that expects a StopIteration -- it's not like > > you're ignorant of the whole idea. > > Not necessarily. Can we get someone here who knows asyncio and > coroutines, and can comment on the use of such generators? What about them? I don't understand your question. -- Steven From rosuav at gmail.com Sat Nov 22 11:57:26 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 22 Nov 2014 21:57:26 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: <20141122104827.GQ2748@ando.pearwood.info> References: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com> <20141121163032.GN2748@ando.pearwood.info> <20141122104827.GQ2748@ando.pearwood.info> Message-ID: On Sat, Nov 22, 2014 at 9:48 PM, Steven D'Aprano wrote: > On Sat, Nov 22, 2014 at 10:40:37AM +1100, Chris Angelico wrote: > >> Let's suppose you use a Python-like macro language to generate Python >> code. In the macro language, you can write stuff like this: > > Is there really any point in hypothesing imaginary macro languages when > we have a concrete and existing language (Python itself) to look at? > > [snip made-up example] > >> This is a reasonably plausible macro language, right? It's basically >> still a class definition, but it lets you leave out a whole bunch of >> boilerplate. Now, the question is: As you write the simplified >> version, should you ever need to concern yourself with StopIteration? > > Sure, why not? It is part of the concrete protocol: iterators raise > StopIteration to halt. That's not a secret, and it is not an > implementation detail, it is a concrete, public part of the API. > > >> I posit no, you should not; it's not a part of the macro language at >> all. > > This is why talking about imaginary macro languages is pointless. You > say it is not part of the macro language. I say it is. Since the > language doesn't actually exist, who is to say which is right? A generator function is exactly the same thing: it's a way to create an iterator, but it's not a class with a __next__ function. I could write an iterator-creation function in many ways, none of which involve StopIteration: def gen(): if condition: raise StopIteration # Wrong return iter([1,2,3]) >> > and of you are writing a generator, presumably you know how it's going to >> > get use -- i.e. by somethign that expects a StopIteration -- it's not like >> > you're ignorant of the whole idea. >> >> Not necessarily. Can we get someone here who knows asyncio and >> coroutines, and can comment on the use of such generators? > > What about them? I don't understand your question. If you have a lengthy nested chain of coroutines, and one of them unexpectedly raises StopIteration, is it right for something to quietly terminate, or should the exception bubble up and be printed to console? ChrisA From raymond.hettinger at gmail.com Sat Nov 22 16:47:51 2014 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sat, 22 Nov 2014 07:47:51 -0800 Subject: [Python-ideas] Counter comparison In-Reply-To: References: Message-ID: <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com> > On Nov 21, 2014, at 3:55 PM, Antony Lee wrote: > > As Counter objects effectively behave like multi-sets, it seems reasonable to overload <, <=, >, >= as for set objects to check whether a Counter is a sub/super-set of another Counter: > > c < d <===> all(c[k] < d[k] for k in c) > > Thoughts This is something that could be done, but I think the demonstrated need is nearly zero. It doesn't seem to arise in practice. Conceptually, a counter is just a dictionary that returns zero instead of raising a KeyError. So, while they can be used as multisets, they have other uses as well (for example, allowing negative counts or fractional counts). Another thought, is that overloading comparison operators risks isn't always a good thing. Even for regular sets, people sometimes get surprised that the sort order isn't deterministic (because sets have a partial ordering instead of a total ordering). The idea of adding counter comparisons was considered from the outset and was intentionally not included. If compelling use cases were arising in practice, we could add comparison support, but until then it is more prudent to be conservative with the API. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sat Nov 22 17:06:19 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 22 Nov 2014 17:06:19 +0100 Subject: [Python-ideas] Counter comparison References: <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com> Message-ID: <20141122170619.07722252@fsol> On Sat, 22 Nov 2014 07:47:51 -0800 Raymond Hettinger wrote: > > > On Nov 21, 2014, at 3:55 PM, Antony Lee wrote: > > > > As Counter objects effectively behave like multi-sets, it seems reasonable to overload <, <=, >, >= as for set objects to check whether a Counter is a sub/super-set of another Counter: > > > > c < d <===> all(c[k] < d[k] for k in c) > > > > Thoughts > > This is something that could be done, but I think the demonstrated need is nearly zero. It doesn't seem to arise in practice. You say this while you closed an issue where the poster asked for this very feature, and clearly stated what his needs were: http://bugs.python.org/issue22515 > Conceptually, a counter is just a dictionary that returns zero instead of raising a KeyError. Come on. If that were the case, it wouldn't expose other operators such as addition, subtraction, intersection and union. https://docs.python.org/3.3/library/collections.html#collections.Counter > So, while they can be used as multisets, they have other uses as well > (for example, allowing negative counts or fractional counts). The demonstrated need for those "other uses" is nearly zero. It doesn't seem to arise in practice. Your insistance on defending irrational API decisions is staggering. Regards Antoine. From jeanpierreda at gmail.com Sat Nov 22 17:43:42 2014 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Sat, 22 Nov 2014 08:43:42 -0800 Subject: [Python-ideas] Counter comparison In-Reply-To: <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com> References: <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com> Message-ID: On Sat, Nov 22, 2014 at 7:47 AM, Raymond Hettinger wrote: > Conceptually, a counter is just a dictionary that returns zero instead of > raising a KeyError. So, while they can be used as multisets, they have > other uses as well (for example, allowing negative counts or fractional > counts). This does little to nothing to explain most of the behavior of counters, like that of &. They are most naturally thought of as multisets. > Another thought, is that overloading comparison operators risks isn't always > a good thing. Even for regular sets, people sometimes get surprised that > the sort order isn't deterministic (because sets have a partial ordering > instead of a total ordering). Sure. OTOH it is possible to still be conservative while adding this: these risks do not exist for the set comparison operators: isdisjoint, issubset, issuperset. -- Devin From storchaka at gmail.com Sat Nov 22 19:03:02 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 22 Nov 2014 20:03:02 +0200 Subject: [Python-ideas] Counter comparison In-Reply-To: References: Message-ID: On 22.11.14 01:55, Antony Lee wrote: > As Counter objects effectively behave like multi-sets, it seems > reasonable to overload <, <=, >, >= as for set objects to check whether > a Counter is a sub/super-set of another Counter: > > c < d <===> all(c[k] < d[k] for k in c) > > Thoughts? http://bugs.python.org/issue22515 From raymond.hettinger at gmail.com Sun Nov 23 03:51:30 2014 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sat, 22 Nov 2014 18:51:30 -0800 Subject: [Python-ideas] Counter comparison In-Reply-To: <20141122170619.07722252@fsol> References: <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com> <20141122170619.07722252@fsol> Message-ID: > On Nov 22, 2014, at 8:06 AM, Antoine Pitrou wrote: > > Your insistance on defending irrational API decisions is staggering. Gee, that's a little harsh. I'm trying to exercise good judgment over API design decisions. You may not agree, but I should at least share my thought process: * Even for regular sets, overriding the comparison operators was a bit dodgy leading to some oddities -- one because it is partial ordering that confuses sort() and another because of people's propensity to write "set_a < set_b" when they really meant "set_a <= set_b". * Also, I have reservations about having added any multi-set operations to counters in the first place. That seems to be the only the part of counter API that has agitated or confused anyone. The problems arise the counter is an extension of the dict API rather than being a fully encapsulated bag or multiset that can prevent counts from being zero, negative, or non-integral. In other words, the multi-set operations may have added some value but they weren't a perfect fit. * With those two thoughts in mind, I had some reluctance to add anything "doubly dodgy" by adding multi-set comparisons. * Next, there is some question of what the comparison operations should do in the presence of zeros or negative counts. Inspired by the design principles in the decimal module (all numbers are exact and rounding is only applied after operations rather than before), the current multi-set operations eliminate non-negative results after the operation rather than before. That said, I don't know whether it makes sense multi-set comparison to follow that model (for consistency) or to clean-up the inputs prior to the operation. For example, should Counter(a=-1, b=0, c=1) be considered a proper subset of Counter(c=2, d=3, e=0, f=-1)? There are reasons to say yes and reasons you could say no. To inform the best possible decision, it would be great to have real use cases to guide the design process. * Which gets us to use cases. When Ram suggested multiset operations in a feature request, all he said was "I suggest implementing `Counter.__lt__` which will be a partial order, similarly to `set.__lt__`.". There was no mention of use case until later where all he said was "I needed it for an internal calculation in a combinatorics package I'm writing. " and he did not elaborate further. As the discussion evolved, no further use case detail emerged. However, later in the discussion it because clear that several of the posters either thought the idea didn't make sense or were confused about what the it should do. Near the end of the discussion, the OP said he had given up on the idea because of the issues regarding the "hybrid mapping/multiset" design. After the -1s, examples of people being confused, and the OP's withdrawal, I stepped in and closed the feature request. * This current thread did not begin with a use case either. It was more along the lines of "in for a penny, in for a pound". Since some multiset operations were implemented, maybe you should do them all. For the reasons listed above, I prefer to be conservative and not expand the API further. For the most part, people seem to use counters for counting. They are good at that task and popular. I don't feel a strong need to expand the API into dodgy territory unless really good use cases arise to guide the design and make it feel like the added benefits were worth moving further on to shaky ground. Also, I've taken some comfort in knowing that since a counter is a dictionary subclass, it is trivially easy to extend for users. If the user can already meet any perceived need by writing "all(c[k] < d[k] for k in c)", then I'm not sure we can make much of a value add. You may not agree with the outcome of my thought process, but it is unfair to call it irrational. not-everything-that-can-be-done-should-be-done-ly yours, Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Nov 23 06:33:05 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 23 Nov 2014 15:33:05 +1000 Subject: [Python-ideas] Counter comparison In-Reply-To: References: <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com> <20141122170619.07722252@fsol> Message-ID: On 23 November 2014 at 12:51, Raymond Hettinger wrote: [snip] > For the reasons listed above, I prefer to be conservative and not expand the > API further. For the most part, people seem to use counters for counting. > They are good at that task and popular. I don't feel a strong need to > expand the API into dodgy territory unless really good use cases arise to > guide the design and make it feel like the added benefits were worth moving > further on to shaky ground. > > Also, I've taken some comfort in knowing that since a counter is a > dictionary subclass, it is trivially easy to extend for users. If the user > can already meet any perceived need by writing "all(c[k] < d[k] for k in > c)", then I'm not sure we can make much of a value add. > > You may not agree with the outcome of my thought process, but it is unfair > to call it irrational. Thanks for the detailed explanation. > not-everything-that-can-be-done-should-be-done-ly yours, This is a key point. Providing APIs that have odd edge case behaviour can be worse than not providing a particular feature at all. When the feature doesn't exist, folks are more inclined to just write their own custom class or function that does exactly what they need. It's generally feasible to keep those quite concise, since they don't need to handle all possible variations that may arise. With the existing Counter-as-multiset features already offering some potential for confusion, and the potential for further confusing interactions between additional multiset features and Counter's support for non-integer values, zero values (distinct from keys being missing entirely) and negative values, there may be scope for a separate true multiset API that draws more heavily on the set API design than the dict API design. A dedicated multiset API would potentially be able to avoid the confusing aspects of Counters-as-multisets by only allowing non-negative integer values. Is there sufficient value in such an API to justify adding it? Or would it just create confusion as folks tried to decide between using Counter or using the new multiset/bag container for their algorithm? That's an open question, but at the very least, it's worth considering as an alternative to further elaborating on an already confusing aspect of the collections.Counter design. There's at least one such example of a bag API already available on PyPI: https://pypi.python.org/pypi/data-structures/0.1.4#bag (you need "-Counter" in a Google search to find that, as most current hits describe the use of Counter as a multiset, rather than using a dedicated container type) Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ethan at stoneleaf.us Sun Nov 23 16:40:46 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 23 Nov 2014 07:40:46 -0800 Subject: [Python-ideas] Counter comparison In-Reply-To: References: <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com> <20141122170619.07722252@fsol> Message-ID: <5471FFFE.4080309@stoneleaf.us> On 11/22/2014 09:33 PM, Nick Coghlan wrote: > > With the existing Counter-as-multiset features already offering some > potential for confusion, and the potential for further confusing > interactions between additional multiset features and Counter's > support for non-integer values, zero values (distinct from keys being > missing entirely) and negative values, there may be scope for a > separate true multiset API that draws more heavily on the set API > design than the dict API design. > > A dedicated multiset API would potentially be able to avoid the > confusing aspects of Counters-as-multisets by only allowing > non-negative integer values. Is there sufficient value in such an API > to justify adding it? Or would it just create confusion as folks tried > to decide between using Counter or using the new multiset/bag > container for their algorithm? > > That's an open question, but at the very least, it's worth considering > as an alternative to further elaborating on an already confusing > aspect of the collections.Counter design. There's at least one such > example of a bag API already available on PyPI: > https://pypi.python.org/pypi/data-structures/0.1.4#bag (you need > "-Counter" in a Google search to find that, as most current hits > describe the use of Counter as a multiset, rather than using a > dedicated container type) I have also wondered about the feasibility of separating out the multiset features into a distinct type. Seems like that would avoid a bunch of confusion. -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From antony.lee at berkeley.edu Sun Nov 23 20:30:25 2014 From: antony.lee at berkeley.edu (Antony Lee) Date: Sun, 23 Nov 2014 11:30:25 -0800 Subject: [Python-ideas] Counter comparison In-Reply-To: <5471FFFE.4080309@stoneleaf.us> References: <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com> <20141122170619.07722252@fsol> <5471FFFE.4080309@stoneleaf.us> Message-ID: My goal was very simply to check whether it was possible to remove a multi-set of elements from another, without any counts going below 0 (as should be the case for "natural" counters). Antony 2014-11-23 7:40 GMT-08:00 Ethan Furman : > On 11/22/2014 09:33 PM, Nick Coghlan wrote: > > > > With the existing Counter-as-multiset features already offering some > > potential for confusion, and the potential for further confusing > > interactions between additional multiset features and Counter's > > support for non-integer values, zero values (distinct from keys being > > missing entirely) and negative values, there may be scope for a > > separate true multiset API that draws more heavily on the set API > > design than the dict API design. > > > > A dedicated multiset API would potentially be able to avoid the > > confusing aspects of Counters-as-multisets by only allowing > > non-negative integer values. Is there sufficient value in such an API > > to justify adding it? Or would it just create confusion as folks tried > > to decide between using Counter or using the new multiset/bag > > container for their algorithm? > > > > That's an open question, but at the very least, it's worth considering > > as an alternative to further elaborating on an already confusing > > aspect of the collections.Counter design. There's at least one such > > example of a bag API already available on PyPI: > > https://pypi.python.org/pypi/data-structures/0.1.4#bag (you need > > "-Counter" in a Google search to find that, as most current hits > > describe the use of Counter as a multiset, rather than using a > > dedicated container type) > > I have also wondered about the feasibility of separating out the multiset > features into a distinct type. Seems like > that would avoid a bunch of confusion. > > -- > ~Ethan~ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Nov 24 01:04:38 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 24 Nov 2014 11:04:38 +1100 Subject: [Python-ideas] Counter comparison In-Reply-To: References: <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com> <20141122170619.07722252@fsol> <5471FFFE.4080309@stoneleaf.us> Message-ID: <20141124000438.GW2748@ando.pearwood.info> On Sun, Nov 23, 2014 at 11:30:25AM -0800, Antony Lee wrote: > My goal was very simply to check whether it was possible to remove a > multi-set of elements from another, without any counts going below 0 (as > should be the case for "natural" counters). Do you mean this? py> from collections import Counter py> c1 = Counter({'a': 5, 'b': 2}) py> c2 = Counter({'a': 1, 'b': 4}) py> c1 - c2 Counter({'a': 4}) -- Steven From ethan at stoneleaf.us Mon Nov 24 02:03:27 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 23 Nov 2014 17:03:27 -0800 Subject: [Python-ideas] Counter comparison In-Reply-To: <20141124000438.GW2748@ando.pearwood.info> References: <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com> <20141122170619.07722252@fsol> <5471FFFE.4080309@stoneleaf.us> <20141124000438.GW2748@ando.pearwood.info> Message-ID: <547283DF.8030406@stoneleaf.us> On 11/23/2014 04:04 PM, Steven D'Aprano wrote: > On Sun, Nov 23, 2014 at 11:30:25AM -0800, Antony Lee wrote: >> >> My goal was very simply to check whether it was possible to remove a >> multi-set of elements from another, without any counts going below 0 (as >> should be the case for "natural" counters). > > Do you mean this? > > py> from collections import Counter > py> c1 = Counter({'a': 5, 'b': 2}) > py> c2 = Counter({'a': 1, 'b': 4}) > py> c1 - c2 > Counter({'a': 4}) Um, how does that show that 'b' went below zero? -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From steve at pearwood.info Mon Nov 24 02:20:09 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 24 Nov 2014 12:20:09 +1100 Subject: [Python-ideas] Counter comparison In-Reply-To: <547283DF.8030406@stoneleaf.us> References: <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com> <20141122170619.07722252@fsol> <5471FFFE.4080309@stoneleaf.us> <20141124000438.GW2748@ando.pearwood.info> <547283DF.8030406@stoneleaf.us> Message-ID: <20141124012009.GZ2748@ando.pearwood.info> On Sun, Nov 23, 2014 at 05:03:27PM -0800, Ethan Furman wrote: > On 11/23/2014 04:04 PM, Steven D'Aprano wrote: > > On Sun, Nov 23, 2014 at 11:30:25AM -0800, Antony Lee wrote: > >> > >> My goal was very simply to check whether it was possible to remove a > >> multi-set of elements from another, without any counts going below 0 (as > >> should be the case for "natural" counters). > > > > Do you mean this? > > > > py> from collections import Counter > > py> c1 = Counter({'a': 5, 'b': 2}) > > py> c2 = Counter({'a': 1, 'b': 4}) > > py> c1 - c2 > > Counter({'a': 4}) > > Um, how does that show that 'b' went below zero? It doesn't, which is the point. Antony asked for removal WITHOUT counts going below zero. -- Steve From ethan at stoneleaf.us Mon Nov 24 02:30:45 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 23 Nov 2014 17:30:45 -0800 Subject: [Python-ideas] Counter comparison In-Reply-To: <20141124012009.GZ2748@ando.pearwood.info> References: <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com> <20141122170619.07722252@fsol> <5471FFFE.4080309@stoneleaf.us> <20141124000438.GW2748@ando.pearwood.info> <547283DF.8030406@stoneleaf.us> <20141124012009.GZ2748@ando.pearwood.info> Message-ID: <54728A45.6030801@stoneleaf.us> On 11/23/2014 05:20 PM, Steven D'Aprano wrote: > On Sun, Nov 23, 2014 at 05:03:27PM -0800, Ethan Furman wrote: >> On 11/23/2014 04:04 PM, Steven D'Aprano wrote: >>> On Sun, Nov 23, 2014 at 11:30:25AM -0800, Antony Lee wrote: >>>> >>>> My goal was very simply to check whether it was possible to remove a >>>> multi-set of elements from another, without any counts going below 0 (as >>>> should be the case for "natural" counters). >>> >>> Do you mean this? >>> >>> py> from collections import Counter >>> py> c1 = Counter({'a': 5, 'b': 2}) >>> py> c2 = Counter({'a': 1, 'b': 4}) >>> py> c1 - c2 >>> Counter({'a': 4}) >> >> Um, how does that show that 'b' went below zero? > > It doesn't, which is the point. Antony asked for removal WITHOUT counts > going below zero. Ah. I read that as "being able to tell if any count would go below zero" -- as in, counter1 is my inventory, counter2 is an order, can I fill the order? -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From antony.lee at berkeley.edu Mon Nov 24 06:37:20 2014 From: antony.lee at berkeley.edu (Antony Lee) Date: Sun, 23 Nov 2014 21:37:20 -0800 Subject: [Python-ideas] Counter comparison In-Reply-To: <54728A45.6030801@stoneleaf.us> References: <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com> <20141122170619.07722252@fsol> <5471FFFE.4080309@stoneleaf.us> <20141124000438.GW2748@ando.pearwood.info> <547283DF.8030406@stoneleaf.us> <20141124012009.GZ2748@ando.pearwood.info> <54728A45.6030801@stoneleaf.us> Message-ID: I guess my post was unclear but my intent was indeed Ethan's (can I satisfy an order from my inventory?), not Steven's. 2014-11-23 17:30 GMT-08:00 Ethan Furman : > On 11/23/2014 05:20 PM, Steven D'Aprano wrote: > > On Sun, Nov 23, 2014 at 05:03:27PM -0800, Ethan Furman wrote: > >> On 11/23/2014 04:04 PM, Steven D'Aprano wrote: > >>> On Sun, Nov 23, 2014 at 11:30:25AM -0800, Antony Lee wrote: > >>>> > >>>> My goal was very simply to check whether it was possible to remove a > >>>> multi-set of elements from another, without any counts going below 0 > (as > >>>> should be the case for "natural" counters). > >>> > >>> Do you mean this? > >>> > >>> py> from collections import Counter > >>> py> c1 = Counter({'a': 5, 'b': 2}) > >>> py> c2 = Counter({'a': 1, 'b': 4}) > >>> py> c1 - c2 > >>> Counter({'a': 4}) > >> > >> Um, how does that show that 'b' went below zero? > > > > It doesn't, which is the point. Antony asked for removal WITHOUT counts > > going below zero. > > Ah. I read that as "being able to tell if any count would go below zero" > -- as in, counter1 is my inventory, counter2 > is an order, can I fill the order? > > -- > ~Ethan~ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.us Mon Nov 24 07:09:51 2014 From: random832 at fastmail.us (random832 at fastmail.us) Date: Mon, 24 Nov 2014 01:09:51 -0500 Subject: [Python-ideas] Counter comparison In-Reply-To: <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com> References: <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com> Message-ID: <1416809391.2601292.194579353.687F5B82@webmail.messagingengine.com> On Sat, Nov 22, 2014, at 10:47, Raymond Hettinger wrote: > Another thought, is that overloading comparison operators risks isn't > always a good thing. Even for regular sets, people sometimes get > surprised that the sort order isn't deterministic (because sets have a > partial ordering instead of a total ordering). Thank you, incidentally, for an example better than floats (in presence of nan) of a type that will break sorting algorithms. From guido at python.org Mon Nov 24 16:43:41 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 24 Nov 2014 07:43:41 -0800 Subject: [Python-ideas] Counter comparison In-Reply-To: <1416809391.2601292.194579353.687F5B82@webmail.messagingengine.com> References: <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com> <1416809391.2601292.194579353.687F5B82@webmail.messagingengine.com> Message-ID: On Sun, Nov 23, 2014 at 10:09 PM, wrote: > On Sat, Nov 22, 2014, at 10:47, Raymond Hettinger wrote: > > Another thought, is that overloading comparison operators risks isn't > > always a good thing. Even for regular sets, people sometimes get > > surprised that the sort order isn't deterministic (because sets have a > > partial ordering instead of a total ordering). > > Thank you, incidentally, for an example better than floats (in presence > of nan) of a type that will break sorting algorithms. > But does it matter? If you've got a list of sets, and you want it sorted, you have to specify how you want it sorted anyways. You can complain that < is only a partial order, but that is the mathematical root of the situation, not a problem caused by an poorly chosen language default. (Leaving < undefined and sorting by object address, now *that* would be an example of the latter. :-) Math is often surprising. This how you learn. (You may also learn more about sorting algorithms this way.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Nov 24 17:38:49 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 24 Nov 2014 08:38:49 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com> <20141121163032.GN2748@ando.pearwood.info> Message-ID: On Fri, Nov 21, 2014 at 4:56 PM, Alexander Heger wrote: > Maybe a way to obtain equivalency to the generator functions in this > case is to "break" this example for the iterator object as well, in > that StopIteration has to be raised in the frame of the generator > object; if it raised in a different context, e.g., a function called > by __next__, that StopIteration should also be converted to a > RuntimeError similar to what is proposed in the PEP for the generator > functions. Maybe this is not what Chris intends to happen, but it > would make things consistent. > I"mn not sure which Chris you are refering to, but for my part, yes and no: Yes, that would keep iterator classes and generator functions consistent, which would be a good thing. No: I don't think we should do that -- StopIteration is part of the iterator protocol -- generators are another way to write something that complies with the iterator protocol -- generators should handle StopIteration the same way that iterator classes do. Yes, there are some cases that can be confusing and hard to debug -- but them's the breaks. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Nov 24 17:41:46 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 24 Nov 2014 08:41:46 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com> <20141121163032.GN2748@ando.pearwood.info> <20141122104827.GQ2748@ando.pearwood.info> Message-ID: On Sat, Nov 22, 2014 at 2:57 AM, Chris Angelico wrote: > If you have a lengthy nested chain of coroutines, and one of them > unexpectedly raises StopIteration, is it right for something to > quietly terminate, or should the exception bubble up and be printed to > console? Couldn't you have a nested pile of iterator classes as well that would exhibit the exact same behavior? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Nov 24 18:06:28 2014 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 25 Nov 2014 04:06:28 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com> <20141121163032.GN2748@ando.pearwood.info> Message-ID: On Tue, Nov 25, 2014 at 3:38 AM, Chris Barker wrote: > On Fri, Nov 21, 2014 at 4:56 PM, Alexander Heger wrote: >> >> Maybe a way to obtain equivalency to the generator functions in this >> case is to "break" this example for the iterator object as well, in >> that StopIteration has to be raised in the frame of the generator >> object; if it raised in a different context, e.g., a function called >> by __next__, that StopIteration should also be converted to a >> RuntimeError similar to what is proposed in the PEP for the generator >> functions. Maybe this is not what Chris intends to happen, but it >> would make things consistent. > > > I"mn not sure which Chris you are refering to, but for my part, yes and no: That's one of the perils of geeky communities - there'll often be multiple people named Chris. I have a brother named Michael who's a rail enthusiast, and that world has the same issue with his name - he was once in a car with three other people named Michael. > Yes, that would keep iterator classes and generator functions consistent, > which would be a good thing. > > No: I don't think we should do that -- StopIteration is part of the iterator > protocol -- generators are another way to write something that complies with > the iterator protocol -- generators should handle StopIteration the same way > that iterator classes do. I've done my "explain it twice, then shut up" on this subject, so I'll just point you to the list archive, where it's been stated clearly that generators are like __iter__, not like __next__. Please, could you respond to previously-given explanations, rather than simply restating that generators should be like __next__? I'd like to move forward with that discussion, rather than reiterating the same points. ChrisA From rosuav at gmail.com Mon Nov 24 18:07:57 2014 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 25 Nov 2014 04:07:57 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com> <20141121163032.GN2748@ando.pearwood.info> <20141122104827.GQ2748@ando.pearwood.info> Message-ID: On Tue, Nov 25, 2014 at 3:41 AM, Chris Barker wrote: > On Sat, Nov 22, 2014 at 2:57 AM, Chris Angelico wrote: >> >> If you have a lengthy nested chain of coroutines, and one of them >> unexpectedly raises StopIteration, is it right for something to >> quietly terminate, or should the exception bubble up and be printed to >> console? > > > Couldn't you have a nested pile of iterator classes as well that would > exhibit the exact same behavior? Potentially, but that would be a different thing. Also, I don't know of cases where a __next__ function chains to a next() call through arbitrary numbers of levels, but :"yield from" gets a solid work-out in asyncio and related, so it's more likely to come up. But I don't personally use asyncio, so I'd like to hear from someone who does. ChrisA From chris.barker at noaa.gov Mon Nov 24 18:18:30 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 24 Nov 2014 09:18:30 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com> <20141121163032.GN2748@ando.pearwood.info> Message-ID: On Mon, Nov 24, 2014 at 9:06 AM, Chris Angelico wrote: > I've done my "explain it twice, then shut up" on this subject, so I'll > just point you to the list archive, where it's been stated clearly > that generators are like __iter__, not like __next__. Please, could > you respond to previously-given explanations, rather than simply > restating that generators should be like __next__? I'm not sure if I've responded or not to previously given explanations -- but you're right, it's time for me to shut up having made my point, too. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Nov 24 18:23:31 2014 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 25 Nov 2014 04:23:31 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com> <20141121163032.GN2748@ando.pearwood.info> Message-ID: On Tue, Nov 25, 2014 at 4:18 AM, Chris Barker wrote: > On Mon, Nov 24, 2014 at 9:06 AM, Chris Angelico wrote: >> >> I've done my "explain it twice, then shut up" on this subject, so I'll >> just point you to the list archive, where it's been stated clearly >> that generators are like __iter__, not like __next__. Please, could >> you respond to previously-given explanations, rather than simply >> restating that generators should be like __next__? > > > I'm not sure if I've responded or not to previously given explanations -- > but you're right, it's time for me to shut up having made my point, too. Well, there is probably more to be said about this - along the lines of *why* generators ought to be more like iterators. (They're iterables, not iterators.) It's just that we seem to be rehashing the same arguments - or maybe that's just my impression, as there's been discussion on three different channels (-ideas, -dev, and the issue tracker - mercifully very little on the latter). ChrisA From random832 at fastmail.us Mon Nov 24 18:37:39 2014 From: random832 at fastmail.us (random832 at fastmail.us) Date: Mon, 24 Nov 2014 12:37:39 -0500 Subject: [Python-ideas] Counter comparison In-Reply-To: References: <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com> <1416809391.2601292.194579353.687F5B82@webmail.messagingengine.com> Message-ID: <1416850659.2181529.194813961.7C685009@webmail.messagingengine.com> On Mon, Nov 24, 2014, at 10:43, Guido van Rossum wrote: > But does it matter? If you've got a list of sets, and you want it sorted, > you have to specify how you want it sorted anyways. You can complain that > < > is only a partial order, but that is the mathematical root of the > situation, not a problem caused by an poorly chosen language default. > (Leaving < undefined and sorting by object address, now *that* would be > an > example of the latter. :-) You seem to be suggesting that sorting a list of partially ordered items is not a meaningful thing to ask for, rather than merely not being implemented by the list.sort method. From solipsis at pitrou.net Mon Nov 24 18:53:00 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 24 Nov 2014 18:53:00 +0100 Subject: [Python-ideas] Counter comparison References: <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com> <1416809391.2601292.194579353.687F5B82@webmail.messagingengine.com> <1416850659.2181529.194813961.7C685009@webmail.messagingengine.com> Message-ID: <20141124185300.69e8f871@fsol> On Mon, 24 Nov 2014 12:37:39 -0500 random832 at fastmail.us wrote: > On Mon, Nov 24, 2014, at 10:43, Guido van Rossum wrote: > > But does it matter? If you've got a list of sets, and you want it sorted, > > you have to specify how you want it sorted anyways. You can complain that > > < > > is only a partial order, but that is the mathematical root of the > > situation, not a problem caused by an poorly chosen language default. > > (Leaving < undefined and sorting by object address, now *that* would be > > an > > example of the latter. :-) > > You seem to be suggesting that sorting a list of partially ordered items > is not a meaningful thing to ask for, rather than merely not being > implemented by the list.sort method. Since list.sort is guaranteed to be stable, the sort order should ideally be deterministic even in the face of partial ordering. Regards Antoine. From alexander.belopolsky at gmail.com Mon Nov 24 19:02:02 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 24 Nov 2014 13:02:02 -0500 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com> <20141121163032.GN2748@ando.pearwood.info> Message-ID: On Mon, Nov 24, 2014 at 12:23 PM, Chris Angelico wrote: > Well, there is probably more to be said about this - along the lines > of *why* generators ought to be more like iterators. (They're > iterables, not iterators.) > If generators are not iterators, how do you explain this? >>> import collections >>> def g(): ... yield 42 ... >>> isinstance(g(), collections.Iterator) True -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Nov 24 19:11:10 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 24 Nov 2014 10:11:10 -0800 Subject: [Python-ideas] Counter comparison In-Reply-To: <1416850659.2181529.194813961.7C685009@webmail.messagingengine.com> References: <39A1F2C1-03AA-42E8-92EA-156F7C074E7A@gmail.com> <1416809391.2601292.194579353.687F5B82@webmail.messagingengine.com> <1416850659.2181529.194813961.7C685009@webmail.messagingengine.com> Message-ID: On Mon, Nov 24, 2014 at 9:37 AM, wrote: > On Mon, Nov 24, 2014, at 10:43, Guido van Rossum wrote: > > But does it matter? If you've got a list of sets, and you want it sorted, > > you have to specify how you want it sorted anyways. You can complain that > > < > > is only a partial order, but that is the mathematical root of the > > situation, not a problem caused by an poorly chosen language default. > > (Leaving < undefined and sorting by object address, now *that* would be > > an > > example of the latter. :-) > > You seem to be suggesting that sorting a list of partially ordered items > is not a meaningful thing to ask for, rather than merely not being > implemented by the list.sort method. > I assume you want a topological sort. But isn't that topic by itself a useful math lesson? -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Nov 24 19:25:04 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 24 Nov 2014 10:25:04 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com> <20141121163032.GN2748@ando.pearwood.info> Message-ID: On Mon, Nov 24, 2014 at 10:02 AM, Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > > On Mon, Nov 24, 2014 at 12:23 PM, Chris Angelico wrote: > >> Well, there is probably more to be said about this - along the lines >> of *why* generators ought to be more like iterators. (They're >> iterables, not iterators.) >> > > If generators are not iterators, how do you explain this? > > >>> import collections > >>> def g(): > ... yield 42 > ... > >>> isinstance(g(), collections.Iterator) > True > I think Chris A was overzealous here. The word "generator" is ambiguous; it can refer to either a generator function (a function definition containing at least one "yield") or to the object you obtain by calling a generator function. The latter is definitely an iterator (it has a __next__ method). You can't really call a generator function an iterable (since calling iter() on it raises TypeError) but it's not an iterator either. For the rest see my explanation in response to Mark Shannon in python-dev: http://code.activestate.com/lists/python-dev/133428/ -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Nov 24 23:48:15 2014 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 25 Nov 2014 09:48:15 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com> <20141121163032.GN2748@ando.pearwood.info> Message-ID: On Tue, Nov 25, 2014 at 5:02 AM, Alexander Belopolsky wrote: > On Mon, Nov 24, 2014 at 12:23 PM, Chris Angelico wrote: >> >> Well, there is probably more to be said about this - along the lines >> of *why* generators ought to be more like iterators. (They're >> iterables, not iterators.) > > > If generators are not iterators, how do you explain this? > >>>> import collections >>>> def g(): > ... yield 42 > ... >>>> isinstance(g(), collections.Iterator) > True My apologies. As Guido said, "generator" is ambiguous; though I was inaccurate as well. A generator *object* is, as you show above, an iterator; a generator *function* is not actually iterable, but it is an iterator factory. An iterator class is also an iterator factory. ChrisA From wolfgang.maier at biologie.uni-freiburg.de Mon Nov 24 23:53:11 2014 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Mon, 24 Nov 2014 23:53:11 +0100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: On 15.11.2014 10:29, Chris Angelico wrote: > PEP: 479 > Title: Change StopIteration handling inside generators > Version: $Revision$ > Last-Modified: $Date$ > Author: Chris Angelico > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 15-Nov-2014 > Python-Version: 3.5 > Post-History: 15-Nov-2014 > > > Abstract > ======== > > This PEP proposes a semantic change to ``StopIteration`` when raised > inside a generator, unifying the behaviour of list comprehensions and > generator expressions somewhat. > > > Rationale > ========= > > The interaction of generators and ``StopIteration`` is currently > somewhat surprising, and can conceal obscure bugs. An unexpected > exception should not result in subtly altered behaviour, but should > cause a noisy and easily-debugged traceback. Currently, > ``StopIteration`` can be absorbed by the generator construct. > > > Proposal > ======== > > If a ``StopIteration`` is about to bubble out of a generator frame, it > is replaced with some other exception (maybe ``RuntimeError``, maybe a > new custom ``Exception`` subclass, but *not* deriving from > ``StopIteration``) which causes the ``next()`` call (which invoked the > generator) to fail, passing that exception out. From then on it's > just like any old exception. [3]_ > Now that this PEP is going to be accepted, I'm not sure how much sense it still makes to suggest an amendment to it, but anyway: As stated in the abstract one of the goals of the PEP is to unify further the behaviour of comprehensions and generator expressions. With the PEP in place the following example (taken from Steven d'Aprano's post on python-list): iterable = [iter([])] list(next(x) for x in iterable) would raise an error just like [next(x) for x in iterable] already does today. However the comprehension currently raises StopIteration, while the proposed error for the generator expression would be of a different class (supposedly RuntimeError) - so comprehensions and generator expressions would still behave a bit (though much less) differently. In addition, the PEP leaves an iterator's __next__() method as the only reasonable place where user-code should raise StopIteration. So I would like to argue that instead of just turning StopIteration into some other error when it's about to bubble out of a generator frame, it should be converted whenever it bubbles out of *anything except an iterator's __next__()*. This would include comprehensions, but also any other code. (On the side, I guess the current form of the PEP does address hard-to-debug bugs caused by nested generators, but what about nested __next__ in iterators ? Shouldn't it using the same logic also be an error if a next call inside a __next__ method raises an uncaught StopIteration ?) I think such general behavior would make it much clearer that StopIteration is considered special and reserved for the iterator protocol. Of course, it might mean more broken code if people use StopIteration or a subclass for error signaling outside generator/iterators, but this PEP will mean backwards incompatibility anyway so why not go all the way and do it consistently. I'm not sure I'd like the pretty general RuntimeError for this (even though Guido favors it for simplicity), instead one could call it UnhandledStopIteration ? I imagine that a dedicated class would help in porting, for example, python2 code to python3 (which this PEP does not really simplify otherwise) since people/scripts could watch out for something specific ? Thoughts? Wolfgang From rosuav at gmail.com Tue Nov 25 00:03:46 2014 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 25 Nov 2014 10:03:46 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: On Tue, Nov 25, 2014 at 9:53 AM, Wolfgang Maier wrote: > In addition, the PEP leaves an iterator's __next__() method as the only > reasonable place where user-code should raise StopIteration. > So I would like to argue that instead of just turning StopIteration into > some other error when it's about to bubble out of a generator frame, it > should be converted whenever it bubbles out of *anything except an > iterator's __next__()*. This would include comprehensions, but also any > other code. There'd have to be a special case for next(), where StopIteration is part of the definition of the function. The question then becomes, what's the boundary where StopIteration is converted? The current proposal is quite simple. All the conversion happens in the one function that (re)starts a generator frame, gen_send_ex() in Objects/genobject.c. To do this for other functions, there'd need to be some way of saying which ones are allowed to raise StopIteration and which aren't. Additionally, the scope for problems is smaller. A StopIteration raised anywhere outside of a loop header won't cause silent behavioral change; with generators, anywhere in the body of the function will have that effect. So, while I do agree in principle that it'd be nice, I don't know that it's practical; however, it might be worth raising a dependent proposal to extend this exception conversion. ChrisA From alexander.belopolsky at gmail.com Tue Nov 25 00:20:11 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 24 Nov 2014 18:20:11 -0500 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com> <20141121163032.GN2748@ando.pearwood.info> Message-ID: On Mon, Nov 24, 2014 at 5:48 PM, Chris Angelico wrote: > > As Guido said, "generator" is ambiguous; though I was > inaccurate as well. A generator *object* is, as you show above, an > iterator; a generator *function* is not actually iterable, but it is > an iterator factory. An iterator class is also an iterator factory. This is correct, and I don't think there is any ambiguity: >>> def g(): yield 42 ... >>> type(g) >>> type(g()) As explained in PEP 255, "a Python generator is a kind of Python iterator[1], but of an especially powerful kind." The other term introduced by PEP 255 is "generator function": "A function that contains a yield statement is called a generator function." In my view, PEP 479 naturally follows from careful reading of PEP 225. All one needs to understand is the difference between a function that returns an iterator and its value. -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at 2sn.net Tue Nov 25 00:23:22 2014 From: python at 2sn.net (Alexander Heger) Date: Tue, 25 Nov 2014 10:23:22 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <6D2153E4-0768-41BD-8BFC-B3F711A82088@gmail.com> <20141121163032.GN2748@ando.pearwood.info> Message-ID: >> Maybe a way to obtain equivalency to the generator functions in this >> case is to "break" this example for the iterator object as well, in >> that StopIteration has to be raised in the frame of the generator >> object; if it raised in a different context, e.g., a function called >> by __next__, that StopIteration should also be converted to a >> RuntimeError similar to what is proposed in the PEP for the generator >> functions. Maybe this is not what Chris intends to happen, but it >> would make things consistent. > > I"mn not sure which Chris you are refering to, but for my part, yes and no: yes, this was a reply to your post > Yes, that would keep iterator classes and generator functions consistent, > which would be a good thing. I think the main goal was to have a consistent interface that is easy to debug and deals with StopIteration bubbling up - hence such Exception from other scopes should convert to RuntimeError when crossing the iterator interface boundary originating from other scopes. > No: I don't think we should do that -- StopIteration is part of the iterator > protocol -- generators are another way to write something that complies with > the iterator protocol -- generators should handle StopIteration the same way > that iterator classes do. You'd Keep StopIteration in the protocol, but only allow it in the local scope. -Alexander From wolfgang.maier at biologie.uni-freiburg.de Tue Nov 25 18:30:07 2014 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Tue, 25 Nov 2014 18:30:07 +0100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: On 11/25/2014 12:03 AM, Chris Angelico wrote: > On Tue, Nov 25, 2014 at 9:53 AM, Wolfgang Maier > wrote: >> In addition, the PEP leaves an iterator's __next__() method as the only >> reasonable place where user-code should raise StopIteration. >> So I would like to argue that instead of just turning StopIteration into >> some other error when it's about to bubble out of a generator frame, it >> should be converted whenever it bubbles out of *anything except an >> iterator's __next__()*. This would include comprehensions, but also any >> other code. > > There'd have to be a special case for next(), where StopIteration is > part of the definition of the function. The question then becomes, > what's the boundary where StopIteration is converted? > > The current proposal is quite simple. All the conversion happens in > the one function that (re)starts a generator frame, gen_send_ex() in > Objects/genobject.c. To do this for other functions, there'd need to > be some way of saying which ones are allowed to raise StopIteration > and which aren't. > Well, I'm not familiar with every implementation detail of the interpreter so I can't judge how difficult to implement certain things would be, but one solution that I could think of is: allow StopIteration to be raised anywhere, but let it bubble up only *one* frame. So if the next outer frame does not deal with it, the exception would be converted to UnhandledStopIteration (or something else) when it's about to bubble out of that outer frame. The builtin next() would simply reset the frame count by catching and reraising StopIteration raised inside its argument (whether that's an iterator's __next__ or a generator; note that in this scenario using raise StopIteration instead of return inside a generator would remain possible). Examples of what would happen: using next on a generator that raises StopIteration explicitly: => next catches the error and reraises StopIteration using next on a generator that returns: => next behaves like currently, raising StopIteration using next on the __next__ method of an iterator: => next catches the error and reraises StopIteration every direct call of an iterator's __next__ method: => has to be guarded by a try/except StopIteration Likewise in the first three cases, the calling frame, which resumes when next returns, (and only this frame) is given a chance to handle the error. If that doesn't happen (i.e. the error would bubble out) it gets converted. So different from the current PEP where a StopIteration must be dealt with explicitly using try/except only inside generators, but bubbles up everywhere else, here StopIteration will be special everywhere, i.e., it must be passed upwards explicitly through all frames or will get converted. Back to Steven's generator expression vs comprehension example: iterable = [iter([])] list(next(x) for x in iterable) would raise UnhandledStopIteration since there is no way, inside the generator expression to catch the StopIteration raised by next(x). ... and if that's all complete nonsense because of some technical detail I'm not aware of, then please excuse my ignorance. Wolfgang From rosuav at gmail.com Tue Nov 25 18:48:16 2014 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 26 Nov 2014 04:48:16 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: On Wed, Nov 26, 2014 at 4:30 AM, Wolfgang Maier wrote: > Well, I'm not familiar with every implementation detail of the interpreter > so I can't judge how difficult to implement certain things would be, but one > solution that I could think of is: I don't know much about the internal details of CPython either, but let's just ignore that for the moment and consider specs for the Python language. AFAIK, not one of the concerns raised (by PEP 479 or your proposal here) is CPython-specific. > allow StopIteration to be raised anywhere, but let it bubble up only *one* > frame. > So if the next outer frame does not deal with it, the exception would be > converted to UnhandledStopIteration (or something else) when it's about to > bubble out of that outer frame. > The builtin next() would simply reset the frame count by catching and > reraising StopIteration raised inside its argument (whether that's an > iterator's __next__ or a generator; note that in this scenario using raise > StopIteration instead of return inside a generator would remain possible). Interesting. Makes a measure of sense, and doesn't have much magic to it. > So different from the current PEP where a StopIteration must be dealt with > explicitly using try/except only inside generators, but bubbles up > everywhere else, here StopIteration will be special everywhere, i.e., it > must be passed upwards explicitly through all frames or will get converted. > > Back to Steven's generator expression vs comprehension example: > > iterable = [iter([])] > list(next(x) for x in iterable) > > would raise UnhandledStopIteration since there is no way, inside the > generator expression to catch the StopIteration raised by next(x). Downside of this is that it's harder to consciously chain iterators, but maybe that's a cost that has to be paid. Suggestion for this: Have a new way of "raise-and-return". It's mostly like raise, except that (a) it can't be caught by a try/except block in the current function (because there's no point), and (b) it bypasses the "this exception must not pass unnoticed". It could then also be used for anything else that needs the "return any object, or signal lack of return value" option, covering AttributeError and so on. So it'd be something like this: class X: def __iter__(self): return self def __next__(self): if condition: return value signal StopIteration The 'signal' statement would promptly terminate the function (not sure exactly how it'd interact with context managers and try/finally, but something would be worked out), and then raise StopIteration in the calling function. Any other StopIteration which passes out of a function would become a RuntimeError. Magic required: Some way of knowing which exceptions should be covered by this ban on bubbling; also, preferably, some way to raise StopIteration in the calling function, without losing the end of the backtrace. This could be a viable proposal. It'd be rather more complicated than PEP 479, though, and would require a minimum of five hundred bikeshedding posts before it comes to any kind of conclusion, but if you feel this issue is worth it, I'd certainly be an interested participant in the discussion. ChrisA From chris.barker at noaa.gov Tue Nov 25 18:47:52 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 25 Nov 2014 09:47:52 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: On Tue, Nov 25, 2014 at 9:30 AM, Wolfgang Maier < wolfgang.maier at biologie.uni-freiburg.de> wrote: > allow StopIteration to be raised anywhere, but let it bubble up only *one* > frame. > So if the next outer frame does not deal with it, the exception would be > converted to UnhandledStopIteration (or something else) when it's about to > bubble out of that outer frame. > The builtin next() would simply reset the frame count by catching and > reraising StopIteration raised inside its argument (whether that's an > iterator's __next__ or a generator; note that in this scenario using raise > StopIteration instead of return inside a generator would remain possible). > I also have no idea if this is practical from an implementation perspective, but I like how it support my goal of keeping the behavior of iterator classes and generators consistent. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Nov 25 18:53:23 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 25 Nov 2014 09:53:23 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: On Tue, Nov 25, 2014 at 9:48 AM, Chris Angelico wrote: > On Wed, Nov 26, 2014 at 4:30 AM, Wolfgang Maier > wrote: > > Well, I'm not familiar with every implementation detail of the > interpreter > > so I can't judge how difficult to implement certain things would be, but > one > > solution that I could think of is: > > I don't know much about the internal details of CPython either, but > let's just ignore that for the moment and consider specs for the > Python language. AFAIK, not one of the concerns raised (by PEP 479 or > your proposal here) is CPython-specific. > > > allow StopIteration to be raised anywhere, but let it bubble up only > *one* > > frame. > > So if the next outer frame does not deal with it, the exception would be > > converted to UnhandledStopIteration (or something else) when it's about > to > > bubble out of that outer frame. > > The builtin next() would simply reset the frame count by catching and > > reraising StopIteration raised inside its argument (whether that's an > > iterator's __next__ or a generator; note that in this scenario using > raise > > StopIteration instead of return inside a generator would remain > possible). > > Interesting. Makes a measure of sense, and doesn't have much magic to it. > > > So different from the current PEP where a StopIteration must be dealt > with > > explicitly using try/except only inside generators, but bubbles up > > everywhere else, here StopIteration will be special everywhere, i.e., it > > must be passed upwards explicitly through all frames or will get > converted. > > > > Back to Steven's generator expression vs comprehension example: > > > > iterable = [iter([])] > > list(next(x) for x in iterable) > > > > would raise UnhandledStopIteration since there is no way, inside the > > generator expression to catch the StopIteration raised by next(x). > > Downside of this is that it's harder to consciously chain iterators, > but maybe that's a cost that has to be paid. > > Suggestion for this: Have a new way of "raise-and-return". It's mostly > like raise, except that (a) it can't be caught by a try/except block > in the current function (because there's no point), and (b) it > bypasses the "this exception must not pass unnoticed". It could then > also be used for anything else that needs the "return any object, or > signal lack of return value" option, covering AttributeError and so > on. > > So it'd be something like this: > > class X: > def __iter__(self): return self > def __next__(self): > if condition: return value > signal StopIteration > > The 'signal' statement would promptly terminate the function (not sure > exactly how it'd interact with context managers and try/finally, but > something would be worked out), and then raise StopIteration in the > calling function. Any other StopIteration which passes out of a > function would become a RuntimeError. > > Magic required: Some way of knowing which exceptions should be covered > by this ban on bubbling; also, preferably, some way to raise > StopIteration in the calling function, without losing the end of the > backtrace. > > This could be a viable proposal. It'd be rather more complicated than > PEP 479, though, and would require a minimum of five hundred > bikeshedding posts before it comes to any kind of conclusion, but if > you feel this issue is worth it, I'd certainly be an interested > participant in the discussion. > It's not viable. It will break more code than PEP 479, and it will incur a larger interpreter overhead (every time any exception bubbles out of any frame we'd have to check whether it is (derived from) StopIteration and replace it, rather than only when exiting a generator frame. (The check for StopIteration is relatively expensive -- it's easy to determine that an exception *is* StopIteration, but in order that it doesn't derive from StopIteration you have to walk the inheritance tree.) Please stop panicking. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Tue Nov 25 18:58:16 2014 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 26 Nov 2014 04:58:16 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: On Wed, Nov 26, 2014 at 4:53 AM, Guido van Rossum wrote: > It's not viable. It will break more code than PEP 479, and it will incur a > larger interpreter overhead (every time any exception bubbles out of any > frame we'd have to check whether it is (derived from) StopIteration and > replace it, rather than only when exiting a generator frame. (The check for > StopIteration is relatively expensive -- it's easy to determine that an > exception *is* StopIteration, but in order that it doesn't derive from > StopIteration you have to walk the inheritance tree.) > > Please stop panicking. Heh. If that last comment was aimed at me, I wasn't panicking... I just thought it an intellectually curious concept, having a measure of interpreter support for the "not-a-return-value" concept. ChrisA From guido at python.org Tue Nov 25 18:59:14 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 25 Nov 2014 09:59:14 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: On Tue, Nov 25, 2014 at 9:47 AM, Chris Barker wrote: > > [...] I like how it support my goal of keeping the behavior of iterator > classes and generators consistent. > This is a great summary of the general confusion I am trying to clear up. The behavior of all types of iterators (including generators) from the *caller's* perspective is not in question and is not changing. It is very simple: you call next(it) (or it.__next__(), and it returns either the next value or raises StopIteration (and any other exception is, indeed, an exception). >From the POV of *implementor*, iterator classes and generators have never been at all alike -- one uses *return* to produce a new value, the other uses *yield*. Returning without a value from a __next__() method is seen as producing a None value by the caller; returning from a generator is translated into a StopIteration which will be interpreted by the caller as the end of series. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Nov 25 19:00:26 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 25 Nov 2014 10:00:26 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: On Tue, Nov 25, 2014 at 9:58 AM, Chris Angelico wrote: > On Wed, Nov 26, 2014 at 4:53 AM, Guido van Rossum > wrote: > > It's not viable. It will break more code than PEP 479, and it will incur > a > > larger interpreter overhead (every time any exception bubbles out of any > > frame we'd have to check whether it is (derived from) StopIteration and > > replace it, rather than only when exiting a generator frame. (The check > for > > StopIteration is relatively expensive -- it's easy to determine that an > > exception *is* StopIteration, but in order that it doesn't derive from > > StopIteration you have to walk the inheritance tree.) > > > > Please stop panicking. > > Heh. If that last comment was aimed at me, I wasn't panicking... I > just thought it an intellectually curious concept, having a measure of > interpreter support for the "not-a-return-value" concept. > OK, then I will just ignore this thread. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Tue Nov 25 21:31:11 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 25 Nov 2014 12:31:11 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: On Tue, Nov 25, 2014 at 9:59 AM, Guido van Rossum wrote: > On Tue, Nov 25, 2014 at 9:47 AM, Chris Barker > wrote: > >> >> [...] I like how it support my goal of keeping the behavior of iterator >> classes and generators consistent. >> > > This is a great summary of the general confusion I am trying to clear up. > The behavior of all types of iterators (including generators) from the > *caller's* perspective is not in question and is not changing. It is very > simple: you call next(it) (or it.__next__(), and it returns either the next > value or raises StopIteration (and any other exception is, indeed, an > exception). > > From the POV of *implementor*, iterator classes and generators have never > been at all alike -- one uses *return* to produce a new value, the other > uses *yield*. Returning without a value from a __next__() method is seen as > producing a None value by the caller; returning from a generator is > translated into a StopIteration which will be interpreted by the caller as > the end of series. > Once you start nesting these things, the distinction between "implementor" and "caller" gets mingled. And I think this is all about how nested generators behave, yes? If I am implementing a iterator of some sort (generator function or iterator class), and I call next() inside my code, then I am both a implementor and caller. And if I'm also writing helper functions, then I need to know about how StopIteration will be handled, and it will be handled a bit differently by generators and iterator classes. But not a big deal, agreed, probably a much smaller deal that all the other stuff you'd better understand to write this kind of code anyway. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Nov 25 22:05:00 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 25 Nov 2014 13:05:00 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: On Tue, Nov 25, 2014 at 12:31 PM, Chris Barker wrote: > On Tue, Nov 25, 2014 at 9:59 AM, Guido van Rossum > wrote: > >> On Tue, Nov 25, 2014 at 9:47 AM, Chris Barker >> wrote: >> >>> >>> [...] I like how it support my goal of keeping the behavior of iterator >>> classes and generators consistent. >>> >> >> This is a great summary of the general confusion I am trying to clear up. >> The behavior of all types of iterators (including generators) from the >> *caller's* perspective is not in question and is not changing. It is very >> simple: you call next(it) (or it.__next__(), and it returns either the next >> value or raises StopIteration (and any other exception is, indeed, an >> exception). >> >> From the POV of *implementor*, iterator classes and generators have never >> been at all alike -- one uses *return* to produce a new value, the other >> uses *yield*. Returning without a value from a __next__() method is seen as >> producing a None value by the caller; returning from a generator is >> translated into a StopIteration which will be interpreted by the caller as >> the end of series. >> > > Once you start nesting these things, the distinction between "implementor" > and "caller" gets mingled. > Hm. An implementer of one protocol is likely the caller of many other protocols. It's not clear that calling something that implements the *same* protocol should deserve special status. For example I could be implementing an iterator processing the lines of a CSV file. Inside this iterator I may be using another iterator that loops over the fields of the current line. (This doesn't sound too far-fetched.) But if I run out of fields in a line, why should that translate into terminating the outer iterator? And the outer iterator may itself be called by another, more outer iterator that iterators over a list of files. > And I think this is all about how nested generators behave, yes? > The inner iterator doesn't have to be a generator (apart from send() and throw(), they have the same interface). And the point of the PEP is that an exhausted inner iterator shouldn't be taken to automatically terminate the outer one. (You could point out that I don't do anything about the similar problem when the outer iterator is implemented as a class with a __next__() method. If I could, I would -- but that case is different because there you *must* raise StopIteration to terminate the iteration, so it becomes more similar to an accidental KeyError being masked when it occurs inside a __getitem__() method.) > If I am implementing a iterator of some sort (generator function or > iterator class), and I call next() inside my code, then I am both a > implementor and caller. And if I'm also writing helper functions, then I > need to know about how StopIteration will be handled, and it will be > handled a bit differently by generators and iterator classes. > A helper function also defines an interface. If you are writing a helper function for a generator (and the helper function is participating in the same iteration as the outer generator, i.e. not in the CSV fies / lines / fields example), the best way to do it is probably to write it as a helper generator, and use "yield from" in the outer generator. > > But not a big deal, agreed, probably a much smaller deal that all the > other stuff you'd better understand to write this kind of code anyway. > Which I'm sorry to see is much less widely understood than I had assumed. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Tue Nov 25 22:18:07 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 25 Nov 2014 13:18:07 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: On Tue, Nov 25, 2014 at 1:05 PM, Guido van Rossum wrote: > (You could point out that I don't do anything about the similar problem > when the outer iterator is implemented as a class with a __next__() method. > Indeed -- that is the key point here -- but you were pretty clear about how special casing StopIteration is a non-starter. > If I could, I would -- but that case is different because there you *must* > raise StopIteration to terminate the iteration, so it becomes more similar > to an accidental KeyError being masked when it occurs inside a > __getitem__() method. > Well, I guess it a good thing to make things easier/clearer where you can -- even if you can't do it everywhere. I suppose if you think of generator functions as an easier way to write an iterator (where it makes sense) then this is one more thing that makes it even easier easier / safer. It does even more of the book keeping for you. So a consistent win-win. Thanks for the time taken clarifying your point. But not a big deal, agreed, probably a much smaller deal that all the other >> stuff you'd better understand to write this kind of code anyway. >> > > Which I'm sorry to see is much less widely understood than I had assumed. > Well, this PEP does make for one less detail you need to understand (or more to the point, keep in mind) when writing generator functions -- so that's a good thing. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.us Tue Nov 25 22:56:48 2014 From: random832 at fastmail.us (random832 at fastmail.us) Date: Tue, 25 Nov 2014 16:56:48 -0500 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: Message-ID: <1416952608.2652417.195393269.2A836E0E@webmail.messagingengine.com> On Tue, Nov 25, 2014, at 15:31, Chris Barker wrote: > Once you start nesting these things, the distinction between > "implementor" > and "caller" gets mingled. And I think this is all about how nested > generators behave, yes? For something more concrete, we can consider a naive implementation of iteration over adjacent pairs: def pairs(x): i = iter(x) while True: yield next(i), next(i) >>> list(pairs(range(1))) [] >>> list(pairs(range(2))) [(0, 1)] >>> list(pairs(range(3))) [(0, 1)] >>> list(pairs(range(4))) [(0, 1), (2, 3)] >>> list(pairs(range(5))) [(0, 1), (2, 3)] >>> list(pairs(range(6))) [(0, 1), (2, 3), (4, 5)] To work under the new paradigm, you need to catch StopIteration explicitly: def pairs(x): i = iter(x) while True: try: a = next(i) b = next(i) except StopIteration: return yield a, b From ian at feete.org Wed Nov 26 00:19:01 2014 From: ian at feete.org (Ian Foote) Date: Tue, 25 Nov 2014 23:19:01 +0000 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: <1416952608.2652417.195393269.2A836E0E@webmail.messagingengine.com> References: <1416952608.2652417.195393269.2A836E0E@webmail.messagingengine.com> Message-ID: <54750E65.7010602@feete.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 25/11/14 21:56, random832 at fastmail.us wrote: > On Tue, Nov 25, 2014, at 15:31, Chris Barker wrote: >> Once you start nesting these things, the distinction between >> "implementor" and "caller" gets mingled. And I think this is all >> about how nested generators behave, yes? > > For something more concrete, we can consider a naive implementation > of iteration over adjacent pairs: > > def pairs(x): i = iter(x) while True: yield next(i), next(i) > > > To work under the new paradigm, you need to catch StopIteration > explicitly: > > def pairs(x): i = iter(x) while True: try: a = next(i) b = next(i) > except StopIteration: return yield a, b > You're right that you need to catch the StopIteration, but it seems to me the natural way to write your second example is: def pairs(x): i = iter(x) while True: try: yield next(i), next(i) except StopIteration: return Adding the extra variables a and b is unnecessary and distracts from the change actually required. Regards, Ian F -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJUdQ5lAAoJEODsV4MF7PWz/zsIALLI67/W+HAG3l0Xe+kd2/Xw QEI5NrOyT/izRHbV69K3zvOVKKCfiUXjkK5rPGxFiBmF96hOmQyro7Z4UiCSYzsT N+8dy8M6/gAWolEbD1EZoXZorNHa9nsZ8q3hBltl824CAl4Kx7FFKssUVIFWjyrD IgPjI4PIJBl12uX9F9VLMaBjfEy+QiCUa3a8s7ZdqS1asm1M4udei/qvt1t/NaIL uoYGuBO/mzxP9sdWtP4z53sX07gOMPUWdBTXFX91+G1pCaUuGCHpDcHwexatxVgk JxgfajyadFj+44QeHpsY10pu0HhaRH+Cbg7vADa2KQ6N3kQic1qHR9FkcHCvaA0= =SdtN -----END PGP SIGNATURE----- From rosuav at gmail.com Wed Nov 26 00:31:38 2014 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 26 Nov 2014 10:31:38 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: <1416952608.2652417.195393269.2A836E0E@webmail.messagingengine.com> References: <1416952608.2652417.195393269.2A836E0E@webmail.messagingengine.com> Message-ID: On Wed, Nov 26, 2014 at 8:56 AM, wrote: > For something more concrete, we can consider a naive implementation of > iteration over adjacent pairs: > > def pairs(x): > i = iter(x) > while True: > yield next(i), next(i) Okay, it's simple and naive. How about this version: def pairs(x): i = iter(x) for val in i: yield val, next(i) Also simple, but subtly different from your version. What's the difference? Will it be obvious to everyone who reads it? ChrisA From ethan at stoneleaf.us Wed Nov 26 00:46:00 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 25 Nov 2014 15:46:00 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <1416952608.2652417.195393269.2A836E0E@webmail.messagingengine.com> Message-ID: <547514B8.4090703@stoneleaf.us> On 11/25/2014 03:31 PM, Chris Angelico wrote: > On Wed, Nov 26, 2014 at 8:56 AM, wrote: >> For something more concrete, we can consider a naive implementation of >> iteration over adjacent pairs: >> >> def pairs(x): >> i = iter(x) >> while True: >> yield next(i), next(i) > > Okay, it's simple and naive. How about this version: > > def pairs(x): > i = iter(x) > for val in i: > yield val, next(i) > > Also simple, but subtly different from your version. What's the > difference? Will it be obvious to everyone who reads it? I don't see the difference being subtle enough -- if an odd number of items is tossed in, that `next(i)` is still going to raise a StopIteration, which under PEP 479 will become a RunTimeError. Or did you mean that even numbered iterators will work fine, but odd-numbered ones will still raise? Nice. :) -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From rosuav at gmail.com Wed Nov 26 01:37:17 2014 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 26 Nov 2014 11:37:17 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: <547514B8.4090703@stoneleaf.us> References: <1416952608.2652417.195393269.2A836E0E@webmail.messagingengine.com> <547514B8.4090703@stoneleaf.us> Message-ID: On Wed, Nov 26, 2014 at 10:46 AM, Ethan Furman wrote: > I don't see the difference being subtle enough -- if an odd number of items is tossed in, that `next(i)` is still going > to raise a StopIteration, which under PEP 479 will become a RunTimeError. > > Or did you mean that even numbered iterators will work fine, but odd-numbered ones will still raise? Nice. :) Yes, I meant the silent termination of an iterable with an odd number of items. That's pretty subtle. ChrisA From ethan at stoneleaf.us Wed Nov 26 01:46:31 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 25 Nov 2014 16:46:31 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <1416952608.2652417.195393269.2A836E0E@webmail.messagingengine.com> <547514B8.4090703@stoneleaf.us> Message-ID: <547522E7.6050301@stoneleaf.us> On 11/25/2014 04:37 PM, Chris Angelico wrote: > On Wed, Nov 26, 2014 at 10:46 AM, Ethan Furman wrote: >> I don't see the difference being subtle enough -- if an odd number of items is tossed in, that `next(i)` is still going >> to raise a StopIteration, which under PEP 479 will become a RunTimeError. >> >> Or did you mean that even numbered iterators will work fine, but odd-numbered ones will still raise? Nice. :) > > Yes, I meant the silent termination of an iterable with an odd number > of items. That's pretty subtle. You mean an even number of items, yes? -- ~Ethan~ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From rosuav at gmail.com Wed Nov 26 01:56:46 2014 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 26 Nov 2014 11:56:46 +1100 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: <547522E7.6050301@stoneleaf.us> References: <1416952608.2652417.195393269.2A836E0E@webmail.messagingengine.com> <547514B8.4090703@stoneleaf.us> <547522E7.6050301@stoneleaf.us> Message-ID: On Wed, Nov 26, 2014 at 11:46 AM, Ethan Furman wrote: > On 11/25/2014 04:37 PM, Chris Angelico wrote: >> On Wed, Nov 26, 2014 at 10:46 AM, Ethan Furman wrote: >>> I don't see the difference being subtle enough -- if an odd number of items is tossed in, that `next(i)` is still going >>> to raise a StopIteration, which under PEP 479 will become a RunTimeError. >>> >>> Or did you mean that even numbered iterators will work fine, but odd-numbered ones will still raise? Nice. :) >> >> Yes, I meant the silent termination of an iterable with an odd number >> of items. That's pretty subtle. > > You mean an even number of items, yes? Presumably the even case is the correct one. It's intended to work that way. If you give it an odd number of items, pre-479 they'll both silently terminate. (Post-479, the next(),next() one will always raise RuntimeError, which indicates clearly that it's not written appropriately, but that's not subtle.) ChrisA From apalala at gmail.com Wed Nov 26 04:51:25 2014 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Tue, 25 Nov 2014 23:21:25 -0430 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: <1416952608.2652417.195393269.2A836E0E@webmail.messagingengine.com> References: <1416952608.2652417.195393269.2A836E0E@webmail.messagingengine.com> Message-ID: On Tue, Nov 25, 2014 at 5:26 PM, wrote: > def pairs(x): > i = iter(x) > while True: > yield next(i), next(i) > That's not too pythonic, and trying to support non-pythonic code while evolving the language is a dead-end street. The web documents pythonic ways to obtain pairs from an iterable: def pairs(x): i = iter(x) return zip(i, i) Or even: def pairs(x): return zip(*[iter(x)]*2) The usual way of dealing with an odd number of elements is to use zip_longest. I don't remember seeing documented that raising StopIteration will cancel more than one iterator. If it's undocumented, then code that relies on the non-feature is broken. Cheers, -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.us Wed Nov 26 14:59:34 2014 From: random832 at fastmail.us (random832 at fastmail.us) Date: Wed, 26 Nov 2014 08:59:34 -0500 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: References: <1416952608.2652417.195393269.2A836E0E@webmail.messagingengine.com> Message-ID: <1417010374.2937928.195655729.58930BAB@webmail.messagingengine.com> On Tue, Nov 25, 2014, at 22:51, Juancarlo A?ez wrote: > On Tue, Nov 25, 2014 at 5:26 PM, wrote: > > > def pairs(x): > > i = iter(x) > > while True: > > yield next(i), next(i) > > > > That's not too pythonic, and trying to support non-pythonic code while > evolving the language is a dead-end street. Out of curiosity, what explicit uses of next are pythonic? > The web documents pythonic ways to obtain pairs from an iterable: > > def pairs(x): > i = iter(x) > return zip(i, i) > > Or even: > > def pairs(x): > return zip(*[iter(x)]*2) See, both of those seem way too implicit to me. From guido at python.org Wed Nov 26 18:28:59 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 26 Nov 2014 09:28:59 -0800 Subject: [Python-ideas] PEP 479: Change StopIteration handling inside generators In-Reply-To: <1417010374.2937928.195655729.58930BAB@webmail.messagingengine.com> References: <1416952608.2652417.195393269.2A836E0E@webmail.messagingengine.com> <1417010374.2937928.195655729.58930BAB@webmail.messagingengine.com> Message-ID: On Wed, Nov 26, 2014 at 5:59 AM, wrote: > Out of curiosity, what explicit uses of next are pythonic? Ones immediately enclosed in try/except StopIteration, e.g.: try: x = next(it) print(x) except StopIteration: print('nothing') You could rewrite this particular one as follows: for x in it: print(x) break else: print('nothing') But if you have multiple next() calls you might be able to have a single try/except catching StopIteration from all of them, so the first pattern is more general. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul at colomiets.name Wed Nov 26 18:35:23 2014 From: paul at colomiets.name (Paul Colomiets) Date: Wed, 26 Nov 2014 19:35:23 +0200 Subject: [Python-ideas] Asynchronous IO ideas for Python Message-ID: Hi, I've written an article about how I perceive the future of asynchronous I/O in Python. It's not something that should directly be incorporated into python now, but I believe it's useful for python-ideas list. https://medium.com/@paulcolomiets/the-future-of-asynchronous-io-in-python-ce200536d847 And a place for comments at Hacker News: https://news.ycombinator.com/item?id=8662782 I hope being helpful with this writeup :) -- Paul From trent at snakebite.org Wed Nov 26 21:56:07 2014 From: trent at snakebite.org (Trent Nelson) Date: Wed, 26 Nov 2014 12:56:07 -0800 Subject: [Python-ideas] Asynchronous IO ideas for Python References: Message-ID: <5E3A89398864414E8EE2CB192E0C965C19A3508EDB@EXMBX10.exchhosting.com> Relevant part of the video with the normal Python stats on the left and PyParallel on the right: https://www.youtube.com/watch?v=4L4Ww3ROuro#t=838 Transcribed stats: Regular Python HTTP server: Thread Stats Avg Stdev Max Latency 4.93ms 714us 10ms Req/Seq 552 154 1.1k 10,480 requests in 10s, 69MB 1048 reqs/sec, 6.9MB/s PyParallel (4 core Windows VM): Thread Stats Avg Stdev Max Latency 2.41ms 531us 10ms Req/Seq 1.74k 183 2.33k 32,831 requests in 10s, 216MB 3263 reqs/sec, 21MB/s So basically a bit less than linear scaling with more cores, which isn't too bad for a full debug build running on a VM. -----Original Message----- From: Trent Nelson Sent: Wednesday, November 26, 2014 3:36 PM To: 'Paul Colomiets'; python-ideas Subject: RE: [Python-ideas] Asynchronous IO ideas for Python Have you seen this?: https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploited-all-cores I spend the first 80-ish slides on async I/O. (That was a year ago. I've done 2-3 sprints on it since then and have gotten it to a point where I can back up the claims with hard numbers on load testing benchmarks, demonstrated in the most recent video: https://www.youtube.com/watch?v=4L4Ww3ROuro.) Trent. -----Original Message----- From: Python-ideas [mailto:python-ideas-bounces+trent=snakebite.org at python.org] On Behalf Of Paul Colomiets Sent: Wednesday, November 26, 2014 12:35 PM To: python-ideas Subject: [Python-ideas] Asynchronous IO ideas for Python Hi, I've written an article about how I perceive the future of asynchronous I/O in Python. It's not something that should directly be incorporated into python now, but I believe it's useful for python-ideas list. https://medium.com/@paulcolomiets/the-future-of-asynchronous-io-in-python-ce200536d847 And a place for comments at Hacker News: https://news.ycombinator.com/item?id=8662782 I hope being helpful with this writeup :) -- Paul _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ From guido at python.org Wed Nov 26 22:48:41 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 26 Nov 2014 13:48:41 -0800 Subject: [Python-ideas] Asynchronous IO ideas for Python In-Reply-To: <5E3A89398864414E8EE2CB192E0C965C19A3508EDB@EXMBX10.exchhosting.com> References: <5E3A89398864414E8EE2CB192E0C965C19A3508EDB@EXMBX10.exchhosting.com> Message-ID: Trent, Can you post source for the regular and pyparallel HTTP servers you used? On Wed, Nov 26, 2014 at 12:56 PM, Trent Nelson wrote: > Relevant part of the video with the normal Python stats on the left and > PyParallel on the right: > > https://www.youtube.com/watch?v=4L4Ww3ROuro#t=838 > > Transcribed stats: > > Regular Python HTTP server: > > Thread Stats Avg Stdev Max > Latency 4.93ms 714us 10ms > Req/Seq 552 154 1.1k > 10,480 requests in 10s, 69MB > 1048 reqs/sec, 6.9MB/s > > PyParallel (4 core Windows VM): > > Thread Stats Avg Stdev Max > Latency 2.41ms 531us 10ms > Req/Seq 1.74k 183 2.33k > 32,831 requests in 10s, 216MB > 3263 reqs/sec, 21MB/s > > So basically a bit less than linear scaling with more cores, which isn't > too bad for a full debug build running on a VM. > > -----Original Message----- > From: Trent Nelson > Sent: Wednesday, November 26, 2014 3:36 PM > To: 'Paul Colomiets'; python-ideas > Subject: RE: [Python-ideas] Asynchronous IO ideas for Python > > Have you seen this?: > > > https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploited-all-cores > > I spend the first 80-ish slides on async I/O. > > (That was a year ago. I've done 2-3 sprints on it since then and have > gotten it to a point where I can back up the claims with hard numbers on > load testing benchmarks, demonstrated in the most recent video: > https://www.youtube.com/watch?v=4L4Ww3ROuro.) > > > Trent. > > -----Original Message----- > From: Python-ideas [mailto:python-ideas-bounces+trent= > snakebite.org at python.org] On Behalf Of Paul Colomiets > Sent: Wednesday, November 26, 2014 12:35 PM > To: python-ideas > Subject: [Python-ideas] Asynchronous IO ideas for Python > > Hi, > > I've written an article about how I perceive the future of asynchronous > I/O in Python. It's not something that should directly be incorporated into > python now, but I believe it's useful for python-ideas list. > > > https://medium.com/@paulcolomiets/the-future-of-asynchronous-io-in-python-ce200536d847 > > And a place for comments at Hacker News: > > https://news.ycombinator.com/item?id=8662782 > > I hope being helpful with this writeup :) > > -- > Paul > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertc at robertcollins.net Thu Nov 27 00:45:39 2014 From: robertc at robertcollins.net (Robert Collins) Date: Thu, 27 Nov 2014 12:45:39 +1300 Subject: [Python-ideas] Extending traceback to (optionally) format and show locals. Message-ID: For context please see http://bugs.python.org/issue22937 and http://bugs.python.org/issue22936. I have two questions I'm hoping to get answered through this thread: - does the change in question need a PEP? Antoine seemed to think it didn't, as long as it was opt-in (via the unittest CLI etc) - Is my implementation approach sound (for traceback, unittest I think I have covered :))? Implementation wise, I think its useful to work in the current traceback module layout - that is to alter extract_stack to (optionally) include rendered data about locals and then look for that and format it in format_list. I'm sure there is code out there that depends on the quadruple nature of extract_stack though, so I think we need to preserve that. Three strategies occured to me; one is to have parallel functions, one quadruple, one quintuple. A second one is to have the return value of extract_stack be a quintuple when a new keyword parameter include_locals is included. Lastly, and this is my preferred one, if we return a tuple subclass with an attribute containing a dict with the rendered data on the locals; this can be present but None, or even just absent when extract_stack was not asked to include locals. The last option is my preferred one because the other two both imply having a data structure which is likely to break existing code - and while you'd have to opt into having them it seems likely to require a systematic set of upgrades vs having an optional attribute that can be looked up. So - thoughts? -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From paul at colomiets.name Thu Nov 27 01:16:36 2014 From: paul at colomiets.name (Paul Colomiets) Date: Thu, 27 Nov 2014 02:16:36 +0200 Subject: [Python-ideas] Asynchronous IO ideas for Python In-Reply-To: <52564bb8db944de58d5d5df2357a2f05@BLUPR02MB341.namprd02.prod.outlook.com> References: <52564bb8db944de58d5d5df2357a2f05@BLUPR02MB341.namprd02.prod.outlook.com> Message-ID: Hi Trent, On Wed, Nov 26, 2014 at 10:36 PM, Trent Nelson wrote: > Have you seen this?: > > https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploited-all-cores > > I spend the first 80-ish slides on async I/O. > > (That was a year ago. I've done 2-3 sprints on it since then and have gotten it to a point where I can back up the claims with hard numbers on load testing benchmarks, demonstrated in the most recent video: https://www.youtube.com/watch?v=4L4Ww3ROuro.) > > Thanks. Have never seen it before. Do you have some detailed description about how I/O subsystem works? From what I see, pyparallel relies on offloading I/O operations to the IOCP. And it seems that I/O kernel which I'm describing can replace IOCP on unix. And allowing the C code to do more high-level stuff (like reconnecting and message delimitation), would be of a benefit for pyparallel too, as it would allow less heap allocations, smaller number of heap snasphot, etc. Still PyParallel, is much bigger departure from current Python. As it doesn't allow arbitrary code to run in threads, so it's much longer perspective, than I/O kernel which will be just a library on PyPI. -- Paul From abarnert at yahoo.com Thu Nov 27 02:12:45 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 26 Nov 2014 17:12:45 -0800 Subject: [Python-ideas] Extending traceback to (optionally) format and show locals. In-Reply-To: References: Message-ID: On Nov 26, 2014, at 15:45, Robert Collins wrote: > For context please see http://bugs.python.org/issue22937 and > http://bugs.python.org/issue22936. > > I have two questions I'm hoping to get answered through this thread: > - does the change in question need a PEP? Antoine seemed to think it > didn't, as long as it was opt-in (via the unittest CLI etc) > - Is my implementation approach sound (for traceback, unittest I > think I have covered :))? > > > Implementation wise, I think its useful to work in the current > traceback module layout - that is to alter extract_stack to > (optionally) include rendered data about locals and then look for that > and format it in format_list. > > I'm sure there is code out there that depends on the quadruple nature > of extract_stack though, so I think we need to preserve that. Three > strategies occured to me; one is to have parallel functions, one > quadruple, one quintuple. A second one is to have the return value of > extract_stack be a quintuple when a new keyword parameter > include_locals is included. Lastly, and this is my preferred one, if > we return a tuple subclass with an attribute containing a dict with > the rendered data on the locals; this can be present but None, or even > just absent when extract_stack was not asked to include locals. There are lots of other cases in the stdlib where something is usable as a tuple of n fields or as a structseq/namedtuple of >n fields: stat results, struct_tm, etc. So, why not do the same thing here? > The last option is my preferred one because the other two both imply > having a data structure which is likely to break existing code - and > while you'd have to opt into having them it seems likely to require a > systematic set of upgrades vs having an optional attribute that can be > looked up. > > So - thoughts? > > -Rob > > > -- > Robert Collins > Distinguished Technologist > HP Converged Cloud > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From ncoghlan at gmail.com Thu Nov 27 03:30:51 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 27 Nov 2014 12:30:51 +1000 Subject: [Python-ideas] Asynchronous IO ideas for Python In-Reply-To: References: Message-ID: On 27 November 2014 at 03:35, Paul Colomiets wrote: > Hi, > > I've written an article about how I perceive the future of > asynchronous I/O in Python. It's not something that should directly be > incorporated into python now, but I believe it's useful for > python-ideas list. > > https://medium.com/@paulcolomiets/the-future-of-asynchronous-io-in-python-ce200536d847 Thanks Paul, that's an interesting write-up. Another couple of potentially relevant projects in the higher level "service discovery" space (beyond Zookeeper, which you already mention) are Fedora's fedmsg (http://www.fedmsg.com/en/latest/ - since also adopted by Debian I believe) and the Zato ESB project (https://zato.io/). Those are the kinds of things a service discovery plugin system would want to be able to handle. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Thu Nov 27 03:34:04 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 26 Nov 2014 21:34:04 -0500 Subject: [Python-ideas] Asynchronous IO ideas for Python In-Reply-To: References: Message-ID: <23D1E7AE-8C1D-4EB1-B11E-71A5D85B155B@stufft.io> > On Nov 26, 2014, at 9:30 PM, Nick Coghlan wrote: > > On 27 November 2014 at 03:35, Paul Colomiets wrote: >> Hi, >> >> I've written an article about how I perceive the future of >> asynchronous I/O in Python. It's not something that should directly be >> incorporated into python now, but I believe it's useful for >> python-ideas list. >> >> https://medium.com/@paulcolomiets/the-future-of-asynchronous-io-in-python-ce200536d847 > > Thanks Paul, that's an interesting write-up. > > Another couple of potentially relevant projects in the higher level > "service discovery" space (beyond Zookeeper, which you already > mention) are Fedora's fedmsg (http://www.fedmsg.com/en/latest/ - since > also adopted by Debian I believe) and the Zato ESB project > (https://zato.io/). Those are the kinds of things a service discovery > plugin system would want to be able to handle. > We?re using consul for service discovery in psf-salt FWIW. It?s been pretty good for us so far. It implements service discovery along with health checks so that if a service starts failing health checks it gets kicked out of the service discovery rotation for that service. It also implements a DNS resolver so you can use DNS to discover instead of it?s API if you want. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From ncoghlan at gmail.com Thu Nov 27 04:08:42 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 27 Nov 2014 13:08:42 +1000 Subject: [Python-ideas] Extending traceback to (optionally) format and show locals. In-Reply-To: References: Message-ID: On 27 November 2014 at 09:45, Robert Collins wrote: > For context please see http://bugs.python.org/issue22937 and > http://bugs.python.org/issue22936. Another useful bit of context is the current RFE to have a way to extract and save a traceback *summary* that omits the local variable details: http://bugs.python.org/issue17911 Our inclination to resolve that one was to design a new higher level traceback manipulation API, which seems relevant to this proposal as well. > I have two questions I'm hoping to get answered through this thread: > - does the change in question need a PEP? Antoine seemed to think it > didn't, as long as it was opt-in (via the unittest CLI etc) I agree this isn't a PEP level change, just a normal RFE. That said, if the API design details get too confusing, a PEP may still end up being a useful way of clarifying specific details. > - Is my implementation approach sound (for traceback, unittest I > think I have covered :))? > > Implementation wise, I think its useful to work in the current > traceback module layout - that is to alter extract_stack to > (optionally) include rendered data about locals and then look for that > and format it in format_list. For 17911, we're not so sure about that - there's a strong case to be made for exposing a new object-oriented API, rather than continuing with the C-style "records + functions that work on them" model that the traceback module currently uses. We made similar additions over the last couple of releases for both inspect (via inspect.Signature & inspect.Parameter) and dis (via dis.ByteCode & dis.Instruction). > I'm sure there is code out there that depends on the quadruple nature > of extract_stack though, so I think we need to preserve that. Three > strategies occured to me; one is to have parallel functions, one > quadruple, one quintuple. A second one is to have the return value of > extract_stack be a quintuple when a new keyword parameter > include_locals is included. Lastly, and this is my preferred one, if > we return a tuple subclass with an attribute containing a dict with > the rendered data on the locals; this can be present but None, or even > just absent when extract_stack was not asked to include locals. Fourth, expand the new ExceptionSummary API proposed in 17911 to also include the ability to *optionally* preserve the local variable data from the stack frame, rather than always discarding it. > The last option is my preferred one because the other two both imply > having a data structure which is likely to break existing code - and > while you'd have to opt into having them it seems likely to require a > systematic set of upgrades vs having an optional attribute that can be > looked up. > > So - thoughts? I don't see a lot of value in adhering too strictly to the current API design model - I think it would make more sense to design a new higher level API, and then look at including some elements to make it easy to adopt the new tools in existing code without having to rewrite the world (e.g. the inspect module work in Python 3.4 that switched the legacy APIs to actually call the new inspect.signature API internally, greatly expanding the scope of the types they could handle). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From trent at snakebite.org Thu Nov 27 02:18:05 2014 From: trent at snakebite.org (Trent Nelson) Date: Wed, 26 Nov 2014 17:18:05 -0800 Subject: [Python-ideas] Asynchronous IO ideas for Python In-Reply-To: References: <5E3A89398864414E8EE2CB192E0C965C19A3508EDB@EXMBX10.exchhosting.com> Message-ID: <5E3A89398864414E8EE2CB192E0C965C19A3508EE1@EXMBX10.exchhosting.com> Sure can! I dumped the entire contents of my PyParallel source repository (including the full build .exes/dlls, Visual Studio files, etc) to here about an hour before I gave that presentation: https://github.com/pyparallel/release So to replicate that test you'd clone that, then run one of the helpers, python33-http-server.bat or pyparallel-http-server.bat. (There's also http://trent.snakebite.net/pyparallel-0.1-3.3.5.zip, but it's 117MB, so... I'd recommend github.com.) The Python 3.3 version is literally `python -m http.server`; the only change I made to the stdlib http/server.py is this: % diff -u cpython.hg/Lib/http/server.py pyparallel-lib-http-server.py --- cpython.hg/Lib/http/server.py 2014-08-14 16:00:29.000000000 -0400 +++ pyparallel-lib-http-server.py 2014-11-26 18:12:58.000000000 -0500 @@ -328,7 +328,7 @@ conntype = self.headers.get('Connection', "") if conntype.lower() == 'close': self.close_connection = 1 - elif (conntype.lower() == 'keep-alive' and + elif (conntype.lower() == 'keep-alive' or self.protocol_version >= "HTTP/1.1"): self.close_connection = 0 # Examine the headers and look for an Expect directive @@ -440,7 +440,7 @@ version and the current date. """ - self.log_request(code) + #self.log_request(code) self.send_response_only(code, message) self.send_header('Server', self.version_string()) self.send_header('Date', self.date_time_string()) The PyParallel version is basically `python_d -m async.http.server`, which is this: https://github.com/pyparallel/release/blob/master/Lib/async/http/server.py#L251 I started with the stdlib version and mostly refactored it a bit for personal style reasons, then made it work for PyParallel. The only PyParallel-specific piece is actually this line: https://github.com/pyparallel/release/blob/master/Lib/async/http/server.py#L437 return response.transport.sendfile(before, path, None) Everything else is just normal Python, nothing special -- it just conforms to the current constraints of PyParallel. Basically, the HttpServer.data_received() method will be invoked from parallel threads, not the main interpreter thread. To give you an idea how the protocol/transport stuff is wired up, the standalone launcher stuff is at the bottom of that file: ipaddr = socket.gethostbyname(socket.gethostname()) server = async.server(ipaddr, 8080) async.register(transport=server, protocol=HttpServer) async.run() As for why I haven't publicized this stuff until now, to quote myself from that video... "It currently crashes, a lot. I know why it's crashing, I just haven't had the time to fix it yet. But hey, it's super fast until it does crash ;-)" By crash, I mean I'm hitting an assert() in my code -- it happens after the benchmark runs and has to do with the asynchronous socket disconnect logic. I tried fixing it properly before giving that talk, but ran out of time (https://bitbucket.org/tpn/pyparallel/branch/3.3-px-pygotham-2014-sprint). I'll fix all of that the next sprint... which will be... heh, hopefully around Christmas? Oh, actually, the big takeaway from the PyGotham sprint was that I spent an evening re-applying all the wild commit and hackery I'd accumulated to a branch created from the 3.3.5 tag: https://bitbucket.org/tpn/pyparallel/commits/branch/3.3-px. So diff that against the 3.3.5 tag to get an idea of what interpreter changes I needed to make to get to this point. (I have no idea why I didn't pick a tag to work off when I first started -- I literally just started hacking on whatever my local tip was on, which was some indeterminate state between... 3.2 and 3.3?) Side note: I'm really happy with how everything has worked out so far, it is exactly how I envisioned it way back in those python-ideas@ discussions that resulted in tulip/asyncio. I was seeing ridiculously good scaling on my beefier machine at home (8 core, running native) -- to the point where I was maxing out the client machine at about 50,000 requests/sec (~100MB/s) and the PyParallel box was only at about 40% CPU use. Oh, and it appears to be much faster than node.js's http-server too (`npm install http-server`, cd into the website directory, `http-server -s .` to get an equivalent HTTP server from node.js), which I thought was cute. Well, I expected it to be, that's the whole point of being able to exploit all cores and not doing single threaded multiplexing -- so it was good to see that being the case. Node wasn't actually that much faster than Python's normal http.server if I remember correctly. It definitely used less CPU overall than the Python one -- basically what I'm seeing is that Python will be maxing out one core, which should only be 25% CPU (4 core VM), but actual CPU use is up around 50%, and it's mostly kernel time making up the other half. Node will also max out a core, but overall CPU use is ~30%. I attribute this to Python's http.server using select(), whereas I believe node.js ends up using IOCP in a single-threaded event loop. So, you could expect Python asyncio to get similar performance to node, but they're both crushed by PyParallel (until it crashes, heh) as soon as you've got more than one core, which was the point I've been vehemently making from day one ;-) And I just realized I'm writing this e-mail on the same laptop that did that demo, so I can actually back all of this up with a quick run now. Python 3.3 On Windows: C:\Users\Trent\src\pyparallel-0.1-3.3.5 ? python33-http-server.bat Serving HTTP on 0.0.0.0 port 8000 ... On Mac: (trent at raptor:ttys003) (Wed/19:06) .. (~s/wrk) % ./wrk -c 8 -t 2 -d 10 --latency http://192.168.46.131:8000/index.html Running 10s test @ http://192.168.46.131:8000/index.html 2 threads and 8 connections Thread Stats Avg Stdev Max +/- Stdev Latency 6.33ms 1.74ms 18.42ms 75.65% Req/Sec 419.77 119.93 846.00 67.43% Latency Distribution 50% 6.26ms 75% 7.15ms 90% 8.21ms 99% 12.42ms 8100 requests in 10.00s, 53.48MB read Requests/sec: 809.92 Transfer/sec: 5.35MB Node.js On Windows: C:\Users\Trent\src\pyparallel-0.1-3.3.5\website ? http-server -s . On Mac: (trent at raptor:ttys003) (Wed/19:07) .. (~s/wrk) % ./wrk -c 8 -t 2 -d 10 --latency http://192.168.46.131:8080/index.html Running 10s test @ http://192.168.46.131:8080/index.html 2 threads and 8 connections Thread Stats Avg Stdev Max +/- Stdev Latency 6.44ms 2.40ms 19.70ms 84.77% Req/Sec 621.94 124.26 0.94k 68.05% Latency Distribution 50% 5.93ms 75% 7.00ms 90% 8.97ms 99% 16.17ms 12021 requests in 10.00s, 80.84MB read Requests/sec: 1201.98 Transfer/sec: 8.08MB PyParallel On Windows: C:\Users\Trent\src\pyparallel-0.1-3.3.5 ? pyparallel-http-server.bat Serving HTTP on 192.168.46.131 port 8080 ... Traceback (most recent call last): File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\runpy.py", line 160, in _run_module_as_main "__main__", fname, loader, pkg_name) File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\runpy.py", line 73, in _run_code exec(code, run_globals) File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\ctk\cli.py", line 518, in cli = run(*args) File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\ctk\cli.py", line 488, in run return CLI(*args, **kwds) File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\ctk\cli.py", line 272, in __init__ self.run() File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\ctk\cli.py", line 278, in run self._process_commandline() File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\ctk\cli.py", line 424, in _process_commandline cl.run(args) File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\ctk\cli.py", line 217, in run self.command.start() File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\ctk\command.py", line 455, in start self.run() File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\px\commands.py", line 90, in run async.run() OSError: [WinError 8] Not enough storage is available to process this command _PyParallel_Finalize(): px->contexts_active: 462 [92105 refs] _PyParallel_DeletingThreadState(): px->contexts_active: 462 Oh dear :-) Hadn't seen that before. The VM has 4GB allocated to it... I checked taskmgr and it was reporting ~90% physical memory use. Closed a bunch of things and got it down to 54%, then re-ran, that did the trick. Including this info in case anyone else runs into this. Re-run: C:\Users\Trent\src\pyparallel-0.1-3.3.5 ? pyparallel-http-server.bat Serving HTTP on 192.168.46.131 port 8080 ... On Mac: (trent at raptor:ttys003) (Wed/19:16) .. (~s/wrk) % ./wrk -c 8 -t 2 -d 10 --latency http://192.168.46.131:8080/index.html Running 10s test @ http://192.168.46.131:8080/index.html 2 threads and 8 connections Thread Stats Avg Stdev Max +/- Stdev Latency 4.04ms 1.80ms 23.35ms 91.16% Req/Sec 1.07k 191.81 1.54k 75.00% Latency Distribution 50% 3.68ms 75% 4.41ms 90% 5.40ms 99% 13.04ms 20317 requests in 10.00s, 134.22MB read Requests/sec: 2031.33 Transfer/sec: 13.42MB And then back on Windows after the benchmark completes: C:\Users\Trent\src\pyparallel-0.1-3.3.5 ? pyparallel-http-server.bat Serving HTTP on 192.168.46.131 port 8080 ... Assertion failed: s->io_op == PxSocket_IO_SEND, file ..\Python\pyparallel.c, line 6311 Assertion failed: s->io_op == PxSocket_IO_SEND, file ..\Python\pyparallel.c, line 6311 Assertion failed: s->io_op == PxSocket_IO_SEND, file ..\Python\pyparallel.c, line 6311 Assertion failed: s->io_op == PxSocket_IO_SEND, file ..\Python\pyparallel.c, line 6311 Assertion failed: s->io_op == PxSocket_IO_SEND, file ..\Python\pyparallel.c, line 6311 Assertion failed: s->io_op == PxSocket_IO_SEND, file ..\Python\pyparallel.c, line 6311 Heh. That's the crashing I was referring to. So basically, it's better in every category (lowest latency, lowest jitter (stddev), highest throughput) for the duration of the benchmark, then crashes :-) (Basically, my DisconnectEx assumptions regarding overlapped sockets, socket resource reuse, I/O completion ports, and thread pools were... not correct apparently.) I remain committed to the assertion that the Windows' kernel approach to asynchronous I/O via I/O completion ports is fundamentally superior to the UNIX/POSIX approach in every aspect if you want to optimally use contemporary multicore hardware (exploit all cores as efficiently as possible). I talk about this in more detail here: https://speakerdeck.com/trent/parallelism-and-concurrency-with-python?slide=26. But good grief, it is orders of magnitude more complex at every level. A less stubborn version of me would have given up waaaay earlier. Glad I stuck with it though, really happy with results so far. Trent. From: gvanrossum at gmail.com [mailto:gvanrossum at gmail.com] On Behalf Of Guido van Rossum Sent: Wednesday, November 26, 2014 4:49 PM To: Trent Nelson Cc: Paul Colomiets; python-ideas Subject: Re: [Python-ideas] Asynchronous IO ideas for Python Trent, Can you post source for the regular and pyparallel HTTP servers you used? On Wed, Nov 26, 2014 at 12:56 PM, Trent Nelson > wrote: Relevant part of the video with the normal Python stats on the left and PyParallel on the right: https://www.youtube.com/watch?v=4L4Ww3ROuro#t=838 Transcribed stats: Regular Python HTTP server: Thread Stats Avg Stdev Max Latency 4.93ms 714us 10ms Req/Seq 552 154 1.1k 10,480 requests in 10s, 69MB 1048 reqs/sec, 6.9MB/s PyParallel (4 core Windows VM): Thread Stats Avg Stdev Max Latency 2.41ms 531us 10ms Req/Seq 1.74k 183 2.33k 32,831 requests in 10s, 216MB 3263 reqs/sec, 21MB/s So basically a bit less than linear scaling with more cores, which isn't too bad for a full debug build running on a VM. -----Original Message----- From: Trent Nelson Sent: Wednesday, November 26, 2014 3:36 PM To: 'Paul Colomiets'; python-ideas Subject: RE: [Python-ideas] Asynchronous IO ideas for Python Have you seen this?: https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploited-all-cores I spend the first 80-ish slides on async I/O. (That was a year ago. I've done 2-3 sprints on it since then and have gotten it to a point where I can back up the claims with hard numbers on load testing benchmarks, demonstrated in the most recent video: https://www.youtube.com/watch?v=4L4Ww3ROuro.) Trent. -----Original Message----- From: Python-ideas [mailto:python-ideas-bounces+trent=snakebite.org at python.org] On Behalf Of Paul Colomiets Sent: Wednesday, November 26, 2014 12:35 PM To: python-ideas Subject: [Python-ideas] Asynchronous IO ideas for Python Hi, I've written an article about how I perceive the future of asynchronous I/O in Python. It's not something that should directly be incorporated into python now, but I believe it's useful for python-ideas list. https://medium.com/@paulcolomiets/the-future-of-asynchronous-io-in-python-ce200536d847 And a place for comments at Hacker News: https://news.ycombinator.com/item?id=8662782 I hope being helpful with this writeup :) -- Paul _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefano.borini at ferrara.linux.it Thu Nov 27 13:09:34 2014 From: stefano.borini at ferrara.linux.it (Stefano Borini) Date: Thu, 27 Nov 2014 13:09:34 +0100 Subject: [Python-ideas] Fast context creation Message-ID: <20141127120934.GA2433@ferrara.linux.it> I have a situation where, no matter how the routine ends, I need to return the focus to a widget. It would be rather clever to do this with a context, but I would expect something like this to be a possible strategy with contextlib.context(enter=None, exit=lambda *args: my_widget.setFocus()): do what I need to do as far as I know, at the moment it's not possible. Am I right? I think it would be an easy and practical addition to the contextlib module to quickly register two routines for enter and exit. From rosuav at gmail.com Thu Nov 27 13:30:40 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 27 Nov 2014 23:30:40 +1100 Subject: [Python-ideas] Fast context creation In-Reply-To: <20141127120934.GA2433@ferrara.linux.it> References: <20141127120934.GA2433@ferrara.linux.it> Message-ID: On Thu, Nov 27, 2014 at 11:09 PM, Stefano Borini wrote: > I have a situation where, no matter how the routine ends, I need to return the focus to a widget. It would be > rather clever to do this with a context, but I would expect something like this to be a possible strategy > > with contextlib.context(enter=None, exit=lambda *args: my_widget.setFocus()): > do what I need to do > > as far as I know, at the moment it's not possible. Am I right? I think it would be an easy and practical > addition to the contextlib module to quickly register two routines for enter and exit. You could abuse try/finally for this purpose. try: whatever you need to do finally: my_widget.setFocus() ChrisA From 4kir4.1i at gmail.com Thu Nov 27 14:04:21 2014 From: 4kir4.1i at gmail.com (Akira Li) Date: Thu, 27 Nov 2014 16:04:21 +0300 Subject: [Python-ideas] Fast context creation References: <20141127120934.GA2433@ferrara.linux.it> Message-ID: <87zjbcztey.fsf@gmail.com> Stefano Borini writes: > I have a situation where, no matter how the routine ends, I need to return the focus to a widget. It would be > rather clever to do this with a context, but I would expect something like this to be a possible strategy > > with contextlib.context(enter=None, exit=lambda *args: my_widget.setFocus()): > do what I need to do > > as far as I know, at the moment it's not possible. Am I right? I think it would be an easy and practical > addition to the contextlib module to quickly register two routines for enter and exit. with ExitStack() as stack: stack.callback(my_widget.setFocus) cm = stack.enter_context(run_your_enter_routine()) https://docs.python.org/3/library/contextlib.html#contextlib.ExitStack -- Akira From apalala at gmail.com Thu Nov 27 14:16:02 2014 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Thu, 27 Nov 2014 08:46:02 -0430 Subject: [Python-ideas] Fast context creation In-Reply-To: References: <20141127120934.GA2433@ferrara.linux.it> Message-ID: On Thu, Nov 27, 2014 at 8:00 AM, Chris Angelico wrote: > You could abuse try/finally for this purpose. > > try: > whatever you need to do > finally: > my_widget.setFocus() > There's no abuse there. "finally" is there precisely for that sort of intention. I think that what the OP wants is to "announce" the final action at the -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Thu Nov 27 14:17:34 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 27 Nov 2014 14:17:34 +0100 Subject: [Python-ideas] Fast context creation References: <20141127120934.GA2433@ferrara.linux.it> Message-ID: <20141127141734.4162ae1f@fsol> On Thu, 27 Nov 2014 23:30:40 +1100 Chris Angelico wrote: > On Thu, Nov 27, 2014 at 11:09 PM, Stefano Borini > wrote: > > I have a situation where, no matter how the routine ends, I need to return the focus to a widget. It would be > > rather clever to do this with a context, but I would expect something like this to be a possible strategy > > > > with contextlib.context(enter=None, exit=lambda *args: my_widget.setFocus()): > > do what I need to do > > > > as far as I know, at the moment it's not possible. Am I right? I think it would be an easy and practical > > addition to the contextlib module to quickly register two routines for enter and exit. > > You could abuse try/finally for this purpose. Or just use it, since it's a normal use of try/finally ;-) Regards Antoine. > > try: > whatever you need to do > finally: > my_widget.setFocus() > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From apalala at gmail.com Thu Nov 27 14:18:39 2014 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Thu, 27 Nov 2014 08:48:39 -0430 Subject: [Python-ideas] Fast context creation In-Reply-To: <20141127120934.GA2433@ferrara.linux.it> References: <20141127120934.GA2433@ferrara.linux.it> Message-ID: On Thu, Nov 27, 2014 at 7:39 AM, Stefano Borini < stefano.borini at ferrara.linux.it> wrote: > > with contextlib.context(enter=None, exit=lambda *args: > my_widget.setFocus()): > do what I need to do > You can easily define your own context manager to do what you want: @contextmanager def finally_focus(widget): try: yield finally: widget.setFocus() with finally_focus(my_widget): # do what I need to do -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Thu Nov 27 14:50:19 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 28 Nov 2014 00:50:19 +1100 Subject: [Python-ideas] Fast context creation In-Reply-To: <20141127141734.4162ae1f@fsol> References: <20141127120934.GA2433@ferrara.linux.it> <20141127141734.4162ae1f@fsol> Message-ID: On Fri, Nov 28, 2014 at 12:17 AM, Antoine Pitrou wrote: >> You could abuse try/finally for this purpose. > > Or just use it, since it's a normal use of try/finally ;-) > I guess so. I normally think of try/finally as an implication that an exception is expected (rather than a function-level "atexit" type of thing), but that's what's happening here. So yes, take off the "ab" and just use try/finally. ChrisA From stefano.borini at ferrara.linux.it Thu Nov 27 15:24:13 2014 From: stefano.borini at ferrara.linux.it (Stefano Borini) Date: Thu, 27 Nov 2014 15:24:13 +0100 Subject: [Python-ideas] Fast context creation In-Reply-To: References: <20141127120934.GA2433@ferrara.linux.it> Message-ID: <20141127142413.GA5579@ferrara.linux.it> On Thu, Nov 27, 2014 at 11:30:40PM +1100, Chris Angelico wrote: > You could abuse try/finally for this purpose. > > try: > whatever you need to do > finally: > my_widget.setFocus() I thought about this solution, but I am concerned about communication of intent. Using the try communicates to a code reader that the code inside is expected to throw, which is not the case. However, I agree there are good alternative strategies. From ncoghlan at gmail.com Thu Nov 27 15:29:54 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 28 Nov 2014 00:29:54 +1000 Subject: [Python-ideas] Fast context creation In-Reply-To: <87zjbcztey.fsf@gmail.com> References: <20141127120934.GA2433@ferrara.linux.it> <87zjbcztey.fsf@gmail.com> Message-ID: On 27 November 2014 at 23:04, Akira Li <4kir4.1i at gmail.com> wrote: > Stefano Borini > writes: > >> I have a situation where, no matter how the routine ends, I need to return the focus to a widget. It would be >> rather clever to do this with a context, but I would expect something like this to be a possible strategy >> >> with contextlib.context(enter=None, exit=lambda *args: my_widget.setFocus()): >> do what I need to do >> >> as far as I know, at the moment it's not possible. Am I right? I think it would be an easy and practical >> addition to the contextlib module to quickly register two routines for enter and exit. > > > with ExitStack() as stack: > stack.callback(my_widget.setFocus) > cm = stack.enter_context(run_your_enter_routine()) > > https://docs.python.org/3/library/contextlib.html#contextlib.ExitStack Right, while the underlying try/except/finally construct is generally a better option for simple cases of inline ad hoc exception handling, ExitStack is useful when you really do want to reproduce the context management protocols "pass the exception details to a callback" behaviour. For example, the feature requested by the OP can be implemented as: @contextmanager def context(enter=None, exit=None): enter_result = enter() if enter is not None else None with ExitStack() as stack: if exit is not None: stack.push(exit) yield enter_result However, I'm hard pressed to think of a case where using such a construct would be clearer than writing out a suitable try/except/finally block. A lot of the value of context managers lies in our ability to give them *names*, such that it's immediately clear what they're doing (or which docs to look up to find out more). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From clint.hepner at gmail.com Thu Nov 27 17:01:09 2014 From: clint.hepner at gmail.com (Clint Hepner) Date: Thu, 27 Nov 2014 11:01:09 -0500 Subject: [Python-ideas] Fast context creation In-Reply-To: <20141127142413.GA5579@ferrara.linux.it> References: <20141127120934.GA2433@ferrara.linux.it> <20141127142413.GA5579@ferrara.linux.it> Message-ID: On Thu, Nov 27, 2014 at 9:24 AM, Stefano Borini < stefano.borini at ferrara.linux.it> wrote: > On Thu, Nov 27, 2014 at 11:30:40PM +1100, Chris Angelico wrote: > > You could abuse try/finally for this purpose. > > > > try: > > whatever you need to do > > finally: > > my_widget.setFocus() > > I thought about this solution, but I am concerned about communication of > intent. Using the try > communicates to a code reader that the code inside is expected to throw, > which is not the case. > > If there is absolutely no possibility of an exception, you don't need any special construct. Just use whatever you need to do my_widget.setFocus() Otherwise, you are tacitly agreeing that an exception *might* occur, so try/finally is not misleading. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cf.natali at gmail.com Thu Nov 27 22:59:45 2014 From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=) Date: Thu, 27 Nov 2014 21:59:45 +0000 Subject: [Python-ideas] Asynchronous IO ideas for Python In-Reply-To: <5E3A89398864414E8EE2CB192E0C965C19A3508EE1@EXMBX10.exchhosting.com> References: <5E3A89398864414E8EE2CB192E0C965C19A3508EDB@EXMBX10.exchhosting.com> <5E3A89398864414E8EE2CB192E0C965C19A3508EE1@EXMBX10.exchhosting.com> Message-ID: 2014-11-27 1:18 GMT+00:00 Trent Nelson : > > Everything else is just normal Python, nothing special -- it just conforms > to the current constraints of PyParallel. Basically, the > HttpServer.data_received() method will be invoked from parallel threads, not > the main interpreter thread. So, still no garbage collection from the threads? > To give you an idea how the protocol/transport stuff is wired up, the > standalone launcher stuff is at the bottom of that file: > > > > ipaddr = socket.gethostbyname(socket.gethostname()) > > server = async.server(ipaddr, 8080) > > async.register(transport=server, protocol=HttpServer) > > async.run() > > > > > > As for why I haven't publicized this stuff until now, to quote myself from > that video... "It currently crashes, a lot. I know why it's crashing, I > just haven't had the time to fix it yet. But hey, it's super fast until it > does crash ;-)" > > > > By crash, I mean I'm hitting an assert() in my code -- it happens after the > benchmark runs and has to do with the asynchronous socket disconnect logic. > I tried fixing it properly before giving that talk, but ran out of time > (https://bitbucket.org/tpn/pyparallel/branch/3.3-px-pygotham-2014-sprint). > > > > I'll fix all of that the next sprint... which will be... heh, hopefully > around Christmas? > > > > Oh, actually, the big takeaway from the PyGotham sprint was that I spent an > evening re-applying all the wild commit and hackery I'd accumulated to a > branch created from the 3.3.5 tag: > https://bitbucket.org/tpn/pyparallel/commits/branch/3.3-px. So diff that > against the 3.3.5 tag to get an idea of what interpreter changes I needed to > make to get to this point. (I have no idea why I didn't pick a tag to work > off when I first started -- I literally just started hacking on whatever my > local tip was on, which was some indeterminate state between... 3.2 and > 3.3?) > > > > Side note: I'm really happy with how everything has worked out so far, it is > exactly how I envisioned it way back in those python-ideas@ discussions that > resulted in tulip/asyncio. I was seeing ridiculously good scaling on my > beefier machine at home (8 core, running native) -- to the point where I was > maxing out the client machine at about 50,000 requests/sec (~100MB/s) and > the PyParallel box was only at about 40% CPU use. > > > > Oh, and it appears to be much faster than node.js's http-server too (`npm > install http-server`, cd into the website directory, `http-server -s .` to > get an equivalent HTTP server from node.js), which I thought was cute. > Well, I expected it to be, that's the whole point of being able to exploit > all cores and not doing single threaded multiplexing -- so it was good to > see that being the case. > > > > Node wasn't actually that much faster than Python's normal http.server if I > remember correctly. It definitely used less CPU overall than the Python one > -- basically what I'm seeing is that Python will be maxing out one core, > which should only be 25% CPU (4 core VM), but actual CPU use is up around > 50%, and it's mostly kernel time making up the other half. Node will also > max out a core, but overall CPU use is ~30%. I attribute this to Python's > http.server using select(), whereas I believe node.js ends up using IOCP in > a single-threaded event loop. So, you could expect Python asyncio to get > similar performance to node, but they're both crushed by PyParallel (until > it crashes, heh) as soon as you've got more than one core, which was the > point I've been vehemently making from day one ;-) > > > > And I just realized I'm writing this e-mail on the same laptop that did that > demo, so I can actually back all of this up with a quick run now. > > > > Python 3.3 > > On Windows: > > C:\Users\Trent\src\pyparallel-0.1-3.3.5 > ? python33-http-server.bat > Serving HTTP on 0.0.0.0 port 8000 ... > > On Mac: > > > > (trent at raptor:ttys003) (Wed/19:06) .. (~s/wrk) > > % ./wrk -c 8 -t 2 -d 10 --latency http://192.168.46.131:8000/index.html > > Running 10s test @ http://192.168.46.131:8000/index.html > > 2 threads and 8 connections > > Thread Stats Avg Stdev Max +/- Stdev > > Latency 6.33ms 1.74ms 18.42ms 75.65% > > Req/Sec 419.77 119.93 846.00 67.43% > > Latency Distribution > > 50% 6.26ms > > 75% 7.15ms > > 90% 8.21ms > > 99% 12.42ms > > 8100 requests in 10.00s, 53.48MB read > > Requests/sec: 809.92 > > Transfer/sec: 5.35MB > > > > Node.js > > On Windows: > > > > C:\Users\Trent\src\pyparallel-0.1-3.3.5\website > ? http-server -s . > > > > On Mac: > > (trent at raptor:ttys003) (Wed/19:07) .. (~s/wrk) > > % ./wrk -c 8 -t 2 -d 10 --latency http://192.168.46.131:8080/index.html > > Running 10s test @ http://192.168.46.131:8080/index.html > > 2 threads and 8 connections > > Thread Stats Avg Stdev Max +/- Stdev > > Latency 6.44ms 2.40ms 19.70ms 84.77% > > Req/Sec 621.94 124.26 0.94k 68.05% > > Latency Distribution > > 50% 5.93ms > > 75% 7.00ms > > 90% 8.97ms > > 99% 16.17ms > > 12021 requests in 10.00s, 80.84MB read > > Requests/sec: 1201.98 > > Transfer/sec: 8.08MB > > > > > > PyParallel > > On Windows: > > > > C:\Users\Trent\src\pyparallel-0.1-3.3.5 > ? pyparallel-http-server.bat > Serving HTTP on 192.168.46.131 port 8080 ... > Traceback (most recent call last): > File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\runpy.py", line 160, in > _run_module_as_main > "__main__", fname, loader, pkg_name) > File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\runpy.py", line 73, in > _run_code > exec(code, run_globals) > File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\ctk\cli.py", line 518, > in > cli = run(*args) > File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\ctk\cli.py", line 488, > in run > return CLI(*args, **kwds) > File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\ctk\cli.py", line 272, > in __init__ > self.run() > File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\ctk\cli.py", line 278, > in run > self._process_commandline() > File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\ctk\cli.py", line 424, > in _process_commandline > cl.run(args) > File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\ctk\cli.py", line 217, > in run > self.command.start() > File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\ctk\command.py", line > 455, in start > self.run() > File "C:\Users\Trent\src\pyparallel-0.1-3.3.5\Lib\px\commands.py", line > 90, in run > async.run() > OSError: [WinError 8] Not enough storage is available to process this > command > _PyParallel_Finalize(): px->contexts_active: 462 > [92105 refs] > _PyParallel_DeletingThreadState(): px->contexts_active: 462 > > > > Oh dear :-) Hadn't seen that before. The VM has 4GB allocated to it... I > checked taskmgr and it was reporting ~90% physical memory use. Closed a > bunch of things and got it down to 54%, then re-ran, that did the trick. > Including this info in case anyone else runs into this. > > > > Re-run: > > C:\Users\Trent\src\pyparallel-0.1-3.3.5 > ? pyparallel-http-server.bat > Serving HTTP on 192.168.46.131 port 8080 ... > > > > On Mac: > > (trent at raptor:ttys003) (Wed/19:16) .. (~s/wrk) > > % ./wrk -c 8 -t 2 -d 10 --latency http://192.168.46.131:8080/index.html > > Running 10s test @ http://192.168.46.131:8080/index.html > > 2 threads and 8 connections > > Thread Stats Avg Stdev Max +/- Stdev > > Latency 4.04ms 1.80ms 23.35ms 91.16% > > Req/Sec 1.07k 191.81 1.54k 75.00% > > Latency Distribution > > 50% 3.68ms > > 75% 4.41ms > > 90% 5.40ms > > 99% 13.04ms > > 20317 requests in 10.00s, 134.22MB read > > Requests/sec: 2031.33 > > Transfer/sec: 13.42MB > > > > And then back on Windows after the benchmark completes: > > > > C:\Users\Trent\src\pyparallel-0.1-3.3.5 > ? pyparallel-http-server.bat > Serving HTTP on 192.168.46.131 port 8080 ... > Assertion failed: s->io_op == PxSocket_IO_SEND, file ..\Python\pyparallel.c, > line 6311 > Assertion failed: s->io_op == PxSocket_IO_SEND, file ..\Python\pyparallel.c, > line 6311 > Assertion failed: s->io_op == PxSocket_IO_SEND, file ..\Python\pyparallel.c, > line 6311 > Assertion failed: s->io_op == PxSocket_IO_SEND, file ..\Python\pyparallel.c, > line 6311 > Assertion failed: s->io_op == PxSocket_IO_SEND, file ..\Python\pyparallel.c, > line 6311 > Assertion failed: s->io_op == PxSocket_IO_SEND, file ..\Python\pyparallel.c, > line 6311 > > Heh. That's the crashing I was referring to. > > > > So basically, it's better in every category (lowest latency, lowest jitter > (stddev), highest throughput) for the duration of the benchmark, then > crashes :-) > > > > (Basically, my DisconnectEx assumptions regarding overlapped sockets, socket > resource reuse, I/O completion ports, and thread pools were... not correct > apparently.) > > > > I remain committed to the assertion that the Windows' kernel approach to > asynchronous I/O via I/O completion ports is fundamentally superior to the > UNIX/POSIX approach in every aspect if you want to optimally use > contemporary multicore hardware (exploit all cores as efficiently as > possible). I talk about this in more detail here: > https://speakerdeck.com/trent/parallelism-and-concurrency-with-python?slide=26. > > > > But good grief, it is orders of magnitude more complex at every level. A > less stubborn version of me would have given up waaaay earlier. Glad I > stuck with it though, really happy with results so far. > > > > Trent. > > > > > > > > From: gvanrossum at gmail.com [mailto:gvanrossum at gmail.com] On Behalf Of Guido > van Rossum > Sent: Wednesday, November 26, 2014 4:49 PM > To: Trent Nelson > Cc: Paul Colomiets; python-ideas > Subject: Re: [Python-ideas] Asynchronous IO ideas for Python > > > > Trent, > > Can you post source for the regular and pyparallel HTTP servers you used? > > > > On Wed, Nov 26, 2014 at 12:56 PM, Trent Nelson wrote: > > Relevant part of the video with the normal Python stats on the left and > PyParallel on the right: > > https://www.youtube.com/watch?v=4L4Ww3ROuro#t=838 > > Transcribed stats: > > Regular Python HTTP server: > > Thread Stats Avg Stdev Max > Latency 4.93ms 714us 10ms > Req/Seq 552 154 1.1k > 10,480 requests in 10s, 69MB > 1048 reqs/sec, 6.9MB/s > > PyParallel (4 core Windows VM): > > Thread Stats Avg Stdev Max > Latency 2.41ms 531us 10ms > Req/Seq 1.74k 183 2.33k > 32,831 requests in 10s, 216MB > 3263 reqs/sec, 21MB/s > > So basically a bit less than linear scaling with more cores, which isn't too > bad for a full debug build running on a VM. > > -----Original Message----- > From: Trent Nelson > Sent: Wednesday, November 26, 2014 3:36 PM > To: 'Paul Colomiets'; python-ideas > Subject: RE: [Python-ideas] Asynchronous IO ideas for Python > > Have you seen this?: > > > https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploited-all-cores > > I spend the first 80-ish slides on async I/O. > > (That was a year ago. I've done 2-3 sprints on it since then and have > gotten it to a point where I can back up the claims with hard numbers on > load testing benchmarks, demonstrated in the most recent video: > https://www.youtube.com/watch?v=4L4Ww3ROuro.) > > > Trent. > > > -----Original Message----- > From: Python-ideas > [mailto:python-ideas-bounces+trent=snakebite.org at python.org] On Behalf Of > Paul Colomiets > Sent: Wednesday, November 26, 2014 12:35 PM > To: python-ideas > Subject: [Python-ideas] Asynchronous IO ideas for Python > > Hi, > > I've written an article about how I perceive the future of asynchronous I/O > in Python. It's not something that should directly be incorporated into > python now, but I believe it's useful for python-ideas list. > > https://medium.com/@paulcolomiets/the-future-of-asynchronous-io-in-python-ce200536d847 > > And a place for comments at Hacker News: > > https://news.ycombinator.com/item?id=8662782 > > I hope being helpful with this writeup :) > > -- > Paul > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > -- > > --Guido van Rossum (python.org/~guido) > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From trent at snakebite.org Fri Nov 28 00:11:20 2014 From: trent at snakebite.org (Trent Nelson) Date: Thu, 27 Nov 2014 18:11:20 -0500 Subject: [Python-ideas] Asynchronous IO ideas for Python] Message-ID: <20141127231119.GA7729@snakebite.org> On Thu, Nov 27, 2014 at 09:59:45PM +0000, Charles-Fran?ois Natali wrote: > 2014-11-27 1:18 GMT+00:00 Trent Nelson : > > > > Everything else is just normal Python, nothing special -- it just conforms > > to the current constraints of PyParallel. Basically, the > > HttpServer.data_received() method will be invoked from parallel threads, not > > the main interpreter thread. > > So, still no garbage collection from the threads? Correct. Not having garbage collection has surprisingly not gotten in the way so far, so it's not even on the radar anymore. There are other means available for persisting objects past the lifetime of the parallel context, and you could always do an @async.call_from_main_thread if you want to have the main thread's memory allocator (and thus, GC) kick in. At one point, all these tests passed, just to give you an idea of some of the facilities that are available: https://bitbucket.org/tpn/pyparallel/src/89576c868a3f41747d138a473b090e0f2c6fef61/Lib/async/test/test_primitives.py?at=3.3-px (I haven't removed any of those facilities, I just haven't spent any time on them since switching over to the async socket stuff, so I can't comment on their current state.) Trent. ----- End forwarded message ----- From robertc at robertcollins.net Fri Nov 28 01:48:15 2014 From: robertc at robertcollins.net (Robert Collins) Date: Fri, 28 Nov 2014 13:48:15 +1300 Subject: [Python-ideas] Extending traceback to (optionally) format and show locals. In-Reply-To: References: Message-ID: On 27 November 2014 at 14:12, Andrew Barnert wrote: > On Nov 26, 2014, at 15:45, Robert Collins wrote: >> I'm sure there is code out there that depends on the quadruple nature >> of extract_stack though, so I think we need to preserve that. Three >> strategies occured to me; one is to have parallel functions, one >> quadruple, one quintuple. A second one is to have the return value of >> extract_stack be a quintuple when a new keyword parameter >> include_locals is included. Lastly, and this is my preferred one, if >> we return a tuple subclass with an attribute containing a dict with >> the rendered data on the locals; this can be present but None, or even >> just absent when extract_stack was not asked to include locals. > > There are lots of other cases in the stdlib where something is usable as a tuple of n fields or as a structseq/namedtuple of >n fields: stat results, struct_tm, etc. So, why not do the same thing here? Because backwards compatibility. Moving to a namedtuple is fine - changing the length of the tuple is a problem. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From robertc at robertcollins.net Fri Nov 28 01:54:35 2014 From: robertc at robertcollins.net (Robert Collins) Date: Fri, 28 Nov 2014 13:54:35 +1300 Subject: [Python-ideas] Extending traceback to (optionally) format and show locals. In-Reply-To: References: Message-ID: On 27 November 2014 at 16:08, Nick Coghlan wrote: > On 27 November 2014 at 09:45, Robert Collins wrote: >> For context please see http://bugs.python.org/issue22937 and >> http://bugs.python.org/issue22936. > > Another useful bit of context is the current RFE to have a way to > extract and save a traceback *summary* that omits the local variable > details: http://bugs.python.org/issue17911 AIUI the specific desire is to allow a minimal cost capture of tracebacks without frames (so as to allow gc), with enough data to render a traceback later should the thing turn out to escape the system. So largely skipping linecache lookup etc. I think thats a good thing to do :). > Our inclination to resolve that one was to design a new higher level > traceback manipulation API, which seems relevant to this proposal as > well. .... >> - Is my implementation approach sound (for traceback, unittest I >> think I have covered :))? >> >> Implementation wise, I think its useful to work in the current >> traceback module layout - that is to alter extract_stack to >> (optionally) include rendered data about locals and then look for that >> and format it in format_list. > > For 17911, we're not so sure about that - there's a strong case to be > made for exposing a new object-oriented API, rather than continuing > with the C-style "records + functions that work on them" model that > the traceback module currently uses. There's a nice functional feel there more than a C thing, IMO - in that there is no hidden state, and everything is laid out for direct view. OTOH I've no objection to a more objects-and-methods feel, though we'll want a thunk through to the new code which will essentially just be constructing objects just-in-time. Seems straight forward enough to write though. > Fourth, expand the new ExceptionSummary API proposed in 17911 to also > include the ability to *optionally* preserve the local variable data > from the stack frame, rather than always discarding it. A couple of other related things I should be clear about: I want something I can backport successfully via traceback2, for use in unittest2. I don't see any issue with the proposal so far, other than the linecache API change needed to support __loader__, which implies a linecache2 backport as well. >> The last option is my preferred one because the other two both imply >> having a data structure which is likely to break existing code - and >> while you'd have to opt into having them it seems likely to require a >> systematic set of upgrades vs having an optional attribute that can be >> looked up. >> >> So - thoughts? > > I don't see a lot of value in adhering too strictly to the current API > design model - I think it would make more sense to design a new higher > level API, and then look at including some elements to make it easy to > adopt the new tools in existing code without having to rewrite the > world (e.g. the inspect module work in Python 3.4 that switched the > legacy APIs to actually call the new inspect.signature API internally, > greatly expanding the scope of the types they could handle). Sure. AIUI noone is actively pushing on the new thing, so I'll put my hand up for it now and we'll see where I get to in my available cycles. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From abarnert at yahoo.com Fri Nov 28 03:43:37 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 27 Nov 2014 18:43:37 -0800 Subject: [Python-ideas] Extending traceback to (optionally) format and show locals. In-Reply-To: References: Message-ID: <7F8C6BEE-BB37-4C09-8A81-5053FC747B39@yahoo.com> On Nov 27, 2014, at 16:48, Robert Collins wrote: > On 27 November 2014 at 14:12, Andrew Barnert wrote: >> On Nov 26, 2014, at 15:45, Robert Collins wrote: > >>> I'm sure there is code out there that depends on the quadruple nature >>> of extract_stack though, so I think we need to preserve that. Three >>> strategies occured to me; one is to have parallel functions, one >>> quadruple, one quintuple. A second one is to have the return value of >>> extract_stack be a quintuple when a new keyword parameter >>> include_locals is included. Lastly, and this is my preferred one, if >>> we return a tuple subclass with an attribute containing a dict with >>> the rendered data on the locals; this can be present but None, or even >>> just absent when extract_stack was not asked to include locals. >> >> There are lots of other cases in the stdlib where something is usable as a tuple of n fields or as a structseq/namedtuple of >n fields: stat results, struct_tm, etc. So, why not do the same thing here? > > Because backwards compatibility. Moving to a namedtuple is fine - > changing the length of the tuple is a problem. That's the whole point: you're _not_ changing the length. Again, look at the examples that are already in the stdlib: they have a fixed length as a tuple, with extra fields accessible by name only. And it's dead easy to do with structseq. From liam.marsh.home at gmail.com Fri Nov 28 17:29:32 2014 From: liam.marsh.home at gmail.com (Liam Marsh) Date: Fri, 28 Nov 2014 17:29:32 +0100 Subject: [Python-ideas] improve compatibility Message-ID: hello, the problem is that even with extreme precaution, it is impossible to keep ALL modules compatible from a version to another. what I want to ask is this: -some "packs" which can, like py-compile, generate .pyc files, but using "old" versions of the default library, and of __builtins__. -One will go with every minor version and will be optionnal in the installation -any imported py file will be able to choose which version it wants with the "#! py recommended version X.X" or "#! py mandatory version X.X" commentaries at the begining of the file. thank you and have a nice day/evening/night. -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sat Nov 29 06:07:24 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 28 Nov 2014 21:07:24 -0800 Subject: [Python-ideas] improve compatibility In-Reply-To: References: Message-ID: <00DC45BE-A51D-4EB7-BF90-D4BA2F4B3189@yahoo.com> On Nov 28, 2014, at 8:29, Liam Marsh wrote: > hello, > the problem is that even with extreme precaution, it is impossible to keep ALL modules compatible from a version to another. Do you have a specific library or app that you've had a problem with? There were a handful of modules that had a problem with the 3.2 to 3.3 conversion, but every one I saw was caused by language and implementation changes, not stdlib changes. I don't think I've seen anything that works with 3.3 but not 3.4. I'm sure it's not impossible for such a thing to happen, but it would be helpful to have at least one real-life example. > what I want to ask is this: > -some "packs" which can, like py-compile, generate .pyc files, but using "old" versions of the default library, and of __builtins__. But how would this work? The same changes that broke a handful of third-party modules between 3.2 and 3.3 probably also mean that the 3.2 stdlib wouldn't work in 3.3 without minor changes. And as for builtins, most of those are exposing internals of the implementation, so trying to make the 3.2 builtins work with 3.3 would take a lot more work than just building the 3.2 code against 3.3. > -One will go with every minor version and will be optionnal in the installation > -any imported py file will be able to choose which version it wants with the > "#! py recommended version X.X" or > "#! py mandatory version X.X" commentaries at the begining of the file. > > thank you and have a nice day/evening/night. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From liam.marsh.home at gmail.com Sat Nov 29 16:26:30 2014 From: liam.marsh.home at gmail.com (Liam Marsh) Date: Sat, 29 Nov 2014 16:26:30 +0100 Subject: [Python-ideas] improve compatibility In-Reply-To: References: <00DC45BE-A51D-4EB7-BF90-D4BA2F4B3189@yahoo.com> Message-ID: oops.. I sent an answer to andrew barnert, who have sent the first answer to my question, and not to the list... however , I do have an answer to my question. thank you! 2014-11-29 11:51 GMT+01:00 Andrew Barnert : > On Nov 29, 2014, at 0:46, Liam Marsh wrote: > > hello, > > yes, the bug which gave me this idea is a py3.2 to py3.3 bug with Vpython, > a 3D graphs library. > > > And is the bug a stdlib bug, or a language bug? > > I also thought this will be the reconciliation between py2 and py3. (this > is why I thought the packs will include a version of the stdlibs) > > > The language differences between 2.x and 3.x are huge, and most of the > stdlib differences between 2.6 and 3.1 or 2.7 and 3.2 are related to those > language differences. Porting code to 3.x is primarily about fixing Unicode > stuff, or code that was already deprecated in 2.5 or 2.6 but wasn't broken > until 3.x. Having the 2.7 stdlib in 3.x would be a huge amount of work for > almost no benefit. > > in fact, how do the .pyc work? were them modified by the "language and > implementation changes"? how do them import other modules? > > > .pyc files are just compiled bytecode, with a version-specific header. > If you fixed the 2.7 stdlib to work in 3.x (which, again, would be a huge > amount of work), you could compile it with 3.4. > > But you're missing the fact that large chunks of the stdlib are written in > C, and compiler against the Python C API. And parts of the stdlib > (especially builtins and the sys module) are exposing types and functions > written in C that are part of the core implementation, so the 2.x version > of the sys module wouldn't be compatible with 2.x code anyway. > > Also, did you mean to write just to me instead of to the list? > > > thank you! > > > > 2014-11-29 6:07 GMT+01:00 Andrew Barnert : > >> On Nov 28, 2014, at 8:29, Liam Marsh wrote: >> >> > hello, >> > the problem is that even with extreme precaution, it is impossible to >> keep ALL modules compatible from a version to another. >> >> Do you have a specific library or app that you've had a problem with? >> There were a handful of modules that had a problem with the 3.2 to 3.3 >> conversion, but every one I saw was caused by language and implementation >> changes, not stdlib changes. I don't think I've seen anything that works >> with 3.3 but not 3.4. I'm sure it's not impossible for such a thing to >> happen, but it would be helpful to have at least one real-life example. >> >> > what I want to ask is this: >> > -some "packs" which can, like py-compile, generate .pyc files, but >> using "old" versions of the default library, and of __builtins__. >> >> But how would this work? The same changes that broke a handful of >> third-party modules between 3.2 and 3.3 probably also mean that the 3.2 >> stdlib wouldn't work in 3.3 without minor changes. And as for builtins, >> most of those are exposing internals of the implementation, so trying to >> make the 3.2 builtins work with 3.3 would take a lot more work than just >> building the 3.2 code against 3.3. >> >> > -One will go with every minor version and will be optionnal in the >> installation >> > -any imported py file will be able to choose which version it wants >> with the >> > "#! py recommended version X.X" or >> > "#! py mandatory version X.X" commentaries at the begining of the >> file. >> > >> > thank you and have a nice day/evening/night. >> > _______________________________________________ >> > Python-ideas mailing list >> > Python-ideas at python.org >> > https://mail.python.org/mailman/listinfo/python-ideas >> > Code of Conduct: http://python.org/psf/codeofconduct/ >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Nov 29 17:05:11 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 30 Nov 2014 02:05:11 +1000 Subject: [Python-ideas] improve compatibility In-Reply-To: <00DC45BE-A51D-4EB7-BF90-D4BA2F4B3189@yahoo.com> References: <00DC45BE-A51D-4EB7-BF90-D4BA2F4B3189@yahoo.com> Message-ID: On 29 November 2014 at 15:07, Andrew Barnert wrote: > On Nov 28, 2014, at 8:29, Liam Marsh wrote: > >> hello, >> the problem is that even with extreme precaution, it is impossible to keep ALL modules compatible from a version to another. > > Do you have a specific library or app that you've had a problem with? There were a handful of modules that had a problem with the 3.2 to 3.3 conversion, but every one I saw was caused by language and implementation changes, not stdlib changes. I don't think I've seen anything that works with 3.3 but not 3.4. I'm sure it's not impossible for such a thing to happen, but it would be helpful to have at least one real-life example. The desire for doing selective upgrades on a stable base is actually a pretty common one (e.g. using modern unittest features on Python 2.6 or 2.7). The hard part is figuring out a way to provide those selective upgrades cost effectively. (In the context of unpaid development, the costs are measured in contributor time) >> what I want to ask is this: >> -some "packs" which can, like py-compile, generate .pyc files, but using "old" versions of the default library, and of __builtins__. > > But how would this work? The same changes that broke a handful of third-party modules between 3.2 and 3.3 probably also mean that the 3.2 stdlib wouldn't work in 3.3 without minor changes. And as for builtins, most of those are exposing internals of the implementation, so trying to make the 3.2 builtins work with 3.3 would take a lot more work than just building the 3.2 code against 3.3. The feature Liam is looking for effectively already exists, in the form of updated standard library modules that have been backported to earlier versions via PyPI. There's a (likely incomplete) list at https://wiki.python.org/moin/Python2orPython3#Supporting_Python_2_and_Python_3_in_a_common_code_base. This tackles the problem in the opposite direction, by ensuring particular modules remain compatible with older versions, rather than only running on the latest release. One of our reasons for making pip more readily available to Python 2 users in Python 2.7.9 is to make those modules easier to access (together with Python 3 migration tools like six, modernize, future, and caniusepython3). Cheers, Nick. P.S. While some of those backports are maintained directly by core developers, the PSF license actually allows anyone that wants to (and has the necessary time available) to backport modules. A number of the backports originated in users wanting particular features on earlier versions for their own use, and deciding to make their backport generally available, rather than keeping it to themselves. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From random832 at fastmail.us Sun Nov 30 22:30:24 2014 From: random832 at fastmail.us (random832 at fastmail.us) Date: Sun, 30 Nov 2014 16:30:24 -0500 Subject: [Python-ideas] Extending traceback to (optionally) format and show locals. In-Reply-To: References: Message-ID: <1417383024.1352065.197040833.4FFABE62@webmail.messagingengine.com> On Thu, Nov 27, 2014, at 19:48, Robert Collins wrote: > On 27 November 2014 at 14:12, Andrew Barnert wrote: > > On Nov 26, 2014, at 15:45, Robert Collins wrote: > > >> I'm sure there is code out there that depends on the quadruple nature > >> of extract_stack though, so I think we need to preserve that. Three > >> strategies occured to me; one is to have parallel functions, one > >> quadruple, one quintuple. A second one is to have the return value of > >> extract_stack be a quintuple when a new keyword parameter > >> include_locals is included. Lastly, and this is my preferred one, if > >> we return a tuple subclass with an attribute containing a dict with > >> the rendered data on the locals; this can be present but None, or even > >> just absent when extract_stack was not asked to include locals. > > > > There are lots of other cases in the stdlib where something is usable as a tuple of n fields or as a structseq/namedtuple of >n fields: stat results, struct_tm, etc. So, why not do the same thing here? > > Because backwards compatibility. Moving to a namedtuple is fine - > changing the length of the tuple is a problem. Er, but what is being suggested is to do the same backwards-compatible thing: move to a namedtuple-like object with extra non-tuple fields, just like those others. I'm confused as to what is the conflict here.