Python-ideas November 2014

python-ideas@python.org

50 participants
24 discussions

PEP 479: Change StopIteration handling inside generators
by Chris Angelico 26 Nov '14

26 Nov '14

PEP: 479 Title: Change StopIteration handling inside generators Version: $Revision$ Last-Modified: $Date$ Author: Chris Angelico <rosuav(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 15-Nov-2014 Python-Version: 3.5 Post-History: 15-Nov-2014 Abstract ======== This PEP proposes a semantic change to ``StopIteration`` when raised inside a generator, unifying the behaviour of list comprehensions and generator expressions somewhat. Rationale ========= The interaction of generators and ``StopIteration`` is currently somewhat surprising, and can conceal obscure bugs. An unexpected exception should not result in subtly altered behaviour, but should cause a noisy and easily-debugged traceback. Currently, ``StopIteration`` can be absorbed by the generator construct. Proposal ======== If a ``StopIteration`` is about to bubble out of a generator frame, it is replaced with some other exception (maybe ``RuntimeError``, maybe a new custom ``Exception`` subclass, but *not* deriving from ``StopIteration``) which causes the ``next()`` call (which invoked the generator) to fail, passing that exception out. From then on it's just like any old exception. [3]_ Consequences to existing code ============================= This change will affect existing code that depends on ``StopIteration`` bubbling up. The pure Python reference implementation of ``groupby`` [1]_ currently has comments "Exit on ``StopIteration``" where it is expected that the exception will propagate and then be handled. This will be unusual, but not unknown, and such constructs will fail. (Nick Coghlan comments: """If you wanted to factor out a helper function that terminated the generator you'd have to do "return yield from helper()" rather than just "helper()".""") As this can break code, it is proposed to utilize the ``__future__`` mechanism to introduce this, finally making it standard in Python 3.6 or 3.7. Alternate proposals =================== Supplying a specific exception to raise on return ------------------------------------------------- Nick Coghlan suggested a means of providing a specific ``StopIteration`` instance to the generator; if any other instance of ``StopIteration`` is raised, it is an error, but if that particular one is raised, the generator has properly completed. Making return-triggered StopIterations obvious ---------------------------------------------- For certain situations, a simpler and fully backward-compatible solution may be sufficient: when a generator returns, instead of raising ``StopIteration``, it raises a specific subclass of ``StopIteration`` which can then be detected. If it is not that subclass, it is an escaping exception rather than a return statement. Criticism ========= Unofficial and apocryphal statistics suggest that this is seldom, if ever, a problem. [4]_ Code does exist which relies on the current behaviour, and there is the concern that this would be unnecessary code churn to achieve little or no gain. References ========== .. [1] Initial mailing list comment (https://mail.python.org/pipermail/python-ideas/2014-November/029906.html) .. [2] Pure Python implementation of groupby (https://docs.python.org/3/library/itertools.html#itertools.groupby) .. [3] Proposal by GvR (https://mail.python.org/pipermail/python-ideas/2014-November/029953.html) .. [4] Response by Steven D'Aprano (https://mail.python.org/pipermail/python-ideas/2014-November/029994.html) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

23 132

Counter comparison
by Antony Lee 24 Nov '14

24 Nov '14

As Counter objects effectively behave like multi-sets, it seems reasonable to overload <, <=, >, >= as for set objects to check whether a Counter is a sub/super-set of another Counter: c < d <===> all(c[k] < d[k] for k in c) Thoughts? Antony

11 19

Re: [Python-ideas] Counter comparison
by MRAB 21 Nov '14

21 Nov '14

On 2014-11-22 01:36, Antony Lee wrote: > I guess it depends on whether you think {"a": 1} < {"a": 2, "b": -1} > or not. I actually thought about it and initially decided that the > inequality should hold, but given that most_common works fine with > negative counts, perhaps it shouldn't, indeed. > The other point is that if the 'b' entry was in the first, then it would be checked, and the negative count would matter, but as it wasn't, it won't, and it won't. > 2014-11-21 16:27 GMT-08:00 MRAB <python(a)mrabarnett.plus.com > <mailto:python@mrabarnett.plus.com>>: > > On 2014-11-21 23:55, Antony Lee wrote: > > As Counter objects effectively behave like multi-sets, it seems > reasonable to overload <, <=, >, >= as for set objects to > check whether > a Counter is a sub/super-set of another Counter: > > c < d <===> all(c[k] < d[k] for k in c) > > Thoughts? > > More correctly, it would be: > > all(c[k] < d[k] for k in set(c) | set(d)) >

1 0

Re: [Python-ideas] PEP 479: Change StopIteration handling inside generators
by Chris Barker 21 Nov '14

21 Nov '14

I htink I'm getting closer to clarity here, but: let's say I have this silly helper function (similar to the one in the PEP): ## (sorry -- python2) def helper(x): if x > 2: raise StopIteration else: return x # I call that helper from both an iterator class and a generator function. class Iter(object): def __init__(self): self.x = 0 def __iter__(self): return self def next(self): if self.x == 3: raise StopIteration self.x += 1 return helper(self.x) def gen(): yield helper(1) yield helper(2) yield helper(3) # first just loop through them: for i in Iter(): print i for i in gen(): print i # they act exactly the same # then put them inside generator expression and loop through that: for j in ( i for i in gen() ): print j for j in ( i for i in Iter() ): print j # they still act exactly the same. Would this PEP chance that? would that StopIteration bubble up through the iterator class, but not through the generator function? If so then I'm -1 on the PEP. The only change made by this proposal is that StopIteration becomes, > in a generator, like any other unexpected exception. but from the user's perspective, an iterator and a generator should look the same. And I think from the perspective of the author of a given generator function, they should look as much the same as possible-- i.e. if a StopIteration is raised internally, it will behave the same way as it would in a iterator class. It creates a > separation between "iterator protocol" (which is implemented by > __next__) and "generator protocol" (which is written in the body of a > function with 'yield' in it). > terminology messing me up here -- I understood the "iterator protocol" top mean the __iter__ and __next__ -- and a generator satisfies that protocol. Is there a "generator protocol" that means somethign different? Maybe that's what this whole sub-thread is about: I never thought there was such a thing as a "generator protocol", and I don't see why there should be. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

2 1

Return expressions
by Nick Coghlan 21 Nov '14

21 Nov '14

This idea occurred to me in context of PEP 479 and the "or stop()" hack for generator expressions. I'm not a big enough fan of the idea to pursue it myself, but if anyone is bothered by the prospect of PEP 479 taking the "or stop()" technique away, this may be worth pursuing further. As a reminder of how the hack works, the following generator: >>> def takewhile(iterable, pred): ... for x in iter(iterable): ... if not pred(x): ... return ... yield x ... >>> list(takewhile(range(10), (lambda x: x < 5))) [0, 1, 2, 3, 4] Can currently be converted to a generator expression with the aid of a helper function: >>> def stop(): ... raise StopIteration ... >>> list(x for x in range(10) if x < 5 or stop()) [0, 1, 2, 3, 4] Under PEP 479, that will raise RuntimeError instead. If return was an expression rather than a statement (ala the yield statement -> expression conversion in PEP 342), then the following would be functionally equivalent to the current "or stop()" trick: >>> list(x for x in range(10) if x < 5 or return) [0, 1, 2, 3, 4] This is emphatically *not* executable pseudocode - it only makes sense in terms of the translation to the corresponding full generator function definition. However, the translation remains exact in terms of the normal semantics of the "return" keyword, unlike various other proposals to use the "while" keyword to provide comparable functionality. For comprehensions, the meaning of a bare return would likely need to be tweaked slightly to be the same as reaching the end of the frame: returning the object being constructed by the comprehension, rather than None. >>> [x for x in range(10) if x < 5 or return] [0, 1, 2, 3, 4] >>> {x for x in range(10) if x < 5 or return} {0, 1, 2, 3, 4} >>> {x:x for x in range(10) if x < 5 or return} {0:0, 1:1, 2:2, 3:3, 4:4} Regards, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

13 23

Re: [Python-ideas] Implement `itertools.permutations.__getitem__` and `itertools.permutations.index`
by Ram Rachum 17 Nov '14

17 Nov '14

Hi guys, I just wanted to give an update on this: I just released my own code that does this to PyPI: https://pypi.python.org/pypi/combi Thanks, Ram. On Tue, Jun 10, 2014 at 9:24 AM, Ram Rachum <ram(a)rachum.com> wrote: > I'll email you if/when it's released :) > > > On Tue, Jun 10, 2014 at 9:04 AM, Neil Girdhar <mistersheik(a)gmail.com> > wrote: > >> I really like this and hope that it eventually makes it into the stdlib. >> It's also a good argument for your other suggestion whereby some of the >> itertools to return Iterables rather than Iterators like range does. >> >> Best, >> >> Neil >> >> On Wednesday, May 7, 2014 1:43:20 PM UTC-4, Ram Rachum wrote: >> >>> I'm probably going to implement it in my python_toolbox package. I >>> already implemented 30% and it's really cool. It's at the point where I >>> doubt that I want it in the stdlib because I've gotten so much awesome >>> functionality into it and I'd hate to (a) have 80% of it stripped and (b) >>> have the class names changed to be non-Pythonic :) >>> >>> >>> On Wed, May 7, 2014 at 8:40 PM, Tal Einat <tale...(a)gmail.com> wrote: >>> >>> On Wed, May 7, 2014 at 8:21 PM, Ram Rachum <ram.r...(a)gmail.com> wrote: >>>> > Hi Tal, >>>> > >>>> > I'm using it for a project of my own (optimizing keyboard layout) but >>>> I >>>> > can't make the case that it's useful for the stdlib. I'd understand >>>> if it >>>> > would be omitted for not being enough of a common need. >>>> >>>> At the least, this (a function for getting a specific permutation by >>>> lexicographical-order index) could make a nice cookbook recipe. >>>> >>>> - Tal >>>> >>> >>> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas(a)python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> -- >> >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "python-ideas" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/python-ideas/in2gUQMFUzA/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> python-ideas+unsubscribe(a)googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> >> >

3 9

Re: [Python-ideas] Change how Generator Expressions handle StopIteration
by Chris Angelico 15 Nov '14

15 Nov '14

On Sat, Nov 15, 2014 at 11:36 PM, Skip Montanaro <skip.montanaro(a)gmail.com> wrote: > On Sat, Nov 15, 2014 at 1:06 AM, Chris Angelico <rosuav(a)gmail.com> wrote: >> >> Draft PEP exists, and now has your concerns incorporated. >> >> https://raw.githubusercontent.com/Rosuav/GenStopIter/master/pep-xxx.txt > > > Chris, > > 404 for me... (replying on-list as this will affect everyone now) It's now been allocated an actual number: https://raw.githubusercontent.com/Rosuav/GenStopIter/master/pep-0479.txt ChrisA

1 0

Change how Generator Expressions handle StopIteration
by yotam vaknin 14 Nov '14

14 Nov '14

Hi, I would like to purpose that generator expressions will not catch StopIteration exception, if this exception did not come from the iterated object's __next__ function specifically. So generator expressions will be able to raise StopIteration by calculating the current value of the Generator. Here is an example of a use-case : def izip(*args): iters = [iter(obj) for obj in args] while True: yield tuple(next(it) for it in iters) a = izip([1,2],[3,4]) print(next(a),next(a),next(a)) #Currently prints : (1, 3) (2, 4) () list(izip([1,2],[3,4])) #Currently never returns Even thought this is the PEP described behaviour, I think this is an unwanted behaviour. I think Generator Expressions should work like List Comprehension in that sense: def iizip(*args): iters = [iter(obj) for obj in args] while True: yield tuple([next(it) for it in iters]) tuple(iizip([1,2],[3,4])) #Returns [(1, 3), (2, 4)]

17 56

Python IDE (New IDE for Python scripting)
by Danilo Peixoto 14 Nov '14

14 Nov '14

An alternative IDE with a new dark and modern interface. http://s27.postimg.org/r4cjxf00z/Python_IDE.png (Concept) Concept shows the Python IDE functionality to organize projects and files in own software and quick information on the status bar. Some additional features: AutocompleteFind and replaceIncrease and decrease line indentand more...

7 7

A way to tell the gc it's OK to delete an object with a __del__
by Chris Barker 14 Nov '14

14 Nov '14

I've been poking into reference counting, circular references, etc, trying to understand how the garbage collector works. In practice, I'm trying to figure out a way to get a couple modules I work with not to leak memory: one is an extension to wxPython that I wrote, that creates a circular reference involving wx objects -- these, as is often the case with wrappers, have a custom __del__ method. The other is the python netcdf4 lib -- which has a circular reference built in, so that Datasets (which map to a file on disk) know about Variables (which map to data arrays within the file) and vice versa. We tried using weak references here, but it turns out that some users like to work with Variables without a reference to the Dataset, so that didn't work. Anyway, in my poking around, I think I understand that the gc won't delete "unreachable" objects if the have a custom __del__ method, because __del__ methods often handle resources that may depend on other objects, and bad (really bad) things can happen if they are deleted out of order, and the gc has no idea what order to delete. However -- some __del__ methods are perfectly safe regardless of delete order. So it would be nice to be able to somehow let the gc know that a particular object is safe to delete at any time. Is there a technical reason this can't be done? It seems something as simple as a __delete_in_any_order__ attribute would do it. Yes, this is dangerous, but authors of such extension objects need to know what they are doing anyway. Is there a technical complication I'm not thinking of? It seems if we could pull this off, we could eliminate one lasting ugly wart in python memory management. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

6 10

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Python-ideas November 2014