I just finished debugging some code that broke after upgrading to Python 2.4 (from 2.3). Turns out the code was testing list iterators for their boolean value (to distinguish them from None). In 2.3, a list iterator (like any iterator) is always true. In 2.4, an exhausted list iterator is false; probably by virtue of having a __len__() method that returns the number of remaining items. I realize that this was a deliberate feature, and that it exists in 2.4 as well as in 2.4.1 and will in 2.4.2; yet, I'm not sure I *like* it. Was this breakage (which is not theoretical!) considered at all? -- --Guido van Rossum (home page: http://www.python.org/~guido/)
On Tuesday 20 September 2005 17:49, Guido van Rossum wrote:
I realize that this was a deliberate feature, and that it exists in 2.4 as well as in 2.4.1 and will in 2.4.2; yet, I'm not sure I *like*
I wasn't paying any attention at the time, so I don't know what was discussed. Some discussion here just now leads me to believe that at least two of us here (including myself) think iterators shouldn't have length at all: they're *not* containers and shouldn't act that way. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org>
[Guido]
I just finished debugging some code that broke after upgrading to Python 2.4 (from 2.3). Turns out the code was testing list iterators for their boolean value (to distinguish them from None). In 2.3, a list iterator (like any iterator) is always true. In 2.4, an exhausted list iterator is false; probably by virtue of having a __len__() method that returns the number of remaining items.
I realize that this was a deliberate feature, and that it exists in 2.4 as well as in 2.4.1 and will in 2.4.2; yet, I'm not sure I *like* it. Was this breakage (which is not theoretical!) considered at all?
It was not considered. AFAICT, 2.3 code assuming the Boolean value of an iterator being true was relying on an accidental implementation detail that may not also be true in Jython, PyPy, etc. Likewise, it is not universally true for arbitrary class based iterators which may have other methods including __nonzero__ or __len__. The Boolean value of an iterator is certainly not promised by the iterator protocol as specified in the docs or the PEP. The code, bool(it), is not really clear about its intent and seems a little weird to me. The reason it wasn't considered was that it wasn't on the radar screen as even a possible use case. On a tangential note, I think in 2.2 or 2.3, we found a number of bugs related to None testing. IIRC, the outcome of that conversation was a specific recommendation to NOT determine Noneness by Boolean tests. That recommendation ended-up making it into PEP 290: http://www.python.org/peps/pep-0290.html#testing-for-none [Fred]
think iterators shouldn't have length at all: they're *not* containers and shouldn't act that way.
Some iterators can usefully report their length with the invariant: len(it) == len(list(it)). There are some use cases for having the length when available. Also, there has been plenty of interest in being able to tell, when possible, if an iterator is empty without having to call it. AFAICT, the only downside was Guido's bool(it) situation. FWIW, the origin of the idea came from reading a comp-sci paper about ways to overcome the limitations of linking operations together using only iterators (the paper's terminology talked about map/fold operations). The issue was that decoupling benefits were partially offset by the loss of useful information about the input to an operation (i.e. the supplier may know and the consumer may want to know the input size, the input type, whether the elements are unique, whether the data is sorted, its provenance, etc.) Raymond
"Guido van Rossum"
I just finished debugging some code that broke after upgrading to Python 2.4 (from 2.3). Turns out the code was testing list iterators for their boolean value (to distinguish them from None).
This seem unnecessarily indirect. Why not 'it != None' or now, 'it is not None'?
In 2.3, a list iterator (like any iterator) is always true. In 2.4, an exhausted list iterator is false; probably by virtue of having a __len__() method that returns the number of remaining items.
According to 2.4 News, dict iterators also got __len__ method, though I saw no mention of list.
I realize that this was a deliberate feature,
I presume there were two reasons: internal efficiency of preallocations (list(some_it) for example) and letting people differentiate iterator with something left to return versus nothing, just as we can differentiate collections with something versus nothing.
and that it exists in 2.4 as well as in 2.4.1 and will in 2.4.2; yet, I'm not sure I *like* it. Was this breakage (which is not theoretical!) considered at all?
Searching the gmane archive with the link given on the python site for 'iterator len', I could not find any discussion of the __len__ method addition itself. Terry J. Reedy
On 9/20/05, Raymond Hettinger
[Guido]
I just finished debugging some code that broke after upgrading to Python 2.4 (from 2.3). Turns out the code was testing list iterators for their boolean value (to distinguish them from None). In 2.3, a list iterator (like any iterator) is always true. In 2.4, an exhausted list iterator is false; probably by virtue of having a __len__() method that returns the number of remaining items.
I realize that this was a deliberate feature, and that it exists in 2.4 as well as in 2.4.1 and will in 2.4.2; yet, I'm not sure I *like* it. Was this breakage (which is not theoretical!) considered at all?
It was not considered.
That's too bad.
AFAICT, 2.3 code assuming the Boolean value of an iterator being true was relying on an accidental implementation detail that may not also be true in Jython, PyPy, etc.
That's bullshit, and you know it -- you're just using this to justify that you didn't think of this. Whether an object is true or not is well-defined by the language and not by an accident of the implementation. Apart from None, all objects are always true unless they define either __nonzero__() or (in its absence) __len__(). The iterators for builtin sequences were carefully designed to have the minimal API required of iterators -- i.e., next() and __iter__() and nothing more.
Likewise, it is not universally true for arbitrary class based iterators which may have other methods including __nonzero__ or __len__.
And those are the *only* ones that affect the boolean value.
The Boolean value of an iterator is certainly not promised by the iterator protocol as specified in the docs or the PEP.
it was implied by not specifying a __nonzero__ or __len__.
The code, bool(it), is not really clear about its intent and seems a little weird to me.
Of course that's not what the broken code actually looked like. If was something like if ...: iter1 = iter(...) else: iter1 = None if ...: iter2 = iter(...) else: iter2 = None ... if iter1 and iter2: ... Where the arguments to the iter() functions were known to be lists.
The reason it wasn't considered was that it wasn't on the radar screen as even a possible use case.
Could you at least admit that this was an oversight and not try to pretend it was intentional breakage?
On a tangential note, I think in 2.2 or 2.3, we found a number of bugs related to None testing. IIRC, the outcome of that conversation was a specific recommendation to NOT determine Noneness by Boolean tests. That recommendation ended-up making it into PEP 290:
And I agree with that one in general (I was bitten by this in Zope once). But it bears a lot more weight when the type of the object is unknown or partially unknown. In my case, there was no possibility that the iter() argument was anything except a list, so the type of the iterator was fully known.
[Fred]
think iterators shouldn't have length at all: they're *not* containers and shouldn't act that way.
Some iterators can usefully report their length with the invariant: len(it) == len(list(it)).
I still consider this an erroneous hypergeneralization of the concept of iterators. Iterators should be pure iterators and not also act as containers. Which other object type implements __len__ but not __getitem__?
There are some use cases for having the length when available. Also, there has been plenty of interest in being able to tell, when possible, if an iterator is empty without having to call it. AFAICT, the only downside was Guido's bool(it) situation.
Theer is plenty of interest in broken features all the time. IMO giving *some* iterators discoverable length (and other properties like reversability) but not all of them makes the iterator protocol more error-prone -- we're back to the situation where someone codes an algorithm for use with arbitrary iterators but only tests it with list and tuple iterators, and ends up breaking in the field. I know you can work around it, but that requires introspection which is not a great match for this kind of application.
FWIW, the origin of the idea came from reading a comp-sci paper about ways to overcome the limitations of linking operations together using only iterators (the paper's terminology talked about map/fold operations). The issue was that decoupling benefits were partially offset by the loss of useful information about the input to an operation (i.e. the supplier may know and the consumer may want to know the input size, the input type, whether the elements are unique, whether the data is sorted, its provenance, etc.)
Not every idea written up in a comp-sci paper is worth implementing (as acquisition in Zope 2 has amply proved). -- --Guido van Rossum (home page: http://www.python.org/~guido/)
On 9/20/05, Terry Reedy
I presume there were two reasons: internal efficiency of preallocations (list(some_it) for example)
This could have been implemented without making the implementation details public.
and letting people differentiate iterator with something left to return versus nothing, just as we can differentiate collections with something versus nothing.
But this is against the very idea of the iterator protocol -- for general iterators, there may be no way to determine whether there is a next item without actually producing the item, so the proper approach is to code with that in mind. When this type of look-ahead is required, a buffering iterator should be inserted, so that the algorithm can work with all iterators rather than only with iterators over built-in containers. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
I just finished debugging some code that broke after upgrading to Python 2.4 (from 2.3). Turns out the code was testing list iterators for their boolean value (to distinguish them from None). In 2.3, a list iterator (like any iterator) is always true. In 2.4, an exhausted list iterator is false; probably by virtue of having a __len__() method that returns the number of remaining items.
This seems like a misfeature to me. In fact I think an iterator having a len() method at all is a misfeature -- the concept doesn't make sense. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg.ewing@canterbury.ac.nz +--------------------------------------+
Raymond Hettinger wrote:
The Boolean value of an iterator is certainly not promised by the iterator protocol as specified in the docs or the PEP.
But if the docs don't mention anything about true or false values for some particular type, one tends to assume that all values are true, as is the default for user-defined classes. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg.ewing@canterbury.ac.nz +--------------------------------------+
On 9/21/05, Raymond Hettinger
[Guido van Rossum]
Could you at least admit that this was an oversight and not try to pretend it was intentional breakage?
Absolutely. I completely missed this one.
Thanks; spoken like a man. I strongly feel that this needs to be corrected in 2.5. Iterators should have neither __len__ nor __nonzero__. I see mostly agreement that this is a misfeature. We don't really want to start writing code like this: while it: x = it.next() ...process x... when we can already write it like this: for x in it: ...process x... do we? Keeping a special API to allow a more efficient implementation of __reversed__ is fine. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
"Guido van Rossum"
[Fred]
think iterators shouldn't have length at all: they're *not* containers and shouldn't act that way.
Some iterators can usefully report their length with the invariant: len(it) == len(list(it)).
I still consider this an erroneous hypergeneralization of the concept of iterators. Iterators should be pure iterators and not also act as containers. Which other object type implements __len__ but not __getitem__?
Too late, and probably irrelevant by now; the answer though is set([1,2,3])
participants (6)
-
Christos Georgiou -
Fred L. Drake, Jr. -
Greg Ewing -
Guido van Rossum -
Raymond Hettinger -
Terry Reedy